Track HiveGrid Cloud planning reference
This commit is contained in:
david committed
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,274 @@
|
||||
# Deployment: boiler.saas-ctrl.com
|
||||
|
||||
Reference for the public boilerplate-demo deployment. The values below
|
||||
are **what to configure in Coolify**, not what to commit. Real secrets
|
||||
(`SESSION_SECRET`, `CRYPTO_SECRET`, etc.) are generated once and live only
|
||||
in Coolify's encrypted env store and the optional `.env` on the host.
|
||||
|
||||
## Hostnames
|
||||
|
||||
Single web-facing host with split admin/api subdomains so the cookie-scope
|
||||
helper in `@toolctrl/config` can still produce canonical URLs.
|
||||
|
||||
| Var | Value |
|
||||
| ----------------- | ---------------------------------------------------------------- |
|
||||
| `APP_BASE_DOMAIN` | `saas-ctrl.com` |
|
||||
| `WEB_APP_HOST` | `boiler.saas-ctrl.com` |
|
||||
| `ADMIN_APP_HOST` | `admin-boiler.saas-ctrl.com` |
|
||||
| `API_HOST` | `api-boiler.saas-ctrl.com` |
|
||||
| `APP_PROTOCOL` | `https` |
|
||||
| `PORT` | `8080` (Dockerfile default; Coolify routes 443 → container 8080) |
|
||||
| `API_PORT` | `8080` (api service runs separately if api host is wired) |
|
||||
|
||||
DNS records (A or CNAME to the Coolify ingress IP):
|
||||
|
||||
- `boiler.saas-ctrl.com`
|
||||
- `admin-boiler.saas-ctrl.com`
|
||||
- `api-boiler.saas-ctrl.com`
|
||||
|
||||
For a single-host minimal demo the admin/api hosts can resolve to the
|
||||
same IP — the runtime checks the `Host:` header to pick the surface.
|
||||
|
||||
## Required core secrets
|
||||
|
||||
Generate once with `openssl rand -hex 32` and store in Coolify:
|
||||
|
||||
- `SESSION_SECRET` — cookie signing
|
||||
- `CRYPTO_SECRET` — AES key for stored workflow credentials (separate from SESSION_SECRET)
|
||||
- `API_ACCESS_TOKEN` — `openssl rand -base64 32`
|
||||
- `JWT_SECRET` — `openssl rand -hex 32`
|
||||
- `ANALYTICS_TOKEN` — `openssl rand -hex 24`
|
||||
|
||||
## Brand
|
||||
|
||||
`BRAND_*` overrides set the public face. For the demo:
|
||||
|
||||
```
|
||||
BRAND_PRODUCT_NAME=ToolCTRL Boilerplate
|
||||
BRAND_SITE_URL=https://boiler.saas-ctrl.com
|
||||
BRAND_SUPPORT_EMAIL=info@tool-ctrl.de
|
||||
BRAND_SUPPORT_FROM_NAME=ToolCTRL Boilerplate
|
||||
BRAND_LEGAL_IMPRINT_URL=https://saas-ctrl.com/impressum
|
||||
BRAND_LEGAL_PRIVACY_URL=https://saas-ctrl.com/datenschutz
|
||||
BRAND_LEGAL_TERMS_URL=https://saas-ctrl.com/agb
|
||||
```
|
||||
|
||||
## Database
|
||||
|
||||
Provision via Coolify's managed Postgres (or external):
|
||||
|
||||
- `DATABASE_URL` — `<pooled-postgres-url>` for application/runtime traffic
|
||||
- `DIRECT_URL` — optional `<direct-postgres-url>` for migrations against the primary DB
|
||||
|
||||
## Mail (optional but expected for boilerplate demo)
|
||||
|
||||
```
|
||||
MAILJET_API_KEY=<from Mailjet>
|
||||
MAILJET_FROM_EMAIL=noreply@saas-ctrl.com
|
||||
# Deployment policy/evidence marker only; the mail provider does not read it.
|
||||
MAILJET_SECRET_KEY=<mailjet-secret-key>
|
||||
MAILJET_WEBHOOK_SECRET=<mailjet-webhook-secret>
|
||||
```
|
||||
|
||||
The managed `worker-mail` Coolify readiness check verifies that the mail queue
|
||||
runtime can boot, reach Redis/Postgres, and, when fail-loud mail mode is
|
||||
enabled, see the required Mailjet environment variables. Treat actual
|
||||
transactional mail delivery as ready only after sender-domain/provider
|
||||
verification and a real delivery smoke have passed.
|
||||
|
||||
## Seed (production bootstrap)
|
||||
|
||||
```
|
||||
SEED_ADMIN_EMAIL=admin@saas-ctrl.com
|
||||
SEED_ADMIN_PASSWORD=<one-time bootstrap; rotate on first login>
|
||||
SEED_DEMO_DATA=false
|
||||
NODE_ENV=production
|
||||
```
|
||||
|
||||
## Storage (optional)
|
||||
|
||||
If file uploads should work in the demo:
|
||||
|
||||
```
|
||||
STORAGE_ENDPOINT=https://s3.eu-central-1.wasabisys.com
|
||||
STORAGE_REGION=eu-central-1
|
||||
STORAGE_BUCKET=boiler-saas-ctrl-com-uploads
|
||||
STORAGE_ACCESS_KEY_ID=<from Wasabi>
|
||||
STORAGE_SECRET_ACCESS_KEY=<from Wasabi>
|
||||
STORAGE_PUBLIC_BASE_URL=https://uploads.boiler.saas-ctrl.com
|
||||
STORAGE_FORCE_PATH_STYLE=false
|
||||
```
|
||||
|
||||
## Workers
|
||||
|
||||
The active Gitea/Coolify deploy path expects separate worker applications
|
||||
for general, mail, import, webhook, and plugin queue groups.
|
||||
|
||||
Create five persistent Coolify worker applications from the same worker
|
||||
image. They are internal runtime apps only: do not configure public FQDNs
|
||||
for any worker app.
|
||||
|
||||
This deployment-specific table is a summary. The canonical worker topology,
|
||||
queue ownership, concurrency, and scaling guide is
|
||||
[Worker queue groups](../ops/worker-queues.md).
|
||||
|
||||
| Coolify app | Required group | Health port | Admin/metrics port |
|
||||
| ---------------- | ---------------------------- | ----------- | ------------------ |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | `9101` | `9201` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | `9102` | `9202` |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | `9103` | `9203` |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | `9104` | `9204` |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | `9105` | `9205` |
|
||||
|
||||
Keep all five services present in staging and production-like environments.
|
||||
`worker-mail` is a permanent service for transactional mail and mail-backed
|
||||
outbox/webhook work; it must not be treated as a manual workaround started only
|
||||
after mail backlog appears.
|
||||
`worker-plugin` is also permanent topology, but production plugin execution
|
||||
remains blocked until external WASM sandbox signoff and an explicit
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` rollout. Before that flip, a healthy
|
||||
`worker-plugin` only proves the idle service can boot; it is not tenant plugin
|
||||
runtime approval. During the rollout, run `pnpm run verify:plugin-runtime-live`
|
||||
inside both `worker-general` and `worker-plugin` before enabling tenant plugins.
|
||||
`worker-import` is also permanent topology but stays idle until import/export
|
||||
processing is fully wired. Leave `IMPORT_QUEUE_ENABLED` and
|
||||
`IMPORT_QUEUE_RUNTIME_READY` unset on `worker-import` until the worker bootstrap
|
||||
injects import, export, and scan services; the service may then report healthy
|
||||
with an empty queue list instead of consuming jobs it cannot process.
|
||||
|
||||
Each worker needs the same database and queue connection, plus its own queue group/ports:
|
||||
|
||||
```
|
||||
WORKER_ENABLED=true
|
||||
DATABASE_URL=<pooled-postgres-url>
|
||||
BULLMQ_REDIS_URL=<redacted-redis-url>
|
||||
WORKER_QUEUE_GROUP=<general|mail|import|webhook|plugin>
|
||||
WORKER_HEALTH_PORT=<9101|9102|9103|9104|9105>
|
||||
WORKER_ADMIN_PORT=<9201|9202|9203|9204|9205>
|
||||
```
|
||||
|
||||
`EXPORT_RETENTION_QUEUE_ENABLED=true` is a legacy standalone-worker fallback.
|
||||
Do not set it as part of the Coolify queue-group contract; when
|
||||
`WORKER_QUEUE_GROUP` is valid, export-retention ownership comes from
|
||||
`packages/queue/src/groups.ts`.
|
||||
|
||||
Add these only on `worker-general` in staging/production so outbox dispatch and
|
||||
workflow trigger fanout fail loud instead of appearing healthy while rows pile
|
||||
up:
|
||||
|
||||
```
|
||||
OUTBOX_DISPATCHER_ENABLED=true
|
||||
OUTBOX_DISPATCHER_REQUIRED=true
|
||||
WORKFLOW_TRIGGER_OUTBOX_ENABLED=true
|
||||
WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true
|
||||
WORKFLOW_TRIGGER_EXECUTOR_MODE=http
|
||||
WORKFLOW_TRIGGER_EXECUTOR_URL=<app-origin> # or APP_URL/WEB_ORIGIN=<app-origin>
|
||||
API_ACCESS_TOKEN=<same value as web API_ACCESS_TOKEN>
|
||||
```
|
||||
|
||||
Keep this unset or false on `worker-mail` until seed data, persisted stats
|
||||
aggregation, Playwright E2E, Mailjet delivery evidence/reconciliation,
|
||||
scheduler/deploy evidence, and the deployment record approve production
|
||||
delivery. The bundled renderer and `@toolctrl/mail` Mailjet adapter exist,
|
||||
but the sequence subscribers do no work unless the value is explicitly `true`;
|
||||
stale-lead/trial-expiring cron runner functions are not production
|
||||
scheduler-registered yet:
|
||||
|
||||
```
|
||||
MARKETING_SEQUENCES_ENABLED=false
|
||||
```
|
||||
|
||||
Add this only on `worker-mail` in staging/production so transactional mail
|
||||
outbox bootstrap fails loud when Redis, database, or mail-outbox wiring is not
|
||||
ready. The worker-group provisioner sets and drift-checks this value for
|
||||
managed `worker-mail` Coolify applications:
|
||||
|
||||
```
|
||||
TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true
|
||||
```
|
||||
|
||||
`scripts/coolify-ensure-worker-groups.mjs` treats missing `DATABASE_URL` or
|
||||
`BULLMQ_REDIS_URL` on any managed worker group (`worker-general`,
|
||||
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) as
|
||||
deployment drift. It also treats missing `worker-general` workflow-trigger
|
||||
bridge configuration (`WORKFLOW_TRIGGER_EXECUTOR_URL`, `APP_URL`, or `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) and missing mail outbox fail-loud configuration as drift. If
|
||||
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` is selected instead of `http`, keep that
|
||||
choice explicit in Coolify and verify the workflow graph uses only the
|
||||
worker-native safe-core blocks.
|
||||
Use `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid` to canary native safe-core execution
|
||||
while routing unsupported workflow graphs through the signed HTTP bridge.
|
||||
|
||||
Coolify worker readiness checks should target `/readyz` on
|
||||
`WORKER_HEALTH_PORT`; `/healthz` remains the lightweight liveness probe for
|
||||
manual inspection. This confirms queue/runtime readiness for the selected
|
||||
worker group, including Redis and database reachability. In fail-loud mail mode,
|
||||
managed `worker-mail` readiness also checks the required Mailjet env
|
||||
presence. It still does not prove sender-domain verification or live provider
|
||||
delivery. Admin/metrics endpoints, when enabled, must remain on
|
||||
`WORKER_ADMIN_PORT` and must not be exposed publicly.
|
||||
|
||||
## Observability (opt-in)
|
||||
|
||||
```
|
||||
SENTRY_DSN=<from Sentry>
|
||||
SENTRY_ENVIRONMENT=production
|
||||
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel.internal:4318/v1/traces
|
||||
OTEL_SERVICE_NAME=boiler-saas-ctrl-com
|
||||
```
|
||||
|
||||
## Coolify configuration checklist
|
||||
|
||||
1. Resource type: Docker Image applications for the published web, api, and worker images.
|
||||
2. Source: this Gitea repository, `main` branch.
|
||||
3. Domain: `boiler.saas-ctrl.com` (Coolify handles Let's Encrypt).
|
||||
4. Port: container `8080` → public `443`.
|
||||
5. Healthcheck: web/api use `GET /healthz`; worker apps use `GET /readyz` on `WORKER_HEALTH_PORT`. Container HEALTHCHECK directives cover liveness; Coolify readiness should wait for the service-specific endpoint.
|
||||
6. Web resource command slots: `pre_deployment_command` runs `cd /myapp && pnpm exec prisma migrate deploy --config prisma.config.ts`; `post_deployment_command` runs `cd /myapp && NODE_ENV=production node packages/db/dist/production-seed.server.js` (staging adds `SEED_DEMO_DATA=true ALLOW_STAGING_DEMO_SEED=true`). Do not combine the seed into the pre-deploy slot.
|
||||
7. Env: paste the block above.
|
||||
8. Worker apps: create `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` as separate persistent Coolify applications with no public FQDN. Set `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, and `WORKER_ADMIN_PORT` per the worker table, and point the Coolify health check at `/readyz`. Do not rely on the worker image's standalone `WORKER_QUEUE_GROUP=general` default for grouped Coolify apps; missing per-app overrides make non-general apps drain the wrong queues.
|
||||
9. Mail delivery: after `worker-mail` is healthy, verify `MAILJET_API_KEY` and `MAILJET_FROM_EMAIL` exist on the mail-capable runtime and run a real transactional-mail smoke. A green worker health check alone is not mail-provider readiness.
|
||||
|
||||
## Trigger a deploy
|
||||
|
||||
- Staging: push to `main` builds and pushes images in Gitea CI, patches the staging Coolify image tags, then triggers staging deploys.
|
||||
- Production: use the Gitea `workflow_dispatch` promote action with a reviewed `staged_sha` and `backup_confirmed=true`. Promotion retags the verified staging images; do not treat a plain `main` push as a production deploy.
|
||||
- Manual emergency deploy: Coolify dashboard → resource → **Deploy**, only after image tags and migration command state are verified against the runbook.
|
||||
|
||||
## Verify the deploy
|
||||
|
||||
After Coolify reports "healthy":
|
||||
|
||||
```sh
|
||||
curl -fsS https://boiler.saas-ctrl.com/healthz
|
||||
curl -fsS https://api-boiler.saas-ctrl.com/healthz
|
||||
# Image-signing — optional, with cosign installed:
|
||||
cosign verify --key cosign.pub {registry}/{namespace}/web:<sha>
|
||||
```
|
||||
|
||||
If web/API `/healthz` returns the expected `revision`, verify Coolify also
|
||||
reports `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`,
|
||||
and `worker-plugin` healthy. The Gitea deploy workflow must also show that
|
||||
each worker passed Coolify execute-based `node apps/worker/dist/health.js`
|
||||
readiness with the expected `revision`, `WORKER_QUEUE_GROUP`, and queue list.
|
||||
`worker-import` may have an empty queue list while its import runtime gates are
|
||||
unset; if it reports `toolctrl.imports` or `toolctrl.exports`, both import gates
|
||||
and service wiring must be intentional.
|
||||
Image alias/tag output is configuration evidence only; it is not an acceptable
|
||||
fallback for worker health. If Coolify execute is unavailable, empty, or
|
||||
non-JSON, fix execute access and rerun the workflow before accepting the
|
||||
worker rollout. Then log into `boiler.saas-ctrl.com` with the seeded admin
|
||||
credentials and confirm the tenant switcher loads.
|
||||
|
||||
## Current deploy latency levers
|
||||
|
||||
Gitea CI pre-builds immutable `web`, `api`, and `worker` images and
|
||||
Coolify deploys those published tags. Production promotion retags the
|
||||
verified staging images and does **not** rebuild them.
|
||||
|
||||
Known remaining latency levers:
|
||||
|
||||
- Docker target builds are already launched in parallel with PID aggregation and per-target cache refs; remaining CI-side gains likely come from cache warmup, runner CPU/network capacity, or splitting buildx work across dedicated runners.
|
||||
- Registry cache is already enabled through `--cache-from/--cache-to` with stable target-specific cache refs, so cache misses usually mean a cold registry cache or Dockerfile layer invalidation.
|
||||
- Coolify pre-pull is best-effort and bounded by the helper timeout; a slow host can still spend time pulling layers during deploy.
|
||||
- Staging and production dependent resources are deployed separately so web health gates worker/API rollout; this is safer, but not the fastest possible rollout.
|
||||
- Optional post-deploy throughput probes remain in `scripts/run-load-tests.mjs`.
|
||||
Reference in new issue
Block a user