HiveGrid-Wireframes/hivegrid-cloud/docs/deployments/boiler-saas-ctrl-com.md

Deployment: boiler.saas-ctrl.com

Reference for the public boilerplate-demo deployment. The values below are what to configure in Coolify, not what to commit. Real secrets (SESSION_SECRET, CRYPTO_SECRET, etc.) are generated once and live only in Coolify's encrypted env store and the optional .env on the host.

Hostnames

Single web-facing host with split admin/api subdomains so the cookie-scope helper in @toolctrl/config can still produce canonical URLs.

Var	Value
APP_BASE_DOMAIN	saas-ctrl.com
WEB_APP_HOST	boiler.saas-ctrl.com
ADMIN_APP_HOST	admin-boiler.saas-ctrl.com
API_HOST	api-boiler.saas-ctrl.com
APP_PROTOCOL	https
PORT	8080 (Dockerfile default; Coolify routes 443 → container 8080)
API_PORT	8080 (api service runs separately if api host is wired)

DNS records (A or CNAME to the Coolify ingress IP):

• boiler.saas-ctrl.com
• admin-boiler.saas-ctrl.com
• api-boiler.saas-ctrl.com

For a single-host minimal demo the admin/api hosts can resolve to the same IP — the runtime checks the Host: header to pick the surface.

Required core secrets

Generate once with openssl rand -hex 32 and store in Coolify:

• SESSION_SECRET — cookie signing
• CRYPTO_SECRET — AES key for stored workflow credentials (separate from SESSION_SECRET)
• API_ACCESS_TOKEN — openssl rand -base64 32
• JWT_SECRET — openssl rand -hex 32
• ANALYTICS_TOKEN — openssl rand -hex 24

Brand

BRAND_* overrides set the public face. For the demo:

BRAND_PRODUCT_NAME=ToolCTRL Boilerplate
BRAND_SITE_URL=https://boiler.saas-ctrl.com
BRAND_SUPPORT_EMAIL=info@tool-ctrl.de
BRAND_SUPPORT_FROM_NAME=ToolCTRL Boilerplate
BRAND_LEGAL_IMPRINT_URL=https://saas-ctrl.com/impressum
BRAND_LEGAL_PRIVACY_URL=https://saas-ctrl.com/datenschutz
BRAND_LEGAL_TERMS_URL=https://saas-ctrl.com/agb

Database

Provision via Coolify's managed Postgres (or external):

• DATABASE_URL — <pooled-postgres-url> for application/runtime traffic
• DIRECT_URL — optional <direct-postgres-url> for migrations against the primary DB

Mail (optional but expected for boilerplate demo)

MAILJET_API_KEY=<from Mailjet>
MAILJET_FROM_EMAIL=noreply@saas-ctrl.com
# Deployment policy/evidence marker only; the mail provider does not read it.
MAILJET_SECRET_KEY=<mailjet-secret-key>
MAILJET_WEBHOOK_SECRET=<mailjet-webhook-secret>

The managed worker-mail Coolify readiness check verifies that the mail queue runtime can boot, reach Redis/Postgres, and, when fail-loud mail mode is enabled, see the required Mailjet environment variables. Treat actual transactional mail delivery as ready only after sender-domain/provider verification and a real delivery smoke have passed.

Seed (production bootstrap)

SEED_ADMIN_EMAIL=admin@saas-ctrl.com
SEED_ADMIN_PASSWORD=<one-time bootstrap; rotate on first login>
SEED_DEMO_DATA=false
NODE_ENV=production

Storage (optional)

If file uploads should work in the demo:

STORAGE_ENDPOINT=https://s3.eu-central-1.wasabisys.com
STORAGE_REGION=eu-central-1
STORAGE_BUCKET=boiler-saas-ctrl-com-uploads
STORAGE_ACCESS_KEY_ID=<from Wasabi>
STORAGE_SECRET_ACCESS_KEY=<from Wasabi>
STORAGE_PUBLIC_BASE_URL=https://uploads.boiler.saas-ctrl.com
STORAGE_FORCE_PATH_STYLE=false

Workers

The active Gitea/Coolify deploy path expects separate worker applications for general, mail, import, webhook, and plugin queue groups.

Create five persistent Coolify worker applications from the same worker image. They are internal runtime apps only: do not configure public FQDNs for any worker app.

This deployment-specific table is a summary. The canonical worker topology, queue ownership, concurrency, and scaling guide is Worker queue groups.

Coolify app	Required group	Health port	Admin/metrics port
worker-general	WORKER_QUEUE_GROUP=general	9101	9201
worker-mail	WORKER_QUEUE_GROUP=mail	9102	9202
worker-import	WORKER_QUEUE_GROUP=import	9103	9203
worker-webhook	WORKER_QUEUE_GROUP=webhook	9104	9204
worker-plugin	WORKER_QUEUE_GROUP=plugin	9105	9205

Keep all five services present in staging and production-like environments. worker-mail is a permanent service for transactional mail and mail-backed outbox/webhook work; it must not be treated as a manual workaround started only after mail backlog appears. worker-plugin is also permanent topology, but production plugin execution remains blocked until external WASM sandbox signoff and an explicit PLUGIN_RUNTIME_PRODUCTION_ENABLED=true rollout. Before that flip, a healthy worker-plugin only proves the idle service can boot; it is not tenant plugin runtime approval. During the rollout, run pnpm run verify:plugin-runtime-live inside both worker-general and worker-plugin before enabling tenant plugins. worker-import is also permanent topology but stays idle until import/export processing is fully wired. Leave IMPORT_QUEUE_ENABLED and IMPORT_QUEUE_RUNTIME_READY unset on worker-import until the worker bootstrap injects import, export, and scan services; the service may then report healthy with an empty queue list instead of consuming jobs it cannot process.

Each worker needs the same database and queue connection, plus its own queue group/ports:

WORKER_ENABLED=true
DATABASE_URL=<pooled-postgres-url>
BULLMQ_REDIS_URL=<redacted-redis-url>
WORKER_QUEUE_GROUP=<general|mail|import|webhook|plugin>
WORKER_HEALTH_PORT=<9101|9102|9103|9104|9105>
WORKER_ADMIN_PORT=<9201|9202|9203|9204|9205>

EXPORT_RETENTION_QUEUE_ENABLED=true is a legacy standalone-worker fallback. Do not set it as part of the Coolify queue-group contract; when WORKER_QUEUE_GROUP is valid, export-retention ownership comes from packages/queue/src/groups.ts.

Add these only on worker-general in staging/production so outbox dispatch and workflow trigger fanout fail loud instead of appearing healthy while rows pile up:

OUTBOX_DISPATCHER_ENABLED=true
OUTBOX_DISPATCHER_REQUIRED=true
WORKFLOW_TRIGGER_OUTBOX_ENABLED=true
WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true
WORKFLOW_TRIGGER_EXECUTOR_MODE=http
WORKFLOW_TRIGGER_EXECUTOR_URL=<app-origin> # or APP_URL/WEB_ORIGIN=<app-origin>
API_ACCESS_TOKEN=<same value as web API_ACCESS_TOKEN>

Keep this unset or false on worker-mail until seed data, persisted stats aggregation, Playwright E2E, Mailjet delivery evidence/reconciliation, scheduler/deploy evidence, and the deployment record approve production delivery. The bundled renderer and @toolctrl/mail Mailjet adapter exist, but the sequence subscribers do no work unless the value is explicitly true; stale-lead/trial-expiring cron runner functions are not production scheduler-registered yet:

MARKETING_SEQUENCES_ENABLED=false

Add this only on worker-mail in staging/production so transactional mail outbox bootstrap fails loud when Redis, database, or mail-outbox wiring is not ready. The worker-group provisioner sets and drift-checks this value for managed worker-mail Coolify applications:

TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true

scripts/coolify-ensure-worker-groups.mjs treats missing DATABASE_URL or BULLMQ_REDIS_URL on any managed worker group (worker-general, worker-mail, worker-import, worker-webhook, worker-plugin) as deployment drift. It also treats missing worker-general workflow-trigger bridge configuration (WORKFLOW_TRIGGER_EXECUTOR_URL, APP_URL, or WEB_ORIGIN plus API_ACCESS_TOKEN) and missing mail outbox fail-loud configuration as drift. If WORKFLOW_TRIGGER_EXECUTOR_MODE=native is selected instead of http, keep that choice explicit in Coolify and verify the workflow graph uses only the worker-native safe-core blocks. Use WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid to canary native safe-core execution while routing unsupported workflow graphs through the signed HTTP bridge.

Coolify worker readiness checks should target /readyz on WORKER_HEALTH_PORT; /healthz remains the lightweight liveness probe for manual inspection. This confirms queue/runtime readiness for the selected worker group, including Redis and database reachability. In fail-loud mail mode, managed worker-mail readiness also checks the required Mailjet env presence. It still does not prove sender-domain verification or live provider delivery. Admin/metrics endpoints, when enabled, must remain on WORKER_ADMIN_PORT and must not be exposed publicly.

Observability (opt-in)

SENTRY_DSN=<from Sentry>
SENTRY_ENVIRONMENT=production
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel.internal:4318/v1/traces
OTEL_SERVICE_NAME=boiler-saas-ctrl-com

Coolify configuration checklist

1. Resource type: Docker Image applications for the published web, api, and worker images.
2. Source: this Gitea repository, main branch.
3. Domain: boiler.saas-ctrl.com (Coolify handles Let's Encrypt).
4. Port: container 8080 → public 443.
5. Healthcheck: web/api use GET /healthz; worker apps use GET /readyz on WORKER_HEALTH_PORT. Container HEALTHCHECK directives cover liveness; Coolify readiness should wait for the service-specific endpoint.
6. Web resource command slots: pre_deployment_command runs cd /myapp && pnpm exec prisma migrate deploy --config prisma.config.ts; post_deployment_command runs cd /myapp && NODE_ENV=production node packages/db/dist/production-seed.server.js (staging adds SEED_DEMO_DATA=true ALLOW_STAGING_DEMO_SEED=true). Do not combine the seed into the pre-deploy slot.
7. Env: paste the block above.
8. Worker apps: create worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin as separate persistent Coolify applications with no public FQDN. Set WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, and WORKER_ADMIN_PORT per the worker table, and point the Coolify health check at /readyz. Do not rely on the worker image's standalone WORKER_QUEUE_GROUP=general default for grouped Coolify apps; missing per-app overrides make non-general apps drain the wrong queues.
9. Mail delivery: after worker-mail is healthy, verify MAILJET_API_KEY and MAILJET_FROM_EMAIL exist on the mail-capable runtime and run a real transactional-mail smoke. A green worker health check alone is not mail-provider readiness.

Trigger a deploy

• Staging: push to main builds and pushes images in Gitea CI, patches the staging Coolify image tags, then triggers staging deploys.
• Production: use the Gitea workflow_dispatch promote action with a reviewed staged_sha and backup_confirmed=true. Promotion retags the verified staging images; do not treat a plain main push as a production deploy.
• Manual emergency deploy: Coolify dashboard → resource → Deploy, only after image tags and migration command state are verified against the runbook.

Verify the deploy

After Coolify reports "healthy":

curl -fsS https://boiler.saas-ctrl.com/healthz
curl -fsS https://api-boiler.saas-ctrl.com/healthz
# Image-signing — optional, with cosign installed:
cosign verify --key cosign.pub {registry}/{namespace}/web:<sha>

If web/API /healthz returns the expected revision, verify Coolify also reports worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin healthy. The Gitea deploy workflow must also show that each worker passed Coolify execute-based node apps/worker/dist/health.js readiness with the expected revision, WORKER_QUEUE_GROUP, and queue list. worker-import may have an empty queue list while its import runtime gates are unset; if it reports toolctrl.imports or toolctrl.exports, both import gates and service wiring must be intentional. Image alias/tag output is configuration evidence only; it is not an acceptable fallback for worker health. If Coolify execute is unavailable, empty, or non-JSON, fix execute access and rerun the workflow before accepting the worker rollout. Then log into boiler.saas-ctrl.com with the seeded admin credentials and confirm the tenant switcher loads.

Current deploy latency levers

Gitea CI pre-builds immutable web, api, and worker images and Coolify deploys those published tags. Production promotion retags the verified staging images and does not rebuild them.

Known remaining latency levers:

• Docker target builds are already launched in parallel with PID aggregation and per-target cache refs; remaining CI-side gains likely come from cache warmup, runner CPU/network capacity, or splitting buildx work across dedicated runners.
• Registry cache is already enabled through --cache-from/--cache-to with stable target-specific cache refs, so cache misses usually mean a cold registry cache or Dockerfile layer invalidation.
• Coolify pre-pull is best-effort and bounded by the helper timeout; a slow host can still spend time pulling layers during deploy.
• Staging and production dependent resources are deployed separately so web health gates worker/API rollout; this is safer, but not the fastest possible rollout.
• Optional post-deploy throughput probes remain in scripts/run-load-tests.mjs.