# Deployment: boiler.saas-ctrl.com

Reference for the public boilerplate-demo deployment. The values below
are **what to configure in Coolify**, not what to commit. Real secrets
(`SESSION_SECRET`, `CRYPTO_SECRET`, etc.) are generated once and live only
in Coolify's encrypted env store and the optional `.env` on the host.

## Hostnames

Single web-facing host with split admin/api subdomains so the cookie-scope
helper in `@toolctrl/config` can still produce canonical URLs.

| Var               | Value                                                            |
| ----------------- | ---------------------------------------------------------------- |
| `APP_BASE_DOMAIN` | `saas-ctrl.com`                                                  |
| `WEB_APP_HOST`    | `boiler.saas-ctrl.com`                                           |
| `ADMIN_APP_HOST`  | `admin-boiler.saas-ctrl.com`                                     |
| `API_HOST`        | `api-boiler.saas-ctrl.com`                                       |
| `APP_PROTOCOL`    | `https`                                                          |
| `PORT`            | `8080` (Dockerfile default; Coolify routes 443 → container 8080) |
| `API_PORT`        | `8080` (api service runs separately if api host is wired)        |

DNS records (A or CNAME to the Coolify ingress IP):

- `boiler.saas-ctrl.com`
- `admin-boiler.saas-ctrl.com`
- `api-boiler.saas-ctrl.com`

For a single-host minimal demo the admin/api hosts can resolve to the
same IP — the runtime checks the `Host:` header to pick the surface.

## Required core secrets

Generate once with `openssl rand -hex 32` and store in Coolify:

- `SESSION_SECRET` — cookie signing
- `CRYPTO_SECRET` — AES key for stored workflow credentials (separate from SESSION_SECRET)
- `API_ACCESS_TOKEN` — `openssl rand -base64 32`
- `JWT_SECRET` — `openssl rand -hex 32`
- `ANALYTICS_TOKEN` — `openssl rand -hex 24`

## Brand

`BRAND_*` overrides set the public face. For the demo:

```
BRAND_PRODUCT_NAME=ToolCTRL Boilerplate
BRAND_SITE_URL=https://boiler.saas-ctrl.com
BRAND_SUPPORT_EMAIL=info@tool-ctrl.de
BRAND_SUPPORT_FROM_NAME=ToolCTRL Boilerplate
BRAND_LEGAL_IMPRINT_URL=https://saas-ctrl.com/impressum
BRAND_LEGAL_PRIVACY_URL=https://saas-ctrl.com/datenschutz
BRAND_LEGAL_TERMS_URL=https://saas-ctrl.com/agb
```

## Database

Provision via Coolify's managed Postgres (or external):

- `DATABASE_URL` — `<pooled-postgres-url>` for application/runtime traffic
- `DIRECT_URL` — optional `<direct-postgres-url>` for migrations against the primary DB

## Mail (optional but expected for boilerplate demo)

```
MAILJET_API_KEY=<from Mailjet>
MAILJET_FROM_EMAIL=noreply@saas-ctrl.com
# Deployment policy/evidence marker only; the mail provider does not read it.
MAILJET_SECRET_KEY=<mailjet-secret-key>
MAILJET_WEBHOOK_SECRET=<mailjet-webhook-secret>
```

The managed `worker-mail` Coolify readiness check verifies that the mail queue
runtime can boot, reach Redis/Postgres, and, when fail-loud mail mode is
enabled, see the required Mailjet environment variables. Treat actual
transactional mail delivery as ready only after sender-domain/provider
verification and a real delivery smoke have passed.

## Seed (production bootstrap)

```
SEED_ADMIN_EMAIL=admin@saas-ctrl.com
SEED_ADMIN_PASSWORD=<one-time bootstrap; rotate on first login>
SEED_DEMO_DATA=false
NODE_ENV=production
```

## Storage (optional)

If file uploads should work in the demo:

```
STORAGE_ENDPOINT=https://s3.eu-central-1.wasabisys.com
STORAGE_REGION=eu-central-1
STORAGE_BUCKET=boiler-saas-ctrl-com-uploads
STORAGE_ACCESS_KEY_ID=<from Wasabi>
STORAGE_SECRET_ACCESS_KEY=<from Wasabi>
STORAGE_PUBLIC_BASE_URL=https://uploads.boiler.saas-ctrl.com
STORAGE_FORCE_PATH_STYLE=false
```

## Workers

The active Gitea/Coolify deploy path expects separate worker applications
for general, mail, import, webhook, and plugin queue groups.

Create five persistent Coolify worker applications from the same worker
image. They are internal runtime apps only: do not configure public FQDNs
for any worker app.

This deployment-specific table is a summary. The canonical worker topology,
queue ownership, concurrency, and scaling guide is
[Worker queue groups](../ops/worker-queues.md).

| Coolify app      | Required group               | Health port | Admin/metrics port |
| ---------------- | ---------------------------- | ----------- | ------------------ |
| `worker-general` | `WORKER_QUEUE_GROUP=general` | `9101`      | `9201`             |
| `worker-mail`    | `WORKER_QUEUE_GROUP=mail`    | `9102`      | `9202`             |
| `worker-import`  | `WORKER_QUEUE_GROUP=import`  | `9103`      | `9203`             |
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | `9104`      | `9204`             |
| `worker-plugin`  | `WORKER_QUEUE_GROUP=plugin`  | `9105`      | `9205`             |

Keep all five services present in staging and production-like environments.
`worker-mail` is a permanent service for transactional mail and mail-backed
outbox/webhook work; it must not be treated as a manual workaround started only
after mail backlog appears.
`worker-plugin` is also permanent topology, but production plugin execution
remains blocked until external WASM sandbox signoff and an explicit
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` rollout. Before that flip, a healthy
`worker-plugin` only proves the idle service can boot; it is not tenant plugin
runtime approval. During the rollout, run `pnpm run verify:plugin-runtime-live`
inside both `worker-general` and `worker-plugin` before enabling tenant plugins.
`worker-import` is also permanent topology but stays idle until import/export
processing is fully wired. Leave `IMPORT_QUEUE_ENABLED` and
`IMPORT_QUEUE_RUNTIME_READY` unset on `worker-import` until the worker bootstrap
injects import, export, and scan services; the service may then report healthy
with an empty queue list instead of consuming jobs it cannot process.

Each worker needs the same database and queue connection, plus its own queue group/ports:

```
WORKER_ENABLED=true
DATABASE_URL=<pooled-postgres-url>
BULLMQ_REDIS_URL=<redacted-redis-url>
WORKER_QUEUE_GROUP=<general|mail|import|webhook|plugin>
WORKER_HEALTH_PORT=<9101|9102|9103|9104|9105>
WORKER_ADMIN_PORT=<9201|9202|9203|9204|9205>
```

`EXPORT_RETENTION_QUEUE_ENABLED=true` is a legacy standalone-worker fallback.
Do not set it as part of the Coolify queue-group contract; when
`WORKER_QUEUE_GROUP` is valid, export-retention ownership comes from
`packages/queue/src/groups.ts`.

Add these only on `worker-general` in staging/production so outbox dispatch and
workflow trigger fanout fail loud instead of appearing healthy while rows pile
up:

```
OUTBOX_DISPATCHER_ENABLED=true
OUTBOX_DISPATCHER_REQUIRED=true
WORKFLOW_TRIGGER_OUTBOX_ENABLED=true
WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true
WORKFLOW_TRIGGER_EXECUTOR_MODE=http
WORKFLOW_TRIGGER_EXECUTOR_URL=<app-origin> # or APP_URL/WEB_ORIGIN=<app-origin>
API_ACCESS_TOKEN=<same value as web API_ACCESS_TOKEN>
```

Keep this unset or false on `worker-mail` until seed data, persisted stats
aggregation, Playwright E2E, Mailjet delivery evidence/reconciliation,
scheduler/deploy evidence, and the deployment record approve production
delivery. The bundled renderer and `@toolctrl/mail` Mailjet adapter exist,
but the sequence subscribers do no work unless the value is explicitly `true`;
stale-lead/trial-expiring cron runner functions are not production
scheduler-registered yet:

```
MARKETING_SEQUENCES_ENABLED=false
```

Add this only on `worker-mail` in staging/production so transactional mail
outbox bootstrap fails loud when Redis, database, or mail-outbox wiring is not
ready. The worker-group provisioner sets and drift-checks this value for
managed `worker-mail` Coolify applications:

```
TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true
```

`scripts/coolify-ensure-worker-groups.mjs` treats missing `DATABASE_URL` or
`BULLMQ_REDIS_URL` on any managed worker group (`worker-general`,
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) as
deployment drift. It also treats missing `worker-general` workflow-trigger
bridge configuration (`WORKFLOW_TRIGGER_EXECUTOR_URL`, `APP_URL`, or `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) and missing mail outbox fail-loud configuration as drift. If
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` is selected instead of `http`, keep that
choice explicit in Coolify and verify the workflow graph uses only the
worker-native safe-core blocks.
Use `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid` to canary native safe-core execution
while routing unsupported workflow graphs through the signed HTTP bridge.

Coolify worker readiness checks should target `/readyz` on
`WORKER_HEALTH_PORT`; `/healthz` remains the lightweight liveness probe for
manual inspection. This confirms queue/runtime readiness for the selected
worker group, including Redis and database reachability. In fail-loud mail mode,
managed `worker-mail` readiness also checks the required Mailjet env
presence. It still does not prove sender-domain verification or live provider
delivery. Admin/metrics endpoints, when enabled, must remain on
`WORKER_ADMIN_PORT` and must not be exposed publicly.

## Observability (opt-in)

```
SENTRY_DSN=<from Sentry>
SENTRY_ENVIRONMENT=production
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel.internal:4318/v1/traces
OTEL_SERVICE_NAME=boiler-saas-ctrl-com
```

## Coolify configuration checklist

1. Resource type: Docker Image applications for the published web, api, and worker images.
2. Source: this Gitea repository, `main` branch.
3. Domain: `boiler.saas-ctrl.com` (Coolify handles Let's Encrypt).
4. Port: container `8080` → public `443`.
5. Healthcheck: web/api use `GET /healthz`; worker apps use `GET /readyz` on `WORKER_HEALTH_PORT`. Container HEALTHCHECK directives cover liveness; Coolify readiness should wait for the service-specific endpoint.
6. Web resource command slots: `pre_deployment_command` runs `cd /myapp && pnpm exec prisma migrate deploy --config prisma.config.ts`; `post_deployment_command` runs `cd /myapp && NODE_ENV=production node packages/db/dist/production-seed.server.js` (staging adds `SEED_DEMO_DATA=true ALLOW_STAGING_DEMO_SEED=true`). Do not combine the seed into the pre-deploy slot.
7. Env: paste the block above.
8. Worker apps: create `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` as separate persistent Coolify applications with no public FQDN. Set `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, and `WORKER_ADMIN_PORT` per the worker table, and point the Coolify health check at `/readyz`. Do not rely on the worker image's standalone `WORKER_QUEUE_GROUP=general` default for grouped Coolify apps; missing per-app overrides make non-general apps drain the wrong queues.
9. Mail delivery: after `worker-mail` is healthy, verify `MAILJET_API_KEY` and `MAILJET_FROM_EMAIL` exist on the mail-capable runtime and run a real transactional-mail smoke. A green worker health check alone is not mail-provider readiness.

## Trigger a deploy

- Staging: push to `main` builds and pushes images in Gitea CI, patches the staging Coolify image tags, then triggers staging deploys.
- Production: use the Gitea `workflow_dispatch` promote action with a reviewed `staged_sha` and `backup_confirmed=true`. Promotion retags the verified staging images; do not treat a plain `main` push as a production deploy.
- Manual emergency deploy: Coolify dashboard → resource → **Deploy**, only after image tags and migration command state are verified against the runbook.

## Verify the deploy

After Coolify reports "healthy":

```sh
curl -fsS https://boiler.saas-ctrl.com/healthz
curl -fsS https://api-boiler.saas-ctrl.com/healthz
# Image-signing — optional, with cosign installed:
cosign verify --key cosign.pub {registry}/{namespace}/web:<sha>
```

If web/API `/healthz` returns the expected `revision`, verify Coolify also
reports `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`,
and `worker-plugin` healthy. The Gitea deploy workflow must also show that
each worker passed Coolify execute-based `node apps/worker/dist/health.js`
readiness with the expected `revision`, `WORKER_QUEUE_GROUP`, and queue list.
`worker-import` may have an empty queue list while its import runtime gates are
unset; if it reports `toolctrl.imports` or `toolctrl.exports`, both import gates
and service wiring must be intentional.
Image alias/tag output is configuration evidence only; it is not an acceptable
fallback for worker health. If Coolify execute is unavailable, empty, or
non-JSON, fix execute access and rerun the workflow before accepting the
worker rollout. Then log into `boiler.saas-ctrl.com` with the seeded admin
credentials and confirm the tenant switcher loads.

## Current deploy latency levers

Gitea CI pre-builds immutable `web`, `api`, and `worker` images and
Coolify deploys those published tags. Production promotion retags the
verified staging images and does **not** rebuild them.

Known remaining latency levers:

- Docker target builds are already launched in parallel with PID aggregation and per-target cache refs; remaining CI-side gains likely come from cache warmup, runner CPU/network capacity, or splitting buildx work across dedicated runners.
- Registry cache is already enabled through `--cache-from/--cache-to` with stable target-specific cache refs, so cache misses usually mean a cold registry cache or Dockerfile layer invalidation.
- Coolify pre-pull is best-effort and bounded by the helper timeout; a slow host can still spend time pulling layers during deploy.
- Staging and production dependent resources are deployed separately so web health gates worker/API rollout; this is safer, but not the fastest possible rollout.
- Optional post-deploy throughput probes remain in `scripts/run-load-tests.mjs`.