# Deployment ## Docker Baseline Health The current baseline is enforced by static Docker, workflow, health, and artifact checks. Historical Docker Baseline evidence is archived in milestone records; this page lists the commands agents should use for the current repository state. ### Baseline Commands The table below is the current command contract and expected result for agents/operators. It is not live-environment proof unless a dated CI or deployment run is attached separately. | Command | Expected exit code | Expected result | Failure fingerprint | | ------------------------------------------ | -----------------: | --------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `node scripts/verify-docker-baseline.mjs` | 0 | Pass | Enforces web, API, MCP, and worker Docker target shape, runtime start commands, healthcheck paths, and runtime `tsx` exclusions. | | `pnpm run verify:runtime-closures` | 0 | Pass | Confirms every bare import in the compiled web, API, MCP, and worker artifacts is declared by that target's isolated runtime manifest. | | `pnpm run lint` | 0 | Pass | ESLint runs with `--max-warnings=0`; warnings are failures. | | `pnpm run test` | 0 | Pass | Runs the repository non-watch Vitest/script suite configured in `package.json`. | | `pnpm typecheck` | 0 | Pass | Prisma client generation, React Router typegen, and `tsc -b` complete successfully. | | `pnpm run verify:production-seed-artifact` | 0 | Pass | Verifies the Coolify production seed artifact build contract and, after build, checks for forbidden bare `@hivegrid/*` runtime imports. | Use focused Vitest files for narrow route, API, worker, and verifier changes; keep browser/E2E checks manual unless explicitly approved for a focused run. ### Platform-lead ingress rollback On the API resource, `PLATFORM_LEADS_INGRESS_ENABLED=false` makes `POST /v1/platform/leads` return a neutral 404 before authentication or persistence. In the web admin, the native lead routes are always present and remain protected by AdminUser status, permissions and Own-/All-RLS. Disable ingress only when producers have been moved back during the documented drain window. The switch requires a redeploy and does not delete data. ### Current Dockerfile Targets and Runtime Commands - `base`: pinned `node:24.18.0-bookworm-slim` on `linux/amd64`, production-oriented environment variables, OpenSSL, Corepack, and `pnpm@11.12.0`. - `deps`: installs build dependencies from the frozen lockfile and every workspace manifest with an offline `pnpm install` behind the BuildKit `/pnpm/store` cache mount. - `build`: copies source once, generates the Prisma client and seed artifacts, and prepares shared SBOM/GeoIP inputs. `web-build`, `api-build`, `mcp-build`, `portal-domain-controller-build`, and `worker-build` compile only their own application. - `production-deps-bundle`: mounts the shared pnpm v2 store with `sharing=locked` and first runs all five runtime-closure deploys offline and in parallel. A complete warm-store hit skips `pnpm fetch`; only a failed wave is fully reaped, cleaned at its five exact `/prod/` paths, repaired by one frozen fetch, and retried once. - `web-production-deps`, `api-production-deps`, `mcp-production-deps`, `portal-domain-controller-production-deps`, and `worker-production-deps`: inherit the common deployed closures, then prune and generate an SBOM for only their matching runtime target. There is no generic whole-workspace production `node_modules` stage. - `web`: copies only the web dependency closure, compiled web/server artifacts, locales, the reviewed migration closure, and GeoIP data; runs `ENTRYPOINT ["./start.sh"]`. It neither contains nor dispatches the separate API application; API-host traffic reaching the web runtime fails closed with 404. - `worker`: copies only the worker dependency closure and compiled `apps/worker/dist`; sets `WORKER_ENABLED=true`; uses `CMD ["node", "apps/worker/dist/index.js"]`; healthcheck runs `node apps/worker/dist/health.js`. - `api`: copies only the API dependency closure and compiled `apps/api/dist`; sets `PORT=8080`; uses `CMD ["node", "apps/api/dist/server.js"]` without `tsx`. - `mcp`: copies only the MCP dependency closure and compiled `apps/mcp/dist`; it remains a separate image/resource. - `hocuspocus`: is built from `apps/hocuspocus/Dockerfile`, runs the compiled collaboration server on port 1234, and exposes a TCP Docker healthcheck. It remains a separate image and Coolify resource. Local closure measurements from `pnpm deploy --prod` (KiB from `du -sk`, excluding compiled application artifacts and base-image layers) show the impact of replacing each app package's broad production deployment with the target runtime manifests: | Target | Previous app closure | Target runtime closure | Reduction | | ------ | --------------------------: | ----------------------: | --------: | | Web | 1,340,304 KiB (1,308.9 MiB) | 835,500 KiB (815.9 MiB) | 37.7% | | API | 1,005,816 KiB (982.2 MiB) | 355,404 KiB (347.1 MiB) | 64.7% | | Worker | 1,159,740 KiB (1,132.6 MiB) | 483,824 KiB (472.5 MiB) | 58.3% | | MCP | 884,936 KiB (864.2 MiB) | 355,012 KiB (346.7 MiB) | 59.9% | These are dependency-closure measurements, not compressed registry image sizes. Repeat the same deploy and `du -sk` commands after dependency changes; `pnpm run verify:runtime-closures` remains the correctness gate. Runtime start command observations: - `start.sh migrate` is a short-lived, `DIRECT_URL`-only migration entrypoint. Before Prisma runs, it executes the semantic role probe and requires a direct `hivegrid_migrator` login with the exact non-superuser, no-`BYPASSRLS`, no-`INHERIT`, and constrained Definer-membership contract. It then adopts the [reviewed Prisma baseline](ops/prisma-baseline.md) only over the complete legacy history: canonical checksums are accepted directly, while Git-pinned historical checksums additionally require the security-sensitive PostgreSQL catalog contracts. After a successful `prisma migrate deploy`, the same one-shot process requires a drift-free live Prisma diff before it reconciles the global permission catalog and global default roles fail-closed; any error prevents rollout. The core web resource requires `DATABASE_URL` for `hivegrid_app`; the customer-portal resource requires the separate `hivegrid_portal` login. Both require `RLS_RUNTIME_MODE=required`, reject migrator/seed variables, select an explicit web surface, and directly `exec` the matching built server runtime. - Root `package.json` defines `start` as `pnpm --filter @hivegrid/web start`; `apps/web/package.json` owns the concrete `node ../../build/server-runtime/server.js` runtime command, so the web Docker target starts the built Production-Runtime artifact instead of `tsx`. - `apps/api/package.json` defines `build` as `node scripts/build.mjs` and `start` as `node dist/server.js`; the Docker `api` target depends only on that compiled artifact and `runtime/api/package.json`'s verified dependency closure. - API and worker tracing use explicit HTTP, Undici, PostgreSQL, ioredis, AWS SDK, and Pino instrumentations. This preserves the remote I/O spans used by the applications without loading the broad Node auto-instrumentation catalog. - `node scripts/verify-docker-baseline.mjs` enforces the locked parallel `pnpm deploy` bundle and its target-specific runtime closures, BuildKit pnpm-store caching, the web/API image boundary, and absence of `tsx` from production commands. `pnpm run verify:runtime-closures` validates each compiled import graph against its runtime manifest. - `apps/worker/package.json` defines production `start` as `node dist/index.js` and `health` as `node dist/health.js`; the worker Docker target depends on these compiled runtime artifacts and must not rely on `tsx` at container runtime. ## Registry Push Baseline Gitea Package Registry Push Contract Registry Push Baseline starts from the existing Gitea CI runner baseline and hardens the S03 deploy workflow contract at `.gitea/workflows/deploy.yml`. The immutable image set is `web`, `api`, `mcp`, `worker`, `hocuspocus`, and `portal-domain-controller`; all six versioned targets are pushed to the Gitea Package Registry on the dedicated self-hosted `toolctrl-deploy` runner before configured Coolify resources are triggered. `hivegrid-web` and `hivegrid-portal` are separate Coolify resources from the same immutable web image and registry digest; each must pass its own redacted `/healthz` and `/readyz` surface/service/revision contract before the pair is accepted. API follows its own health contract. MCP must pass both its external `/readyz` dependency/service/revision gate and an enabled Coolify `cmd` healthcheck using `node apps/mcp/dist/deploy-health.js <40-character lowercase Git SHA>`. Hocuspocus must be present in the environment's explicit resource allowlist, use `hivegrid_app` with `RLS_RUNTIME_MODE=required`, report Coolify `running:healthy` through its image-baked TCP healthcheck, and retain the exact configured image revision. Worker resources are separate Coolify applications for `WORKER_QUEUE_GROUP=general`, `mail`, `import`, `webhook`, and `plugin`. Before rollout, the worker provisioner atomically configures the immutable image tag and `node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> `. Both command-health entrypoints compare a root-owned image-baked revision with `APP_REVISION` before probing the running process over loopback `/readyz`; the worker command additionally requires enabled processing and the exact queue group. The workflow accepts a resource only after its target-specific health contract passes. Image alias/tag checks prove only resource configuration; they are not a pass-open live-revision or readiness fallback. These static and workflow checks are deployment contract proof; they are still not browser-flow or manual production acceptance evidence. Auto-Mode-safe verification commands: | Command | Expected result | | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `node scripts/verify-gitea-ci-workflow.mjs` | Passes with every pull-request CI job using `runs-on: toolctrl-ci` and with forbidden browser/E2E deploy commands absent. The deploy workflow is verified separately on `toolctrl-deploy`. | | `node scripts/verify-gitea-deploy-workflow.mjs` | Emits deterministic PASS/FAIL lines for the `.gitea/workflows/deploy.yml` static contract, including runner, registry-login, image-target, tag-policy, and forbidden-command checks. | | `pnpm run verify:deploy-readiness` | Confirms the package scripts, CI workflow, deploy workflow, deployment docs, worker matrix docs, and worker-group provisioner stay consistent while distinguishing the lightweight deploy preflight from CI/readiness coverage for staging/promotion, health, i18n, hot-swap, Docker, seed, and migration gates. | | `pnpm run verify:tenant-policy-contract` | Regenerates the tenant-policy contract in check mode and fails on stale or missing RLS policy coverage. | | `pnpm run verify:rls-release-readiness` | Verifies the release-level RLS role, policy, migration, Docker, and workflow contract. | | `pnpm run verify:rls-live` | Proves tenant A/B isolation against an ephemeral PostgreSQL instance using the runtime roles. | | `pnpm exec vitest run scripts/verify-gitea-deploy-workflow.test.mjs` | Exercises verifier fixtures for missing deploy contract pieces, forbidden literal secret exposure, forbidden browser/E2E commands, required Coolify trigger coverage, and required image tag behavior. | | `grep -R "ubuntu-latest" .gitea` | Returns no matches; any match means the self-hosted runner baseline has regressed. | | `grep -n "verify:deploy-workflow" docs/deployment.md && grep -n "main" docs/deployment.md` | Confirms this deployment document keeps the deploy verifier command and tag-policy language visible. | The package script `verify:deploy-workflow` is the supported shorthand for the deploy workflow verifier. The deploy workflow follows a stage-and-promote contract on the dedicated self-hosted `toolctrl-deploy` runner. One retained `stage-or-promote` job runs lightweight static deploy wiring checks against the workflow revision selected by Gitea before a promotion detaches to its exact staged application revision; this avoids releasing and reacquiring a runner between preflight and rollout. The fail-closed checks remain the Gitea deploy workflow verifier, deploy-readiness docs/worker/provisioner consistency verifier, Docker baseline, Docker non-root runtime verifier, Coolify helper syntax checks, worker-group provisioner syntax checks, and pre-pull script syntax checks. Broader gates such as API inventory, OpenAPI security, env drift, health baseline, i18n, Phase C boundaries, Host Aware URL baseline, removed-module and plugin-isolation gates, hot-swap, UI visual, build, and production seed artifact verification belong to the pull-request CI/readiness contract. Therefore Gitea main branch protection remains mandatory: direct pushes to `main` must be disabled and merges to `main` must require the pull-request CI workflow to pass. The stage-or-promote job independently binds its security proof to the exact deployment SHA before any registry login, image mutation, migration, or Coolify mutation. It fetches and checks out `IMAGE_TAG`, confirms the checked-out commit and ancestry, installs that revision's frozen dependency graph, and then runs `pnpm run verify:tenant-policy-contract`, `pnpm run verify:rls-release-readiness`, and `pnpm run verify:rls-live`. A failure in any gate stops the deployment before external state changes. Pushes to `main` then build all six image targets, push immutable full-SHA tags plus the `staging` alias, run the reviewed web image once with the step-scoped staging migrator credential, verify every Coolify runtime has only its approved least-privilege capability, pair `hivegrid-web` and `hivegrid-portal` on the same web SHA/digest, and independently deploy and health-check the private portal-domain controller before triggering dependent API, MCP, Hocuspocus, `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` resources. MCP and Hocuspocus are separate public image targets and Coolify resources; neither may be folded into `hivegrid-api`. Production promotion is an input-free `workflow_dispatch`: a manual run from `main` always promotes the release that is currently live on the canonical staging core surface. Starting that production-only run is the operator's audited confirmation that production backup or restore-point readiness has been checked. The workflow reads the full staged commit SHA from staging health, validates the expected service and surface plus main ancestry, runs the same exact-revision gates, verifies the full-SHA image set against `staging`, moves the production `main` alias without rebuilding, runs the same web image as a one-shot migrator with the production migrator credential scoped to that step, verifies production runtime credentials, deploys production web first, waits for web health/revision proof, verifies migration status, and only then deploys dependent API, MCP, Hocuspocus, plus all five worker-group resources. These names identify required CI variable or secret slots only; documentation must not include real deploy endpoint structures, UUIDs, tokens, passwords, credentials, or secret-shaped examples. The deploy image tag contract is static and versioned. Every staging run must publish full-SHA immutable tags for each image target and update the `staging` alias. Promotion must move the `main` alias to the exact full-SHA image set verified by that workflow run; it must not rebuild from source. Images are recorded by safe target labels such as `web`, `api`, `mcp`, `worker`, and `hocuspocus`; do not paste registry URLs, project IDs, credentials, or token-shaped examples into docs or tickets. The mutable `latest` alias is not part of the deployment contract. Each staging build imports the previous target-specific `staging` image. BuildKit exports `type=inline`, so the portable cache metadata travels with the already required immutable SHA image push and remains available when a fresh deploy runner starts with an empty local cache. Legacy `build-cache:$TARGET` references are forbidden because the workflow never publishes them; removing those dead references avoids six useless registry lookups per staging build. A separate `type=registry` cache export is also forbidden: it adds another registry write after the image build and previously allowed registry-side 499/context-deadline failures to turn an otherwise successful release red. Until the homeserver Docker Engine is upgraded from 29.2.1 to the fixed 29.4+ line, staging builds must explicitly select the persistent `hivegrid-deploy-v0312-gc1` `docker-container` builder. Its BuildKit v0.31.2 image is manifest-digest pinned, its history and OCI worker are bounded by `scripts/ci/deploy-buildkitd.toml`, and the workflow starts it only around the image-build window. The always-run shutdown stops the privileged builder after normal or failed staging builds without deleting its bounded package-cache mounts. Every subsequent serialized deploy first stops an orphaned host container, and `restart-policy=no` prevents Docker from reviving it after a daemon restart. A staging run validates the host container's exact image, restart policy, and loaded TOML independently of the fresh job container's Buildx client metadata before any build starts. A hard cancellation can still leave only this single stable builder running until that next deploy; it cannot create another cache volume. Bump the builder's generation suffix when its pinned image or daemon configuration changes. This keeps deploy builds away from the affected Engine-embedded BuildKit history listener without changing the build-once/promote-without-rebuild contract. Coolify deployment is runner-side and follows successful image publication. The `.gitea/workflows/deploy.yml` stage path runs on pushes to `main`; an input-free manual `workflow_dispatch` from `main` is exclusively the promote path. It resolves the full 40-character SHA from the live staging core health response, validates it and verifies all current `staging` image aliases before any production mutation. Staging Coolify is called only after all staging images publish and the one-shot migration succeeds. The workflow then clears the dead `pre_deployment_command`/`post_deployment_command` slots, checks web/API/MCP/Hocuspocus runtime env with `scripts/coolify-deploy.mjs assert-runtime-db-env`, reconciles worker resources with `hivegrid_worker` credentials and SHA-bound `cmd` healthchecks, triggers staging web first, and then waits on every dependent resource. The explicit staging and production resource allowlists must both include the Hocuspocus application; a missing resource fails closed before rollout. Production uses the same one-shot-before-rollout and credential-verification sequence after the resolved full-SHA images have been verified and promoted. The trigger must read endpoint and resource information from configured CI variables and secrets and must not expose deploy URLs, UUIDs, bearer tokens, passwords, or registry credentials in docs, logs, tickets, or verification summaries. The private portal-domain controller is an explicit Coolify resource, not a side effect of the general resource resolver. The current rollout is deliberately staging-only: configure its UUID through `COOLIFY_STAGING_PORTAL_DOMAIN_CONTROLLER_RESOURCE_UUID`, include it in the staging resource allowlist, and leave its Coolify route empty. Its only host port is the Prometheus endpoint bound to the staging WireGuard address (`10.42.0.3:19120:8080`); a wildcard, loopback-only, missing, or additional mapping is rejected. The staging workflow validates its runtime before mutation, pins the exact full-SHA image while clearing deployment hooks, and reconciles a revision-bound Node `cmd` healthcheck because Coolify's `dockerimage` build pack does not reliably expose the Dockerfile healthcheck. The command proves `/healthz`, `/readyz`, the controller service identity, and equality between the expected, baked, and runtime revision without adding `curl` or `wget` to the minimal image. The workflow requires Coolify's custom-image-health marker to remain false, triggers and waits on the exact returned deployment UUID, and accepts the release only as `running:healthy` with that exact command configuration. The staging `hivegrid-web` and `hivegrid-portal` resources must both set `PORTAL_CUSTOM_DOMAIN_RUNTIME_ENABLED=true` exactly. The controller deployment helper verifies both values before any mutation. Production deliberately leaves this opt-in unset or false while no authorised production controller exists: the tenant-admin domain route, its admin link, and every custom-host request then fail closed with 404, while the standard portal host remains available. Enabling the production variable is part of the separately reviewed controller rollout and must never happen implicitly during staging promotion. Production remains inactive until a separately authorised controller resource, host reachability and observability contract exist. The helper already reserves the fail-closed production CNAME and mesh binding (`10.42.0.8:19120:8080`), but the deploy workflow must not reference or require a production controller UUID before that provisioning is complete. Activating it is a separate reviewed rollout, not an implicit consequence of the staging change. The controller receives exactly two writable bind mounts: the directory `/data/coolify/proxy/dynamic/hivegrid-portal-domains` and the single sentinel file `/data/coolify/proxy/dynamic/.hivegrid-portal-domains.reload`. Never mount the complete Traefik dynamic-config root. Set `PORTAL_DOMAIN_CONTROLLER_CONFIG_DIR` and `PORTAL_DOMAIN_CONTROLLER_RELOAD_TRIGGER_FILE` to those exact paths. Supply `PORTAL_DOMAIN_CONTROLLER_BACKEND_SERVICE` as the environment-specific provider-qualified Docker service name ending in `@docker`; `PORTAL_DOMAIN_CONTROLLER_ENTRYPOINT` is optional and defaults to `https`. Staging domain-flow tests use the exact expected CNAME `staging-domains.portal.hivegrid.de`; production uses `domains.portal.hivegrid.de` and remains separately configured. The controller and `worker-general` share only the portal-domain command HMAC secret and the same Redis queue endpoint/database. `worker-general` must set `PORTAL_DOMAIN_ORCHESTRATION_REQUIRED=true`; it must not receive the controller metrics token. The controller additionally requires a distinct metrics token and its expected CNAME. Missing, duplicate, malformed, mismatched, or privileged runtime values fail the deployment before the controller resource is changed. Values remain CI/Coolify secrets and must never be recorded in the repository. If the self-hosted runner needs an internal hosts override for the Gitea registry, provide `HIVEGRID_GITEA_HOST_IP` as a CI variable. The workflows leave `/etc/hosts` untouched when that variable is absent. Migrations run in a short-lived container from the exact immutable web image being rolled out: `start.sh migrate` accepts only `DIRECT_URL`, verifies that it authenticates directly as the exact `hivegrid_migrator` role with no superuser, database-creation, role-creation, inheritance, replication, or `BYPASSRLS` capability and only the constrained `hivegrid_rls_definer` membership, runs the fail-closed Prisma-baseline adoption, aliases the URL to `DATABASE_URL` inside that process for Prisma CLI compatibility, runs `prisma migrate deploy`, verifies the resulting live schema against the current Prisma schema, runs the global permission-catalog reconcile only after both checks succeed, and then exits. It rejects runtime role markers and every `SEED_*` value. The migration step is fail-closed: a missing/unreachable credential, a semantic role mismatch, an incomplete or unknown legacy migration history, security-sensitive catalog drift behind a reviewed historical checksum, a failed migration, post-migration schema drift, a missing externally provisioned managed role, or a failed catalog reconcile stops the deployment before any Coolify resource is changed. Run `pnpm run db:roles:provision` with all five distinct login passwords (`HIVEGRID_RLS_MIGRATOR_PASSWORD`, `HIVEGRID_RLS_APP_PASSWORD`, `HIVEGRID_RLS_PORTAL_PASSWORD`, `HIVEGRID_RLS_WORKER_PASSWORD`, and `HIVEGRID_RLS_AUTHENTIK_JIT_PASSWORD`) before the cutover migration; the NOCREATEROLE migrator never creates or alters roles. Long-lived web/API/MCP/Hocuspocus containers receive only the `hivegrid_app` DSN; the customer-portal runtime receives only `hivegrid_portal`; workers receive only `hivegrid_worker`; all set `DATABASE_RUNTIME_ROLE` and `RLS_RUNTIME_MODE=required`. The core web resource additionally receives `AUTHENTIK_ADMIN_JIT_DATABASE_URL` only when Admin SSO is enabled; that URL must authenticate as the execute-only `hivegrid_authentik_jit` login and must not be copied to the portal, API, MCP, Hocuspocus, or worker resources. `DIRECT_URL`, `MIGRATOR_*`, `MIGRATE_ON_BOOT`, `SEED_ON_BOOT`, and `SEED_*` are forbidden in runtime resources. The PR Docker smokes rehearse that same boundary against an empty database. The deploy workflow keeps both Coolify command slots cleared so the dead hook mechanism is not mistaken for an active one. Docker healthchecks use `/healthz` as process liveness, except the dedicated Hocuspocus image, whose TCP healthcheck covers port 1234. MCP and workers use dependency-free Node `cmd` healthchecks against their loopback `/readyz` endpoints. Der Reconcile bearbeitet ausschließlich globale `Permission`-Einträge, globale Defaultrollen (`tenantId IS NULL`) und deren katalogeigene `RolePermission`-Links. Innerhalb einer kurzen Transaktion erhält `hivegrid_rls_definer` nur die dafür nötigen DML-Rechte; sie werden vor dem Commit wieder entzogen. `FORCE ROW LEVEL SECURITY` bleibt durchgehend aktiv. Jede Abweichung der Sicherheitsgrenze oder Daten-Postcondition führt zum Rollback der gesamten Reconcile-Transaktion. Nach AD-29 sind tenant-eigene Rollen und Rollen mit `isTemplateOrigin = true` von automatischen Deploy-Änderungen ausgeschlossen. Eine kanonische globale Rolle mit diesem Marker lässt den Reconcile vor Rollen-DML fail-closed abbrechen; Tenant-Rollen bleiben unberührt. Nachzüge für bestehende Tenants sind nur als ausdrücklich freigegebene und auditierte Operator-Aktion zulässig. Der Reconcile ist kein allgemeiner Seed und erteilt dafür keine Freigabe. Seeding is deliberately not part of the non-superuser migrator or any long-lived runtime. Existing bootstrap/demo seed tooling requires a separately reviewed, tenant-scoped operator workflow before it may be used under forced RLS. Do not set `SEED_DEMO_DATA=true`, `ALLOW_STAGING_DEMO_SEED=true`, `SEED_DEMO_PRO_MANAGED=true`, seed credentials, or seed flags on Coolify runtime resources. The one-shot migrator runs on the **CI runner host**, not on the database host, so the migrator DSN must point at an address that is routable from the runner. A Coolify container hostname is not: it only resolves inside the target host's docker network. When it is unreachable, `scripts/verify-migrator-role.mjs` reports the generic `PostgreSQL migrator role verification failed` — that message is the catch-all for **connection** errors, not a role-contract violation (a real contract breach reports a specific reason), so check reachability before touching roles. On staging the database is therefore published on the WireGuard mesh address only (`10.42.0.3:5432`, never `0.0.0.0`); see `infra/LESSONS.md`. Clarify the same reachability question for production before opening the cutover window. Signing in with a seeded staging demo account is a separate concern from seeding and therefore has its own runtime gate, `STAGING_DEMO_LOGIN_ENABLED=true` on the staging web resource. It only takes effect together with `SENTRY_ENVIRONMENT=staging`, so it cannot unlock a demo account in production. The gate previously rode on the `SEED_*` flags above; because `assert-runtime-db-env` rejects every `SEED_*`-prefixed key on a runtime resource, enforcing the deploy contract silently locked the staging demo accounts out until the gate was renamed. Set `WORKER_ENABLED=true` only on worker Coolify resources. Leave it unset on web/API resources and avoid sharing a production env group that forces `WORKER_ENABLED=false` into worker containers. The deploy resource list must include separate worker applications with `WORKER_QUEUE_GROUP=general`, `mail`, `import`, `webhook`, and `plugin`; the deploy workflow identifies them by the group env marker or by resource names such as `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin`. `worker-import` always owns `hivegrid.kwp-imports`; generic file imports remain separately gated. Full mode (`IMPORT_QUEUE_ENABLED=true` + `IMPORT_QUEUE_RUNTIME_READY=true`) adds `hivegrid.imports` and `hivegrid.exports` and requires wired import/export/scan **and** portal/privacy GDPR services plus complete HTTPS `STORAGE_*` and canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`; the mutually exclusive internal `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` mode adds only `hivegrid.exports` beside KWP and requires the same encrypted storage/GDPR services. The same SSE-C key must be provisioned on core web and portal web before that image is deployed; it is never exposed through a signed-URL redirect. The narrow mode never enables `hivegrid.imports` or the generic export service. Mixed gates, a non-import group, missing storage/encryption, or missing service wiring fail readiness/startup closed. Treat this deployment paragraph as a summary: `docs/ops/worker-queues.md` is the canonical worker topology and queue ownership reference. `worker-general` additionally owns the provider boundary keys `LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`, `MOLLIE_PROFILE_ID`, and `MOLLIE_TIMEOUT_MS`, plus the environment-specific `BILLING_MOLLIE_WEBHOOK_URL`. `MOLLIE_PROFILE_ID` is optional and both timeouts use their documented defaults when unset. These keys plus `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED` are exclusive to `worker-general`: the Coolify worker reconciler copies them from the approved runtime-env source only into that group, never into `worker-mail`, `worker-import`, `worker-webhook`, or `worker-plugin`. It removes provider keys left on those workers by older provisioning runs through their Coolify env UUID and fails closed before mutation when a key cannot be removed safely. Provider side effects default to required under `RLS_RUNTIME_MODE=required`; in that mode the reconciler rejects missing `LAGO_API_URL`, `LAGO_API_KEY`, `MOLLIE_API_KEY` or `BILLING_MOLLIE_WEBHOOK_URL` and malformed configured values before deploy. During an approved pre-live window, `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` is the only supported override: it disables the subscriber and waives the provider-key preflight without weakening RLS or any other worker dependency. Any value other than `true`, `false` or unset fails closed. Provider values remain in Coolify/secret storage and must not be printed by preflight or runbooks. The plugin worker belongs to the topology even while production plugin execution is gated: with `PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, `worker-plugin` is an idle service, not evidence that tenant plugin execution is approved. If `COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true` or `COOLIFY_PROVISION_WORKER_GROUPS=true` is set with the matching project, server, environment, and destination UUID variables, the deploy workflow creates any missing worker-group applications through the Coolify Docker Image API before patching image tags. The provisioner copies runtime env from an existing worker source app when it creates a group resource, then overwrites the group-specific worker env, including distinct health and admin ports. It also creates or reconciles the enabled `cmd` healthcheck to the exact immutable SHA used for that deployment, so operators should keep an env-source worker app configured until all group resources have their own Coolify env. B-012 is not a repository code blocker. The deploy workflow and Docker verifiers cover the repository contract, while the permanent fix for runner Docker network exhaustion is operator-owned host hardening: enlarge Docker `default-address-pools` on the self-hosted runner and add scheduled cleanup for orphaned Gitea Actions networks. Deploy secrets, registry credentials, Coolify URLs, UUIDs, personal access tokens, webhook tokens, and other deploy-only values must be supplied through CI secrets or a secret manager. Documentation, workflow logs, tickets, and verification summaries must use placeholder configuration only and must not commit, echo, or paste real URLs, UUIDs, tokens, passwords, registry credentials, or secret-shaped examples. Auto-Mode-safe proof is limited to repository/static checks and the configured CI workflow contract such as `pnpm run verify:deploy-workflow`, `node scripts/verify-gitea-deploy-workflow.mjs`, and focused verifier tests. Browser flows, Playwright, Cypress, and E2E checks remain manual exclusions unless a human explicitly approves the exact focused run. ## Release and rollback contract Deployment proof is layered. Static verifiers such as `pnpm run verify:deploy-workflow`, `node scripts/verify-gitea-deploy-workflow.mjs`, `node scripts/verify-gitea-ci-workflow.mjs` and focused tests prove repository shape and image policy; they do not prove a live registry push, Coolify rollout or browser flow. The production image set is `web`, `api`, `mcp`, `worker`, `hocuspocus`, and `portal-domain-controller`. MCP uses a separate `mcp` target/resource for `hivegrid-mcp`, and Hocuspocus uses its dedicated image/resource; neither must be folded into the `api` or `web` runtime. Staging uses the same target set with isolated runtime configuration in Coolify. Published image references must stay placeholder-only in docs and tickets, using safe target labels and tag labels rather than real registry hosts, namespaces, project IDs, UUIDs, credentials, or token-shaped examples. Every deployable image needs an immutable full-SHA tag; the `staging` alias identifies the currently staged image set, and the `main` alias is updated only by promotion. Operators should inspect workflow output for redacted target names, tag lists, and PASS/FAIL verifier lines only, never raw secrets. Coolify execution is part of the main-branch stage path and the `workflow_dispatch` promote path. The workflow may trigger Coolify only after the exact-revision RLS gates and image publication or promotion prerequisites pass. It deploys the migration-bearing web application first, verifies web health/revision, then deploys dependent API, MCP, Hocuspocus, and worker resources by reading endpoint/resource data from configured CI variables and secrets. Hocuspocus acceptance requires Coolify `running:healthy` from the image TCP healthcheck plus the exact configured image SHA. Documentation must not include Coolify URLs, webhook paths, UUIDs, bearer tokens, passwords, registry credentials, or derived endpoint examples. Pull requests and schedules must not invoke Coolify. Ad hoc live Docker builds, registry pushes, Coolify runs, browser flows, Playwright, Cypress, and E2E checks outside the configured Gitea deploy workflow are manual runtime activities unless a human explicitly approves the exact focused run. The Gitea deploy workflow may run Docker, registry, Coolify, migration, and web/API health steps automatically; record only redacted evidence: deployment status, image target names, immutable tag IDs, web/API/MCP service and revision health, revision-bound command-health status, worker group/queue summary, API `requestId` or `x-request-id` correlation, and queue/job identifiers. Worker live-revision proof uses `node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> `; MCP uses `node apps/mcp/dist/deploy-health.js <40-character lowercase Git SHA>`. An absent, disabled, non-`cmd`, mismatched, or unhealthy command fails the automated rollout; operators must fix the runtime healthcheck contract before accepting the deploy. Do not paste database URLs, Redis URLs, Supabase keys, mail provider credentials, stack traces with secrets, webhook tokens, or container environment values. Rollback uses immutable image tags instead of mutable guesses. Select the previous known-good full 40-character SHA for the affected target. Web and API may follow the approved manual redeploy path. Before invoking an MCP or worker rollback helper, provide the exact HiveGrid image identity through `MCP_IMAGE` / `WORKER_IMAGE` or the complete `REGISTRY` + `IMAGE_NAMESPACE` + `IMAGE_PREFIX` tuple; suffix-only target matching is intentionally rejected. For MCP and worker images built with the revision-bound deployment-health entrypoints, treat `docker_registry_image_tag` and `health_check_command` as one atomic configuration unit: changing only the image tag leaves the command bound to the newer revision and must fail closed. Reconcile MCP with `node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`; reconcile workers with `IMAGE_TAG="$OLD_FULL_SHA" node scripts/coolify-ensure-worker-groups.mjs `, preserving each resource's exact `general`, `mail`, `import`, `webhook`, or `plugin` group. The worker reconciler is fleet-wide: after it succeeds, force-deploy all five worker resources and wait on every exact returned deployment UUID. Never reconcile the old SHA and trigger only one group, because the persisted configuration of the other groups already points at the rollback. MCP is triggered separately and likewise waited by its exact deployment UUID. There is one bootstrap exception: images from before the revision-bound entrypoints existed do not contain `apps/mcp/dist/deploy-health.js`, `apps/worker/dist/deploy-health.js`, or `/myapp/.image-revision`. Never point the new command at such an image. Use `node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$RESOURCE_UUID" "$OLD_FULL_SHA" [worker-group]`; the optional group is mandatory for workers. The helper validates the full SHA, rejects a Coolify `APP_REVISION` override or group drift, atomically sets the old tag, disables command health, clears the command, and rereads the resource fail-closed. Then force-deploy and wait on the exact deployment UUID. This is a degraded, emergency-only bootstrap rollback: Coolify cannot use the disabled command as its candidate gate, so brief downtime is possible. Accept the legacy MCP only after Coolify reports `running:healthy` and public `/readyz` reports `service=mcp`, the exact old revision, and successful dependency checks. Accept each legacy worker only after Coolify reports `running:healthy`, the live container digest/OCI revision resolves to the old SHA image through the approved read-only host check, `node apps/worker/dist/health.js` reports the image-baked old revision, enabled processing, exact group, and ready dependencies, and the Coolify `APP_REVISION` override remains absent. This compatibility path is for rollback only. Re-enable and verify the SHA-bound `cmd` healthcheck before the next roll-forward to a current image. All identifiers above are placeholders; never paste real UUIDs, endpoints, tokens, or credentials into documentation or logs. Confirm web/API health with `GET /healthz` and `GET /readyz` plus the documented API health endpoint. For worker incidents, inspect redacted health output, queue names, failed-job counts, and correlation IDs; pause or drain queues before replaying jobs. If readiness fails, capture only the non-secret diagnostic code, target name, tag, and correlation ID. ## Health Baseline Health Baseline Gate S02 enforces CI-safe static contracts for the web and API liveness/readiness endpoints. The web app exposes `GET /healthz` through `apps/web/app/routes/healthz.tsx` as a lightweight liveness route with no database dependency, and `GET /readyz` through `apps/web/app/routes/readyz.tsx` as a database-backed readiness route; both return the non-secret runtime service, surface, and image-baked revision, while readiness failures remain redacted. The API exposes unauthenticated `GET /healthz` and `GET /readyz`; successful responses include non-secret JSON with `service: "api"`, `revision`, and `requestId`, and `/readyz` returns HTTP 503 with a redacted dependency diagnostic code when readiness fails. Auto-Mode-safe verification commands: | Command | Purpose | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `pnpm exec vitest run apps/web/app/routes/healthz.test.ts apps/web/app/routes/readyz.test.ts` | Focused web route coverage for liveness/readiness response behavior. | | `pnpm exec vitest run apps/api/src/app.test.ts apps/api/src/openapi.test.ts` | Focused API route and OpenAPI coverage for `/healthz` and `/readyz`. | | `pnpm exec vitest run scripts/verify-health-baseline.test.mjs` | Static verifier fixture coverage, including missing web/API endpoint regressions and worker DB-readiness regressions. | | `node scripts/verify-health-baseline.mjs` | Static repository baseline check with deterministic path-specific `PASS`, `OBSERVE`, and `FAIL` lines for web, API, and worker health contracts. | The verifier output must no longer contain web/API observations saying `/healthz` or `/readyz` is not present, and worker DB readiness must be reported as an enforced pass/fail baseline rather than an observation. Missing web routes, missing API routes, web liveness database usage, missing web readiness database checks, missing API `/readyz` 503 dependency diagnostics, and missing worker DB readiness are all baseline failures. Manual runtime inspection remains separate from Auto-Mode. A human may inspect a deployed web or API runtime with `GET /healthz` and `GET /readyz` and may compare the API `x-request-id` response header with the JSON `requestId` value. Readiness diagnostics are intentionally non-secret: do not record database URLs, Supabase keys, Redis URLs, tokens, passwords, stack traces, or other secret values in docs, logs, tickets, or verification summaries. Live Docker smokes, browser flows, Cypress, Playwright, and other E2E checks remain manual runtime verification and are not part of this Auto-Mode health baseline gate unless a human explicitly approves the exact focused run. ### Worker Docker Target Verification and Inspection The worker target is statically/package verified by these CI-safe commands: | Command | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pnpm --filter @hivegrid/worker build` | Confirms the worker package emits compiled runtime files for `node dist/index.js` and `node dist/health.js`. | | `pnpm exec vitest run apps/worker/src/index.test.ts apps/worker/src/health.test.ts scripts/verify-docker-baseline.test.mjs scripts/verify-health-baseline.test.mjs` | Exercises worker runtime/package tests and static verifier coverage, including worker `tsx` leakage and DB-readiness regressions. | | `node scripts/verify-docker-baseline.mjs` | Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for worker Docker target copy paths, start scripts, health scripts, and runtime `tsx` usage. | | `node scripts/verify-health-baseline.mjs` | Enforces that worker health statically reports Redis parse state and DB readiness without requiring live DB access. | Manual Docker smoke verification is intentionally separate from Auto-Mode. A human may run a focused build such as `docker build --target worker -t hivegrid-worker:smoke .`, then run the image with safe environment key names only, for example `WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, `WORKER_ADMIN_PORT`, `DATABASE_URL`, `BULLMQ_REDIS_URL`, fallback `REDIS_URL`, group-specific fail-loud keys, and mail provider keys supplied by the local shell or secret manager rather than written into docs. To inspect the worker healthcheck inside a container, run `node apps/worker/dist/health.js`; the JSON output is expected to report whether the worker is enabled/configured, whether Redis configuration parses, the DB readiness state, queue names, and redacted dependency codes, without echoing Redis URLs, passwords, database connection strings, stack traces, tokens, or other deployment secret values. Auto-Mode did not run live Docker build/run smoke checks, browser flows, Playwright, Cypress, or E2E commands for S04; the recorded proof is limited to static/package verification. ### API Docker Target Verification and Inspection The API target is statically/package verified by these task-scoped commands: | Command | Purpose | | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `pnpm --filter @hivegrid/api build` | Confirms the API package can compile `apps/api/src` into `apps/api/dist`, including `dist/server.js` for the Docker runtime start path. | | `pnpm exec vitest run scripts/verify-docker-baseline.test.mjs` | Exercises the Docker baseline verifier test coverage, including API-target PASS/FAIL cases for required copies and `tsx` runtime leakage. | | `node scripts/verify-docker-baseline.mjs` | Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for the API target. | Runtime inspection for a manually run API container should use `GET /v1/health`. A healthy response is expected to identify `service: "api"` and include a `requestId`; startup logs should include the structured `API listening` message with a redacted `port` field. Safe environment keys that may be supplied by deployment automation include `PORT`, `NODE_ENV`, database connection variables, Supabase keys, and JWT/password-hash configuration keys; documentation and logs must not include secret values. ### Scope Notes Docker build/run smoke verification belongs to focused manual target checks. A manual Docker smoke check may optionally build and run the focused API target with `docker build --target api ...`, start the container, and request `GET /v1/health`, or build the focused worker target with `docker build --target worker ...` and inspect `node apps/worker/dist/health.js` inside the container. Auto-Mode must not run live Docker smoke, Playwright, Cypress, browser, or E2E commands unless a human explicitly approves that exact run. The recorded Auto-Mode verification for S03 and S04 is static/package verification only; it does not claim a live Docker runtime proof.