HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/deployment.md

Deployment

Docker Baseline Health

The current baseline is enforced by static Docker, workflow, health, and artifact checks. Historical Docker Baseline evidence is archived in milestone records; this page lists the commands agents should use for the current repository state.

Baseline Commands

The table below is the current command contract and expected result for agents/operators. It is not live-environment proof unless a dated CI or deployment run is attached separately.

Command	Expected exit code	Expected result	Failure fingerprint
node scripts/verify-docker-baseline.mjs	0	Pass	Enforces web, API, MCP, and worker Docker target shape, runtime start commands, healthcheck paths, and runtime tsx exclusions.
pnpm run verify:runtime-closures	0	Pass	Confirms every bare import in the compiled web, API, MCP, and worker artifacts is declared by that target's isolated runtime manifest.
pnpm run lint	0	Pass	ESLint runs with --max-warnings=0; warnings are failures.
pnpm run test	0	Pass	Runs the repository non-watch Vitest/script suite configured in package.json.
pnpm typecheck	0	Pass	Prisma client generation, React Router typegen, and tsc -b complete successfully.
pnpm run verify:production-seed-artifact	0	Pass	Verifies the Coolify production seed artifact build contract and, after build, checks for forbidden bare @hivegrid/* runtime imports.

Use focused Vitest files for narrow route, API, worker, and verifier changes; keep browser/E2E checks manual unless explicitly approved for a focused run.

Platform-lead ingress rollback

On the API resource, PLATFORM_LEADS_INGRESS_ENABLED=false makes POST /v1/platform/leads return a neutral 404 before authentication or persistence. In the web admin, the native lead routes are always present and remain protected by AdminUser status, permissions and Own-/All-RLS. Disable ingress only when producers have been moved back during the documented drain window. The switch requires a redeploy and does not delete data.

Current Dockerfile Targets and Runtime Commands

• base: pinned node:24.18.0-bookworm-slim on linux/amd64, production-oriented environment variables, OpenSSL, Corepack, and pnpm@11.12.0.
• deps: installs build dependencies from the frozen lockfile and every workspace manifest with an offline pnpm install behind the BuildKit /pnpm/store cache mount.
• build: copies source once, generates the Prisma client and seed artifacts, and prepares shared SBOM/GeoIP inputs. web-build, api-build, mcp-build, portal-domain-controller-build, and worker-build compile only their own application.
• production-deps-bundle: mounts the shared pnpm v2 store with sharing=locked and first runs all five runtime-closure deploys offline and in parallel. A complete warm-store hit skips pnpm fetch; only a failed wave is fully reaped, cleaned at its five exact /prod/<target> paths, repaired by one frozen fetch, and retried once.
• web-production-deps, api-production-deps, mcp-production-deps, portal-domain-controller-production-deps, and worker-production-deps: inherit the common deployed closures, then prune and generate an SBOM for only their matching runtime target. There is no generic whole-workspace production node_modules stage.
• web: copies only the web dependency closure, compiled web/server artifacts, locales, the reviewed migration closure, and GeoIP data; runs ENTRYPOINT ["./start.sh"]. It neither contains nor dispatches the separate API application; API-host traffic reaching the web runtime fails closed with 404.
• worker: copies only the worker dependency closure and compiled apps/worker/dist; sets WORKER_ENABLED=true; uses CMD ["node", "apps/worker/dist/index.js"]; healthcheck runs node apps/worker/dist/health.js.
• api: copies only the API dependency closure and compiled apps/api/dist; sets PORT=8080; uses CMD ["node", "apps/api/dist/server.js"] without tsx.
• mcp: copies only the MCP dependency closure and compiled apps/mcp/dist; it remains a separate image/resource.
• hocuspocus: is built from apps/hocuspocus/Dockerfile, runs the compiled collaboration server on port 1234, and exposes a TCP Docker healthcheck. It remains a separate image and Coolify resource.

Local closure measurements from pnpm deploy --prod (KiB from du -sk, excluding compiled application artifacts and base-image layers) show the impact of replacing each app package's broad production deployment with the target runtime manifests:

Target	Previous app closure	Target runtime closure	Reduction
Web	1,340,304 KiB (1,308.9 MiB)	835,500 KiB (815.9 MiB)	37.7%
API	1,005,816 KiB (982.2 MiB)	355,404 KiB (347.1 MiB)	64.7%
Worker	1,159,740 KiB (1,132.6 MiB)	483,824 KiB (472.5 MiB)	58.3%
MCP	884,936 KiB (864.2 MiB)	355,012 KiB (346.7 MiB)	59.9%

These are dependency-closure measurements, not compressed registry image sizes. Repeat the same deploy and du -sk commands after dependency changes; pnpm run verify:runtime-closures remains the correctness gate.

Runtime start command observations:

• start.sh migrate is a short-lived, DIRECT_URL-only migration entrypoint. Before Prisma runs, it executes the semantic role probe and requires a direct hivegrid_migrator login with the exact non-superuser, no-BYPASSRLS, no-INHERIT, and constrained Definer-membership contract. It then adopts the reviewed Prisma baseline only over the complete legacy history: canonical checksums are accepted directly, while Git-pinned historical checksums additionally require the security-sensitive PostgreSQL catalog contracts. After a successful prisma migrate deploy, the same one-shot process requires a drift-free live Prisma diff before it reconciles the global permission catalog and global default roles fail-closed; any error prevents rollout. The core web resource requires DATABASE_URL for hivegrid_app; the customer-portal resource requires the separate hivegrid_portal login. Both require RLS_RUNTIME_MODE=required, reject migrator/seed variables, select an explicit web surface, and directly exec the matching built server runtime.
• Root package.json defines start as pnpm --filter @hivegrid/web start; apps/web/package.json owns the concrete node ../../build/server-runtime/server.js runtime command, so the web Docker target starts the built Production-Runtime artifact instead of tsx.
• apps/api/package.json defines build as node scripts/build.mjs and start as node dist/server.js; the Docker api target depends only on that compiled artifact and runtime/api/package.json's verified dependency closure.
• API and worker tracing use explicit HTTP, Undici, PostgreSQL, ioredis, AWS SDK, and Pino instrumentations. This preserves the remote I/O spans used by the applications without loading the broad Node auto-instrumentation catalog.
• node scripts/verify-docker-baseline.mjs enforces the locked parallel pnpm deploy bundle and its target-specific runtime closures, BuildKit pnpm-store caching, the web/API image boundary, and absence of tsx from production commands. pnpm run verify:runtime-closures validates each compiled import graph against its runtime manifest.
• apps/worker/package.json defines production start as node dist/index.js and health as node dist/health.js; the worker Docker target depends on these compiled runtime artifacts and must not rely on tsx at container runtime.

Registry Push Baseline Gitea Package Registry Push Contract

Registry Push Baseline starts from the existing Gitea CI runner baseline and hardens the S03 deploy workflow contract at .gitea/workflows/deploy.yml. The immutable image set is web, api, mcp, worker, hocuspocus, and portal-domain-controller; all six versioned targets are pushed to the Gitea Package Registry on the dedicated self-hosted toolctrl-deploy runner before configured Coolify resources are triggered. hivegrid-web and hivegrid-portal are separate Coolify resources from the same immutable web image and registry digest; each must pass its own redacted /healthz and /readyz surface/service/revision contract before the pair is accepted. API follows its own health contract. MCP must pass both its external /readyz dependency/service/revision gate and an enabled Coolify cmd healthcheck using node apps/mcp/dist/deploy-health.js <40-character lowercase Git SHA>. Hocuspocus must be present in the environment's explicit resource allowlist, use hivegrid_app with RLS_RUNTIME_MODE=required, report Coolify running:healthy through its image-baked TCP healthcheck, and retain the exact configured image revision. Worker resources are separate Coolify applications for WORKER_QUEUE_GROUP=general, mail, import, webhook, and plugin. Before rollout, the worker provisioner atomically configures the immutable image tag and node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>. Both command-health entrypoints compare a root-owned image-baked revision with APP_REVISION before probing the running process over loopback /readyz; the worker command additionally requires enabled processing and the exact queue group. The workflow accepts a resource only after its target-specific health contract passes. Image alias/tag checks prove only resource configuration; they are not a pass-open live-revision or readiness fallback. These static and workflow checks are deployment contract proof; they are still not browser-flow or manual production acceptance evidence.

Auto-Mode-safe verification commands:

Command	Expected result
node scripts/verify-gitea-ci-workflow.mjs	Passes with every pull-request CI job using runs-on: toolctrl-ci and with forbidden browser/E2E deploy commands absent. The deploy workflow is verified separately on toolctrl-deploy.
node scripts/verify-gitea-deploy-workflow.mjs	Emits deterministic PASS/FAIL lines for the .gitea/workflows/deploy.yml static contract, including runner, registry-login, image-target, tag-policy, and forbidden-command checks.
pnpm run verify:deploy-readiness	Confirms the package scripts, CI workflow, deploy workflow, deployment docs, worker matrix docs, and worker-group provisioner stay consistent while distinguishing the lightweight deploy preflight from CI/readiness coverage for staging/promotion, health, i18n, hot-swap, Docker, seed, and migration gates.
pnpm run verify:tenant-policy-contract	Regenerates the tenant-policy contract in check mode and fails on stale or missing RLS policy coverage.
pnpm run verify:rls-release-readiness	Verifies the release-level RLS role, policy, migration, Docker, and workflow contract.
pnpm run verify:rls-live	Proves tenant A/B isolation against an ephemeral PostgreSQL instance using the runtime roles.
pnpm exec vitest run scripts/verify-gitea-deploy-workflow.test.mjs	Exercises verifier fixtures for missing deploy contract pieces, forbidden literal secret exposure, forbidden browser/E2E commands, required Coolify trigger coverage, and required image tag behavior.
grep -R "ubuntu-latest" .gitea	Returns no matches; any match means the self-hosted runner baseline has regressed.
grep -n "verify:deploy-workflow" docs/deployment.md && grep -n "main" docs/deployment.md	Confirms this deployment document keeps the deploy verifier command and tag-policy language visible.

The package script verify:deploy-workflow is the supported shorthand for the deploy workflow verifier. The deploy workflow follows a stage-and-promote contract on the dedicated self-hosted toolctrl-deploy runner. One retained stage-or-promote job runs lightweight static deploy wiring checks against the workflow revision selected by Gitea before a promotion detaches to its exact staged application revision; this avoids releasing and reacquiring a runner between preflight and rollout. The fail-closed checks remain the Gitea deploy workflow verifier, deploy-readiness docs/worker/provisioner consistency verifier, Docker baseline, Docker non-root runtime verifier, Coolify helper syntax checks, worker-group provisioner syntax checks, and pre-pull script syntax checks. Broader gates such as API inventory, OpenAPI security, env drift, health baseline, i18n, Phase C boundaries, Host Aware URL baseline, removed-module and plugin-isolation gates, hot-swap, UI visual, build, and production seed artifact verification belong to the pull-request CI/readiness contract. Therefore Gitea main branch protection remains mandatory: direct pushes to main must be disabled and merges to main must require the pull-request CI workflow to pass.

The stage-or-promote job independently binds its security proof to the exact deployment SHA before any registry login, image mutation, migration, or Coolify mutation. It fetches and checks out IMAGE_TAG, confirms the checked-out commit and ancestry, installs that revision's frozen dependency graph, and then runs pnpm run verify:tenant-policy-contract, pnpm run verify:rls-release-readiness, and pnpm run verify:rls-live. A failure in any gate stops the deployment before external state changes. Pushes to main then build all six image targets, push immutable full-SHA tags plus the staging alias, run the reviewed web image once with the step-scoped staging migrator credential, verify every Coolify runtime has only its approved least-privilege capability, pair hivegrid-web and hivegrid-portal on the same web SHA/digest, and independently deploy and health-check the private portal-domain controller before triggering dependent API, MCP, Hocuspocus, worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin resources. MCP and Hocuspocus are separate public image targets and Coolify resources; neither may be folded into hivegrid-api.

Production promotion is an input-free workflow_dispatch: a manual run from main always promotes the release that is currently live on the canonical staging core surface. Starting that production-only run is the operator's audited confirmation that production backup or restore-point readiness has been checked. The workflow reads the full staged commit SHA from staging health, validates the expected service and surface plus main ancestry, runs the same exact-revision gates, verifies the full-SHA image set against staging, moves the production main alias without rebuilding, runs the same web image as a one-shot migrator with the production migrator credential scoped to that step, verifies production runtime credentials, deploys production web first, waits for web health/revision proof, verifies migration status, and only then deploys dependent API, MCP, Hocuspocus, plus all five worker-group resources. These names identify required CI variable or secret slots only; documentation must not include real deploy endpoint structures, UUIDs, tokens, passwords, credentials, or secret-shaped examples.

The deploy image tag contract is static and versioned. Every staging run must publish full-SHA immutable tags for each image target and update the staging alias. Promotion must move the main alias to the exact full-SHA image set verified by that workflow run; it must not rebuild from source. Images are recorded by safe target labels such as web, api, mcp, worker, and hocuspocus; do not paste registry URLs, project IDs, credentials, or token-shaped examples into docs or tickets. The mutable latest alias is not part of the deployment contract.

Each staging build imports the previous target-specific staging image. BuildKit exports type=inline, so the portable cache metadata travels with the already required immutable SHA image push and remains available when a fresh deploy runner starts with an empty local cache. Legacy build-cache:$TARGET references are forbidden because the workflow never publishes them; removing those dead references avoids six useless registry lookups per staging build. A separate type=registry cache export is also forbidden: it adds another registry write after the image build and previously allowed registry-side 499/context-deadline failures to turn an otherwise successful release red.

Until the homeserver Docker Engine is upgraded from 29.2.1 to the fixed 29.4+ line, staging builds must explicitly select the persistent hivegrid-deploy-v0312-gc1 docker-container builder. Its BuildKit v0.31.2 image is manifest-digest pinned, its history and OCI worker are bounded by scripts/ci/deploy-buildkitd.toml, and the workflow starts it only around the image-build window. The always-run shutdown stops the privileged builder after normal or failed staging builds without deleting its bounded package-cache mounts. Every subsequent serialized deploy first stops an orphaned host container, and restart-policy=no prevents Docker from reviving it after a daemon restart. A staging run validates the host container's exact image, restart policy, and loaded TOML independently of the fresh job container's Buildx client metadata before any build starts. A hard cancellation can still leave only this single stable builder running until that next deploy; it cannot create another cache volume. Bump the builder's generation suffix when its pinned image or daemon configuration changes. This keeps deploy builds away from the affected Engine-embedded BuildKit history listener without changing the build-once/promote-without-rebuild contract.

Coolify deployment is runner-side and follows successful image publication. The .gitea/workflows/deploy.yml stage path runs on pushes to main; an input-free manual workflow_dispatch from main is exclusively the promote path. It resolves the full 40-character SHA from the live staging core health response, validates it and verifies all current staging image aliases before any production mutation. Staging Coolify is called only after all staging images publish and the one-shot migration succeeds. The workflow then clears the dead pre_deployment_command/post_deployment_command slots, checks web/API/MCP/Hocuspocus runtime env with scripts/coolify-deploy.mjs assert-runtime-db-env, reconciles worker resources with hivegrid_worker credentials and SHA-bound cmd healthchecks, triggers staging web first, and then waits on every dependent resource. The explicit staging and production resource allowlists must both include the Hocuspocus application; a missing resource fails closed before rollout. Production uses the same one-shot-before-rollout and credential-verification sequence after the resolved full-SHA images have been verified and promoted. The trigger must read endpoint and resource information from configured CI variables and secrets and must not expose deploy URLs, UUIDs, bearer tokens, passwords, or registry credentials in docs, logs, tickets, or verification summaries.

The private portal-domain controller is an explicit Coolify resource, not a side effect of the general resource resolver. The current rollout is deliberately staging-only: configure its UUID through COOLIFY_STAGING_PORTAL_DOMAIN_CONTROLLER_RESOURCE_UUID, include it in the staging resource allowlist, and leave its Coolify route empty. Its only host port is the Prometheus endpoint bound to the staging WireGuard address (10.42.0.3:19120:8080); a wildcard, loopback-only, missing, or additional mapping is rejected. The staging workflow validates its runtime before mutation, pins the exact full-SHA image while clearing deployment hooks, and reconciles a revision-bound Node cmd healthcheck because Coolify's dockerimage build pack does not reliably expose the Dockerfile healthcheck. The command proves /healthz, /readyz, the controller service identity, and equality between the expected, baked, and runtime revision without adding curl or wget to the minimal image. The workflow requires Coolify's custom-image-health marker to remain false, triggers and waits on the exact returned deployment UUID, and accepts the release only as running:healthy with that exact command configuration.

The staging hivegrid-web and hivegrid-portal resources must both set PORTAL_CUSTOM_DOMAIN_RUNTIME_ENABLED=true exactly. The controller deployment helper verifies both values before any mutation. Production deliberately leaves this opt-in unset or false while no authorised production controller exists: the tenant-admin domain route, its admin link, and every custom-host request then fail closed with 404, while the standard portal host remains available. Enabling the production variable is part of the separately reviewed controller rollout and must never happen implicitly during staging promotion.

Production remains inactive until a separately authorised controller resource, host reachability and observability contract exist. The helper already reserves the fail-closed production CNAME and mesh binding (10.42.0.8:19120:8080), but the deploy workflow must not reference or require a production controller UUID before that provisioning is complete. Activating it is a separate reviewed rollout, not an implicit consequence of the staging change.

The controller receives exactly two writable bind mounts: the directory /data/coolify/proxy/dynamic/hivegrid-portal-domains and the single sentinel file /data/coolify/proxy/dynamic/.hivegrid-portal-domains.reload. Never mount the complete Traefik dynamic-config root. Set PORTAL_DOMAIN_CONTROLLER_CONFIG_DIR and PORTAL_DOMAIN_CONTROLLER_RELOAD_TRIGGER_FILE to those exact paths. Supply PORTAL_DOMAIN_CONTROLLER_BACKEND_SERVICE as the environment-specific provider-qualified Docker service name ending in @docker; PORTAL_DOMAIN_CONTROLLER_ENTRYPOINT is optional and defaults to https. Staging domain-flow tests use the exact expected CNAME staging-domains.portal.hivegrid.de; production uses domains.portal.hivegrid.de and remains separately configured.

The controller and worker-general share only the portal-domain command HMAC secret and the same Redis queue endpoint/database. worker-general must set PORTAL_DOMAIN_ORCHESTRATION_REQUIRED=true; it must not receive the controller metrics token. The controller additionally requires a distinct metrics token and its expected CNAME. Missing, duplicate, malformed, mismatched, or privileged runtime values fail the deployment before the controller resource is changed. Values remain CI/Coolify secrets and must never be recorded in the repository. If the self-hosted runner needs an internal hosts override for the Gitea registry, provide HIVEGRID_GITEA_HOST_IP as a CI variable. The workflows leave /etc/hosts untouched when that variable is absent.

Migrations run in a short-lived container from the exact immutable web image being rolled out: start.sh migrate accepts only DIRECT_URL, verifies that it authenticates directly as the exact hivegrid_migrator role with no superuser, database-creation, role-creation, inheritance, replication, or BYPASSRLS capability and only the constrained hivegrid_rls_definer membership, runs the fail-closed Prisma-baseline adoption, aliases the URL to DATABASE_URL inside that process for Prisma CLI compatibility, runs prisma migrate deploy, verifies the resulting live schema against the current Prisma schema, runs the global permission-catalog reconcile only after both checks succeed, and then exits. It rejects runtime role markers and every SEED_* value. The migration step is fail-closed: a missing/unreachable credential, a semantic role mismatch, an incomplete or unknown legacy migration history, security-sensitive catalog drift behind a reviewed historical checksum, a failed migration, post-migration schema drift, a missing externally provisioned managed role, or a failed catalog reconcile stops the deployment before any Coolify resource is changed. Run pnpm run db:roles:provision with all five distinct login passwords (HIVEGRID_RLS_MIGRATOR_PASSWORD, HIVEGRID_RLS_APP_PASSWORD, HIVEGRID_RLS_PORTAL_PASSWORD, HIVEGRID_RLS_WORKER_PASSWORD, and HIVEGRID_RLS_AUTHENTIK_JIT_PASSWORD) before the cutover migration; the NOCREATEROLE migrator never creates or alters roles. Long-lived web/API/MCP/Hocuspocus containers receive only the hivegrid_app DSN; the customer-portal runtime receives only hivegrid_portal; workers receive only hivegrid_worker; all set DATABASE_RUNTIME_ROLE and RLS_RUNTIME_MODE=required. The core web resource additionally receives AUTHENTIK_ADMIN_JIT_DATABASE_URL only when Admin SSO is enabled; that URL must authenticate as the execute-only hivegrid_authentik_jit login and must not be copied to the portal, API, MCP, Hocuspocus, or worker resources. DIRECT_URL, MIGRATOR_*, MIGRATE_ON_BOOT, SEED_ON_BOOT, and SEED_* are forbidden in runtime resources. The PR Docker smokes rehearse that same boundary against an empty database. The deploy workflow keeps both Coolify command slots cleared so the dead hook mechanism is not mistaken for an active one. Docker healthchecks use /healthz as process liveness, except the dedicated Hocuspocus image, whose TCP healthcheck covers port 1234. MCP and workers use dependency-free Node cmd healthchecks against their loopback /readyz endpoints.

Der Reconcile bearbeitet ausschließlich globale Permission-Einträge, globale Defaultrollen (tenantId IS NULL) und deren katalogeigene RolePermission-Links. Innerhalb einer kurzen Transaktion erhält hivegrid_rls_definer nur die dafür nötigen DML-Rechte; sie werden vor dem Commit wieder entzogen. FORCE ROW LEVEL SECURITY bleibt durchgehend aktiv. Jede Abweichung der Sicherheitsgrenze oder Daten-Postcondition führt zum Rollback der gesamten Reconcile-Transaktion.

Nach AD-29 sind tenant-eigene Rollen und Rollen mit isTemplateOrigin = true von automatischen Deploy-Änderungen ausgeschlossen. Eine kanonische globale Rolle mit diesem Marker lässt den Reconcile vor Rollen-DML fail-closed abbrechen; Tenant-Rollen bleiben unberührt. Nachzüge für bestehende Tenants sind nur als ausdrücklich freigegebene und auditierte Operator-Aktion zulässig. Der Reconcile ist kein allgemeiner Seed und erteilt dafür keine Freigabe. Seeding is deliberately not part of the non-superuser migrator or any long-lived runtime. Existing bootstrap/demo seed tooling requires a separately reviewed, tenant-scoped operator workflow before it may be used under forced RLS. Do not set SEED_DEMO_DATA=true, ALLOW_STAGING_DEMO_SEED=true, SEED_DEMO_PRO_MANAGED=true, seed credentials, or seed flags on Coolify runtime resources.

The one-shot migrator runs on the CI runner host, not on the database host, so the migrator DSN must point at an address that is routable from the runner. A Coolify container hostname is not: it only resolves inside the target host's docker network. When it is unreachable, scripts/verify-migrator-role.mjs reports the generic PostgreSQL migrator role verification failed — that message is the catch-all for connection errors, not a role-contract violation (a real contract breach reports a specific reason), so check reachability before touching roles. On staging the database is therefore published on the WireGuard mesh address only (10.42.0.3:5432, never 0.0.0.0); see infra/LESSONS.md. Clarify the same reachability question for production before opening the cutover window.

Signing in with a seeded staging demo account is a separate concern from seeding and therefore has its own runtime gate, STAGING_DEMO_LOGIN_ENABLED=true on the staging web resource. It only takes effect together with SENTRY_ENVIRONMENT=staging, so it cannot unlock a demo account in production. The gate previously rode on the SEED_* flags above; because assert-runtime-db-env rejects every SEED_*-prefixed key on a runtime resource, enforcing the deploy contract silently locked the staging demo accounts out until the gate was renamed. Set WORKER_ENABLED=true only on worker Coolify resources. Leave it unset on web/API resources and avoid sharing a production env group that forces WORKER_ENABLED=false into worker containers. The deploy resource list must include separate worker applications with WORKER_QUEUE_GROUP=general, mail, import, webhook, and plugin; the deploy workflow identifies them by the group env marker or by resource names such as worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin. worker-import always owns hivegrid.kwp-imports; generic file imports remain separately gated. Full mode (IMPORT_QUEUE_ENABLED=true + IMPORT_QUEUE_RUNTIME_READY=true) adds hivegrid.imports and hivegrid.exports and requires wired import/export/scan and portal/privacy GDPR services plus complete HTTPS STORAGE_* and canonical GDPR_EXPORT_SSE_C_KEY_BASE64; the mutually exclusive internal PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true mode adds only hivegrid.exports beside KWP and requires the same encrypted storage/GDPR services. The same SSE-C key must be provisioned on core web and portal web before that image is deployed; it is never exposed through a signed-URL redirect. The narrow mode never enables hivegrid.imports or the generic export service. Mixed gates, a non-import group, missing storage/encryption, or missing service wiring fail readiness/startup closed. Treat this deployment paragraph as a summary: docs/ops/worker-queues.md is the canonical worker topology and queue ownership reference. worker-general additionally owns the provider boundary keys LAGO_API_URL, LAGO_API_KEY, LAGO_TIMEOUT_MS, MOLLIE_API_KEY, MOLLIE_PROFILE_ID, and MOLLIE_TIMEOUT_MS, plus the environment-specific BILLING_MOLLIE_WEBHOOK_URL. MOLLIE_PROFILE_ID is optional and both timeouts use their documented defaults when unset. These keys plus PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED are exclusive to worker-general: the Coolify worker reconciler copies them from the approved runtime-env source only into that group, never into worker-mail, worker-import, worker-webhook, or worker-plugin. It removes provider keys left on those workers by older provisioning runs through their Coolify env UUID and fails closed before mutation when a key cannot be removed safely. Provider side effects default to required under RLS_RUNTIME_MODE=required; in that mode the reconciler rejects missing LAGO_API_URL, LAGO_API_KEY, MOLLIE_API_KEY or BILLING_MOLLIE_WEBHOOK_URL and malformed configured values before deploy. During an approved pre-live window, PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false is the only supported override: it disables the subscriber and waives the provider-key preflight without weakening RLS or any other worker dependency. Any value other than true, false or unset fails closed. Provider values remain in Coolify/secret storage and must not be printed by preflight or runbooks. The plugin worker belongs to the topology even while production plugin execution is gated: with PLUGIN_RUNTIME_PRODUCTION_ENABLED unset or false, worker-plugin is an idle service, not evidence that tenant plugin execution is approved. If COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true or COOLIFY_PROVISION_WORKER_GROUPS=true is set with the matching project, server, environment, and destination UUID variables, the deploy workflow creates any missing worker-group applications through the Coolify Docker Image API before patching image tags. The provisioner copies runtime env from an existing worker source app when it creates a group resource, then overwrites the group-specific worker env, including distinct health and admin ports. It also creates or reconciles the enabled cmd healthcheck to the exact immutable SHA used for that deployment, so operators should keep an env-source worker app configured until all group resources have their own Coolify env.

B-012 is not a repository code blocker. The deploy workflow and Docker verifiers cover the repository contract, while the permanent fix for runner Docker network exhaustion is operator-owned host hardening: enlarge Docker default-address-pools on the self-hosted runner and add scheduled cleanup for orphaned Gitea Actions networks.

Deploy secrets, registry credentials, Coolify URLs, UUIDs, personal access tokens, webhook tokens, and other deploy-only values must be supplied through CI secrets or a secret manager. Documentation, workflow logs, tickets, and verification summaries must use placeholder configuration only and must not commit, echo, or paste real URLs, UUIDs, tokens, passwords, registry credentials, or secret-shaped examples. Auto-Mode-safe proof is limited to repository/static checks and the configured CI workflow contract such as pnpm run verify:deploy-workflow, node scripts/verify-gitea-deploy-workflow.mjs, and focused verifier tests. Browser flows, Playwright, Cypress, and E2E checks remain manual exclusions unless a human explicitly approves the exact focused run.

Release and rollback contract

Deployment proof is layered. Static verifiers such as pnpm run verify:deploy-workflow, node scripts/verify-gitea-deploy-workflow.mjs, node scripts/verify-gitea-ci-workflow.mjs and focused tests prove repository shape and image policy; they do not prove a live registry push, Coolify rollout or browser flow.

The production image set is web, api, mcp, worker, hocuspocus, and portal-domain-controller. MCP uses a separate mcp target/resource for hivegrid-mcp, and Hocuspocus uses its dedicated image/resource; neither must be folded into the api or web runtime. Staging uses the same target set with isolated runtime configuration in Coolify. Published image references must stay placeholder-only in docs and tickets, using safe target labels and tag labels rather than real registry hosts, namespaces, project IDs, UUIDs, credentials, or token-shaped examples. Every deployable image needs an immutable full-SHA tag; the staging alias identifies the currently staged image set, and the main alias is updated only by promotion. Operators should inspect workflow output for redacted target names, tag lists, and PASS/FAIL verifier lines only, never raw secrets.

Coolify execution is part of the main-branch stage path and the workflow_dispatch promote path. The workflow may trigger Coolify only after the exact-revision RLS gates and image publication or promotion prerequisites pass. It deploys the migration-bearing web application first, verifies web health/revision, then deploys dependent API, MCP, Hocuspocus, and worker resources by reading endpoint/resource data from configured CI variables and secrets. Hocuspocus acceptance requires Coolify running:healthy from the image TCP healthcheck plus the exact configured image SHA. Documentation must not include Coolify URLs, webhook paths, UUIDs, bearer tokens, passwords, registry credentials, or derived endpoint examples. Pull requests and schedules must not invoke Coolify.

Ad hoc live Docker builds, registry pushes, Coolify runs, browser flows, Playwright, Cypress, and E2E checks outside the configured Gitea deploy workflow are manual runtime activities unless a human explicitly approves the exact focused run. The Gitea deploy workflow may run Docker, registry, Coolify, migration, and web/API health steps automatically; record only redacted evidence: deployment status, image target names, immutable tag IDs, web/API/MCP service and revision health, revision-bound command-health status, worker group/queue summary, API requestId or x-request-id correlation, and queue/job identifiers. Worker live-revision proof uses node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>; MCP uses node apps/mcp/dist/deploy-health.js <40-character lowercase Git SHA>. An absent, disabled, non-cmd, mismatched, or unhealthy command fails the automated rollout; operators must fix the runtime healthcheck contract before accepting the deploy. Do not paste database URLs, Redis URLs, Supabase keys, mail provider credentials, stack traces with secrets, webhook tokens, or container environment values.

Rollback uses immutable image tags instead of mutable guesses. Select the previous known-good full 40-character SHA for the affected target. Web and API may follow the approved manual redeploy path. Before invoking an MCP or worker rollback helper, provide the exact HiveGrid image identity through MCP_IMAGE / WORKER_IMAGE or the complete REGISTRY + IMAGE_NAMESPACE + IMAGE_PREFIX tuple; suffix-only target matching is intentionally rejected. For MCP and worker images built with the revision-bound deployment-health entrypoints, treat docker_registry_image_tag and health_check_command as one atomic configuration unit: changing only the image tag leaves the command bound to the newer revision and must fail closed. Reconcile MCP with node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"; reconcile workers with IMAGE_TAG="$OLD_FULL_SHA" node scripts/coolify-ensure-worker-groups.mjs <staging-or-production>, preserving each resource's exact general, mail, import, webhook, or plugin group. The worker reconciler is fleet-wide: after it succeeds, force-deploy all five worker resources and wait on every exact returned deployment UUID. Never reconcile the old SHA and trigger only one group, because the persisted configuration of the other groups already points at the rollback. MCP is triggered separately and likewise waited by its exact deployment UUID.

There is one bootstrap exception: images from before the revision-bound entrypoints existed do not contain apps/mcp/dist/deploy-health.js, apps/worker/dist/deploy-health.js, or /myapp/.image-revision. Never point the new command at such an image. Use node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$RESOURCE_UUID" "$OLD_FULL_SHA" [worker-group]; the optional group is mandatory for workers. The helper validates the full SHA, rejects a Coolify APP_REVISION override or group drift, atomically sets the old tag, disables command health, clears the command, and rereads the resource fail-closed. Then force-deploy and wait on the exact deployment UUID. This is a degraded, emergency-only bootstrap rollback: Coolify cannot use the disabled command as its candidate gate, so brief downtime is possible. Accept the legacy MCP only after Coolify reports running:healthy and public /readyz reports service=mcp, the exact old revision, and successful dependency checks. Accept each legacy worker only after Coolify reports running:healthy, the live container digest/OCI revision resolves to the old SHA image through the approved read-only host check, node apps/worker/dist/health.js reports the image-baked old revision, enabled processing, exact group, and ready dependencies, and the Coolify APP_REVISION override remains absent. This compatibility path is for rollback only. Re-enable and verify the SHA-bound cmd healthcheck before the next roll-forward to a current image. All identifiers above are placeholders; never paste real UUIDs, endpoints, tokens, or credentials into documentation or logs. Confirm web/API health with GET /healthz and GET /readyz plus the documented API health endpoint. For worker incidents, inspect redacted health output, queue names, failed-job counts, and correlation IDs; pause or drain queues before replaying jobs. If readiness fails, capture only the non-secret diagnostic code, target name, tag, and correlation ID.

Health Baseline Health Baseline Gate

S02 enforces CI-safe static contracts for the web and API liveness/readiness endpoints. The web app exposes GET /healthz through apps/web/app/routes/healthz.tsx as a lightweight liveness route with no database dependency, and GET /readyz through apps/web/app/routes/readyz.tsx as a database-backed readiness route; both return the non-secret runtime service, surface, and image-baked revision, while readiness failures remain redacted. The API exposes unauthenticated GET /healthz and GET /readyz; successful responses include non-secret JSON with service: "api", revision, and requestId, and /readyz returns HTTP 503 with a redacted dependency diagnostic code when readiness fails.

Auto-Mode-safe verification commands:

Command	Purpose
pnpm exec vitest run apps/web/app/routes/healthz.test.ts apps/web/app/routes/readyz.test.ts	Focused web route coverage for liveness/readiness response behavior.
pnpm exec vitest run apps/api/src/app.test.ts apps/api/src/openapi.test.ts	Focused API route and OpenAPI coverage for /healthz and /readyz.
pnpm exec vitest run scripts/verify-health-baseline.test.mjs	Static verifier fixture coverage, including missing web/API endpoint regressions and worker DB-readiness regressions.
node scripts/verify-health-baseline.mjs	Static repository baseline check with deterministic path-specific PASS, OBSERVE, and FAIL lines for web, API, and worker health contracts.

The verifier output must no longer contain web/API observations saying /healthz or /readyz is not present, and worker DB readiness must be reported as an enforced pass/fail baseline rather than an observation. Missing web routes, missing API routes, web liveness database usage, missing web readiness database checks, missing API /readyz 503 dependency diagnostics, and missing worker DB readiness are all baseline failures.

Manual runtime inspection remains separate from Auto-Mode. A human may inspect a deployed web or API runtime with GET /healthz and GET /readyz and may compare the API x-request-id response header with the JSON requestId value. Readiness diagnostics are intentionally non-secret: do not record database URLs, Supabase keys, Redis URLs, tokens, passwords, stack traces, or other secret values in docs, logs, tickets, or verification summaries. Live Docker smokes, browser flows, Cypress, Playwright, and other E2E checks remain manual runtime verification and are not part of this Auto-Mode health baseline gate unless a human explicitly approves the exact focused run.

Worker Docker Target Verification and Inspection

The worker target is statically/package verified by these CI-safe commands:

Command	Purpose
pnpm --filter @hivegrid/worker build	Confirms the worker package emits compiled runtime files for node dist/index.js and node dist/health.js.
pnpm exec vitest run apps/worker/src/index.test.ts apps/worker/src/health.test.ts scripts/verify-docker-baseline.test.mjs scripts/verify-health-baseline.test.mjs	Exercises worker runtime/package tests and static verifier coverage, including worker tsx leakage and DB-readiness regressions.
node scripts/verify-docker-baseline.mjs	Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for worker Docker target copy paths, start scripts, health scripts, and runtime tsx usage.
node scripts/verify-health-baseline.mjs	Enforces that worker health statically reports Redis parse state and DB readiness without requiring live DB access.

Manual Docker smoke verification is intentionally separate from Auto-Mode. A human may run a focused build such as docker build --target worker -t hivegrid-worker:smoke ., then run the image with safe environment key names only, for example WORKER_ENABLED, WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, WORKER_ADMIN_PORT, DATABASE_URL, BULLMQ_REDIS_URL, fallback REDIS_URL, group-specific fail-loud keys, and mail provider keys supplied by the local shell or secret manager rather than written into docs. To inspect the worker healthcheck inside a container, run node apps/worker/dist/health.js; the JSON output is expected to report whether the worker is enabled/configured, whether Redis configuration parses, the DB readiness state, queue names, and redacted dependency codes, without echoing Redis URLs, passwords, database connection strings, stack traces, tokens, or other deployment secret values. Auto-Mode did not run live Docker build/run smoke checks, browser flows, Playwright, Cypress, or E2E commands for S04; the recorded proof is limited to static/package verification.

API Docker Target Verification and Inspection

The API target is statically/package verified by these task-scoped commands:

Command	Purpose
pnpm --filter @hivegrid/api build	Confirms the API package can compile apps/api/src into apps/api/dist, including dist/server.js for the Docker runtime start path.
pnpm exec vitest run scripts/verify-docker-baseline.test.mjs	Exercises the Docker baseline verifier test coverage, including API-target PASS/FAIL cases for required copies and tsx runtime leakage.
node scripts/verify-docker-baseline.mjs	Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for the API target.

Runtime inspection for a manually run API container should use GET /v1/health. A healthy response is expected to identify service: "api" and include a requestId; startup logs should include the structured API listening message with a redacted port field. Safe environment keys that may be supplied by deployment automation include PORT, NODE_ENV, database connection variables, Supabase keys, and JWT/password-hash configuration keys; documentation and logs must not include secret values.

Scope Notes

Docker build/run smoke verification belongs to focused manual target checks. A manual Docker smoke check may optionally build and run the focused API target with docker build --target api ..., start the container, and request GET /v1/health, or build the focused worker target with docker build --target worker ... and inspect node apps/worker/dist/health.js inside the container. Auto-Mode must not run live Docker smoke, Playwright, Cypress, browser, or E2E commands unless a human explicitly approves that exact run. The recorded Auto-Mode verification for S03 and S04 is static/package verification only; it does not claim a live Docker runtime proof.