HiveGrid-Wireframes/hivegrid-cloud/docs/deployment.md

Deployment

Docker Baseline Health

The current baseline is enforced by static Docker, workflow, health, and artifact checks. Historical Docker Baseline evidence is archived in milestone records; this page lists the commands agents should use for the current repository state.

Baseline Commands

The table below is the current command contract and expected result for agents/operators. It is not live-environment proof unless a dated CI or deployment run is attached separately.

Command	Expected exit code	Expected result	Failure fingerprint
node scripts/verify-docker-baseline.mjs	0	Pass	Enforces web, API, and worker Docker target shape, runtime start commands, healthcheck paths, and runtime tsx exclusions.
pnpm run lint	0	Pass	ESLint runs with --max-warnings=0; warnings are failures.
pnpm run test	0	Pass	Runs the repository non-watch Vitest/script suite configured in package.json.
pnpm typecheck	0	Pass	Prisma client generation, React Router typegen, and tsc -b complete successfully.
pnpm run verify:production-seed-artifact	0	Pass	Verifies the Coolify production seed artifact build contract and, after build, checks for forbidden bare @toolctrl/* runtime imports.

Use focused Vitest files for narrow route, API, worker, and verifier changes; keep browser/E2E checks manual unless explicitly approved for a focused run.

Current Dockerfile Targets and Runtime Commands

• base: node:20-bookworm-slim on linux/amd64, production-oriented environment variables, OpenSSL, Corepack, and pnpm@10.33.0.
• deps: installs all workspace dependencies from package.json, pnpm-lock.yaml, pnpm-workspace.yaml, prisma, and each workspace package.json manifest with pnpm install --frozen-lockfile --prod=false --prefer-offline behind a BuildKit /pnpm/store cache mount.
• production-deps: copies the deps filesystem and runs pnpm prune --prod for the final runtime images.
• build: generates Prisma client, runs the root pnpm build command, and runs pnpm --filter @toolctrl/api build plus pnpm --filter @toolctrl/worker build; the root build produces the React Router build and web server runtime artifact at build/server-runtime/server.js, while the API and worker builds produce apps/api/dist/server.js and apps/worker/dist.
• web: copies production node_modules, build, build/server-runtime, apps/web/build, apps/web/public, package.json, start.sh, workspace package runtime dependencies, and prisma; runs ENTRYPOINT ["./start.sh"].
• worker: copies production node_modules, workspace manifests, apps/worker/package.json, apps/worker/dist, queue/mail/db packages, generated Prisma client artifacts, and prisma; sets WORKER_ENABLED=true; uses CMD ["node", "apps/worker/dist/index.js"]; healthcheck runs node apps/worker/dist/health.js.
• api: copies production node_modules, workspace manifests, apps/api/package.json, apps/api/dist, packages/config, packages/queue, packages/db, packages/db/generated/prisma/client, and prisma; sets PORT=8080; uses CMD ["node", "apps/api/dist/server.js"] without tsx.

Runtime start command observations:

• start.sh delegates to root pnpm start.
• Root package.json defines start as pnpm --filter @toolctrl/web start; apps/web/package.json owns the concrete node ../../build/server-runtime/server.js runtime command, so the web Docker target starts the built Production-Runtime artifact instead of tsx.
• apps/api/package.json defines build as node scripts/build.mjs and start as node dist/server.js; the Docker api target depends on that built apps/api/dist artifact and on the package-owned Prisma client copied from packages/db/generated/prisma/client.
• node scripts/verify-docker-baseline.mjs enforces the web Production-Runtime contract and the S03 API Docker contract, including the explicit Dockerfile api target, required API/package/Prisma copy paths, BuildKit pnpm-store cache usage in the deps target, and absence of tsx from the API runtime start command.
• apps/worker/package.json defines production start as node dist/index.js and health as node dist/health.js; the worker Docker target depends on these compiled runtime artifacts and must not rely on tsx at container runtime.

Registry Push Baseline Gitea Package Registry Push Contract

Registry Push Baseline starts from the existing Gitea CI runner baseline and hardens the S03 deploy workflow contract at .gitea/workflows/deploy.yml. The active workflow builds and pushes versioned web, api, and worker Docker image targets to the Gitea Package Registry on the self-hosted toolctrl-ci runner, then triggers configured Coolify resources. Web and API resources must pass redacted /healthz service-and-revision checks where those resources expose HTTP health URLs; worker resources are separate Coolify applications for WORKER_QUEUE_GROUP=general, mail, import, webhook, and plugin, and each worker rollout must pass Coolify execute-based node apps/worker/dist/health.js readiness with matching revision, enabled worker state, expected group, and expected queue. Image alias/tag checks prove only non-HTTP resource configuration; they are not a pass-open worker health fallback. These static and workflow checks are deployment contract proof; they are still not browser-flow or manual production acceptance evidence.

Auto-Mode-safe verification commands:

Command	Expected result
node scripts/verify-gitea-ci-workflow.mjs	Passes with every .gitea workflow job using runs-on: toolctrl-ci and with forbidden browser/E2E deploy commands absent.
node scripts/verify-gitea-deploy-workflow.mjs	Emits deterministic PASS/FAIL lines for the .gitea/workflows/deploy.yml static contract, including runner, registry-login, image-target, tag-policy, and forbidden-command checks.
pnpm run verify:deploy-readiness	Confirms the package scripts, CI workflow, deploy workflow, deployment docs, worker matrix docs, and worker-group provisioner stay consistent while distinguishing the lightweight deploy preflight from CI/readiness coverage for staging/promotion, health, i18n, hot-swap, Docker, seed, and migration gates.
pnpm exec vitest run scripts/verify-gitea-deploy-workflow.test.mjs	Exercises verifier fixtures for missing deploy contract pieces, forbidden literal secret exposure, forbidden browser/E2E commands, required Coolify trigger coverage, and required image tag behavior.
grep -R "ubuntu-latest" .gitea	Returns no matches; any match means the self-hosted runner baseline has regressed.
grep -n "verify:deploy-workflow" docs/deployment.md && grep -n "main" docs/deployment.md	Confirms this deployment document keeps the deploy verifier command and tag-policy language visible.

The package script verify:deploy-workflow is the supported shorthand for the deploy workflow verifier. The deploy workflow follows a stage-and-promote contract on the self-hosted toolctrl-ci runner. Every deploy job depends on deploy-preflight, which intentionally runs only lightweight static deploy wiring checks before any image is published or promoted: the Gitea deploy workflow verifier, deploy-readiness docs/worker/provisioner consistency verifier, Docker baseline, Docker non-root runtime verifier, Coolify helper syntax checks, worker-group provisioner syntax checks, and pre-pull script syntax checks. Broader gates such as API inventory, OpenAPI security, env drift, health baseline, i18n, Phase C boundaries, Host Aware URL baseline, Feature Manifest plugin-removal gates, hot-swap, UI visual, build, and production seed artifact verification belong to the CI/readiness contract rather than the deploy preflight job. Because .gitea/workflows/ci.yml intentionally does not run on push, production readiness requires Gitea main branch protection: direct pushes to main must be disabled and merges to main must require the pull-request CI workflow to pass. Without that repository setting, a direct main push can stage after only the lightweight deploy preflight. Pushes to main then build web, api, and worker, push immutable full-SHA tags plus the staging alias, trigger the staging web resource first, wait for web health/revision proof, then trigger dependent API plus worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin resources. Production promotion is a separate workflow_dispatch action named promote; operators must provide the full staged commit SHA as staged_sha and set backup_confirmed=true only after production backup or restore-point readiness is confirmed. The workflow verifies that full-SHA image set against staging, moves the production main alias without rebuilding, configures the production web pre-deployment migration command, deploys the production web resource first, waits for web health/revision proof, verifies production migration status with the database URL scoped to that step, and only then deploys dependent API plus all five worker-group resources. These names identify required CI variable or secret slots only; documentation must not include real deploy endpoint structures, UUIDs, tokens, passwords, credentials, or secret-shaped examples.

The deploy image tag contract is static and versioned. Every staging run must publish full-SHA immutable tags for each image target and update the staging alias. Promotion must move the main alias to the exact full-SHA image set verified by that workflow run; it must not rebuild from source. Images are recorded by safe target labels such as web, api, and worker; do not paste registry URLs, project IDs, credentials, or token-shaped examples into docs or tickets. The mutable latest alias is not part of the deployment contract.

Coolify deployment is runner-side and follows successful image publication. The .gitea/workflows/deploy.yml stage path runs on pushes to main; the promote path runs only through workflow_dispatch with action=promote, staged_sha=<full 40-character staged SHA>, and backup_confirmed=true. Staging Coolify must be called only after web, api, and worker staging images publish successfully: the workflow patches the staging web resource so pre_deployment_command runs Prisma migrations and post_deployment_command runs the guarded seed, triggers the staging web resource first, waits for web health/revision proof, then triggers and waits on dependent API plus worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin resources. Production follows the same web-first/dependent-resource order after the runner verifies the requested full-SHA image set and moves the main aliases; migration status is confirmed after the production web resource has run its pre-deployment migration command and post-deployment seed command and before dependent API/worker deploys start. The trigger must read endpoint and resource information from configured CI variables and secrets and must not expose deploy URLs, UUIDs, bearer tokens, passwords, or registry credentials in docs, logs, tickets, or verification summaries. If the self-hosted runner needs an internal hosts override for the Gitea registry, provide TOOLCTRL_GITEA_HOST_IP as a CI variable. The workflows leave /etc/hosts untouched when that variable is absent.

The web application uses separate Coolify command slots. pre_deployment_command must run only reviewed Prisma migrations from the final image: cd /myapp && pnpm exec prisma migrate deploy --config prisma.config.ts. post_deployment_command must run the guarded production seed artifact from the final image: cd /myapp && NODE_ENV=production node packages/db/dist/production-seed.server.js. Docker healthchecks use /healthz; /readyz remains the stricter readiness endpoint for manual/runtime probes after dependencies are available. That production seed path creates only the configured bootstrap admin plus roles, permissions, and entitlements, and requires SEED_ADMIN_EMAIL / SEED_ADMIN_PASSWORD in the Coolify environment. Staging may opt into the CRM SaaS demo seed through the guarded command SEED_DEMO_DATA=true ALLOW_STAGING_DEMO_SEED=true NODE_ENV=production node packages/db/dist/production-seed.server.js. That staging-only path adds demo Starter, Pro, and Business tenants, users, plan assignments, product blueprint configuration, tenant-owned roles, KB fixtures, and support tickets. The staging web runtime must also keep SEED_DEMO_DATA=true and ALLOW_STAGING_DEMO_SEED=true so those seeded demo users can log in while NODE_ENV=production. Do not set SEED_DEMO_DATA=true or ALLOW_STAGING_DEMO_SEED=true on production resources. Set WORKER_ENABLED=true only on worker Coolify resources. Leave it unset on web/API resources and avoid sharing a production env group that forces WORKER_ENABLED=false into worker containers. The deploy resource list must include separate worker applications with WORKER_QUEUE_GROUP=general, mail, import, webhook, and plugin; the deploy workflow identifies them by the group env marker or by resource names such as worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin. worker-import is allowed to be a healthy idle topology placeholder with an empty queue list until IMPORT_QUEUE_ENABLED=true and IMPORT_QUEUE_RUNTIME_READY=true are backed by wired import, export, and scan services. Treat this deployment paragraph as a summary: docs/ops/worker-queues.md is the canonical worker topology and queue ownership reference. The plugin worker belongs to the topology even while production plugin execution is gated: with PLUGIN_RUNTIME_PRODUCTION_ENABLED unset or false, worker-plugin is an idle service, not evidence that tenant plugin execution is approved. If COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true or COOLIFY_PROVISION_WORKER_GROUPS=true is set with the matching project, server, environment, and destination UUID variables, the deploy workflow creates any missing worker-group applications through the Coolify Docker Image API before patching image tags. The provisioner copies runtime env from an existing worker source app when it creates a group resource, then overwrites the group-specific worker env, including distinct health and admin ports, so operators should keep an env-source worker app configured until all group resources have their own Coolify env.

B-012 is not a repository code blocker. The deploy workflow and Docker verifiers cover the repository contract, while the permanent fix for runner Docker network exhaustion is operator-owned host hardening: enlarge Docker default-address-pools on the self-hosted runner and add scheduled cleanup for orphaned Gitea Actions networks.

Deploy secrets, registry credentials, Coolify URLs, UUIDs, personal access tokens, webhook tokens, and other deploy-only values must be supplied through CI secrets or a secret manager. Documentation, workflow logs, tickets, and verification summaries must use placeholder configuration only and must not commit, echo, or paste real URLs, UUIDs, tokens, passwords, registry credentials, or secret-shaped examples. Auto-Mode-safe proof is limited to repository/static checks and the configured CI workflow contract such as pnpm run verify:deploy-workflow, node scripts/verify-gitea-deploy-workflow.mjs, and focused verifier tests. Browser flows, Playwright, Cypress, and E2E checks remain manual exclusions unless a human explicitly approves the exact focused run.

Production Docs Production Deployment Runbook

Production Docs/S04 treats deployment proof as a layered contract. Auto-Mode may run pnpm run verify:production-docs to verify the production documentation set, plus existing static deploy verifiers such as pnpm run verify:deploy-workflow, node scripts/verify-gitea-deploy-workflow.mjs, node scripts/verify-gitea-ci-workflow.mjs, and focused verifier tests. These checks prove repository shape, documentation coverage, image-target policy, placeholder usage, and forbidden automatic browser/E2E command exclusions; they do not prove that a live Docker build, registry push, Coolify execution, or browser runtime succeeded.

The production image set remains the web, api, and worker Docker targets; staging uses the same target set with isolated runtime configuration in Coolify. Published image references must stay placeholder-only in docs and tickets, using safe target labels and tag labels rather than real registry hosts, namespaces, project IDs, UUIDs, credentials, or token-shaped examples. Every deployable image needs an immutable full-SHA tag; the staging alias identifies the currently staged image set, and the main alias is updated only by promotion. Operators should inspect workflow output for redacted target names, tag lists, and PASS/FAIL verifier lines only, never raw secrets.

Coolify execution is part of the main-branch stage path and the workflow_dispatch promote path. The workflow may trigger Coolify only after image publication or promotion prerequisites pass, and it deploys the migration-bearing web application first, verifies web health/revision, then deploys dependent API/worker resources by reading endpoint/resource data from configured CI variables and secrets. Documentation must not include Coolify URLs, webhook paths, UUIDs, bearer tokens, passwords, registry credentials, or derived endpoint examples. Pull requests and schedules must not invoke Coolify.

Ad hoc live Docker builds, registry pushes, Coolify runs, browser flows, Playwright, Cypress, and E2E checks outside the configured Gitea deploy workflow are manual runtime activities unless a human explicitly approves the exact focused run. The Gitea deploy workflow may run Docker, registry, Coolify, migration, and web/API health steps automatically; record only redacted evidence: deployment status, image target names, immutable tag IDs, web/API service and revision health, worker execute-health status, worker group/queue summary, API requestId or x-request-id correlation, and queue/job identifiers. Worker live-revision proof is checked automatically through Coolify execute returning worker health JSON with revision data; if execute is unavailable, empty, or non-JSON, the automated rollout fails closed and operators must fix Coolify execute access before accepting the worker deploy. Do not paste database URLs, Redis URLs, Supabase keys, mail provider credentials, stack traces with secrets, webhook tokens, or container environment values.

Rollback uses immutable image tags instead of mutable guesses. Select the previous known-good SHA-derived tag for the affected target, redeploy it through the approved manual path, and confirm web/API health with GET /healthz and GET /readyz plus the documented API health endpoint while correlating request IDs across logs. For worker incidents, inspect the redacted worker health output, queue names, failed-job counts, and job/request correlation IDs; pause or drain queues through the operational tooling before replaying jobs. If readiness fails, capture the non-secret dependency diagnostic code, target name, tag, and correlation ID, then escalate with secret values omitted.

Health Baseline Health Baseline Gate

S02 enforces CI-safe static contracts for the web and API liveness/readiness endpoints. The web app exposes GET /healthz through apps/web/app/routes/healthz.tsx as a lightweight liveness route with no database dependency, and GET /readyz through apps/web/app/routes/readyz.tsx as a database-backed readiness route. The API exposes unauthenticated GET /healthz and GET /readyz; successful responses include non-secret JSON with service: "api", revision, and requestId, and /readyz returns HTTP 503 with a redacted dependency diagnostic code when readiness fails.

Auto-Mode-safe verification commands:

Command	Purpose
pnpm exec vitest run apps/web/app/routes/healthz.test.ts apps/web/app/routes/readyz.test.ts	Focused web route coverage for liveness/readiness response behavior.
pnpm exec vitest run apps/api/src/app.test.ts apps/api/src/openapi.test.ts	Focused API route and OpenAPI coverage for /healthz and /readyz.
pnpm exec vitest run scripts/verify-health-baseline.test.mjs	Static verifier fixture coverage, including missing web/API endpoint regressions and worker DB-readiness regressions.
node scripts/verify-health-baseline.mjs	Static repository baseline check with deterministic path-specific PASS, OBSERVE, and FAIL lines for web, API, and worker health contracts.

The verifier output must no longer contain web/API observations saying /healthz or /readyz is not present, and worker DB readiness must be reported as an enforced pass/fail baseline rather than an observation. Missing web routes, missing API routes, web liveness database usage, missing web readiness database checks, missing API /readyz 503 dependency diagnostics, and missing worker DB readiness are all baseline failures.

Manual runtime inspection remains separate from Auto-Mode. A human may inspect a deployed web or API runtime with GET /healthz and GET /readyz and may compare the API x-request-id response header with the JSON requestId value. Readiness diagnostics are intentionally non-secret: do not record database URLs, Supabase keys, Redis URLs, tokens, passwords, stack traces, or other secret values in docs, logs, tickets, or verification summaries. Live Docker smokes, browser flows, Cypress, Playwright, and other E2E checks remain manual runtime verification and are not part of this Auto-Mode health baseline gate unless a human explicitly approves the exact focused run.

Worker Docker Target Verification and Inspection

The worker target is statically/package verified by these CI-safe commands:

Command	Purpose
pnpm --filter @toolctrl/worker build	Confirms the worker package emits compiled runtime files for node dist/index.js and node dist/health.js.
pnpm exec vitest run apps/worker/src/index.test.ts apps/worker/src/health.test.ts scripts/verify-docker-baseline.test.mjs scripts/verify-health-baseline.test.mjs	Exercises worker runtime/package tests and static verifier coverage, including worker tsx leakage and DB-readiness regressions.
node scripts/verify-docker-baseline.mjs	Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for worker Docker target copy paths, start scripts, health scripts, and runtime tsx usage.
node scripts/verify-health-baseline.mjs	Enforces that worker health statically reports Redis parse state and DB readiness without requiring live DB access.

Manual Docker smoke verification is intentionally separate from Auto-Mode. A human may run a focused build such as docker build --target worker -t toolctrl-worker:smoke ., then run the image with safe environment key names only, for example WORKER_ENABLED, WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, WORKER_ADMIN_PORT, DATABASE_URL, BULLMQ_REDIS_URL, fallback REDIS_URL, group-specific fail-loud keys, and mail provider keys supplied by the local shell or secret manager rather than written into docs. To inspect the worker healthcheck inside a container, run node apps/worker/dist/health.js; the JSON output is expected to report whether the worker is enabled/configured, whether Redis configuration parses, the DB readiness state, queue names, and redacted dependency codes, without echoing Redis URLs, passwords, database connection strings, stack traces, tokens, or other deployment secret values. Auto-Mode did not run live Docker build/run smoke checks, browser flows, Playwright, Cypress, or E2E commands for S04; the recorded proof is limited to static/package verification.

API Docker Target Verification and Inspection

The API target is statically/package verified by these task-scoped commands:

Command	Purpose
pnpm --filter @toolctrl/api build	Confirms the API package can compile apps/api/src into apps/api/dist, including dist/server.js for the Docker runtime start path.
pnpm exec vitest run scripts/verify-docker-baseline.test.mjs	Exercises the Docker baseline verifier test coverage, including API-target PASS/FAIL cases for required copies and tsx runtime leakage.
node scripts/verify-docker-baseline.mjs	Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for the API target.

Runtime inspection for a manually run API container should use GET /v1/health. A healthy response is expected to identify service: "api" and include a requestId; startup logs should include the structured API listening message with a redacted port field. Safe environment keys that may be supplied by deployment automation include PORT, NODE_ENV, database connection variables, Supabase keys, and JWT/password-hash configuration keys; documentation and logs must not include secret values.

Scope Notes

Docker build/run smoke verification belongs to focused manual target checks. A manual Docker smoke check may optionally build and run the focused API target with docker build --target api ..., start the container, and request GET /v1/health, or build the focused worker target with docker build --target worker ... and inspect node apps/worker/dist/health.js inside the container. Auto-Mode must not run live Docker smoke, Playwright, Cypress, browser, or E2E commands unless a human explicitly approves that exact run. The recorded Auto-Mode verification for S03 and S04 is static/package verification only; it does not claim a live Docker runtime proof.