135 lines
25 KiB
Markdown
135 lines
25 KiB
Markdown
# Deployment
|
|
|
|
## Docker Baseline Health
|
|
|
|
The current baseline is enforced by static Docker, workflow, health, and artifact checks. Historical Docker Baseline evidence is archived in milestone records; this page lists the commands agents should use for the current repository state.
|
|
|
|
### Baseline Commands
|
|
|
|
The table below is the current command contract and expected result for agents/operators. It is not live-environment proof unless a dated CI or deployment run is attached separately.
|
|
|
|
| Command | Expected exit code | Expected result | Failure fingerprint |
|
|
| --- | ---: | --- | --- |
|
|
| `node scripts/verify-docker-baseline.mjs` | 0 | Pass | Enforces web, API, and worker Docker target shape, runtime start commands, healthcheck paths, and runtime `tsx` exclusions. |
|
|
| `pnpm run lint` | 0 | Pass | ESLint runs with `--max-warnings=0`; warnings are failures. |
|
|
| `pnpm run test` | 0 | Pass | Runs the repository non-watch Vitest/script suite configured in `package.json`. |
|
|
| `pnpm typecheck` | 0 | Pass | Prisma client generation, React Router typegen, and `tsc -b` complete successfully. |
|
|
| `pnpm run verify:production-seed-artifact` | 0 | Pass | Verifies the Coolify production seed artifact build contract and, after build, checks for forbidden bare `@toolctrl/*` runtime imports. |
|
|
|
|
Use focused Vitest files for narrow route, API, worker, and verifier changes; keep browser/E2E checks manual unless explicitly approved for a focused run.
|
|
|
|
### Current Dockerfile Targets and Runtime Commands
|
|
|
|
- `base`: `node:20-bookworm-slim` on `linux/amd64`, production-oriented environment variables, OpenSSL, Corepack, and `pnpm@10.33.0`.
|
|
- `deps`: installs all workspace dependencies from `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`, `prisma`, and each workspace `package.json` manifest with `pnpm install --frozen-lockfile --prod=false --prefer-offline` behind a BuildKit `/pnpm/store` cache mount.
|
|
- `production-deps`: copies the `deps` filesystem and runs `pnpm prune --prod` for the final runtime images.
|
|
- `build`: generates Prisma client, runs the root `pnpm build` command, and runs `pnpm --filter @toolctrl/api build` plus `pnpm --filter @toolctrl/worker build`; the root build produces the React Router build and web server runtime artifact at `build/server-runtime/server.js`, while the API and worker builds produce `apps/api/dist/server.js` and `apps/worker/dist`.
|
|
- `web`: copies production node_modules, `build`, `build/server-runtime`, `apps/web/build`, `apps/web/public`, `package.json`, `start.sh`, workspace package runtime dependencies, and `prisma`; runs `ENTRYPOINT ["./start.sh"]`.
|
|
- `worker`: copies production node_modules, workspace manifests, `apps/worker/package.json`, `apps/worker/dist`, queue/mail/db packages, generated Prisma client artifacts, and `prisma`; sets `WORKER_ENABLED=true`; uses `CMD ["node", "apps/worker/dist/index.js"]`; healthcheck runs `node apps/worker/dist/health.js`.
|
|
- `api`: copies production node_modules, workspace manifests, `apps/api/package.json`, `apps/api/dist`, `packages/config`, `packages/queue`, `packages/db`, `packages/db/generated/prisma/client`, and `prisma`; sets `PORT=8080`; uses `CMD ["node", "apps/api/dist/server.js"]` without `tsx`.
|
|
|
|
Runtime start command observations:
|
|
|
|
- `start.sh` delegates to root `pnpm start`.
|
|
- Root `package.json` defines `start` as `pnpm --filter @toolctrl/web start`; `apps/web/package.json` owns the concrete `node ../../build/server-runtime/server.js` runtime command, so the web Docker target starts the built Production-Runtime artifact instead of `tsx`.
|
|
- `apps/api/package.json` defines `build` as `node scripts/build.mjs` and `start` as `node dist/server.js`; the Docker `api` target depends on that built `apps/api/dist` artifact and on the package-owned Prisma client copied from `packages/db/generated/prisma/client`.
|
|
- `node scripts/verify-docker-baseline.mjs` enforces the web Production-Runtime contract and the S03 API Docker contract, including the explicit Dockerfile `api` target, required API/package/Prisma copy paths, BuildKit pnpm-store cache usage in the `deps` target, and absence of `tsx` from the API runtime start command.
|
|
- `apps/worker/package.json` defines production `start` as `node dist/index.js` and `health` as `node dist/health.js`; the worker Docker target depends on these compiled runtime artifacts and must not rely on `tsx` at container runtime.
|
|
|
|
## Registry Push Baseline Gitea Package Registry Push Contract
|
|
|
|
Registry Push Baseline starts from the existing Gitea CI runner baseline and hardens the S03 deploy workflow contract at `.gitea/workflows/deploy.yml`. The active workflow builds and pushes versioned `web`, `api`, and `worker` Docker image targets to the Gitea Package Registry on the self-hosted `toolctrl-ci` runner, then triggers configured Coolify resources. Web and API resources must pass redacted `/healthz` service-and-revision checks where those resources expose HTTP health URLs; worker resources are separate Coolify applications for `WORKER_QUEUE_GROUP=general`, `mail`, `import`, `webhook`, and `plugin`, and each worker rollout must pass Coolify execute-based `node apps/worker/dist/health.js` readiness with matching revision, enabled worker state, expected group, and expected queue. Image alias/tag checks prove only non-HTTP resource configuration; they are not a pass-open worker health fallback. These static and workflow checks are deployment contract proof; they are still not browser-flow or manual production acceptance evidence.
|
|
|
|
Auto-Mode-safe verification commands:
|
|
|
|
| Command | Expected result |
|
|
| --- | --- |
|
|
| `node scripts/verify-gitea-ci-workflow.mjs` | Passes with every `.gitea` workflow job using `runs-on: toolctrl-ci` and with forbidden browser/E2E deploy commands absent. |
|
|
| `node scripts/verify-gitea-deploy-workflow.mjs` | Emits deterministic PASS/FAIL lines for the `.gitea/workflows/deploy.yml` static contract, including runner, registry-login, image-target, tag-policy, and forbidden-command checks. |
|
|
| `pnpm run verify:deploy-readiness` | Confirms the package scripts, CI workflow, deploy workflow, deployment docs, worker matrix docs, and worker-group provisioner stay consistent while distinguishing the lightweight deploy preflight from CI/readiness coverage for staging/promotion, health, i18n, hot-swap, Docker, seed, and migration gates. |
|
|
| `pnpm exec vitest run scripts/verify-gitea-deploy-workflow.test.mjs` | Exercises verifier fixtures for missing deploy contract pieces, forbidden literal secret exposure, forbidden browser/E2E commands, required Coolify trigger coverage, and required image tag behavior. |
|
|
| `grep -R "ubuntu-latest" .gitea` | Returns no matches; any match means the self-hosted runner baseline has regressed. |
|
|
| `grep -n "verify:deploy-workflow" docs/deployment.md && grep -n "main" docs/deployment.md` | Confirms this deployment document keeps the deploy verifier command and tag-policy language visible. |
|
|
|
|
The package script `verify:deploy-workflow` is the supported shorthand for the deploy workflow verifier. The deploy workflow follows a stage-and-promote contract on the self-hosted `toolctrl-ci` runner. Every deploy job depends on `deploy-preflight`, which intentionally runs only lightweight static deploy wiring checks before any image is published or promoted: the Gitea deploy workflow verifier, deploy-readiness docs/worker/provisioner consistency verifier, Docker baseline, Docker non-root runtime verifier, Coolify helper syntax checks, worker-group provisioner syntax checks, and pre-pull script syntax checks. Broader gates such as API inventory, OpenAPI security, env drift, health baseline, i18n, Phase C boundaries, Host Aware URL baseline, Feature Manifest plugin-removal gates, hot-swap, UI visual, build, and production seed artifact verification belong to the CI/readiness contract rather than the deploy preflight job. Because `.gitea/workflows/ci.yml` intentionally does not run on `push`, production readiness requires Gitea main branch protection: direct pushes to `main` must be disabled and merges to `main` must require the pull-request CI workflow to pass. Without that repository setting, a direct `main` push can stage after only the lightweight deploy preflight. Pushes to `main` then build `web`, `api`, and `worker`, push immutable full-SHA tags plus the `staging` alias, trigger the staging web resource first, wait for web health/revision proof, then trigger dependent API plus `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` resources. Production promotion is a separate `workflow_dispatch` action named `promote`; operators must provide the full staged commit SHA as `staged_sha` and set `backup_confirmed=true` only after production backup or restore-point readiness is confirmed. The workflow verifies that full-SHA image set against `staging`, moves the production `main` alias without rebuilding, configures the production web pre-deployment migration command, deploys the production web resource first, waits for web health/revision proof, verifies production migration status with the database URL scoped to that step, and only then deploys dependent API plus all five worker-group resources. These names identify required CI variable or secret slots only; documentation must not include real deploy endpoint structures, UUIDs, tokens, passwords, credentials, or secret-shaped examples.
|
|
|
|
The deploy image tag contract is static and versioned. Every staging run must publish full-SHA immutable tags for each image target and update the `staging` alias. Promotion must move the `main` alias to the exact full-SHA image set verified by that workflow run; it must not rebuild from source. Images are recorded by safe target labels such as `web`, `api`, and `worker`; do not paste registry URLs, project IDs, credentials, or token-shaped examples into docs or tickets. The mutable `latest` alias is not part of the deployment contract.
|
|
|
|
Coolify deployment is runner-side and follows successful image publication. The `.gitea/workflows/deploy.yml` stage path runs on pushes to `main`; the promote path runs only through `workflow_dispatch` with `action=promote`, `staged_sha=<full 40-character staged SHA>`, and `backup_confirmed=true`. Staging Coolify must be called only after `web`, `api`, and `worker` staging images publish successfully: the workflow patches the staging web resource so `pre_deployment_command` runs Prisma migrations and `post_deployment_command` runs the guarded seed, triggers the staging web resource first, waits for web health/revision proof, then triggers and waits on dependent API plus `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` resources. Production follows the same web-first/dependent-resource order after the runner verifies the requested full-SHA image set and moves the `main` aliases; migration status is confirmed after the production web resource has run its pre-deployment migration command and post-deployment seed command and before dependent API/worker deploys start. The trigger must read endpoint and resource information from configured CI variables and secrets and must not expose deploy URLs, UUIDs, bearer tokens, passwords, or registry credentials in docs, logs, tickets, or verification summaries.
|
|
If the self-hosted runner needs an internal hosts override for the Gitea registry, provide `TOOLCTRL_GITEA_HOST_IP` as a CI variable. The workflows leave `/etc/hosts` untouched when that variable is absent.
|
|
|
|
The web application uses separate Coolify command slots. `pre_deployment_command` must run only reviewed Prisma migrations from the final image: `cd /myapp && pnpm exec prisma migrate deploy --config prisma.config.ts`. `post_deployment_command` must run the guarded production seed artifact from the final image: `cd /myapp && NODE_ENV=production node packages/db/dist/production-seed.server.js`. Docker healthchecks use `/healthz`; `/readyz` remains the stricter readiness endpoint for manual/runtime probes after dependencies are available. That production seed path creates only the configured bootstrap admin plus roles, permissions, and entitlements, and requires `SEED_ADMIN_EMAIL` / `SEED_ADMIN_PASSWORD` in the Coolify environment.
|
|
Staging may opt into the CRM SaaS demo seed through the guarded command `SEED_DEMO_DATA=true ALLOW_STAGING_DEMO_SEED=true NODE_ENV=production node packages/db/dist/production-seed.server.js`. That staging-only path adds demo Starter, Pro, and Business tenants, users, plan assignments, product blueprint configuration, tenant-owned roles, KB fixtures, and support tickets. The staging web runtime must also keep `SEED_DEMO_DATA=true` and `ALLOW_STAGING_DEMO_SEED=true` so those seeded demo users can log in while `NODE_ENV=production`. Do not set `SEED_DEMO_DATA=true` or `ALLOW_STAGING_DEMO_SEED=true` on production resources.
|
|
Set `WORKER_ENABLED=true` only on worker Coolify resources. Leave it unset on web/API resources and avoid sharing a production env group that forces `WORKER_ENABLED=false` into worker containers. The deploy resource list must include separate worker applications with `WORKER_QUEUE_GROUP=general`, `mail`, `import`, `webhook`, and `plugin`; the deploy workflow identifies them by the group env marker or by resource names such as `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin`. `worker-import` is allowed to be a healthy idle topology placeholder with an empty queue list until `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` are backed by wired import, export, and scan services. Treat this deployment paragraph as a summary: `docs/ops/worker-queues.md` is the canonical worker topology and queue ownership reference.
|
|
The plugin worker belongs to the topology even while production plugin
|
|
execution is gated: with `PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`,
|
|
`worker-plugin` is an idle service, not evidence that tenant plugin execution is
|
|
approved.
|
|
If `COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true` or `COOLIFY_PROVISION_WORKER_GROUPS=true` is set with the matching project, server, environment, and destination UUID variables, the deploy workflow creates any missing worker-group applications through the Coolify Docker Image API before patching image tags. The provisioner copies runtime env from an existing worker source app when it creates a group resource, then overwrites the group-specific worker env, including distinct health and admin ports, so operators should keep an env-source worker app configured until all group resources have their own Coolify env.
|
|
|
|
B-012 is not a repository code blocker. The deploy workflow and Docker
|
|
verifiers cover the repository contract, while the permanent fix for runner
|
|
Docker network exhaustion is operator-owned host hardening: enlarge Docker
|
|
`default-address-pools` on the self-hosted runner and add scheduled cleanup for
|
|
orphaned Gitea Actions networks.
|
|
|
|
Deploy secrets, registry credentials, Coolify URLs, UUIDs, personal access tokens, webhook tokens, and other deploy-only values must be supplied through CI secrets or a secret manager. Documentation, workflow logs, tickets, and verification summaries must use placeholder configuration only and must not commit, echo, or paste real URLs, UUIDs, tokens, passwords, registry credentials, or secret-shaped examples. Auto-Mode-safe proof is limited to repository/static checks and the configured CI workflow contract such as `pnpm run verify:deploy-workflow`, `node scripts/verify-gitea-deploy-workflow.mjs`, and focused verifier tests. Browser flows, Playwright, Cypress, and E2E checks remain manual exclusions unless a human explicitly approves the exact focused run.
|
|
|
|
## Production Docs Production Deployment Runbook
|
|
|
|
Production Docs/S04 treats deployment proof as a layered contract. Auto-Mode may run `pnpm run verify:production-docs` to verify the production documentation set, plus existing static deploy verifiers such as `pnpm run verify:deploy-workflow`, `node scripts/verify-gitea-deploy-workflow.mjs`, `node scripts/verify-gitea-ci-workflow.mjs`, and focused verifier tests. These checks prove repository shape, documentation coverage, image-target policy, placeholder usage, and forbidden automatic browser/E2E command exclusions; they do not prove that a live Docker build, registry push, Coolify execution, or browser runtime succeeded.
|
|
|
|
The production image set remains the `web`, `api`, and `worker` Docker targets; staging uses the same target set with isolated runtime configuration in Coolify. Published image references must stay placeholder-only in docs and tickets, using safe target labels and tag labels rather than real registry hosts, namespaces, project IDs, UUIDs, credentials, or token-shaped examples. Every deployable image needs an immutable full-SHA tag; the `staging` alias identifies the currently staged image set, and the `main` alias is updated only by promotion. Operators should inspect workflow output for redacted target names, tag lists, and PASS/FAIL verifier lines only, never raw secrets.
|
|
|
|
Coolify execution is part of the main-branch stage path and the `workflow_dispatch` promote path. The workflow may trigger Coolify only after image publication or promotion prerequisites pass, and it deploys the migration-bearing web application first, verifies web health/revision, then deploys dependent API/worker resources by reading endpoint/resource data from configured CI variables and secrets. Documentation must not include Coolify URLs, webhook paths, UUIDs, bearer tokens, passwords, registry credentials, or derived endpoint examples. Pull requests and schedules must not invoke Coolify.
|
|
|
|
Ad hoc live Docker builds, registry pushes, Coolify runs, browser flows, Playwright, Cypress, and E2E checks outside the configured Gitea deploy workflow are manual runtime activities unless a human explicitly approves the exact focused run. The Gitea deploy workflow may run Docker, registry, Coolify, migration, and web/API health steps automatically; record only redacted evidence: deployment status, image target names, immutable tag IDs, web/API service and revision health, worker execute-health status, worker group/queue summary, API `requestId` or `x-request-id` correlation, and queue/job identifiers. Worker live-revision proof is checked automatically through Coolify execute returning worker health JSON with revision data; if execute is unavailable, empty, or non-JSON, the automated rollout fails closed and operators must fix Coolify execute access before accepting the worker deploy. Do not paste database URLs, Redis URLs, Supabase keys, mail provider credentials, stack traces with secrets, webhook tokens, or container environment values.
|
|
|
|
Rollback uses immutable image tags instead of mutable guesses. Select the previous known-good SHA-derived tag for the affected target, redeploy it through the approved manual path, and confirm web/API health with `GET /healthz` and `GET /readyz` plus the documented API health endpoint while correlating request IDs across logs. For worker incidents, inspect the redacted worker health output, queue names, failed-job counts, and job/request correlation IDs; pause or drain queues through the operational tooling before replaying jobs. If readiness fails, capture the non-secret dependency diagnostic code, target name, tag, and correlation ID, then escalate with secret values omitted.
|
|
|
|
## Health Baseline Health Baseline Gate
|
|
|
|
S02 enforces CI-safe static contracts for the web and API liveness/readiness endpoints. The web app exposes `GET /healthz` through `apps/web/app/routes/healthz.tsx` as a lightweight liveness route with no database dependency, and `GET /readyz` through `apps/web/app/routes/readyz.tsx` as a database-backed readiness route. The API exposes unauthenticated `GET /healthz` and `GET /readyz`; successful responses include non-secret JSON with `service: "api"`, `revision`, and `requestId`, and `/readyz` returns HTTP 503 with a redacted dependency diagnostic code when readiness fails.
|
|
|
|
Auto-Mode-safe verification commands:
|
|
|
|
| Command | Purpose |
|
|
| --- | --- |
|
|
| `pnpm exec vitest run apps/web/app/routes/healthz.test.ts apps/web/app/routes/readyz.test.ts` | Focused web route coverage for liveness/readiness response behavior. |
|
|
| `pnpm exec vitest run apps/api/src/app.test.ts apps/api/src/openapi.test.ts` | Focused API route and OpenAPI coverage for `/healthz` and `/readyz`. |
|
|
| `pnpm exec vitest run scripts/verify-health-baseline.test.mjs` | Static verifier fixture coverage, including missing web/API endpoint regressions and worker DB-readiness regressions. |
|
|
| `node scripts/verify-health-baseline.mjs` | Static repository baseline check with deterministic path-specific `PASS`, `OBSERVE`, and `FAIL` lines for web, API, and worker health contracts. |
|
|
|
|
The verifier output must no longer contain web/API observations saying `/healthz` or `/readyz` is not present, and worker DB readiness must be reported as an enforced pass/fail baseline rather than an observation. Missing web routes, missing API routes, web liveness database usage, missing web readiness database checks, missing API `/readyz` 503 dependency diagnostics, and missing worker DB readiness are all baseline failures.
|
|
|
|
Manual runtime inspection remains separate from Auto-Mode. A human may inspect a deployed web or API runtime with `GET /healthz` and `GET /readyz` and may compare the API `x-request-id` response header with the JSON `requestId` value. Readiness diagnostics are intentionally non-secret: do not record database URLs, Supabase keys, Redis URLs, tokens, passwords, stack traces, or other secret values in docs, logs, tickets, or verification summaries. Live Docker smokes, browser flows, Cypress, Playwright, and other E2E checks remain manual runtime verification and are not part of this Auto-Mode health baseline gate unless a human explicitly approves the exact focused run.
|
|
|
|
### Worker Docker Target Verification and Inspection
|
|
|
|
The worker target is statically/package verified by these CI-safe commands:
|
|
|
|
| Command | Purpose |
|
|
| --- | --- |
|
|
| `pnpm --filter @toolctrl/worker build` | Confirms the worker package emits compiled runtime files for `node dist/index.js` and `node dist/health.js`. |
|
|
| `pnpm exec vitest run apps/worker/src/index.test.ts apps/worker/src/health.test.ts scripts/verify-docker-baseline.test.mjs scripts/verify-health-baseline.test.mjs` | Exercises worker runtime/package tests and static verifier coverage, including worker `tsx` leakage and DB-readiness regressions. |
|
|
| `node scripts/verify-docker-baseline.mjs` | Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for worker Docker target copy paths, start scripts, health scripts, and runtime `tsx` usage. |
|
|
| `node scripts/verify-health-baseline.mjs` | Enforces that worker health statically reports Redis parse state and DB readiness without requiring live DB access. |
|
|
|
|
Manual Docker smoke verification is intentionally separate from Auto-Mode. A human may run a focused build such as `docker build --target worker -t toolctrl-worker:smoke .`, then run the image with safe environment key names only, for example `WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, `WORKER_ADMIN_PORT`, `DATABASE_URL`, `BULLMQ_REDIS_URL`, fallback `REDIS_URL`, group-specific fail-loud keys, and mail provider keys supplied by the local shell or secret manager rather than written into docs. To inspect the worker healthcheck inside a container, run `node apps/worker/dist/health.js`; the JSON output is expected to report whether the worker is enabled/configured, whether Redis configuration parses, the DB readiness state, queue names, and redacted dependency codes, without echoing Redis URLs, passwords, database connection strings, stack traces, tokens, or other deployment secret values. Auto-Mode did not run live Docker build/run smoke checks, browser flows, Playwright, Cypress, or E2E commands for S04; the recorded proof is limited to static/package verification.
|
|
|
|
### API Docker Target Verification and Inspection
|
|
|
|
The API target is statically/package verified by these task-scoped commands:
|
|
|
|
| Command | Purpose |
|
|
| --- | --- |
|
|
| `pnpm --filter @toolctrl/api build` | Confirms the API package can compile `apps/api/src` into `apps/api/dist`, including `dist/server.js` for the Docker runtime start path. |
|
|
| `pnpm exec vitest run scripts/verify-docker-baseline.test.mjs` | Exercises the Docker baseline verifier test coverage, including API-target PASS/FAIL cases for required copies and `tsx` runtime leakage. |
|
|
| `node scripts/verify-docker-baseline.mjs` | Performs the static Dockerfile/package contract check and emits path-specific PASS/FAIL output for the API target. |
|
|
|
|
Runtime inspection for a manually run API container should use `GET /v1/health`. A healthy response is expected to identify `service: "api"` and include a `requestId`; startup logs should include the structured `API listening` message with a redacted `port` field. Safe environment keys that may be supplied by deployment automation include `PORT`, `NODE_ENV`, database connection variables, Supabase keys, and JWT/password-hash configuration keys; documentation and logs must not include secret values.
|
|
|
|
### Scope Notes
|
|
|
|
Docker build/run smoke verification belongs to focused manual target checks. A manual Docker smoke check may optionally build and run the focused API target with `docker build --target api ...`, start the container, and request `GET /v1/health`, or build the focused worker target with `docker build --target worker ...` and inspect `node apps/worker/dist/health.js` inside the container. Auto-Mode must not run live Docker smoke, Playwright, Cypress, browser, or E2E commands unless a human explicitly approves that exact run. The recorded Auto-Mode verification for S03 and S04 is static/package verification only; it does not claim a live Docker runtime proof.
|