Refresh HiveGrid Cloud reference snapshot

This commit is contained in:
david committed 2026-08-03 15:31:14 +02:00
1 parent 5e99e14772
commit 7f88026f79
5955 files changed
+1177325 -188362

No files matched your search

@@ -37,13 +37,13 @@ transactional mail, mail outbox, form-submission webhook delivery, and
email-sequence worker contracts are intentionally isolated from general outbox
fanout.
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
| ---------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing | `9103` | `9203` | KWP dependencies are always required; GDPR modes additionally require HTTPS `STORAGE_*`, canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`, and both GDPR services; full mode also needs both `IMPORT_QUEUE_*` gates plus generic import/export/scan services |
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
The detailed queue mapping below is the source of truth for package queues,
subscriber queues, concurrency, and workload profile. Update this page first
@@ -61,24 +61,66 @@ accept plugin invoke consumption as production-ready. The final live proof is
operator-run, inside both `worker-general` and `worker-plugin`:
`pnpm run verify:plugin-runtime-live`.
`worker-import` follows the same placeholder-first principle. Provision the
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
export, and scan services are wired into the worker bootstrap. In that idle
state `/readyz` may be healthy with an empty queue list, which proves topology
without consuming `hivegrid.imports` or `hivegrid.exports`. When both runtime
gates are set, startup fails closed unless the worker has real import, export,
and scan service dependencies.
`worker-import` always consumes `hivegrid.kwp-imports`. KWP provider work is
kept off the generic file-import queue so automatic project catalog and
binding synchronization remain active while tenant file imports are gated.
Keep `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until
generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C
services are wired into the worker bootstrap. In that KWP-only state `/readyz`
is healthy with exactly the dedicated KWP queue. When both generic runtime
gates are set, startup fails closed unless the worker also has real import,
generic export, scan, portal/privacy GDPR service dependencies and the
complete encrypted storage contract described below.
The internal portal/privacy GDPR closeout uses a narrower alternative on the
same `worker-import` resource. With both `IMPORT_QUEUE_*` gates unset or `false`,
`PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` adds `hivegrid.exports` beside the
always-active `hivegrid.kwp-imports` queue and wires only the portal and privacy
GDPR export services. Complete
`STORAGE_ENDPOINT`, `STORAGE_REGION`, `STORAGE_BUCKET`,
`STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, and a canonical
`GDPR_EXPORT_SSE_C_KEY_BASE64` for exactly 32 random bytes is mandatory. The
same key must be present on core web and portal web. Both writers use SSE-C;
both authorized download routes fetch, fully validate and proxy the object server-side, so
the key never enters a redirect, browser header, URL, access log, or ticket.
The dedicated and full modes are mutually exclusive; mixed gates,
another worker group, missing storage, or missing GDPR service wiring fail
closed. Full mode uses the same encrypted GDPR services in addition to generic
imports/exports/scans. Dedicated mode does not activate `hivegrid.imports` or
generic exports; neither mode can disable the isolated KWP queue.
`worker-general` owns tenant hard-delete. In production, required-RLS, and
staging/production Sentry modes it therefore requires both `MEILI_URL` and the
HiveGrid-scoped `MEILI_API_KEY` in addition to its deletion-only storage
credential. A missing search client aborts startup before queue ownership;
provider failures leave the request retryable and logs contain only the error
class, never the provider message or credentials. Local development/test may
skip external search cleanup only while none of those required-runtime markers
is active.
The privacy and portal writers on `worker-import` and tenant hard-delete on
`worker-general` share one distributed tenant-export barrier. Their effective
SharedStore target is the first non-empty value of `REDIS_URL`, then
`BULLMQ_REDIS_URL`; both resources must resolve to the same Redis protocol,
host, port, and database. Credentials may differ. The deploy preflight rejects
missing, invalid, or drifting targets without printing either URL. The general
worker is delete-only and must never receive `GDPR_EXPORT_SSE_C_KEY_BASE64`.
Key rotation is deliberately fail-closed: pause the export queue, wait until
all 72-hour privacy artifacts have expired (the portal window is 24 hours) or
delete those exact expired artifacts through the retention workflow, update
the same key atomically on both web resources and `worker-import`, then resume
and request fresh exports. There is no plaintext or old-key fallback.
## Group → queue mapping
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
| `import` | `hivegrid.imports`, `hivegrid.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
| `import` | Always: `hivegrid.kwp-imports`. Full adds `hivegrid.imports`, `hivegrid.exports` after both `IMPORT_QUEUE_*` gates. Dedicated portal/privacy mode adds only `hivegrid.exports` after `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` | none | 1 | 1 | Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed |
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
Source of truth: `packages/queue/src/groups.ts`. A test
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
@@ -181,22 +223,43 @@ Coolify worker apps before deploy: every managed worker group must have
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
The same general-worker preflight requires the non-empty Lago/Mollie provider
contract (`LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`,
`MOLLIE_PROFILE_ID`, `MOLLIE_TIMEOUT_MS`, `BILLING_MOLLIE_WEBHOOK_URL`) because payment reconciliation and
durable provider side effects run only in that group. Only `LAGO_API_URL`,
`LAGO_API_KEY`, `MOLLIE_API_KEY`, and the environment-specific HTTPS
`BILLING_MOLLIE_WEBHOOK_URL` are mandatory; `MOLLIE_PROFILE_ID` and both timeout
overrides are optional.
| Variable | Purpose |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
Production may hold this subscriber before the external payment gates are
complete by setting
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`.
That explicit value disables the subscriber even under required RLS and waives
only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default,
`true` forces the provider contract, and any other value fails closed. The gate
must not be copied to another worker group.
Ein mehrdeutiger Mollie-Payment-Start ohne persistierte `providerPaymentId` wird vom
Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere,
auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist
in [`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md) beschrieben. Direktes
BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten.
| Variable | Purpose |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
## Coolify resources
@@ -231,7 +294,14 @@ webhook, `9205` plugin) so the admin server does not collide with the health ser
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
applications during the deploy workflow when provisioning is explicitly
enabled. Staging uses:
enabled. It also reconciles every managed worker app to an enabled Coolify
`cmd` healthcheck with
`node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>`
for the exact immutable image revision and worker resource before rollout. That
command verifies the image-baked revision, `APP_REVISION`, enablement, and group,
then probes the running process over loopback `/readyz`. A mutable alias, wrong
queue group, disabled worker, or healthy stale container cannot satisfy the
deploy gate. Staging uses:
```bash
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
@@ -261,6 +331,7 @@ Optional overrides:
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
| `PORTAL_GDPR_EXPORT_QUEUE_ENABLED` | false | Internal exports-only mode for `WORKER_QUEUE_GROUP=import`; both `IMPORT_QUEUE_*` gates must remain unset/false and complete `STORAGE_*`, `GDPR_EXPORT_SSE_C_KEY_BASE64`, plus GDPR service wiring is mandatory |
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
@@ -278,8 +349,10 @@ Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
for required/expected runtime bridges such as `outbox.mail` on
`WORKER_QUEUE_GROUP=mail`.
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
as the liveness probe.
Use `/readyz` as the Kubernetes and manual HTTP readiness probe and `/healthz`
as the liveness probe. Managed Coolify worker apps instead use the revision-bound
`cmd` healthcheck documented above; this avoids relying on HTTP-probe utilities
inside the minimal worker image and binds health to the exact deployed SHA.
Worker health is primarily a queue-runtime check. For managed `worker-mail`
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
@@ -316,13 +389,13 @@ CLIENT LIST
## Scaling cookbook
| Symptom | Action |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
| Symptom | Action |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
## Operator commands
@@ -378,6 +451,56 @@ supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
standalone default is `general`, so removing the marker during rollback can
silently turn a non-general app into another general worker, while an unknown
group fails startup by design. For an incident rollback, redeploy the previous
known-good image/tag or compose/deploy definition, or keep the five group apps
and roll the affected group back independently. In-flight BullMQ job state
lives in Redis, so jobs are not lost while workers are rolled.
known-good full 40-character SHA or compose/deploy definition. The current
reconciler is fleet-wide: it updates all five managed group applications in one
validated pass. A modern rollback must therefore redeploy and verify all five
groups; it is not safe to reconcile the old SHA and trigger only one group,
because a later restart of another group would consume the already-rolled-back
configuration. For images that contain the revision-bound deployment-health
entrypoint, a Coolify rollback
must never patch only `docker_registry_image_tag`: the image tag and
`health_check_command` are one atomic configuration unit. The command must be
`node apps/worker/dist/deploy-health.js <old-full-sha> <existing-worker-group>`,
where the final argument remains the resource's exact existing `general`,
`mail`, `import`, `webhook`, or `plugin` group.
Before the canonical force-deploy is triggered, set `IMAGE_TAG="$OLD_FULL_SHA"`
and provide the exact HiveGrid runtime image identity through `WORKER_IMAGE` and
`MCP_IMAGE`, or through the complete `REGISTRY`, `IMAGE_NAMESPACE`, and
`IMAGE_PREFIX` tuple. The rollback helpers reject suffix-only image matches so
a valid CUID from another product cannot be mutated accidentally. Then run
`node scripts/coolify-ensure-worker-groups.mjs <staging-or-production>` with the
already configured environment-specific Coolify variables. The reconciler
atomically sets and rereads both fields for every managed worker resource and
rejects group, revision, or persistence drift. Only after it succeeds may the
operator run `node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID"`
for each of the five worker resources and wait on every exact returned
deployment UUID. A scoped single-group rollback requires a future dedicated
helper that leaves the other four persisted configurations untouched.
For an MCP rollback, the equivalent prerequisite is
`node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`;
it atomically binds the MCP image tag and command healthcheck to the same old
SHA before the same canonical force-deploy path is used. All identifiers above
are placeholders; never record real resource UUIDs, endpoints, tokens, or
credentials.
Images built before `apps/worker/dist/deploy-health.js` and
`/myapp/.image-revision` were introduced need the bootstrap-compatible legacy
path. Do not run the current reconciler against such an old SHA. Instead run
`node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" <existing-worker-group>`.
The helper rejects mutable tags, Coolify `APP_REVISION` overrides, and group
drift, then atomically sets the old tag, clears `health_check_command`, disables
command health, and rereads the result. Force-deploy only after it succeeds and
wait on the exact returned deployment UUID.
This is a degraded, emergency-only bootstrap rollback: disabling command health
removes it from Coolify's candidate gate, so brief downtime is possible. Accept
the worker only after Coolify reports `running:healthy`, an approved read-only
host check resolves the live container digest/OCI revision to the old SHA image,
and `node apps/worker/dist/health.js` inside that live container reports the
image-baked old revision, enabled processing, the exact existing group, and
ready dependencies. `APP_REVISION` must be absent only as a Coolify runtime
override; the old image's baked environment value must equal the old SHA.
Re-enable and verify the revision-bound `cmd` healthcheck before the next
roll-forward to a current image. In-flight BullMQ job state lives in Redis, so
jobs are not lost while workers are rolled.