Refresh HiveGrid Cloud reference snapshot
This commit is contained in:
1 parent
5e99e14772
commit
7f88026f79
5955 files changed
+1177325
-188362
No files matched your search
@@ -37,13 +37,13 @@ transactional mail, mail outbox, form-submission webhook delivery, and
|
||||
email-sequence worker contracts are intentionally isolated from general outbox
|
||||
fanout.
|
||||
|
||||
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
||||
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
||||
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
||||
| ---------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing | `9103` | `9203` | KWP dependencies are always required; GDPR modes additionally require HTTPS `STORAGE_*`, canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`, and both GDPR services; full mode also needs both `IMPORT_QUEUE_*` gates plus generic import/export/scan services |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
||||
|
||||
The detailed queue mapping below is the source of truth for package queues,
|
||||
subscriber queues, concurrency, and workload profile. Update this page first
|
||||
@@ -61,24 +61,66 @@ accept plugin invoke consumption as production-ready. The final live proof is
|
||||
operator-run, inside both `worker-general` and `worker-plugin`:
|
||||
`pnpm run verify:plugin-runtime-live`.
|
||||
|
||||
`worker-import` follows the same placeholder-first principle. Provision the
|
||||
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
|
||||
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
|
||||
export, and scan services are wired into the worker bootstrap. In that idle
|
||||
state `/readyz` may be healthy with an empty queue list, which proves topology
|
||||
without consuming `hivegrid.imports` or `hivegrid.exports`. When both runtime
|
||||
gates are set, startup fails closed unless the worker has real import, export,
|
||||
and scan service dependencies.
|
||||
`worker-import` always consumes `hivegrid.kwp-imports`. KWP provider work is
|
||||
kept off the generic file-import queue so automatic project catalog and
|
||||
binding synchronization remain active while tenant file imports are gated.
|
||||
Keep `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until
|
||||
generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C
|
||||
services are wired into the worker bootstrap. In that KWP-only state `/readyz`
|
||||
is healthy with exactly the dedicated KWP queue. When both generic runtime
|
||||
gates are set, startup fails closed unless the worker also has real import,
|
||||
generic export, scan, portal/privacy GDPR service dependencies and the
|
||||
complete encrypted storage contract described below.
|
||||
|
||||
The internal portal/privacy GDPR closeout uses a narrower alternative on the
|
||||
same `worker-import` resource. With both `IMPORT_QUEUE_*` gates unset or `false`,
|
||||
`PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` adds `hivegrid.exports` beside the
|
||||
always-active `hivegrid.kwp-imports` queue and wires only the portal and privacy
|
||||
GDPR export services. Complete
|
||||
`STORAGE_ENDPOINT`, `STORAGE_REGION`, `STORAGE_BUCKET`,
|
||||
`STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, and a canonical
|
||||
`GDPR_EXPORT_SSE_C_KEY_BASE64` for exactly 32 random bytes is mandatory. The
|
||||
same key must be present on core web and portal web. Both writers use SSE-C;
|
||||
both authorized download routes fetch, fully validate and proxy the object server-side, so
|
||||
the key never enters a redirect, browser header, URL, access log, or ticket.
|
||||
The dedicated and full modes are mutually exclusive; mixed gates,
|
||||
another worker group, missing storage, or missing GDPR service wiring fail
|
||||
closed. Full mode uses the same encrypted GDPR services in addition to generic
|
||||
imports/exports/scans. Dedicated mode does not activate `hivegrid.imports` or
|
||||
generic exports; neither mode can disable the isolated KWP queue.
|
||||
|
||||
`worker-general` owns tenant hard-delete. In production, required-RLS, and
|
||||
staging/production Sentry modes it therefore requires both `MEILI_URL` and the
|
||||
HiveGrid-scoped `MEILI_API_KEY` in addition to its deletion-only storage
|
||||
credential. A missing search client aborts startup before queue ownership;
|
||||
provider failures leave the request retryable and logs contain only the error
|
||||
class, never the provider message or credentials. Local development/test may
|
||||
skip external search cleanup only while none of those required-runtime markers
|
||||
is active.
|
||||
|
||||
The privacy and portal writers on `worker-import` and tenant hard-delete on
|
||||
`worker-general` share one distributed tenant-export barrier. Their effective
|
||||
SharedStore target is the first non-empty value of `REDIS_URL`, then
|
||||
`BULLMQ_REDIS_URL`; both resources must resolve to the same Redis protocol,
|
||||
host, port, and database. Credentials may differ. The deploy preflight rejects
|
||||
missing, invalid, or drifting targets without printing either URL. The general
|
||||
worker is delete-only and must never receive `GDPR_EXPORT_SSE_C_KEY_BASE64`.
|
||||
|
||||
Key rotation is deliberately fail-closed: pause the export queue, wait until
|
||||
all 72-hour privacy artifacts have expired (the portal window is 24 hours) or
|
||||
delete those exact expired artifacts through the retention workflow, update
|
||||
the same key atomically on both web resources and `worker-import`, then resume
|
||||
and request fresh exports. There is no plaintext or old-key fallback.
|
||||
|
||||
## Group → queue mapping
|
||||
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | `hivegrid.imports`, `hivegrid.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
||||
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | Always: `hivegrid.kwp-imports`. Full adds `hivegrid.imports`, `hivegrid.exports` after both `IMPORT_QUEUE_*` gates. Dedicated portal/privacy mode adds only `hivegrid.exports` after `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` | none | 1 | 1 | Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed |
|
||||
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
|
||||
Source of truth: `packages/queue/src/groups.ts`. A test
|
||||
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
||||
@@ -181,22 +223,43 @@ Coolify worker apps before deploy: every managed worker group must have
|
||||
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
|
||||
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
|
||||
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
|
||||
The same general-worker preflight requires the non-empty Lago/Mollie provider
|
||||
contract (`LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`,
|
||||
`MOLLIE_PROFILE_ID`, `MOLLIE_TIMEOUT_MS`, `BILLING_MOLLIE_WEBHOOK_URL`) because payment reconciliation and
|
||||
durable provider side effects run only in that group. Only `LAGO_API_URL`,
|
||||
`LAGO_API_KEY`, `MOLLIE_API_KEY`, and the environment-specific HTTPS
|
||||
`BILLING_MOLLIE_WEBHOOK_URL` are mandatory; `MOLLIE_PROFILE_ID` and both timeout
|
||||
overrides are optional.
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
||||
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
||||
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
||||
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
||||
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
||||
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
||||
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
||||
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
||||
Production may hold this subscriber before the external payment gates are
|
||||
complete by setting
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`.
|
||||
That explicit value disables the subscriber even under required RLS and waives
|
||||
only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default,
|
||||
`true` forces the provider contract, and any other value fails closed. The gate
|
||||
must not be copied to another worker group.
|
||||
|
||||
Ein mehrdeutiger Mollie-Payment-Start ohne persistierte `providerPaymentId` wird vom
|
||||
Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere,
|
||||
auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist
|
||||
in [`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md) beschrieben. Direktes
|
||||
BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten.
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
||||
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
||||
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
||||
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
||||
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
||||
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
||||
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
||||
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
||||
|
||||
## Coolify resources
|
||||
|
||||
@@ -231,7 +294,14 @@ webhook, `9205` plugin) so the admin server does not collide with the health ser
|
||||
|
||||
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
|
||||
applications during the deploy workflow when provisioning is explicitly
|
||||
enabled. Staging uses:
|
||||
enabled. It also reconciles every managed worker app to an enabled Coolify
|
||||
`cmd` healthcheck with
|
||||
`node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>`
|
||||
for the exact immutable image revision and worker resource before rollout. That
|
||||
command verifies the image-baked revision, `APP_REVISION`, enablement, and group,
|
||||
then probes the running process over loopback `/readyz`. A mutable alias, wrong
|
||||
queue group, disabled worker, or healthy stale container cannot satisfy the
|
||||
deploy gate. Staging uses:
|
||||
|
||||
```bash
|
||||
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
|
||||
@@ -261,6 +331,7 @@ Optional overrides:
|
||||
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
|
||||
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
|
||||
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
|
||||
| `PORTAL_GDPR_EXPORT_QUEUE_ENABLED` | false | Internal exports-only mode for `WORKER_QUEUE_GROUP=import`; both `IMPORT_QUEUE_*` gates must remain unset/false and complete `STORAGE_*`, `GDPR_EXPORT_SSE_C_KEY_BASE64`, plus GDPR service wiring is mandatory |
|
||||
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
|
||||
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
|
||||
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
|
||||
@@ -278,8 +349,10 @@ Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
|
||||
for required/expected runtime bridges such as `outbox.mail` on
|
||||
`WORKER_QUEUE_GROUP=mail`.
|
||||
|
||||
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
|
||||
as the liveness probe.
|
||||
Use `/readyz` as the Kubernetes and manual HTTP readiness probe and `/healthz`
|
||||
as the liveness probe. Managed Coolify worker apps instead use the revision-bound
|
||||
`cmd` healthcheck documented above; this avoids relying on HTTP-probe utilities
|
||||
inside the minimal worker image and binds health to the exact deployed SHA.
|
||||
|
||||
Worker health is primarily a queue-runtime check. For managed `worker-mail`
|
||||
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
|
||||
@@ -316,13 +389,13 @@ CLIENT LIST
|
||||
|
||||
## Scaling cookbook
|
||||
|
||||
| Symptom | Action |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
||||
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
||||
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
||||
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
||||
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
||||
| Symptom | Action |
|
||||
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
||||
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
||||
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
||||
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
||||
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
||||
|
||||
## Operator commands
|
||||
|
||||
@@ -378,6 +451,56 @@ supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
|
||||
standalone default is `general`, so removing the marker during rollback can
|
||||
silently turn a non-general app into another general worker, while an unknown
|
||||
group fails startup by design. For an incident rollback, redeploy the previous
|
||||
known-good image/tag or compose/deploy definition, or keep the five group apps
|
||||
and roll the affected group back independently. In-flight BullMQ job state
|
||||
lives in Redis, so jobs are not lost while workers are rolled.
|
||||
known-good full 40-character SHA or compose/deploy definition. The current
|
||||
reconciler is fleet-wide: it updates all five managed group applications in one
|
||||
validated pass. A modern rollback must therefore redeploy and verify all five
|
||||
groups; it is not safe to reconcile the old SHA and trigger only one group,
|
||||
because a later restart of another group would consume the already-rolled-back
|
||||
configuration. For images that contain the revision-bound deployment-health
|
||||
entrypoint, a Coolify rollback
|
||||
must never patch only `docker_registry_image_tag`: the image tag and
|
||||
`health_check_command` are one atomic configuration unit. The command must be
|
||||
`node apps/worker/dist/deploy-health.js <old-full-sha> <existing-worker-group>`,
|
||||
where the final argument remains the resource's exact existing `general`,
|
||||
`mail`, `import`, `webhook`, or `plugin` group.
|
||||
|
||||
Before the canonical force-deploy is triggered, set `IMAGE_TAG="$OLD_FULL_SHA"`
|
||||
and provide the exact HiveGrid runtime image identity through `WORKER_IMAGE` and
|
||||
`MCP_IMAGE`, or through the complete `REGISTRY`, `IMAGE_NAMESPACE`, and
|
||||
`IMAGE_PREFIX` tuple. The rollback helpers reject suffix-only image matches so
|
||||
a valid CUID from another product cannot be mutated accidentally. Then run
|
||||
`node scripts/coolify-ensure-worker-groups.mjs <staging-or-production>` with the
|
||||
already configured environment-specific Coolify variables. The reconciler
|
||||
atomically sets and rereads both fields for every managed worker resource and
|
||||
rejects group, revision, or persistence drift. Only after it succeeds may the
|
||||
operator run `node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID"`
|
||||
for each of the five worker resources and wait on every exact returned
|
||||
deployment UUID. A scoped single-group rollback requires a future dedicated
|
||||
helper that leaves the other four persisted configurations untouched.
|
||||
For an MCP rollback, the equivalent prerequisite is
|
||||
`node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`;
|
||||
it atomically binds the MCP image tag and command healthcheck to the same old
|
||||
SHA before the same canonical force-deploy path is used. All identifiers above
|
||||
are placeholders; never record real resource UUIDs, endpoints, tokens, or
|
||||
credentials.
|
||||
|
||||
Images built before `apps/worker/dist/deploy-health.js` and
|
||||
`/myapp/.image-revision` were introduced need the bootstrap-compatible legacy
|
||||
path. Do not run the current reconciler against such an old SHA. Instead run
|
||||
`node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" <existing-worker-group>`.
|
||||
The helper rejects mutable tags, Coolify `APP_REVISION` overrides, and group
|
||||
drift, then atomically sets the old tag, clears `health_check_command`, disables
|
||||
command health, and rereads the result. Force-deploy only after it succeeds and
|
||||
wait on the exact returned deployment UUID.
|
||||
|
||||
This is a degraded, emergency-only bootstrap rollback: disabling command health
|
||||
removes it from Coolify's candidate gate, so brief downtime is possible. Accept
|
||||
the worker only after Coolify reports `running:healthy`, an approved read-only
|
||||
host check resolves the live container digest/OCI revision to the old SHA image,
|
||||
and `node apps/worker/dist/health.js` inside that live container reports the
|
||||
image-baked old revision, enabled processing, the exact existing group, and
|
||||
ready dependencies. `APP_REVISION` must be absent only as a Coolify runtime
|
||||
override; the old image's baked environment value must equal the old SHA.
|
||||
Re-enable and verify the revision-bound `cmd` healthcheck before the next
|
||||
roll-forward to a current image. In-flight BullMQ job state lives in Redis, so
|
||||
jobs are not lost while workers are rolled.
|
||||
Reference in new issue
Block a user