507 lines
52 KiB
Markdown
507 lines
52 KiB
Markdown
# Worker queue groups — operations guide
|
|
|
|
The standalone `apps/worker` runtime is split into five queue groups. Each
|
|
group is the same image and the same code; the only difference is the
|
|
`WORKER_QUEUE_GROUP` env variable that decides which BullMQ queues the
|
|
process subscribes to.
|
|
|
|
This page is the canonical worker topology reference. Shorter worker tables
|
|
in deployment, environment, or feature docs are summaries only and should link
|
|
back here instead of redefining queue ownership.
|
|
|
|
## Why five groups
|
|
|
|
A single worker process processed every queue. That had three operational
|
|
problems:
|
|
|
|
1. **Head-of-line blocking.** A long-running import (minutes) stalled
|
|
transactional mail and notifications that ran on different queues but
|
|
the same event loop.
|
|
2. **Wrong concurrency for the workload.** Webhook delivery needs high
|
|
parallelism (I/O-bound HTTP calls); imports need low parallelism
|
|
(CPU/memory-bound). One `WORKER_CONCURRENCY` value cannot suit both.
|
|
3. **No per-workload scaling.** When mail queues backlog we couldn't scale
|
|
mail capacity without also duplicating the import worker (waste of RAM
|
|
and Redis connections).
|
|
|
|
The split fixes all three by giving each workload its own process and its
|
|
own concurrency. Add replicas to only the group that needs them.
|
|
|
|
## Production worker matrix
|
|
|
|
Production-like Coolify environments must run one persistent worker service per
|
|
queue group. These are long-lived internal services from the same worker image,
|
|
not manual emergency processes and not public web apps. `worker-mail` with
|
|
`WORKER_QUEUE_GROUP=mail` is part of the baseline topology because
|
|
transactional mail, mail outbox, form-submission webhook delivery, and
|
|
email-sequence worker contracts are intentionally isolated from general outbox
|
|
fanout.
|
|
|
|
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
|
| ---------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
|
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
|
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing | `9103` | `9203` | KWP dependencies are always required; GDPR modes additionally require HTTPS `STORAGE_*`, canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`, and both GDPR services; full mode also needs both `IMPORT_QUEUE_*` gates plus generic import/export/scan services |
|
|
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
|
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
|
|
|
The detailed queue mapping below is the source of truth for package queues,
|
|
subscriber queues, concurrency, and workload profile. Update this page first
|
|
when queue ownership changes; deployment-specific docs should keep only the
|
|
short app/group/port summary and link back here.
|
|
|
|
`worker-plugin` is required as a persistent Coolify service so the deploy
|
|
topology is stable, but production plugin execution is still gated. With
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
|
|
as an idle topology placeholder and must not consume `hivegrid.plugin-invoke`.
|
|
Only after the external WASM sandbox signoff in
|
|
`docs/security/plugin-runtime-signoff.md` may operators set
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
|
|
accept plugin invoke consumption as production-ready. The final live proof is
|
|
operator-run, inside both `worker-general` and `worker-plugin`:
|
|
`pnpm run verify:plugin-runtime-live`.
|
|
|
|
`worker-import` always consumes `hivegrid.kwp-imports`. KWP provider work is
|
|
kept off the generic file-import queue so automatic project catalog and
|
|
binding synchronization remain active while tenant file imports are gated.
|
|
Keep `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until
|
|
generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C
|
|
services are wired into the worker bootstrap. In that KWP-only state `/readyz`
|
|
is healthy with exactly the dedicated KWP queue. When both generic runtime
|
|
gates are set, startup fails closed unless the worker also has real import,
|
|
generic export, scan, portal/privacy GDPR service dependencies and the
|
|
complete encrypted storage contract described below.
|
|
|
|
The internal portal/privacy GDPR closeout uses a narrower alternative on the
|
|
same `worker-import` resource. With both `IMPORT_QUEUE_*` gates unset or `false`,
|
|
`PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` adds `hivegrid.exports` beside the
|
|
always-active `hivegrid.kwp-imports` queue and wires only the portal and privacy
|
|
GDPR export services. Complete
|
|
`STORAGE_ENDPOINT`, `STORAGE_REGION`, `STORAGE_BUCKET`,
|
|
`STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, and a canonical
|
|
`GDPR_EXPORT_SSE_C_KEY_BASE64` for exactly 32 random bytes is mandatory. The
|
|
same key must be present on core web and portal web. Both writers use SSE-C;
|
|
both authorized download routes fetch, fully validate and proxy the object server-side, so
|
|
the key never enters a redirect, browser header, URL, access log, or ticket.
|
|
The dedicated and full modes are mutually exclusive; mixed gates,
|
|
another worker group, missing storage, or missing GDPR service wiring fail
|
|
closed. Full mode uses the same encrypted GDPR services in addition to generic
|
|
imports/exports/scans. Dedicated mode does not activate `hivegrid.imports` or
|
|
generic exports; neither mode can disable the isolated KWP queue.
|
|
|
|
`worker-general` owns tenant hard-delete. In production, required-RLS, and
|
|
staging/production Sentry modes it therefore requires both `MEILI_URL` and the
|
|
HiveGrid-scoped `MEILI_API_KEY` in addition to its deletion-only storage
|
|
credential. A missing search client aborts startup before queue ownership;
|
|
provider failures leave the request retryable and logs contain only the error
|
|
class, never the provider message or credentials. Local development/test may
|
|
skip external search cleanup only while none of those required-runtime markers
|
|
is active.
|
|
|
|
The privacy and portal writers on `worker-import` and tenant hard-delete on
|
|
`worker-general` share one distributed tenant-export barrier. Their effective
|
|
SharedStore target is the first non-empty value of `REDIS_URL`, then
|
|
`BULLMQ_REDIS_URL`; both resources must resolve to the same Redis protocol,
|
|
host, port, and database. Credentials may differ. The deploy preflight rejects
|
|
missing, invalid, or drifting targets without printing either URL. The general
|
|
worker is delete-only and must never receive `GDPR_EXPORT_SSE_C_KEY_BASE64`.
|
|
|
|
Key rotation is deliberately fail-closed: pause the export queue, wait until
|
|
all 72-hour privacy artifacts have expired (the portal window is 24 hours) or
|
|
delete those exact expired artifacts through the retention workflow, update
|
|
the same key atomically on both web resources and `worker-import`, then resume
|
|
and request fresh exports. There is no plaintext or old-key fallback.
|
|
|
|
## Group → queue mapping
|
|
|
|
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
|
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
|
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
|
| `import` | Always: `hivegrid.kwp-imports`. Full adds `hivegrid.imports`, `hivegrid.exports` after both `IMPORT_QUEUE_*` gates. Dedicated portal/privacy mode adds only `hivegrid.exports` after `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` | none | 1 | 1 | Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed |
|
|
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
|
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
|
|
|
Source of truth: `packages/queue/src/groups.ts`. A test
|
|
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
|
group — adding a new package queue forces a deliberate group assignment.
|
|
Outbox and auxiliary subscriber queues are wired by `apps/worker/src/index.ts`
|
|
through their dedicated bootstrap modules; update this table when adding a
|
|
new subscriber that is not represented in `packages/queue`.
|
|
|
|
`outbox.workflow` runs on the `general` group. The worker performs
|
|
live-workflow lookup and preflight checks in `apps/worker/src/jobs`, then
|
|
starts matching workflows through the selected executor: signed internal
|
|
workflow-run API by default, or hybrid/native execution when explicitly
|
|
configured. The native executor is a credential-free safe-core path: it must
|
|
not hydrate stored workflow credentials, and any `$credentials.*`,
|
|
`lookup $credentials`, or `#with $credentials` reference is a preflight gap.
|
|
That bridge avoids importing `apps/web` server modules into the worker
|
|
while keeping the existing workflow execution semantics centralized. The
|
|
same receiver also drains `workflow.execution.requested` rows produced by
|
|
manual workflow runs and tenant API workflow runs: the web action creates the
|
|
`WorkflowExecution` and outbox row in one transaction, then worker-general
|
|
starts that existing execution through the signed internal execution-start
|
|
bridge or the native safe-core executor. Waiting execution resumes publish
|
|
`workflow.execution.continue.requested`; worker-general then resumes the
|
|
existing execution through the signed internal continuation bridge or, in
|
|
native/hybrid mode, through the worker-native safe-core continuation path when
|
|
the workflow's only native preflight gap is the current `waitForInput` boundary.
|
|
Unsupported continuation graphs still fall back to the HTTP bridge in `hybrid`
|
|
mode and fail loud in `native` mode. Stream runs remain separate until their
|
|
progress contract moves to a distributed execution path. The
|
|
subscriber/bootstrap path is active from `apps/worker/src/index.ts` when the
|
|
`general` worker is configured with the workflow execution settings.
|
|
Set `OUTBOX_DISPATCHER_ENABLED=true`, `OUTBOX_DISPATCHER_REQUIRED=true`, and
|
|
`WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` on staging/production `worker-general`
|
|
once Redis plus the bridge URL/token are provisioned; in that mode a missing
|
|
dispatcher or bridge dependency fails startup instead of leaving outbox rows or
|
|
`outbox.workflow` silently undrained behind a green process.
|
|
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
|
|
call-sites stay absent and that only the signed internal trigger bridge may
|
|
start `type: "trigger"` executions directly. The generic entity-builder's
|
|
`row.created` / `row.updated` / `row.deleted` route-backed producers were
|
|
**removed with the entity-builder** (R-P6B-β, AD-2); typed fachentities and
|
|
workflow row blocks (via the typed model registry) no longer emit generic
|
|
`row.*` outbox events. `workflow.trigger.event` application events still publish
|
|
through `EventsService`. Other internal requestless mutations remain eventless by
|
|
default to avoid worker feedback loops.
|
|
|
|
`worker-general` also registers the
|
|
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
|
|
old manual/API `WorkflowExecution` rows that are still `running`, have no block
|
|
runs, have no waiting block, and whose matching
|
|
`workflow.execution.requested` outbox event with dispatch key
|
|
`workflow-execution:<executionId>` is already `FAILED`. It does not infer
|
|
failure from `DONE` outbox rows because `DONE` only proves dispatch into BullMQ,
|
|
not workflow completion.
|
|
|
|
Use `pnpm run verify:workflow-outbox-native` as the canonical non-browser gate
|
|
before changing workflow outbox/native execution readiness. The gate composes
|
|
existing focused tests for `outbox.workflow` routing, subscriber bootstrap,
|
|
native/hybrid safe-core execution, worker health required-mode checks, and
|
|
queued manual/API/continuation producer inventory. It does not prove a live
|
|
environment is draining Redis jobs; pair it with the opt-in integration test or
|
|
operator-gated E2E drain probe for live worker readiness.
|
|
|
|
Form submissions use a two-hop worker path. The public submit route stores
|
|
the `FormSubmission` row and emits `form.submission.process`, which the
|
|
outbox dispatcher routes to `outbox.form-submission-route` on
|
|
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
|
|
marks the submission, and publishes follow-up events. Webhook follow-ups use
|
|
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
|
|
`worker-mail`, where the bridge enqueues the existing `hivegrid.email`
|
|
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
|
|
throughput group and leaves the general group responsible only for the
|
|
database-backed routing decision.
|
|
|
|
## Env variables per group
|
|
|
|
Required (per worker container):
|
|
|
|
| Variable | Purpose |
|
|
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKER_QUEUE_GROUP` | `general` / `mail` / `import` / `webhook` / `plugin` |
|
|
| `WORKER_HEALTH_PORT` | TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105 |
|
|
| `BULLMQ_REDIS_URL` | Redis target |
|
|
| `DATABASE_URL` | Postgres (via PgBouncer in staging/production) |
|
|
|
|
Required only on `worker-general`. The default execution adapter still bridges
|
|
through the web runtime; `hybrid` canaries credential-free safe core
|
|
event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped
|
|
rowGet/email blocks, and single-boundary `waitForInput` continuation resumes,
|
|
then falls back to the signed HTTP bridge for unsupported blocks or credential
|
|
references; `native` is reserved for credential-free safe-core-only deployments
|
|
and fails loud for unsupported blocks or credential references.
|
|
The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless
|
|
platform workflow trigger execution is intentionally unsupported in HTTP mode
|
|
and should use native mode once the workflow is within the supported block
|
|
contract.
|
|
`scripts/coolify-ensure-worker-groups.mjs` verifies these settings on the
|
|
Coolify worker apps before deploy: every managed worker group must have
|
|
`DATABASE_URL` and `BULLMQ_REDIS_URL` configured, and `worker-general` must
|
|
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
|
|
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
|
|
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
|
|
The same general-worker preflight requires the non-empty Lago/Mollie provider
|
|
contract (`LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`,
|
|
`MOLLIE_PROFILE_ID`, `MOLLIE_TIMEOUT_MS`, `BILLING_MOLLIE_WEBHOOK_URL`) because payment reconciliation and
|
|
durable provider side effects run only in that group. Only `LAGO_API_URL`,
|
|
`LAGO_API_KEY`, `MOLLIE_API_KEY`, and the environment-specific HTTPS
|
|
`BILLING_MOLLIE_WEBHOOK_URL` are mandatory; `MOLLIE_PROFILE_ID` and both timeout
|
|
overrides are optional.
|
|
|
|
Production may hold this subscriber before the external payment gates are
|
|
complete by setting
|
|
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`.
|
|
That explicit value disables the subscriber even under required RLS and waives
|
|
only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default,
|
|
`true` forces the provider contract, and any other value fails closed. The gate
|
|
must not be copied to another worker group.
|
|
|
|
Ein mehrdeutiger Mollie-Payment-Start ohne persistierte `providerPaymentId` wird vom
|
|
Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere,
|
|
auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist
|
|
in [`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md) beschrieben. Direktes
|
|
BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten.
|
|
|
|
| Variable | Purpose |
|
|
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
|
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
|
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
|
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
|
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
|
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
|
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
|
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
|
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
|
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
|
|
|
## Coolify resources
|
|
|
|
Coolify must run queue groups as separate persistent applications that use
|
|
the same `worker` image with different `WORKER_QUEUE_GROUP` values. The
|
|
deploy workflow requires `worker-general`, `worker-mail`,
|
|
`worker-import`, `worker-webhook`, and `worker-plugin` resources in the staging and
|
|
production resource allow-lists. `worker-mail` is a permanent service, not a
|
|
manual workaround: without it, queued transactional mail and mail-backed
|
|
outbox/webhook work is not drained after deploy. Without the other group
|
|
resources, production traffic falls back into manual starts.
|
|
This is an operator-owned runtime prerequisite: the repository and static
|
|
verifiers document the required topology, but they do not by themselves prove
|
|
that a target Coolify environment currently has every worker group deployed and
|
|
healthy.
|
|
The same operator/code split applies to plugin runtime status: docs and code
|
|
require the `worker-plugin` service, while tenant-facing production execution
|
|
remains blocked on external signoff and the explicit
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` flip. A live rollout must additionally
|
|
pass `pnpm run verify:plugin-runtime-live` in both `worker-general` and
|
|
`worker-plugin` so the general fanout/audit path and isolated invoke worker are
|
|
checked separately.
|
|
|
|
Name the resources with the group marker (`worker-general`,
|
|
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) or expose the
|
|
`WORKER_QUEUE_GROUP=<group>` marker in the resource metadata returned by
|
|
Coolify. Keep `WORKER_ENABLED=true` on each worker resource and assign
|
|
distinct health ports (`9101` general, `9102` mail, `9103` import,
|
|
`9104` webhook, `9105` plugin). When metrics/admin endpoints are enabled, keep them on
|
|
separate ports (`9201` general, `9202` mail, `9203` import, `9204`
|
|
webhook, `9205` plugin) so the admin server does not collide with the health server.
|
|
|
|
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
|
|
applications during the deploy workflow when provisioning is explicitly
|
|
enabled. It also reconciles every managed worker app to an enabled Coolify
|
|
`cmd` healthcheck with
|
|
`node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>`
|
|
for the exact immutable image revision and worker resource before rollout. That
|
|
command verifies the image-baked revision, `APP_REVISION`, enablement, and group,
|
|
then probes the running process over loopback `/readyz`. A mutable alias, wrong
|
|
queue group, disabled worker, or healthy stale container cannot satisfy the
|
|
deploy gate. Staging uses:
|
|
|
|
```bash
|
|
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
|
|
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
|
|
COOLIFY_STAGING_SERVER_UUID=<server uuid>
|
|
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
|
|
COOLIFY_STAGING_ENVIRONMENT_NAME=staging
|
|
```
|
|
|
|
Production uses the same keys without `STAGING_`:
|
|
`COOLIFY_PROVISION_WORKER_GROUPS`, `COOLIFY_PROJECT_UUID`,
|
|
`COOLIFY_SERVER_UUID`, `COOLIFY_DESTINATION_UUID`, and
|
|
`COOLIFY_ENVIRONMENT_NAME`. When it creates a new group app, the script
|
|
copies the existing worker app env from an allowed non-group worker
|
|
resource or from `COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID` /
|
|
`COOLIFY_WORKER_ENV_SOURCE_UUID`, then overwrites only the group-specific
|
|
settings (`WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`,
|
|
`WORKER_ADMIN_PORT`, and group concurrency). Keep that source worker app
|
|
available until every group resource has its own runtime env.
|
|
|
|
Optional overrides:
|
|
|
|
| Variable | Default | Notes |
|
|
| ------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKER_CONCURRENCY_GENERAL` | 10 | Only read when `WORKER_QUEUE_GROUP=general` |
|
|
| `WORKER_CONCURRENCY_MAIL` | 5 | Only read when `WORKER_QUEUE_GROUP=mail` |
|
|
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
|
|
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
|
|
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
|
|
| `PORTAL_GDPR_EXPORT_QUEUE_ENABLED` | false | Internal exports-only mode for `WORKER_QUEUE_GROUP=import`; both `IMPORT_QUEUE_*` gates must remain unset/false and complete `STORAGE_*`, `GDPR_EXPORT_SSE_C_KEY_BASE64`, plus GDPR service wiring is mandatory |
|
|
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
|
|
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
|
|
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
|
|
| `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `false`; managed Coolify `worker-mail` uses `true` | Mail-group only. Set to `true` in staging/production so `worker-mail` exits if the `outbox.mail` bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed `worker-mail` apps. |
|
|
|
|
## Health endpoints
|
|
|
|
Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
|
|
|
|
- `GET /healthz` — liveness. Returns 200 as long as the event loop is
|
|
responsive. No external dependency calls; safe to hammer.
|
|
- `GET /readyz` — readiness. Runs a Redis ping and a Prisma `SELECT 1`
|
|
in parallel with a 2-second timeout each. Returns 200 only when both
|
|
succeed; 503 otherwise. Queue-group subscriber probes are also included
|
|
for required/expected runtime bridges such as `outbox.mail` on
|
|
`WORKER_QUEUE_GROUP=mail`.
|
|
|
|
Use `/readyz` as the Kubernetes and manual HTTP readiness probe and `/healthz`
|
|
as the liveness probe. Managed Coolify worker apps instead use the revision-bound
|
|
`cmd` healthcheck documented above; this avoids relying on HTTP-probe utilities
|
|
inside the minimal worker image and binds health to the exact deployed SHA.
|
|
|
|
Worker health is primarily a queue-runtime check. For managed `worker-mail`
|
|
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
|
|
fail loud when the required outbound provider pair `MAILJET_API_KEY` and
|
|
`MAILJET_FROM_EMAIL` is absent, using key names only. A healthy
|
|
`worker-mail` therefore proves Redis/Postgres reachability, the mail queue
|
|
process, the transactional outbox bridge, and required Mailjet env presence;
|
|
it still does not prove sender-domain verification or live provider delivery.
|
|
Managed Coolify `worker-mail` apps must keep
|
|
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
|
|
`outbox.mail` cannot subscribe; without that bridge, invite and other
|
|
`mail.transactional.*` outbox events can remain pending even if the direct
|
|
`hivegrid.email` worker is alive. The Coolify worker-group provisioner writes
|
|
and drift-checks this env value for managed mail workers.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
curl -fsS http://worker-mail:9102/healthz
|
|
curl -fsS http://worker-mail:9102/readyz
|
|
```
|
|
|
|
## Connection naming
|
|
|
|
Each worker tags its ioredis connection with `worker-<group>-<pid>`
|
|
(via `apps/worker/src/connection.ts`). Run `CLIENT LIST` against Redis
|
|
to attribute traffic to a specific worker group:
|
|
|
|
```redis
|
|
CLIENT LIST
|
|
# id=42 name=worker-mail-218 ...
|
|
# id=43 name=worker-webhook-219 ...
|
|
```
|
|
|
|
## Scaling cookbook
|
|
|
|
| Symptom | Action |
|
|
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
|
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
|
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
|
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
|
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
|
|
|
## Operator commands
|
|
|
|
```bash
|
|
# Inspect all worker-group logs
|
|
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin
|
|
|
|
# Restart a single group
|
|
docker compose restart worker-import
|
|
|
|
# Smoke-test all five readiness endpoints
|
|
for port in 9101 9102 9103 9104 9105; do
|
|
curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
|
|
done
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
**Wrong or missing worker group.** The worker Docker image defaults
|
|
`WORKER_QUEUE_GROUP` to `general` for raw image users. Coolify group apps
|
|
must still set the explicit group marker (`general`, `mail`, `import`,
|
|
`webhook`, or `plugin`) so each app drains the intended queues. A missing
|
|
Coolify override can therefore start as another `general` worker instead of a
|
|
mail/import/webhook/plugin worker; `scripts/coolify-ensure-worker-groups.mjs`
|
|
treats that as env drift. An unknown group still fails worker startup when the
|
|
worker is enabled. Disabled workers (`WORKER_ENABLED=false`) skip this check so
|
|
dev environments without Redis still boot.
|
|
|
|
**`/readyz` returns 503 with `checks.redis="error"`** — BullMQ couldn't
|
|
reach Redis. Confirm `BULLMQ_REDIS_URL` and that the redis container is
|
|
healthy. The error is intentionally short (no host names) so the probe
|
|
response never leaks credentials.
|
|
|
|
**`/readyz` returns 503 with `checks.database="error"`** — Prisma
|
|
couldn't run `SELECT 1`. Check `DATABASE_URL`, PgBouncer health, and
|
|
whether the pool is saturated.
|
|
|
|
**A queue is silent on every worker** — verify the queue is mapped to a
|
|
group in `packages/queue/src/groups.ts`. If you just added a new queue,
|
|
the `groups.test.ts` exhaustiveness check should have failed CI; if it
|
|
didn't, the queue is missing from `names.ts`.
|
|
|
|
**Outbox dispatcher missing.** The outbox dispatcher runs on the
|
|
`general` group only (see `apps/worker/src/index.ts`). If your stack
|
|
runs no `general` replica, outbox events will not dispatch — keep at
|
|
least one `general` replica.
|
|
|
|
## Rollback
|
|
|
|
Current production worker topology requires an explicit `WORKER_QUEUE_GROUP`
|
|
on each Coolify group app whenever `WORKER_ENABLED=true`; there is no
|
|
supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
|
|
standalone default is `general`, so removing the marker during rollback can
|
|
silently turn a non-general app into another general worker, while an unknown
|
|
group fails startup by design. For an incident rollback, redeploy the previous
|
|
known-good full 40-character SHA or compose/deploy definition. The current
|
|
reconciler is fleet-wide: it updates all five managed group applications in one
|
|
validated pass. A modern rollback must therefore redeploy and verify all five
|
|
groups; it is not safe to reconcile the old SHA and trigger only one group,
|
|
because a later restart of another group would consume the already-rolled-back
|
|
configuration. For images that contain the revision-bound deployment-health
|
|
entrypoint, a Coolify rollback
|
|
must never patch only `docker_registry_image_tag`: the image tag and
|
|
`health_check_command` are one atomic configuration unit. The command must be
|
|
`node apps/worker/dist/deploy-health.js <old-full-sha> <existing-worker-group>`,
|
|
where the final argument remains the resource's exact existing `general`,
|
|
`mail`, `import`, `webhook`, or `plugin` group.
|
|
|
|
Before the canonical force-deploy is triggered, set `IMAGE_TAG="$OLD_FULL_SHA"`
|
|
and provide the exact HiveGrid runtime image identity through `WORKER_IMAGE` and
|
|
`MCP_IMAGE`, or through the complete `REGISTRY`, `IMAGE_NAMESPACE`, and
|
|
`IMAGE_PREFIX` tuple. The rollback helpers reject suffix-only image matches so
|
|
a valid CUID from another product cannot be mutated accidentally. Then run
|
|
`node scripts/coolify-ensure-worker-groups.mjs <staging-or-production>` with the
|
|
already configured environment-specific Coolify variables. The reconciler
|
|
atomically sets and rereads both fields for every managed worker resource and
|
|
rejects group, revision, or persistence drift. Only after it succeeds may the
|
|
operator run `node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID"`
|
|
for each of the five worker resources and wait on every exact returned
|
|
deployment UUID. A scoped single-group rollback requires a future dedicated
|
|
helper that leaves the other four persisted configurations untouched.
|
|
For an MCP rollback, the equivalent prerequisite is
|
|
`node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`;
|
|
it atomically binds the MCP image tag and command healthcheck to the same old
|
|
SHA before the same canonical force-deploy path is used. All identifiers above
|
|
are placeholders; never record real resource UUIDs, endpoints, tokens, or
|
|
credentials.
|
|
|
|
Images built before `apps/worker/dist/deploy-health.js` and
|
|
`/myapp/.image-revision` were introduced need the bootstrap-compatible legacy
|
|
path. Do not run the current reconciler against such an old SHA. Instead run
|
|
`node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" <existing-worker-group>`.
|
|
The helper rejects mutable tags, Coolify `APP_REVISION` overrides, and group
|
|
drift, then atomically sets the old tag, clears `health_check_command`, disables
|
|
command health, and rereads the result. Force-deploy only after it succeeds and
|
|
wait on the exact returned deployment UUID.
|
|
|
|
This is a degraded, emergency-only bootstrap rollback: disabling command health
|
|
removes it from Coolify's candidate gate, so brief downtime is possible. Accept
|
|
the worker only after Coolify reports `running:healthy`, an approved read-only
|
|
host check resolves the live container digest/OCI revision to the old SHA image,
|
|
and `node apps/worker/dist/health.js` inside that live container reports the
|
|
image-baked old revision, enabled processing, the exact existing group, and
|
|
ready dependencies. `APP_REVISION` must be absent only as a Coolify runtime
|
|
override; the old image's baked environment value must equal the old SHA.
|
|
Re-enable and verify the revision-bound `cmd` healthcheck before the next
|
|
roll-forward to a current image. In-flight BullMQ job state lives in Redis, so
|
|
jobs are not lost while workers are rolled.
|