# Worker queue groups — operations guide The standalone `apps/worker` runtime is split into five queue groups. Each group is the same image and the same code; the only difference is the `WORKER_QUEUE_GROUP` env variable that decides which BullMQ queues the process subscribes to. This page is the canonical worker topology reference. Shorter worker tables in deployment, environment, or feature docs are summaries only and should link back here instead of redefining queue ownership. ## Why five groups A single worker process processed every queue. That had three operational problems: 1. **Head-of-line blocking.** A long-running import (minutes) stalled transactional mail and notifications that ran on different queues but the same event loop. 2. **Wrong concurrency for the workload.** Webhook delivery needs high parallelism (I/O-bound HTTP calls); imports need low parallelism (CPU/memory-bound). One `WORKER_CONCURRENCY` value cannot suit both. 3. **No per-workload scaling.** When mail queues backlog we couldn't scale mail capacity without also duplicating the import worker (waste of RAM and Redis connections). The split fixes all three by giving each workload its own process and its own concurrency. Add replicas to only the group that needs them. ## Production worker matrix Production-like Coolify environments must run one persistent worker service per queue group. These are long-lived internal services from the same worker image, not manual emergency processes and not public web apps. `worker-mail` with `WORKER_QUEUE_GROUP=mail` is part of the baseline topology because transactional mail, mail outbox, form-submission webhook delivery, and email-sequence worker contracts are intentionally isolated from general outbox fanout. | Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings | | ---------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | | `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production | | `worker-import` | `WORKER_QUEUE_GROUP=import` | Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing | `9103` | `9203` | KWP dependencies are always required; GDPR modes additionally require HTTPS `STORAGE_*`, canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`, and both GDPR services; full mode also needs both `IMPORT_QUEUE_*` gates plus generic import/export/scan services | | `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness | | `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` | The detailed queue mapping below is the source of truth for package queues, subscriber queues, concurrency, and workload profile. Update this page first when queue ownership changes; deployment-specific docs should keep only the short app/group/port summary and link back here. `worker-plugin` is required as a persistent Coolify service so the deploy topology is stable, but production plugin execution is still gated. With `PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts as an idle topology placeholder and must not consume `hivegrid.plugin-invoke`. Only after the external WASM sandbox signoff in `docs/security/plugin-runtime-signoff.md` may operators set `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and accept plugin invoke consumption as production-ready. The final live proof is operator-run, inside both `worker-general` and `worker-plugin`: `pnpm run verify:plugin-runtime-live`. `worker-import` always consumes `hivegrid.kwp-imports`. KWP provider work is kept off the generic file-import queue so automatic project catalog and binding synchronization remain active while tenant file imports are gated. Keep `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C services are wired into the worker bootstrap. In that KWP-only state `/readyz` is healthy with exactly the dedicated KWP queue. When both generic runtime gates are set, startup fails closed unless the worker also has real import, generic export, scan, portal/privacy GDPR service dependencies and the complete encrypted storage contract described below. The internal portal/privacy GDPR closeout uses a narrower alternative on the same `worker-import` resource. With both `IMPORT_QUEUE_*` gates unset or `false`, `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` adds `hivegrid.exports` beside the always-active `hivegrid.kwp-imports` queue and wires only the portal and privacy GDPR export services. Complete `STORAGE_ENDPOINT`, `STORAGE_REGION`, `STORAGE_BUCKET`, `STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, and a canonical `GDPR_EXPORT_SSE_C_KEY_BASE64` for exactly 32 random bytes is mandatory. The same key must be present on core web and portal web. Both writers use SSE-C; both authorized download routes fetch, fully validate and proxy the object server-side, so the key never enters a redirect, browser header, URL, access log, or ticket. The dedicated and full modes are mutually exclusive; mixed gates, another worker group, missing storage, or missing GDPR service wiring fail closed. Full mode uses the same encrypted GDPR services in addition to generic imports/exports/scans. Dedicated mode does not activate `hivegrid.imports` or generic exports; neither mode can disable the isolated KWP queue. `worker-general` owns tenant hard-delete. In production, required-RLS, and staging/production Sentry modes it therefore requires both `MEILI_URL` and the HiveGrid-scoped `MEILI_API_KEY` in addition to its deletion-only storage credential. A missing search client aborts startup before queue ownership; provider failures leave the request retryable and logs contain only the error class, never the provider message or credentials. Local development/test may skip external search cleanup only while none of those required-runtime markers is active. The privacy and portal writers on `worker-import` and tenant hard-delete on `worker-general` share one distributed tenant-export barrier. Their effective SharedStore target is the first non-empty value of `REDIS_URL`, then `BULLMQ_REDIS_URL`; both resources must resolve to the same Redis protocol, host, port, and database. Credentials may differ. The deploy preflight rejects missing, invalid, or drifting targets without printing either URL. The general worker is delete-only and must never receive `GDPR_EXPORT_SSE_C_KEY_BASE64`. Key rotation is deliberately fail-closed: pause the export queue, wait until all 72-hour privacy artifacts have expired (the portal window is 24 hours) or delete those exact expired artifacts through the retention workflow, update the same key atomically on both web resources and `worker-import`, then resume and request fresh exports. There is no plaintext or old-key fallback. ## Group → queue mapping | Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. | | `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. | | `import` | Always: `hivegrid.kwp-imports`. Full adds `hivegrid.imports`, `hivegrid.exports` after both `IMPORT_QUEUE_*` gates. Dedicated portal/privacy mode adds only `hivegrid.exports` after `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` | none | 1 | 1 | Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed | | `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services | | `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. | Source of truth: `packages/queue/src/groups.ts`. A test (`groups.test.ts`) fails CI if a queue exists that isn't mapped to a group — adding a new package queue forces a deliberate group assignment. Outbox and auxiliary subscriber queues are wired by `apps/worker/src/index.ts` through their dedicated bootstrap modules; update this table when adding a new subscriber that is not represented in `packages/queue`. `outbox.workflow` runs on the `general` group. The worker performs live-workflow lookup and preflight checks in `apps/worker/src/jobs`, then starts matching workflows through the selected executor: signed internal workflow-run API by default, or hybrid/native execution when explicitly configured. The native executor is a credential-free safe-core path: it must not hydrate stored workflow credentials, and any `$credentials.*`, `lookup $credentials`, or `#with $credentials` reference is a preflight gap. That bridge avoids importing `apps/web` server modules into the worker while keeping the existing workflow execution semantics centralized. The same receiver also drains `workflow.execution.requested` rows produced by manual workflow runs and tenant API workflow runs: the web action creates the `WorkflowExecution` and outbox row in one transaction, then worker-general starts that existing execution through the signed internal execution-start bridge or the native safe-core executor. Waiting execution resumes publish `workflow.execution.continue.requested`; worker-general then resumes the existing execution through the signed internal continuation bridge or, in native/hybrid mode, through the worker-native safe-core continuation path when the workflow's only native preflight gap is the current `waitForInput` boundary. Unsupported continuation graphs still fall back to the HTTP bridge in `hybrid` mode and fail loud in `native` mode. Stream runs remain separate until their progress contract moves to a distributed execution path. The subscriber/bootstrap path is active from `apps/worker/src/index.ts` when the `general` worker is configured with the workflow execution settings. Set `OUTBOX_DISPATCHER_ENABLED=true`, `OUTBOX_DISPATCHER_REQUIRED=true`, and `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` on staging/production `worker-general` once Redis plus the bridge URL/token are provisioned; in that mode a missing dispatcher or bridge dependency fails startup instead of leaving outbox rows or `outbox.workflow` silently undrained behind a green process. Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)` call-sites stay absent and that only the signed internal trigger bridge may start `type: "trigger"` executions directly. The generic entity-builder's `row.created` / `row.updated` / `row.deleted` route-backed producers were **removed with the entity-builder** (R-P6B-β, AD-2); typed fachentities and workflow row blocks (via the typed model registry) no longer emit generic `row.*` outbox events. `workflow.trigger.event` application events still publish through `EventsService`. Other internal requestless mutations remain eventless by default to avoid worker feedback loops. `worker-general` also registers the `workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only old manual/API `WorkflowExecution` rows that are still `running`, have no block runs, have no waiting block, and whose matching `workflow.execution.requested` outbox event with dispatch key `workflow-execution:` is already `FAILED`. It does not infer failure from `DONE` outbox rows because `DONE` only proves dispatch into BullMQ, not workflow completion. Use `pnpm run verify:workflow-outbox-native` as the canonical non-browser gate before changing workflow outbox/native execution readiness. The gate composes existing focused tests for `outbox.workflow` routing, subscriber bootstrap, native/hybrid safe-core execution, worker health required-mode checks, and queued manual/API/continuation producer inventory. It does not prove a live environment is draining Redis jobs; pair it with the opt-in integration test or operator-gated E2E drain probe for live worker readiness. Form submissions use a two-hop worker path. The public submit route stores the `FormSubmission` row and emits `form.submission.process`, which the outbox dispatcher routes to `outbox.form-submission-route` on `worker-general`. That subscriber performs tenant-scoped CRM/entity routing, marks the submission, and publishes follow-up events. Webhook follow-ups use `webhook.form-submission.deliver`, routed to `outbox.webhook` on `worker-mail`, where the bridge enqueues the existing `hivegrid.email` `deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP throughput group and leaves the general group responsible only for the database-backed routing decision. ## Env variables per group Required (per worker container): | Variable | Purpose | | -------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `WORKER_QUEUE_GROUP` | `general` / `mail` / `import` / `webhook` / `plugin` | | `WORKER_HEALTH_PORT` | TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105 | | `BULLMQ_REDIS_URL` | Redis target | | `DATABASE_URL` | Postgres (via PgBouncer in staging/production) | Required only on `worker-general`. The default execution adapter still bridges through the web runtime; `hybrid` canaries credential-free safe core event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped rowGet/email blocks, and single-boundary `waitForInput` continuation resumes, then falls back to the signed HTTP bridge for unsupported blocks or credential references; `native` is reserved for credential-free safe-core-only deployments and fails loud for unsupported blocks or credential references. The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless platform workflow trigger execution is intentionally unsupported in HTTP mode and should use native mode once the workflow is within the supported block contract. `scripts/coolify-ensure-worker-groups.mjs` verifies these settings on the Coolify worker apps before deploy: every managed worker group must have `DATABASE_URL` and `BULLMQ_REDIS_URL` configured, and `worker-general` must also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` / `APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit `WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`. The same general-worker preflight requires the non-empty Lago/Mollie provider contract (`LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`, `MOLLIE_PROFILE_ID`, `MOLLIE_TIMEOUT_MS`, `BILLING_MOLLIE_WEBHOOK_URL`) because payment reconciliation and durable provider side effects run only in that group. Only `LAGO_API_URL`, `LAGO_API_KEY`, `MOLLIE_API_KEY`, and the environment-specific HTTPS `BILLING_MOLLIE_WEBHOOK_URL` are mandatory; `MOLLIE_PROFILE_ID` and both timeout overrides are optional. Production may hold this subscriber before the external payment gates are complete by setting `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`. That explicit value disables the subscriber even under required RLS and waives only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default, `true` forces the provider contract, and any other value fails closed. The gate must not be copied to another worker group. Ein mehrdeutiger Mollie-Payment-Start ohne persistierte `providerPaymentId` wird vom Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere, auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist in [`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md) beschrieben. Direktes BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten. | Variable | Purpose | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. | | `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge | | `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key | | `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` | | `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher | | `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them | | `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually | | `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. | | `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. | | `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance | | `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout | | `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` | | `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured | ## Coolify resources Coolify must run queue groups as separate persistent applications that use the same `worker` image with different `WORKER_QUEUE_GROUP` values. The deploy workflow requires `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin` resources in the staging and production resource allow-lists. `worker-mail` is a permanent service, not a manual workaround: without it, queued transactional mail and mail-backed outbox/webhook work is not drained after deploy. Without the other group resources, production traffic falls back into manual starts. This is an operator-owned runtime prerequisite: the repository and static verifiers document the required topology, but they do not by themselves prove that a target Coolify environment currently has every worker group deployed and healthy. The same operator/code split applies to plugin runtime status: docs and code require the `worker-plugin` service, while tenant-facing production execution remains blocked on external signoff and the explicit `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` flip. A live rollout must additionally pass `pnpm run verify:plugin-runtime-live` in both `worker-general` and `worker-plugin` so the general fanout/audit path and isolated invoke worker are checked separately. Name the resources with the group marker (`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) or expose the `WORKER_QUEUE_GROUP=` marker in the resource metadata returned by Coolify. Keep `WORKER_ENABLED=true` on each worker resource and assign distinct health ports (`9101` general, `9102` mail, `9103` import, `9104` webhook, `9105` plugin). When metrics/admin endpoints are enabled, keep them on separate ports (`9201` general, `9202` mail, `9203` import, `9204` webhook, `9205` plugin) so the admin server does not collide with the health server. `scripts/coolify-ensure-worker-groups.mjs` can create missing worker applications during the deploy workflow when provisioning is explicitly enabled. It also reconciles every managed worker app to an enabled Coolify `cmd` healthcheck with `node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> ` for the exact immutable image revision and worker resource before rollout. That command verifies the image-baked revision, `APP_REVISION`, enablement, and group, then probes the running process over loopback `/readyz`. A mutable alias, wrong queue group, disabled worker, or healthy stale container cannot satisfy the deploy gate. Staging uses: ```bash COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true COOLIFY_STAGING_PROJECT_UUID= COOLIFY_STAGING_SERVER_UUID= COOLIFY_STAGING_DESTINATION_UUID= COOLIFY_STAGING_ENVIRONMENT_NAME=staging ``` Production uses the same keys without `STAGING_`: `COOLIFY_PROVISION_WORKER_GROUPS`, `COOLIFY_PROJECT_UUID`, `COOLIFY_SERVER_UUID`, `COOLIFY_DESTINATION_UUID`, and `COOLIFY_ENVIRONMENT_NAME`. When it creates a new group app, the script copies the existing worker app env from an allowed non-group worker resource or from `COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID` / `COOLIFY_WORKER_ENV_SOURCE_UUID`, then overwrites only the group-specific settings (`WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, `WORKER_ADMIN_PORT`, and group concurrency). Keep that source worker app available until every group resource has its own runtime env. Optional overrides: | Variable | Default | Notes | | ------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `WORKER_CONCURRENCY_GENERAL` | 10 | Only read when `WORKER_QUEUE_GROUP=general` | | `WORKER_CONCURRENCY_MAIL` | 5 | Only read when `WORKER_QUEUE_GROUP=mail` | | `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` | | `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` | | `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` | | `PORTAL_GDPR_EXPORT_QUEUE_ENABLED` | false | Internal exports-only mode for `WORKER_QUEUE_GROUP=import`; both `IMPORT_QUEUE_*` gates must remain unset/false and complete `STORAGE_*`, `GDPR_EXPORT_SSE_C_KEY_BASE64`, plus GDPR service wiring is mandatory | | `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` | | `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts | | `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size | | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `false`; managed Coolify `worker-mail` uses `true` | Mail-group only. Set to `true` in staging/production so `worker-mail` exits if the `outbox.mail` bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed `worker-mail` apps. | ## Health endpoints Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`: - `GET /healthz` — liveness. Returns 200 as long as the event loop is responsive. No external dependency calls; safe to hammer. - `GET /readyz` — readiness. Runs a Redis ping and a Prisma `SELECT 1` in parallel with a 2-second timeout each. Returns 200 only when both succeed; 503 otherwise. Queue-group subscriber probes are also included for required/expected runtime bridges such as `outbox.mail` on `WORKER_QUEUE_GROUP=mail`. Use `/readyz` as the Kubernetes and manual HTTP readiness probe and `/healthz` as the liveness probe. Managed Coolify worker apps instead use the revision-bound `cmd` healthcheck documented above; this avoids relying on HTTP-probe utilities inside the minimal worker image and binds health to the exact deployed SHA. Worker health is primarily a queue-runtime check. For managed `worker-mail` apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also fail loud when the required outbound provider pair `MAILJET_API_KEY` and `MAILJET_FROM_EMAIL` is absent, using key names only. A healthy `worker-mail` therefore proves Redis/Postgres reachability, the mail queue process, the transactional outbox bridge, and required Mailjet env presence; it still does not prove sender-domain verification or live provider delivery. Managed Coolify `worker-mail` apps must keep `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when `outbox.mail` cannot subscribe; without that bridge, invite and other `mail.transactional.*` outbox events can remain pending even if the direct `hivegrid.email` worker is alive. The Coolify worker-group provisioner writes and drift-checks this env value for managed mail workers. Example: ```bash curl -fsS http://worker-mail:9102/healthz curl -fsS http://worker-mail:9102/readyz ``` ## Connection naming Each worker tags its ioredis connection with `worker--` (via `apps/worker/src/connection.ts`). Run `CLIENT LIST` against Redis to attribute traffic to a specific worker group: ```redis CLIENT LIST # id=42 name=worker-mail-218 ... # id=43 name=worker-webhook-219 ... ``` ## Scaling cookbook | Symptom | Action | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` | | Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix | | AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) | | Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further | | One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed | ## Operator commands ```bash # Inspect all worker-group logs docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin # Restart a single group docker compose restart worker-import # Smoke-test all five readiness endpoints for port in 9101 9102 9103 9104 9105; do curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready" done ``` ## Troubleshooting **Wrong or missing worker group.** The worker Docker image defaults `WORKER_QUEUE_GROUP` to `general` for raw image users. Coolify group apps must still set the explicit group marker (`general`, `mail`, `import`, `webhook`, or `plugin`) so each app drains the intended queues. A missing Coolify override can therefore start as another `general` worker instead of a mail/import/webhook/plugin worker; `scripts/coolify-ensure-worker-groups.mjs` treats that as env drift. An unknown group still fails worker startup when the worker is enabled. Disabled workers (`WORKER_ENABLED=false`) skip this check so dev environments without Redis still boot. **`/readyz` returns 503 with `checks.redis="error"`** — BullMQ couldn't reach Redis. Confirm `BULLMQ_REDIS_URL` and that the redis container is healthy. The error is intentionally short (no host names) so the probe response never leaks credentials. **`/readyz` returns 503 with `checks.database="error"`** — Prisma couldn't run `SELECT 1`. Check `DATABASE_URL`, PgBouncer health, and whether the pool is saturated. **A queue is silent on every worker** — verify the queue is mapped to a group in `packages/queue/src/groups.ts`. If you just added a new queue, the `groups.test.ts` exhaustiveness check should have failed CI; if it didn't, the queue is missing from `names.ts`. **Outbox dispatcher missing.** The outbox dispatcher runs on the `general` group only (see `apps/worker/src/index.ts`). If your stack runs no `general` replica, outbox events will not dispatch — keep at least one `general` replica. ## Rollback Current production worker topology requires an explicit `WORKER_QUEUE_GROUP` on each Coolify group app whenever `WORKER_ENABLED=true`; there is no supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's standalone default is `general`, so removing the marker during rollback can silently turn a non-general app into another general worker, while an unknown group fails startup by design. For an incident rollback, redeploy the previous known-good full 40-character SHA or compose/deploy definition. The current reconciler is fleet-wide: it updates all five managed group applications in one validated pass. A modern rollback must therefore redeploy and verify all five groups; it is not safe to reconcile the old SHA and trigger only one group, because a later restart of another group would consume the already-rolled-back configuration. For images that contain the revision-bound deployment-health entrypoint, a Coolify rollback must never patch only `docker_registry_image_tag`: the image tag and `health_check_command` are one atomic configuration unit. The command must be `node apps/worker/dist/deploy-health.js `, where the final argument remains the resource's exact existing `general`, `mail`, `import`, `webhook`, or `plugin` group. Before the canonical force-deploy is triggered, set `IMAGE_TAG="$OLD_FULL_SHA"` and provide the exact HiveGrid runtime image identity through `WORKER_IMAGE` and `MCP_IMAGE`, or through the complete `REGISTRY`, `IMAGE_NAMESPACE`, and `IMAGE_PREFIX` tuple. The rollback helpers reject suffix-only image matches so a valid CUID from another product cannot be mutated accidentally. Then run `node scripts/coolify-ensure-worker-groups.mjs ` with the already configured environment-specific Coolify variables. The reconciler atomically sets and rereads both fields for every managed worker resource and rejects group, revision, or persistence drift. Only after it succeeds may the operator run `node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID"` for each of the five worker resources and wait on every exact returned deployment UUID. A scoped single-group rollback requires a future dedicated helper that leaves the other four persisted configurations untouched. For an MCP rollback, the equivalent prerequisite is `node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`; it atomically binds the MCP image tag and command healthcheck to the same old SHA before the same canonical force-deploy path is used. All identifiers above are placeholders; never record real resource UUIDs, endpoints, tokens, or credentials. Images built before `apps/worker/dist/deploy-health.js` and `/myapp/.image-revision` were introduced need the bootstrap-compatible legacy path. Do not run the current reconciler against such an old SHA. Instead run `node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" `. The helper rejects mutable tags, Coolify `APP_REVISION` overrides, and group drift, then atomically sets the old tag, clears `health_check_command`, disables command health, and rereads the result. Force-deploy only after it succeeds and wait on the exact returned deployment UUID. This is a degraded, emergency-only bootstrap rollback: disabling command health removes it from Coolify's candidate gate, so brief downtime is possible. Accept the worker only after Coolify reports `running:healthy`, an approved read-only host check resolves the live container digest/OCI revision to the old SHA image, and `node apps/worker/dist/health.js` inside that live container reports the image-baked old revision, enabled processing, the exact existing group, and ready dependencies. `APP_REVISION` must be absent only as a Coolify runtime override; the old image's baked environment value must equal the old SHA. Re-enable and verify the revision-bound `cmd` healthcheck before the next roll-forward to a current image. In-flight BullMQ job state lives in Redis, so jobs are not lost while workers are rolled.