HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/worker-queues.md

Worker queue groups — operations guide

The standalone apps/worker runtime is split into five queue groups. Each group is the same image and the same code; the only difference is the WORKER_QUEUE_GROUP env variable that decides which BullMQ queues the process subscribes to.

This page is the canonical worker topology reference. Shorter worker tables in deployment, environment, or feature docs are summaries only and should link back here instead of redefining queue ownership.

Why five groups

A single worker process processed every queue. That had three operational problems:

1. Head-of-line blocking. A long-running import (minutes) stalled transactional mail and notifications that ran on different queues but the same event loop.
2. Wrong concurrency for the workload. Webhook delivery needs high parallelism (I/O-bound HTTP calls); imports need low parallelism (CPU/memory-bound). One WORKER_CONCURRENCY value cannot suit both.
3. No per-workload scaling. When mail queues backlog we couldn't scale mail capacity without also duplicating the import worker (waste of RAM and Redis connections).

The split fixes all three by giving each workload its own process and its own concurrency. Add replicas to only the group that needs them.

Production worker matrix

Production-like Coolify environments must run one persistent worker service per queue group. These are long-lived internal services from the same worker image, not manual emergency processes and not public web apps. worker-mail with WORKER_QUEUE_GROUP=mail is part of the baseline topology because transactional mail, mail outbox, form-submission webhook delivery, and email-sequence worker contracts are intentionally isolated from general outbox fanout.

Coolify app	Queue group	Owns	Health port	Admin/metrics port	Required fail-loud settings
worker-general	WORKER_QUEUE_GROUP=general	General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing	9101	9201	OUTBOX_DISPATCHER_REQUIRED=true, WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true
worker-mail	WORKER_QUEUE_GROUP=mail	Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge	9102	9202	TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true in staging/production
worker-import	WORKER_QUEUE_GROUP=import	Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing	9103	9203	KWP dependencies are always required; GDPR modes additionally require HTTPS STORAGE_*, canonical GDPR_EXPORT_SSE_C_KEY_BASE64, and both GDPR services; full mode also needs both IMPORT_QUEUE_* gates plus generic import/export/scan services
worker-webhook	WORKER_QUEUE_GROUP=webhook	AI chatbot and browser automation queues	9104	9204	none beyond shared worker readiness
worker-plugin	WORKER_QUEUE_GROUP=plugin	Isolated WASM plugin invocation queue; idle before production plugin signoff	9105	9205	Plugin runtime gates from docs/operations/plugin-lifecycle.md

The detailed queue mapping below is the source of truth for package queues, subscriber queues, concurrency, and workload profile. Update this page first when queue ownership changes; deployment-specific docs should keep only the short app/group/port summary and link back here.

worker-plugin is required as a persistent Coolify service so the deploy topology is stable, but production plugin execution is still gated. With PLUGIN_RUNTIME_PRODUCTION_ENABLED unset or false, the plugin worker starts as an idle topology placeholder and must not consume hivegrid.plugin-invoke. Only after the external WASM sandbox signoff in docs/security/plugin-runtime-signoff.md may operators set PLUGIN_RUNTIME_PRODUCTION_ENABLED=true, provide the trusted public keys, and accept plugin invoke consumption as production-ready. The final live proof is operator-run, inside both worker-general and worker-plugin: pnpm run verify:plugin-runtime-live.

worker-import always consumes hivegrid.kwp-imports. KWP provider work is kept off the generic file-import queue so automatic project catalog and binding synchronization remain active while tenant file imports are gated. Keep IMPORT_QUEUE_ENABLED and IMPORT_QUEUE_RUNTIME_READY unset until generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C services are wired into the worker bootstrap. In that KWP-only state /readyz is healthy with exactly the dedicated KWP queue. When both generic runtime gates are set, startup fails closed unless the worker also has real import, generic export, scan, portal/privacy GDPR service dependencies and the complete encrypted storage contract described below.

The internal portal/privacy GDPR closeout uses a narrower alternative on the same worker-import resource. With both IMPORT_QUEUE_* gates unset or false, PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true adds hivegrid.exports beside the always-active hivegrid.kwp-imports queue and wires only the portal and privacy GDPR export services. Complete STORAGE_ENDPOINT, STORAGE_REGION, STORAGE_BUCKET, STORAGE_ACCESS_KEY_ID, STORAGE_SECRET_ACCESS_KEY, and a canonical GDPR_EXPORT_SSE_C_KEY_BASE64 for exactly 32 random bytes is mandatory. The same key must be present on core web and portal web. Both writers use SSE-C; both authorized download routes fetch, fully validate and proxy the object server-side, so the key never enters a redirect, browser header, URL, access log, or ticket. The dedicated and full modes are mutually exclusive; mixed gates, another worker group, missing storage, or missing GDPR service wiring fail closed. Full mode uses the same encrypted GDPR services in addition to generic imports/exports/scans. Dedicated mode does not activate hivegrid.imports or generic exports; neither mode can disable the isolated KWP queue.

worker-general owns tenant hard-delete. In production, required-RLS, and staging/production Sentry modes it therefore requires both MEILI_URL and the HiveGrid-scoped MEILI_API_KEY in addition to its deletion-only storage credential. A missing search client aborts startup before queue ownership; provider failures leave the request retryable and logs contain only the error class, never the provider message or credentials. Local development/test may skip external search cleanup only while none of those required-runtime markers is active.

The privacy and portal writers on worker-import and tenant hard-delete on worker-general share one distributed tenant-export barrier. Their effective SharedStore target is the first non-empty value of REDIS_URL, then BULLMQ_REDIS_URL; both resources must resolve to the same Redis protocol, host, port, and database. Credentials may differ. The deploy preflight rejects missing, invalid, or drifting targets without printing either URL. The general worker is delete-only and must never receive GDPR_EXPORT_SSE_C_KEY_BASE64.

Key rotation is deliberately fail-closed: pause the export queue, wait until all 72-hour privacy artifacts have expired (the portal window is 24 hours) or delete those exact expired artifacts through the retention workflow, update the same key atomically on both web resources and worker-import, then resume and request fresh exports. There is no plaintext or old-key fallback.

Group → queue mapping

Group	Package queues	Bootstrapped subscriber queues	Default concurrency	Default replicas	Workload profile
general	hivegrid.notifications, hivegrid.maintenance, hivegrid.file-scan	Outbox dispatcher, outbox.ai-cost, outbox.search-index, outbox.rag-index, outbox.audit, outbox.notification, outbox.plugin-hook, outbox.plugin-capability-use, outbox.workflow, outbox.form-submission-route, plugin-audit-retention	10	1	Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The outbox.plugin-capability-use subscriber intentionally defaults to concurrency 1 to keep capability-use hash chains linear; do not raise PLUGIN_CAPABILITY_USE_CONCURRENCY without store-level serialization.
mail	hivegrid.email	outbox.mail, outbox.marketing (email-sequence trigger only), outbox.webhook (form-submission delivery bridge), email-sequence.step	5	2	SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require MARKETING_SEQUENCES_ENABLED=true; keep it unset or false in production until the remaining approval follow-ups in docs/features/email-sequences.md land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in packages/marketing-outbound and apps/worker/src/jobs/marketing-outbound.ts, but it is pure handler foundation and does not claim active ownership of outbox.marketing; general marketing integration delivery is not bootstrapped in apps/worker/src/index.ts yet.
import	Always: hivegrid.kwp-imports. Full adds hivegrid.imports, hivegrid.exports after both IMPORT_QUEUE_* gates. Dedicated portal/privacy mode adds only hivegrid.exports after PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true	none	1	1	Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed
webhook	hivegrid.ai-chatbot, hivegrid.browser-automation	none	20	3	I/O-bound calls to external services
plugin	hivegrid.plugin-invoke	none	5	2	Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until PLUGIN_RUNTIME_PRODUCTION_ENABLED=true is set after external sandbox signoff.

Source of truth: packages/queue/src/groups.ts. A test (groups.test.ts) fails CI if a queue exists that isn't mapped to a group — adding a new package queue forces a deliberate group assignment. Outbox and auxiliary subscriber queues are wired by apps/worker/src/index.ts through their dedicated bootstrap modules; update this table when adding a new subscriber that is not represented in packages/queue.

outbox.workflow runs on the general group. The worker performs live-workflow lookup and preflight checks in apps/worker/src/jobs, then starts matching workflows through the selected executor: signed internal workflow-run API by default, or hybrid/native execution when explicitly configured. The native executor is a credential-free safe-core path: it must not hydrate stored workflow credentials, and any $credentials.*, lookup $credentials, or #with $credentials reference is a preflight gap. That bridge avoids importing apps/web server modules into the worker while keeping the existing workflow execution semantics centralized. The same receiver also drains workflow.execution.requested rows produced by manual workflow runs and tenant API workflow runs: the web action creates the WorkflowExecution and outbox row in one transaction, then worker-general starts that existing execution through the signed internal execution-start bridge or the native safe-core executor. Waiting execution resumes publish workflow.execution.continue.requested; worker-general then resumes the existing execution through the signed internal continuation bridge or, in native/hybrid mode, through the worker-native safe-core continuation path when the workflow's only native preflight gap is the current waitForInput boundary. Unsupported continuation graphs still fall back to the HTTP bridge in hybrid mode and fail loud in native mode. Stream runs remain separate until their progress contract moves to a distributed execution path. The subscriber/bootstrap path is active from apps/worker/src/index.ts when the general worker is configured with the workflow execution settings. Set OUTBOX_DISPATCHER_ENABLED=true, OUTBOX_DISPATCHER_REQUIRED=true, and WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true on staging/production worker-general once Redis plus the bridge URL/token are provisioned; in that mode a missing dispatcher or bridge dependency fails startup instead of leaving outbox rows or outbox.workflow silently undrained behind a green process. Source-inventory tests guard that legacy WorkflowEngine.trigger(...) call-sites stay absent and that only the signed internal trigger bridge may start type: "trigger" executions directly. The generic entity-builder's row.created / row.updated / row.deleted route-backed producers were removed with the entity-builder (R-P6B-β, AD-2); typed fachentities and workflow row blocks (via the typed model registry) no longer emit generic row.* outbox events. workflow.trigger.event application events still publish through EventsService. Other internal requestless mutations remain eventless by default to avoid worker feedback loops.

worker-general also registers the workflow-execution-stalled-reconcile maintenance scheduler. It repairs only old manual/API WorkflowExecution rows that are still running, have no block runs, have no waiting block, and whose matching workflow.execution.requested outbox event with dispatch key workflow-execution:<executionId> is already FAILED. It does not infer failure from DONE outbox rows because DONE only proves dispatch into BullMQ, not workflow completion.

Use pnpm run verify:workflow-outbox-native as the canonical non-browser gate before changing workflow outbox/native execution readiness. The gate composes existing focused tests for outbox.workflow routing, subscriber bootstrap, native/hybrid safe-core execution, worker health required-mode checks, and queued manual/API/continuation producer inventory. It does not prove a live environment is draining Redis jobs; pair it with the opt-in integration test or operator-gated E2E drain probe for live worker readiness.

Form submissions use a two-hop worker path. The public submit route stores the FormSubmission row and emits form.submission.process, which the outbox dispatcher routes to outbox.form-submission-route on worker-general. That subscriber performs tenant-scoped CRM/entity routing, marks the submission, and publishes follow-up events. Webhook follow-ups use webhook.form-submission.deliver, routed to outbox.webhook on worker-mail, where the bridge enqueues the existing hivegrid.email deliver-webhook job. This keeps webhook HTTP delivery in the mail/HTTP throughput group and leaves the general group responsible only for the database-backed routing decision.

Env variables per group

Required (per worker container):

Variable	Purpose
WORKER_QUEUE_GROUP	general / mail / import / webhook / plugin
WORKER_HEALTH_PORT	TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105
BULLMQ_REDIS_URL	Redis target
DATABASE_URL	Postgres (via PgBouncer in staging/production)

Required only on worker-general. The default execution adapter still bridges through the web runtime; hybrid canaries credential-free safe core event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped rowGet/email blocks, and single-boundary waitForInput continuation resumes, then falls back to the signed HTTP bridge for unsupported blocks or credential references; native is reserved for credential-free safe-core-only deployments and fails loud for unsupported blocks or credential references. The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless platform workflow trigger execution is intentionally unsupported in HTTP mode and should use native mode once the workflow is within the supported block contract. scripts/coolify-ensure-worker-groups.mjs verifies these settings on the Coolify worker apps before deploy: every managed worker group must have DATABASE_URL and BULLMQ_REDIS_URL configured, and worker-general must also have either a complete HTTP bridge (WORKFLOW_TRIGGER_EXECUTOR_URL / APP_URL / WEB_ORIGIN plus API_ACCESS_TOKEN) or explicit WORKFLOW_TRIGGER_EXECUTOR_MODE=native / WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid. The same general-worker preflight requires the non-empty Lago/Mollie provider contract (LAGO_API_URL, LAGO_API_KEY, LAGO_TIMEOUT_MS, MOLLIE_API_KEY, MOLLIE_PROFILE_ID, MOLLIE_TIMEOUT_MS, BILLING_MOLLIE_WEBHOOK_URL) because payment reconciliation and durable provider side effects run only in that group. Only LAGO_API_URL, LAGO_API_KEY, MOLLIE_API_KEY, and the environment-specific HTTPS BILLING_MOLLIE_WEBHOOK_URL are mandatory; MOLLIE_PROFILE_ID and both timeout overrides are optional.

Production may hold this subscriber before the external payment gates are complete by setting PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false on worker-general. That explicit value disables the subscriber even under required RLS and waives only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default, true forces the provider contract, and any other value fails closed. The gate must not be copied to another worker group.

Ein mehrdeutiger Mollie-Payment-Start ohne persistierte providerPaymentId wird vom Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere, auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist in docs/ops/product-acquisition-recovery.md beschrieben. Direktes BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten.

Variable	Purpose
WORKFLOW_TRIGGER_EXECUTOR_MODE=http	Explicitly selects the signed HTTP bridge. Use hybrid to canary worker-native credential-free safe-core execution with HTTP fallback; use native only for the worker-native safe-core MVP (event, manual, if, switch, doNothing, log, alertUser, variable, httpRequest, tenant-scoped rowGet/email, and single-boundary waitForInput continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution.
APP_URL, WEB_ORIGIN, or WORKFLOW_TRIGGER_EXECUTOR_URL	Web runtime origin used by outbox.workflow execution bridge
API_ACCESS_TOKEN	Canonical service token used by outbox.workflow; value must match the web runtime's API_ACCESS_TOKEN because the web runtime validates the signed internal bridge against that key
OUTBOX_DISPATCHER_ENABLED=true	Starts the transactional outbox dispatcher on worker-general
OUTBOX_DISPATCHER_REQUIRED=true	Fail startup if the general worker cannot start the dispatcher
OUTBOX_DONE_RETENTION_HOURS=720	Daily worker-general maintenance keeps DONE outbox rows for 30 days before purging them
OUTBOX_DONE_RETENTION_BATCH_SIZE=1000	Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually
WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30	Five-minute worker-general maintenance marks only untouched manual/API workflow executions as error when their initial workflow.execution.requested outbox dispatch is already FAILED and older than this threshold.
WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100	Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually.
WORKFLOW_TRIGGER_OUTBOX_ENABLED=true	Keeps the outbox.workflow subscriber active; set false only for controlled maintenance
WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5	BullMQ concurrency for workflow trigger fanout
WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true	Fail startup if the general worker cannot subscribe to outbox.workflow
PLUGIN_HOOK_FANOUT_REQUIRED=true	Optional staging/canary fail-closed switch for outbox.plugin-hook; production plugin runtime with PLUGIN_RUNTIME_PRODUCTION_ENABLED=true already requires fanout when Redis is configured

Coolify resources

Coolify must run queue groups as separate persistent applications that use the same worker image with different WORKER_QUEUE_GROUP values. The deploy workflow requires worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin resources in the staging and production resource allow-lists. worker-mail is a permanent service, not a manual workaround: without it, queued transactional mail and mail-backed outbox/webhook work is not drained after deploy. Without the other group resources, production traffic falls back into manual starts. This is an operator-owned runtime prerequisite: the repository and static verifiers document the required topology, but they do not by themselves prove that a target Coolify environment currently has every worker group deployed and healthy. The same operator/code split applies to plugin runtime status: docs and code require the worker-plugin service, while tenant-facing production execution remains blocked on external signoff and the explicit PLUGIN_RUNTIME_PRODUCTION_ENABLED=true flip. A live rollout must additionally pass pnpm run verify:plugin-runtime-live in both worker-general and worker-plugin so the general fanout/audit path and isolated invoke worker are checked separately.

Name the resources with the group marker (worker-general, worker-mail, worker-import, worker-webhook, worker-plugin) or expose the WORKER_QUEUE_GROUP=<group> marker in the resource metadata returned by Coolify. Keep WORKER_ENABLED=true on each worker resource and assign distinct health ports (9101 general, 9102 mail, 9103 import, 9104 webhook, 9105 plugin). When metrics/admin endpoints are enabled, keep them on separate ports (9201 general, 9202 mail, 9203 import, 9204 webhook, 9205 plugin) so the admin server does not collide with the health server.

scripts/coolify-ensure-worker-groups.mjs can create missing worker applications during the deploy workflow when provisioning is explicitly enabled. It also reconciles every managed worker app to an enabled Coolify cmd healthcheck with node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group> for the exact immutable image revision and worker resource before rollout. That command verifies the image-baked revision, APP_REVISION, enablement, and group, then probes the running process over loopback /readyz. A mutable alias, wrong queue group, disabled worker, or healthy stale container cannot satisfy the deploy gate. Staging uses:

COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
COOLIFY_STAGING_SERVER_UUID=<server uuid>
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
COOLIFY_STAGING_ENVIRONMENT_NAME=staging

Production uses the same keys without STAGING_: COOLIFY_PROVISION_WORKER_GROUPS, COOLIFY_PROJECT_UUID, COOLIFY_SERVER_UUID, COOLIFY_DESTINATION_UUID, and COOLIFY_ENVIRONMENT_NAME. When it creates a new group app, the script copies the existing worker app env from an allowed non-group worker resource or from COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID / COOLIFY_WORKER_ENV_SOURCE_UUID, then overwrites only the group-specific settings (WORKER_ENABLED, WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, WORKER_ADMIN_PORT, and group concurrency). Keep that source worker app available until every group resource has its own runtime env.

Optional overrides:

Variable	Default	Notes
WORKER_CONCURRENCY_GENERAL	10	Only read when WORKER_QUEUE_GROUP=general
WORKER_CONCURRENCY_MAIL	5	Only read when WORKER_QUEUE_GROUP=mail
WORKER_CONCURRENCY_IMPORT	1	Only read when WORKER_QUEUE_GROUP=import
WORKER_CONCURRENCY_WEBHOOK	20	Only read when WORKER_QUEUE_GROUP=webhook
WORKER_CONCURRENCY_PLUGIN	5	Only read when WORKER_QUEUE_GROUP=plugin
PORTAL_GDPR_EXPORT_QUEUE_ENABLED	false	Internal exports-only mode for WORKER_QUEUE_GROUP=import; both IMPORT_QUEUE_* gates must remain unset/false and complete STORAGE_*, GDPR_EXPORT_SSE_C_KEY_BASE64, plus GDPR service wiring is mandatory
WORKER_ADMIN_PORT	Dockerfile/code default 9101; 9201-9205 in Coolify	Must differ from WORKER_HEALTH_PORT when METRICS_ENABLED=true
WORKER_HEALTH_HOST	0.0.0.0	Override when running outside Docker on multi-NIC hosts
WORKER_REDIS_POOL_SIZE	4/4/2/8/4 (general/mail/import/webhook/plugin)	Advisory ioredis pool size
TRANSACTIONAL_MAIL_OUTBOX_REQUIRED	false; managed Coolify worker-mail uses true	Mail-group only. Set to true in staging/production so worker-mail exits if the outbox.mail bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed worker-mail apps.

Health endpoints

Every worker exposes two HTTP probes on WORKER_HEALTH_PORT:

• GET /healthz — liveness. Returns 200 as long as the event loop is responsive. No external dependency calls; safe to hammer.
• GET /readyz — readiness. Runs a Redis ping and a Prisma SELECT 1 in parallel with a 2-second timeout each. Returns 200 only when both succeed; 503 otherwise. Queue-group subscriber probes are also included for required/expected runtime bridges such as outbox.mail on WORKER_QUEUE_GROUP=mail.

Use /readyz as the Kubernetes and manual HTTP readiness probe and /healthz as the liveness probe. Managed Coolify worker apps instead use the revision-bound cmd healthcheck documented above; this avoids relying on HTTP-probe utilities inside the minimal worker image and binds health to the exact deployed SHA.

Worker health is primarily a queue-runtime check. For managed worker-mail apps with TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true, health diagnostics also fail loud when the required outbound provider pair MAILJET_API_KEY and MAILJET_FROM_EMAIL is absent, using key names only. A healthy worker-mail therefore proves Redis/Postgres reachability, the mail queue process, the transactional outbox bridge, and required Mailjet env presence; it still does not prove sender-domain verification or live provider delivery. Managed Coolify worker-mail apps must keep TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true so the process fails fast when outbox.mail cannot subscribe; without that bridge, invite and other mail.transactional.* outbox events can remain pending even if the direct hivegrid.email worker is alive. The Coolify worker-group provisioner writes and drift-checks this env value for managed mail workers.

Example:

curl -fsS http://worker-mail:9102/healthz
curl -fsS http://worker-mail:9102/readyz

Connection naming

Each worker tags its ioredis connection with worker-<group>-<pid> (via apps/worker/src/connection.ts). Run CLIENT LIST against Redis to attribute traffic to a specific worker group:

CLIENT LIST
# id=42 name=worker-mail-218 ...
# id=43 name=worker-webhook-219 ...

Scaling cookbook

Symptom	Action
Mail queue depth > 1 000 sustained	docker compose up -d --scale worker-mail=4
Import queue depth > 10 sustained	docker compose up -d --scale worker-import=2; review job size first — single jobs taking >10min are usually the real fix
AI/browser external-call queue latency p95 climbs	docker compose up -d --scale worker-webhook=5 (each replica handles 20 parallel)
Form-submission webhook or mail outbox latency climbs	docker compose up -d --scale worker-mail=4; review SMTP/HTTP provider throttling before raising concurrency further
One worker stuck at 100 % CPU	Inspect /readyz — if it's still 200, profile inside the process; if 503, look at the checks payload to see which dep failed

Operator commands

# Inspect all worker-group logs
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin

# Restart a single group
docker compose restart worker-import

# Smoke-test all five readiness endpoints
for port in 9101 9102 9103 9104 9105; do
  curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
done

Troubleshooting

Wrong or missing worker group. The worker Docker image defaults WORKER_QUEUE_GROUP to general for raw image users. Coolify group apps must still set the explicit group marker (general, mail, import, webhook, or plugin) so each app drains the intended queues. A missing Coolify override can therefore start as another general worker instead of a mail/import/webhook/plugin worker; scripts/coolify-ensure-worker-groups.mjs treats that as env drift. An unknown group still fails worker startup when the worker is enabled. Disabled workers (WORKER_ENABLED=false) skip this check so dev environments without Redis still boot.

/readyz returns 503 with checks.redis="error" — BullMQ couldn't reach Redis. Confirm BULLMQ_REDIS_URL and that the redis container is healthy. The error is intentionally short (no host names) so the probe response never leaks credentials.

/readyz returns 503 with checks.database="error" — Prisma couldn't run SELECT 1. Check DATABASE_URL, PgBouncer health, and whether the pool is saturated.

A queue is silent on every worker — verify the queue is mapped to a group in packages/queue/src/groups.ts. If you just added a new queue, the groups.test.ts exhaustiveness check should have failed CI; if it didn't, the queue is missing from names.ts.

Outbox dispatcher missing. The outbox dispatcher runs on the general group only (see apps/worker/src/index.ts). If your stack runs no general replica, outbox events will not dispatch — keep at least one general replica.

Rollback

Current production worker topology requires an explicit WORKER_QUEUE_GROUP on each Coolify group app whenever WORKER_ENABLED=true; there is no supported WORKER_QUEUE_GROUP=all runtime rollback path. The Dockerfile's standalone default is general, so removing the marker during rollback can silently turn a non-general app into another general worker, while an unknown group fails startup by design. For an incident rollback, redeploy the previous known-good full 40-character SHA or compose/deploy definition. The current reconciler is fleet-wide: it updates all five managed group applications in one validated pass. A modern rollback must therefore redeploy and verify all five groups; it is not safe to reconcile the old SHA and trigger only one group, because a later restart of another group would consume the already-rolled-back configuration. For images that contain the revision-bound deployment-health entrypoint, a Coolify rollback must never patch only docker_registry_image_tag: the image tag and health_check_command are one atomic configuration unit. The command must be node apps/worker/dist/deploy-health.js <old-full-sha> <existing-worker-group>, where the final argument remains the resource's exact existing general, mail, import, webhook, or plugin group.

Before the canonical force-deploy is triggered, set IMAGE_TAG="$OLD_FULL_SHA" and provide the exact HiveGrid runtime image identity through WORKER_IMAGE and MCP_IMAGE, or through the complete REGISTRY, IMAGE_NAMESPACE, and IMAGE_PREFIX tuple. The rollback helpers reject suffix-only image matches so a valid CUID from another product cannot be mutated accidentally. Then run node scripts/coolify-ensure-worker-groups.mjs <staging-or-production> with the already configured environment-specific Coolify variables. The reconciler atomically sets and rereads both fields for every managed worker resource and rejects group, revision, or persistence drift. Only after it succeeds may the operator run node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID" for each of the five worker resources and wait on every exact returned deployment UUID. A scoped single-group rollback requires a future dedicated helper that leaves the other four persisted configurations untouched. For an MCP rollback, the equivalent prerequisite is node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"; it atomically binds the MCP image tag and command healthcheck to the same old SHA before the same canonical force-deploy path is used. All identifiers above are placeholders; never record real resource UUIDs, endpoints, tokens, or credentials.

Images built before apps/worker/dist/deploy-health.js and /myapp/.image-revision were introduced need the bootstrap-compatible legacy path. Do not run the current reconciler against such an old SHA. Instead run node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" <existing-worker-group>. The helper rejects mutable tags, Coolify APP_REVISION overrides, and group drift, then atomically sets the old tag, clears health_check_command, disables command health, and rereads the result. Force-deploy only after it succeeds and wait on the exact returned deployment UUID.

This is a degraded, emergency-only bootstrap rollback: disabling command health removes it from Coolify's candidate gate, so brief downtime is possible. Accept the worker only after Coolify reports running:healthy, an approved read-only host check resolves the live container digest/OCI revision to the old SHA image, and node apps/worker/dist/health.js inside that live container reports the image-baked old revision, enabled processing, the exact existing group, and ready dependencies. APP_REVISION must be absent only as a Coolify runtime override; the old image's baked environment value must equal the old SHA. Re-enable and verify the revision-bound cmd healthcheck before the next roll-forward to a current image. In-flight BullMQ job state lives in Redis, so jobs are not lost while workers are rolled.