HiveGrid-Wireframes/hivegrid-cloud/docs/ops/worker-queues.md

Worker queue groups — operations guide

The standalone apps/worker runtime is split into five queue groups. Each group is the same image and the same code; the only difference is the WORKER_QUEUE_GROUP env variable that decides which BullMQ queues the process subscribes to.

This page is the canonical worker topology reference. Shorter worker tables in deployment, environment, or feature docs are summaries only and should link back here instead of redefining queue ownership.

Why five groups

A single worker process processed every queue. That had three operational problems:

1. Head-of-line blocking. A long-running import (minutes) stalled transactional mail and notifications that ran on different queues but the same event loop.
2. Wrong concurrency for the workload. Webhook delivery needs high parallelism (I/O-bound HTTP calls); imports need low parallelism (CPU/memory-bound). One WORKER_CONCURRENCY value cannot suit both.
3. No per-workload scaling. When mail queues backlog we couldn't scale mail capacity without also duplicating the import worker (waste of RAM and Redis connections).

The split fixes all three by giving each workload its own process and its own concurrency. Add replicas to only the group that needs them.

Production worker matrix

Production-like Coolify environments must run one persistent worker service per queue group. These are long-lived internal services from the same worker image, not manual emergency processes and not public web apps. worker-mail with WORKER_QUEUE_GROUP=mail is part of the baseline topology because transactional mail, mail outbox, form-submission webhook delivery, and email-sequence worker contracts are intentionally isolated from general outbox fanout.

Coolify app	Queue group	Owns	Health port	Admin/metrics port	Required fail-loud settings
worker-general	WORKER_QUEUE_GROUP=general	General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing	9101	9201	OUTBOX_DISPATCHER_REQUIRED=true, WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true
worker-mail	WORKER_QUEUE_GROUP=mail	Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge	9102	9202	TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true in staging/production
worker-import	WORKER_QUEUE_GROUP=import	Import and export queues after runtime gates are enabled	9103	9203	IMPORT_QUEUE_ENABLED=true and IMPORT_QUEUE_RUNTIME_READY=true only after services are wired
worker-webhook	WORKER_QUEUE_GROUP=webhook	AI chatbot and browser automation queues	9104	9204	none beyond shared worker readiness
worker-plugin	WORKER_QUEUE_GROUP=plugin	Isolated WASM plugin invocation queue; idle before production plugin signoff	9105	9205	Plugin runtime gates from docs/operations/plugin-lifecycle.md

The detailed queue mapping below is the source of truth for package queues, subscriber queues, concurrency, and workload profile. Update this page first when queue ownership changes; deployment-specific docs should keep only the short app/group/port summary and link back here.

worker-plugin is required as a persistent Coolify service so the deploy topology is stable, but production plugin execution is still gated. With PLUGIN_RUNTIME_PRODUCTION_ENABLED unset or false, the plugin worker starts as an idle topology placeholder and must not consume toolctrl.plugin-invoke. Only after the external WASM sandbox signoff in docs/security/plugin-runtime-signoff.md may operators set PLUGIN_RUNTIME_PRODUCTION_ENABLED=true, provide the trusted public keys, and accept plugin invoke consumption as production-ready. The final live proof is operator-run, inside both worker-general and worker-plugin: pnpm run verify:plugin-runtime-live.

worker-import follows the same placeholder-first principle. Provision the separate Coolify app with WORKER_QUEUE_GROUP=import, but keep IMPORT_QUEUE_ENABLED and IMPORT_QUEUE_RUNTIME_READY unset until import, export, and scan services are wired into the worker bootstrap. In that idle state /readyz may be healthy with an empty queue list, which proves topology without consuming toolctrl.imports or toolctrl.exports. When both runtime gates are set, startup fails closed unless the worker has real import, export, and scan service dependencies.

Group → queue mapping

Group	Package queues	Bootstrapped subscriber queues	Default concurrency	Default replicas	Workload profile
general	toolctrl.notifications, toolctrl.maintenance, toolctrl.file-scan	Outbox dispatcher, outbox.ai-cost, outbox.search-index, outbox.rag-index, outbox.audit, outbox.notification, outbox.plugin-hook, outbox.plugin-capability-use, outbox.workflow, outbox.form-submission-route, plugin-audit-retention	10	1	Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The outbox.plugin-capability-use subscriber intentionally defaults to concurrency 1 to keep capability-use hash chains linear; do not raise PLUGIN_CAPABILITY_USE_CONCURRENCY without store-level serialization.
mail	toolctrl.email	outbox.mail, outbox.marketing (email-sequence trigger only), outbox.webhook (form-submission delivery bridge), email-sequence.step	5	2	SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require MARKETING_SEQUENCES_ENABLED=true; keep it unset or false in production until the remaining approval follow-ups in docs/features/email-sequences.md land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in packages/marketing-outbound and apps/worker/src/jobs/marketing-outbound.ts, but it is pure handler foundation and does not claim active ownership of outbox.marketing; general marketing integration delivery is not bootstrapped in apps/worker/src/index.ts yet.
import	toolctrl.imports, toolctrl.exports after IMPORT_QUEUE_ENABLED=true and IMPORT_QUEUE_RUNTIME_READY=true	none	1	1	CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired
webhook	toolctrl.ai-chatbot, toolctrl.browser-automation	none	20	3	I/O-bound calls to external services
plugin	toolctrl.plugin-invoke	none	5	2	Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until PLUGIN_RUNTIME_PRODUCTION_ENABLED=true is set after external sandbox signoff.

Source of truth: packages/queue/src/groups.ts. A test (groups.test.ts) fails CI if a queue exists that isn't mapped to a group — adding a new package queue forces a deliberate group assignment. Outbox and auxiliary subscriber queues are wired by apps/worker/src/index.ts through their dedicated bootstrap modules; update this table when adding a new subscriber that is not represented in packages/queue.

outbox.workflow runs on the general group. The worker performs live-workflow lookup and preflight checks in apps/worker/src/jobs, then starts matching workflows through the selected executor: signed internal workflow-run API by default, or hybrid/native execution when explicitly configured. The native executor is a credential-free safe-core path: it must not hydrate stored workflow credentials, and any $credentials.*, lookup $credentials, or #with $credentials reference is a preflight gap. That bridge avoids importing apps/web server modules into the worker while keeping the existing workflow execution semantics centralized. The same receiver also drains workflow.execution.requested rows produced by manual workflow runs and tenant API workflow runs: the web action creates the WorkflowExecution and outbox row in one transaction, then worker-general starts that existing execution through the signed internal execution-start bridge or the native safe-core executor. Waiting execution resumes publish workflow.execution.continue.requested; worker-general then resumes the existing execution through the signed internal continuation bridge or, in native/hybrid mode, through the worker-native safe-core continuation path when the workflow's only native preflight gap is the current waitForInput boundary. Unsupported continuation graphs still fall back to the HTTP bridge in hybrid mode and fail loud in native mode. Stream runs remain separate until their progress contract moves to a distributed execution path. The subscriber/bootstrap path is active from apps/worker/src/index.ts when the general worker is configured with the workflow execution settings. Set OUTBOX_DISPATCHER_ENABLED=true, OUTBOX_DISPATCHER_REQUIRED=true, and WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true on staging/production worker-general once Redis plus the bridge URL/token are provisioned; in that mode a missing dispatcher or bridge dependency fails startup instead of leaving outbox rows or outbox.workflow silently undrained behind a green process. Source-inventory tests guard that legacy WorkflowEngine.trigger(...) call-sites stay absent and that only the signed internal trigger bridge may start type: "trigger" executions directly. The row.created, row.updated, and row.deleted route-backed and workflow-context producers now write the row mutation, application event, and workflow.trigger.event outbox row in the same transaction when a request or workflow event context is passed. Other internal requestless mutations remain eventless by default to avoid worker feedback loops.

worker-general also registers the workflow-execution-stalled-reconcile maintenance scheduler. It repairs only old manual/API WorkflowExecution rows that are still running, have no block runs, have no waiting block, and whose matching workflow.execution.requested outbox event with dispatch key workflow-execution:<executionId> is already FAILED. It does not infer failure from DONE outbox rows because DONE only proves dispatch into BullMQ, not workflow completion.

Use pnpm run verify:workflow-outbox-native as the canonical non-browser gate before changing workflow outbox/native execution readiness. The gate composes existing focused tests for outbox.workflow routing, subscriber bootstrap, native/hybrid safe-core execution, worker health required-mode checks, and queued manual/API/continuation producer inventory. It does not prove a live environment is draining Redis jobs; pair it with the opt-in integration test or operator-gated E2E drain probe for live worker readiness.

Form submissions use a two-hop worker path. The public submit route stores the FormSubmission row and emits form.submission.process, which the outbox dispatcher routes to outbox.form-submission-route on worker-general. That subscriber performs tenant-scoped CRM/entity routing, marks the submission, and publishes follow-up events. Webhook follow-ups use webhook.form-submission.deliver, routed to outbox.webhook on worker-mail, where the bridge enqueues the existing toolctrl.email deliver-webhook job. This keeps webhook HTTP delivery in the mail/HTTP throughput group and leaves the general group responsible only for the database-backed routing decision.

Env variables per group

Required (per worker container):

Variable	Purpose
WORKER_QUEUE_GROUP	general / mail / import / webhook / plugin
WORKER_HEALTH_PORT	TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105
BULLMQ_REDIS_URL	Redis target
DATABASE_URL	Postgres (via PgBouncer in staging/production)

Required only on worker-general. The default execution adapter still bridges through the web runtime; hybrid canaries credential-free safe core event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped rowGet/email blocks, and single-boundary waitForInput continuation resumes, then falls back to the signed HTTP bridge for unsupported blocks or credential references; native is reserved for credential-free safe-core-only deployments and fails loud for unsupported blocks or credential references. The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless platform workflow trigger execution is intentionally unsupported in HTTP mode and should use native mode once the workflow is within the supported block contract. scripts/coolify-ensure-worker-groups.mjs verifies these settings on the Coolify worker apps before deploy: every managed worker group must have DATABASE_URL and BULLMQ_REDIS_URL configured, and worker-general must also have either a complete HTTP bridge (WORKFLOW_TRIGGER_EXECUTOR_URL / APP_URL / WEB_ORIGIN plus API_ACCESS_TOKEN) or explicit WORKFLOW_TRIGGER_EXECUTOR_MODE=native / WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid.

Variable	Purpose
WORKFLOW_TRIGGER_EXECUTOR_MODE=http	Explicitly selects the signed HTTP bridge. Use hybrid to canary worker-native credential-free safe-core execution with HTTP fallback; use native only for the worker-native safe-core MVP (event, manual, if, switch, doNothing, log, alertUser, variable, httpRequest, tenant-scoped rowGet/email, and single-boundary waitForInput continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution.
APP_URL, WEB_ORIGIN, or WORKFLOW_TRIGGER_EXECUTOR_URL	Web runtime origin used by outbox.workflow execution bridge
API_ACCESS_TOKEN	Canonical service token used by outbox.workflow; value must match the web runtime's API_ACCESS_TOKEN because the web runtime validates the signed internal bridge against that key
OUTBOX_DISPATCHER_ENABLED=true	Starts the transactional outbox dispatcher on worker-general
OUTBOX_DISPATCHER_REQUIRED=true	Fail startup if the general worker cannot start the dispatcher
OUTBOX_DONE_RETENTION_HOURS=720	Daily worker-general maintenance keeps DONE outbox rows for 30 days before purging them
OUTBOX_DONE_RETENTION_BATCH_SIZE=1000	Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually
WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30	Five-minute worker-general maintenance marks only untouched manual/API workflow executions as error when their initial workflow.execution.requested outbox dispatch is already FAILED and older than this threshold.
WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100	Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually.
WORKFLOW_TRIGGER_OUTBOX_ENABLED=true	Keeps the outbox.workflow subscriber active; set false only for controlled maintenance
WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5	BullMQ concurrency for workflow trigger fanout
WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true	Fail startup if the general worker cannot subscribe to outbox.workflow
PLUGIN_HOOK_FANOUT_REQUIRED=true	Optional staging/canary fail-closed switch for outbox.plugin-hook; production plugin runtime with PLUGIN_RUNTIME_PRODUCTION_ENABLED=true already requires fanout when Redis is configured

Coolify resources

Coolify must run queue groups as separate persistent applications that use the same worker image with different WORKER_QUEUE_GROUP values. The deploy workflow requires worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin resources in the staging and production resource allow-lists. worker-mail is a permanent service, not a manual workaround: without it, queued transactional mail and mail-backed outbox/webhook work is not drained after deploy. Without the other group resources, production traffic falls back into manual starts. This is an operator-owned runtime prerequisite: the repository and static verifiers document the required topology, but they do not by themselves prove that a target Coolify environment currently has every worker group deployed and healthy. The same operator/code split applies to plugin runtime status: docs and code require the worker-plugin service, while tenant-facing production execution remains blocked on external signoff and the explicit PLUGIN_RUNTIME_PRODUCTION_ENABLED=true flip. A live rollout must additionally pass pnpm run verify:plugin-runtime-live in both worker-general and worker-plugin so the general fanout/audit path and isolated invoke worker are checked separately.

Name the resources with the group marker (worker-general, worker-mail, worker-import, worker-webhook, worker-plugin) or expose the WORKER_QUEUE_GROUP=<group> marker in the resource metadata returned by Coolify. Keep WORKER_ENABLED=true on each worker resource and assign distinct health ports (9101 general, 9102 mail, 9103 import, 9104 webhook, 9105 plugin). When metrics/admin endpoints are enabled, keep them on separate ports (9201 general, 9202 mail, 9203 import, 9204 webhook, 9205 plugin) so the admin server does not collide with the health server.

scripts/coolify-ensure-worker-groups.mjs can create missing worker applications during the deploy workflow when provisioning is explicitly enabled. Staging uses:

COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
COOLIFY_STAGING_SERVER_UUID=<server uuid>
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
COOLIFY_STAGING_ENVIRONMENT_NAME=staging

Production uses the same keys without STAGING_: COOLIFY_PROVISION_WORKER_GROUPS, COOLIFY_PROJECT_UUID, COOLIFY_SERVER_UUID, COOLIFY_DESTINATION_UUID, and COOLIFY_ENVIRONMENT_NAME. When it creates a new group app, the script copies the existing worker app env from an allowed non-group worker resource or from COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID / COOLIFY_WORKER_ENV_SOURCE_UUID, then overwrites only the group-specific settings (WORKER_ENABLED, WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, WORKER_ADMIN_PORT, and group concurrency). Keep that source worker app available until every group resource has its own runtime env.

Optional overrides:

Variable	Default	Notes
WORKER_CONCURRENCY_GENERAL	10	Only read when WORKER_QUEUE_GROUP=general
WORKER_CONCURRENCY_MAIL	5	Only read when WORKER_QUEUE_GROUP=mail
WORKER_CONCURRENCY_IMPORT	1	Only read when WORKER_QUEUE_GROUP=import
WORKER_CONCURRENCY_WEBHOOK	20	Only read when WORKER_QUEUE_GROUP=webhook
WORKER_CONCURRENCY_PLUGIN	5	Only read when WORKER_QUEUE_GROUP=plugin
WORKER_ADMIN_PORT	Dockerfile/code default 9101; 9201-9205 in Coolify	Must differ from WORKER_HEALTH_PORT when METRICS_ENABLED=true
WORKER_HEALTH_HOST	0.0.0.0	Override when running outside Docker on multi-NIC hosts
WORKER_REDIS_POOL_SIZE	4/4/2/8/4 (general/mail/import/webhook/plugin)	Advisory ioredis pool size
TRANSACTIONAL_MAIL_OUTBOX_REQUIRED	false; managed Coolify worker-mail uses true	Mail-group only. Set to true in staging/production so worker-mail exits if the outbox.mail bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed worker-mail apps.

Health endpoints

Every worker exposes two HTTP probes on WORKER_HEALTH_PORT:

• GET /healthz — liveness. Returns 200 as long as the event loop is responsive. No external dependency calls; safe to hammer.
• GET /readyz — readiness. Runs a Redis ping and a Prisma SELECT 1 in parallel with a 2-second timeout each. Returns 200 only when both succeed; 503 otherwise. Queue-group subscriber probes are also included for required/expected runtime bridges such as outbox.mail on WORKER_QUEUE_GROUP=mail.

Use /readyz as the Coolify / Kubernetes readiness probe and /healthz as the liveness probe.

Worker health is primarily a queue-runtime check. For managed worker-mail apps with TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true, health diagnostics also fail loud when the required outbound provider pair MAILJET_API_KEY and MAILJET_FROM_EMAIL is absent, using key names only. A healthy worker-mail therefore proves Redis/Postgres reachability, the mail queue process, the transactional outbox bridge, and required Mailjet env presence; it still does not prove sender-domain verification or live provider delivery. Managed Coolify worker-mail apps must keep TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true so the process fails fast when outbox.mail cannot subscribe; without that bridge, invite and other mail.transactional.* outbox events can remain pending even if the direct toolctrl.email worker is alive. The Coolify worker-group provisioner writes and drift-checks this env value for managed mail workers.

Example:

curl -fsS http://worker-mail:9102/healthz
curl -fsS http://worker-mail:9102/readyz

Connection naming

Each worker tags its ioredis connection with worker-<group>-<pid> (via apps/worker/src/connection.ts). Run CLIENT LIST against Redis to attribute traffic to a specific worker group:

CLIENT LIST
# id=42 name=worker-mail-218 ...
# id=43 name=worker-webhook-219 ...

Scaling cookbook

Symptom	Action
Mail queue depth > 1 000 sustained	docker compose up -d --scale worker-mail=4
Import queue depth > 10 sustained	docker compose up -d --scale worker-import=2; review job size first — single jobs taking >10min are usually the real fix
AI/browser external-call queue latency p95 climbs	docker compose up -d --scale worker-webhook=5 (each replica handles 20 parallel)
Form-submission webhook or mail outbox latency climbs	docker compose up -d --scale worker-mail=4; review SMTP/HTTP provider throttling before raising concurrency further
One worker stuck at 100 % CPU	Inspect /readyz — if it's still 200, profile inside the process; if 503, look at the checks payload to see which dep failed

Operator commands

# Inspect all worker-group logs
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin

# Restart a single group
docker compose restart worker-import

# Smoke-test all five readiness endpoints
for port in 9101 9102 9103 9104 9105; do
  curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
done

Troubleshooting

Wrong or missing worker group. The worker Docker image defaults WORKER_QUEUE_GROUP to general for raw image users. Coolify group apps must still set the explicit group marker (general, mail, import, webhook, or plugin) so each app drains the intended queues. A missing Coolify override can therefore start as another general worker instead of a mail/import/webhook/plugin worker; scripts/coolify-ensure-worker-groups.mjs treats that as env drift. An unknown group still fails worker startup when the worker is enabled. Disabled workers (WORKER_ENABLED=false) skip this check so dev environments without Redis still boot.

/readyz returns 503 with checks.redis="error" — BullMQ couldn't reach Redis. Confirm BULLMQ_REDIS_URL and that the redis container is healthy. The error is intentionally short (no host names) so the probe response never leaks credentials.

/readyz returns 503 with checks.database="error" — Prisma couldn't run SELECT 1. Check DATABASE_URL, PgBouncer health, and whether the pool is saturated.

A queue is silent on every worker — verify the queue is mapped to a group in packages/queue/src/groups.ts. If you just added a new queue, the groups.test.ts exhaustiveness check should have failed CI; if it didn't, the queue is missing from names.ts.

Outbox dispatcher missing. The outbox dispatcher runs on the general group only (see apps/worker/src/index.ts). If your stack runs no general replica, outbox events will not dispatch — keep at least one general replica.

Rollback

Current production worker topology requires an explicit WORKER_QUEUE_GROUP on each Coolify group app whenever WORKER_ENABLED=true; there is no supported WORKER_QUEUE_GROUP=all runtime rollback path. The Dockerfile's standalone default is general, so removing the marker during rollback can silently turn a non-general app into another general worker, while an unknown group fails startup by design. For an incident rollback, redeploy the previous known-good image/tag or compose/deploy definition, or keep the five group apps and roll the affected group back independently. In-flight BullMQ job state lives in Redis, so jobs are not lost while workers are rolled.