382 lines
36 KiB
Markdown
382 lines
36 KiB
Markdown
# Worker queue groups — operations guide
|
|
|
|
The standalone `apps/worker` runtime is split into five queue groups. Each
|
|
group is the same image and the same code; the only difference is the
|
|
`WORKER_QUEUE_GROUP` env variable that decides which BullMQ queues the
|
|
process subscribes to.
|
|
|
|
This page is the canonical worker topology reference. Shorter worker tables
|
|
in deployment, environment, or feature docs are summaries only and should link
|
|
back here instead of redefining queue ownership.
|
|
|
|
## Why five groups
|
|
|
|
A single worker process processed every queue. That had three operational
|
|
problems:
|
|
|
|
1. **Head-of-line blocking.** A long-running import (minutes) stalled
|
|
transactional mail and notifications that ran on different queues but
|
|
the same event loop.
|
|
2. **Wrong concurrency for the workload.** Webhook delivery needs high
|
|
parallelism (I/O-bound HTTP calls); imports need low parallelism
|
|
(CPU/memory-bound). One `WORKER_CONCURRENCY` value cannot suit both.
|
|
3. **No per-workload scaling.** When mail queues backlog we couldn't scale
|
|
mail capacity without also duplicating the import worker (waste of RAM
|
|
and Redis connections).
|
|
|
|
The split fixes all three by giving each workload its own process and its
|
|
own concurrency. Add replicas to only the group that needs them.
|
|
|
|
## Production worker matrix
|
|
|
|
Production-like Coolify environments must run one persistent worker service per
|
|
queue group. These are long-lived internal services from the same worker image,
|
|
not manual emergency processes and not public web apps. `worker-mail` with
|
|
`WORKER_QUEUE_GROUP=mail` is part of the baseline topology because
|
|
transactional mail, mail outbox, form-submission webhook delivery, and
|
|
email-sequence worker contracts are intentionally isolated from general outbox
|
|
fanout.
|
|
|
|
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
|
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
|
|
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
|
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
|
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
|
|
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
|
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
|
|
|
The detailed queue mapping below is the source of truth for package queues,
|
|
subscriber queues, concurrency, and workload profile. Update this page first
|
|
when queue ownership changes; deployment-specific docs should keep only the
|
|
short app/group/port summary and link back here.
|
|
|
|
`worker-plugin` is required as a persistent Coolify service so the deploy
|
|
topology is stable, but production plugin execution is still gated. With
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
|
|
as an idle topology placeholder and must not consume `toolctrl.plugin-invoke`.
|
|
Only after the external WASM sandbox signoff in
|
|
`docs/security/plugin-runtime-signoff.md` may operators set
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
|
|
accept plugin invoke consumption as production-ready. The final live proof is
|
|
operator-run, inside both `worker-general` and `worker-plugin`:
|
|
`pnpm run verify:plugin-runtime-live`.
|
|
|
|
`worker-import` follows the same placeholder-first principle. Provision the
|
|
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
|
|
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
|
|
export, and scan services are wired into the worker bootstrap. In that idle
|
|
state `/readyz` may be healthy with an empty queue list, which proves topology
|
|
without consuming `toolctrl.imports` or `toolctrl.exports`. When both runtime
|
|
gates are set, startup fails closed unless the worker has real import, export,
|
|
and scan service dependencies.
|
|
|
|
## Group → queue mapping
|
|
|
|
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
|
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
|
| `mail` | `toolctrl.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
|
| `import` | `toolctrl.imports`, `toolctrl.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
|
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
|
| `plugin` | `toolctrl.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
|
|
|
Source of truth: `packages/queue/src/groups.ts`. A test
|
|
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
|
group — adding a new package queue forces a deliberate group assignment.
|
|
Outbox and auxiliary subscriber queues are wired by `apps/worker/src/index.ts`
|
|
through their dedicated bootstrap modules; update this table when adding a
|
|
new subscriber that is not represented in `packages/queue`.
|
|
|
|
`outbox.workflow` runs on the `general` group. The worker performs
|
|
live-workflow lookup and preflight checks in `apps/worker/src/jobs`, then
|
|
starts matching workflows through the selected executor: signed internal
|
|
workflow-run API by default, or hybrid/native execution when explicitly
|
|
configured. The native executor is a credential-free safe-core path: it must
|
|
not hydrate stored workflow credentials, and any `$credentials.*`,
|
|
`lookup $credentials`, or `#with $credentials` reference is a preflight gap.
|
|
That bridge avoids importing `apps/web` server modules into the worker
|
|
while keeping the existing workflow execution semantics centralized. The
|
|
same receiver also drains `workflow.execution.requested` rows produced by
|
|
manual workflow runs and tenant API workflow runs: the web action creates the
|
|
`WorkflowExecution` and outbox row in one transaction, then worker-general
|
|
starts that existing execution through the signed internal execution-start
|
|
bridge or the native safe-core executor. Waiting execution resumes publish
|
|
`workflow.execution.continue.requested`; worker-general then resumes the
|
|
existing execution through the signed internal continuation bridge or, in
|
|
native/hybrid mode, through the worker-native safe-core continuation path when
|
|
the workflow's only native preflight gap is the current `waitForInput` boundary.
|
|
Unsupported continuation graphs still fall back to the HTTP bridge in `hybrid`
|
|
mode and fail loud in `native` mode. Stream runs remain separate until their
|
|
progress contract moves to a distributed execution path. The
|
|
subscriber/bootstrap path is active from `apps/worker/src/index.ts` when the
|
|
`general` worker is configured with the workflow execution settings.
|
|
Set `OUTBOX_DISPATCHER_ENABLED=true`, `OUTBOX_DISPATCHER_REQUIRED=true`, and
|
|
`WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` on staging/production `worker-general`
|
|
once Redis plus the bridge URL/token are provisioned; in that mode a missing
|
|
dispatcher or bridge dependency fails startup instead of leaving outbox rows or
|
|
`outbox.workflow` silently undrained behind a green process.
|
|
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
|
|
call-sites stay absent and that only the signed internal trigger bridge may
|
|
start `type: "trigger"` executions directly. The `row.created`, `row.updated`,
|
|
and `row.deleted` route-backed and workflow-context producers now write the row
|
|
mutation, application event, and `workflow.trigger.event` outbox row in the same
|
|
transaction when a request or workflow event context is passed. Other internal
|
|
requestless mutations remain eventless by default to avoid worker feedback loops.
|
|
|
|
`worker-general` also registers the
|
|
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
|
|
old manual/API `WorkflowExecution` rows that are still `running`, have no block
|
|
runs, have no waiting block, and whose matching
|
|
`workflow.execution.requested` outbox event with dispatch key
|
|
`workflow-execution:<executionId>` is already `FAILED`. It does not infer
|
|
failure from `DONE` outbox rows because `DONE` only proves dispatch into BullMQ,
|
|
not workflow completion.
|
|
|
|
Use `pnpm run verify:workflow-outbox-native` as the canonical non-browser gate
|
|
before changing workflow outbox/native execution readiness. The gate composes
|
|
existing focused tests for `outbox.workflow` routing, subscriber bootstrap,
|
|
native/hybrid safe-core execution, worker health required-mode checks, and
|
|
queued manual/API/continuation producer inventory. It does not prove a live
|
|
environment is draining Redis jobs; pair it with the opt-in integration test or
|
|
operator-gated E2E drain probe for live worker readiness.
|
|
|
|
Form submissions use a two-hop worker path. The public submit route stores
|
|
the `FormSubmission` row and emits `form.submission.process`, which the
|
|
outbox dispatcher routes to `outbox.form-submission-route` on
|
|
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
|
|
marks the submission, and publishes follow-up events. Webhook follow-ups use
|
|
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
|
|
`worker-mail`, where the bridge enqueues the existing `toolctrl.email`
|
|
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
|
|
throughput group and leaves the general group responsible only for the
|
|
database-backed routing decision.
|
|
|
|
## Env variables per group
|
|
|
|
Required (per worker container):
|
|
|
|
| Variable | Purpose |
|
|
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKER_QUEUE_GROUP` | `general` / `mail` / `import` / `webhook` / `plugin` |
|
|
| `WORKER_HEALTH_PORT` | TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105 |
|
|
| `BULLMQ_REDIS_URL` | Redis target |
|
|
| `DATABASE_URL` | Postgres (via PgBouncer in staging/production) |
|
|
|
|
Required only on `worker-general`. The default execution adapter still bridges
|
|
through the web runtime; `hybrid` canaries credential-free safe core
|
|
event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped
|
|
rowGet/email blocks, and single-boundary `waitForInput` continuation resumes,
|
|
then falls back to the signed HTTP bridge for unsupported blocks or credential
|
|
references; `native` is reserved for credential-free safe-core-only deployments
|
|
and fails loud for unsupported blocks or credential references.
|
|
The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless
|
|
platform workflow trigger execution is intentionally unsupported in HTTP mode
|
|
and should use native mode once the workflow is within the supported block
|
|
contract.
|
|
`scripts/coolify-ensure-worker-groups.mjs` verifies these settings on the
|
|
Coolify worker apps before deploy: every managed worker group must have
|
|
`DATABASE_URL` and `BULLMQ_REDIS_URL` configured, and `worker-general` must
|
|
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
|
|
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
|
|
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
|
|
|
|
| Variable | Purpose |
|
|
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
|
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
|
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
|
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
|
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
|
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
|
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
|
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
|
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
|
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
|
|
|
## Coolify resources
|
|
|
|
Coolify must run queue groups as separate persistent applications that use
|
|
the same `worker` image with different `WORKER_QUEUE_GROUP` values. The
|
|
deploy workflow requires `worker-general`, `worker-mail`,
|
|
`worker-import`, `worker-webhook`, and `worker-plugin` resources in the staging and
|
|
production resource allow-lists. `worker-mail` is a permanent service, not a
|
|
manual workaround: without it, queued transactional mail and mail-backed
|
|
outbox/webhook work is not drained after deploy. Without the other group
|
|
resources, production traffic falls back into manual starts.
|
|
This is an operator-owned runtime prerequisite: the repository and static
|
|
verifiers document the required topology, but they do not by themselves prove
|
|
that a target Coolify environment currently has every worker group deployed and
|
|
healthy.
|
|
The same operator/code split applies to plugin runtime status: docs and code
|
|
require the `worker-plugin` service, while tenant-facing production execution
|
|
remains blocked on external signoff and the explicit
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` flip. A live rollout must additionally
|
|
pass `pnpm run verify:plugin-runtime-live` in both `worker-general` and
|
|
`worker-plugin` so the general fanout/audit path and isolated invoke worker are
|
|
checked separately.
|
|
|
|
Name the resources with the group marker (`worker-general`,
|
|
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) or expose the
|
|
`WORKER_QUEUE_GROUP=<group>` marker in the resource metadata returned by
|
|
Coolify. Keep `WORKER_ENABLED=true` on each worker resource and assign
|
|
distinct health ports (`9101` general, `9102` mail, `9103` import,
|
|
`9104` webhook, `9105` plugin). When metrics/admin endpoints are enabled, keep them on
|
|
separate ports (`9201` general, `9202` mail, `9203` import, `9204`
|
|
webhook, `9205` plugin) so the admin server does not collide with the health server.
|
|
|
|
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
|
|
applications during the deploy workflow when provisioning is explicitly
|
|
enabled. Staging uses:
|
|
|
|
```bash
|
|
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
|
|
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
|
|
COOLIFY_STAGING_SERVER_UUID=<server uuid>
|
|
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
|
|
COOLIFY_STAGING_ENVIRONMENT_NAME=staging
|
|
```
|
|
|
|
Production uses the same keys without `STAGING_`:
|
|
`COOLIFY_PROVISION_WORKER_GROUPS`, `COOLIFY_PROJECT_UUID`,
|
|
`COOLIFY_SERVER_UUID`, `COOLIFY_DESTINATION_UUID`, and
|
|
`COOLIFY_ENVIRONMENT_NAME`. When it creates a new group app, the script
|
|
copies the existing worker app env from an allowed non-group worker
|
|
resource or from `COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID` /
|
|
`COOLIFY_WORKER_ENV_SOURCE_UUID`, then overwrites only the group-specific
|
|
settings (`WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`,
|
|
`WORKER_ADMIN_PORT`, and group concurrency). Keep that source worker app
|
|
available until every group resource has its own runtime env.
|
|
|
|
Optional overrides:
|
|
|
|
| Variable | Default | Notes |
|
|
| ------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `WORKER_CONCURRENCY_GENERAL` | 10 | Only read when `WORKER_QUEUE_GROUP=general` |
|
|
| `WORKER_CONCURRENCY_MAIL` | 5 | Only read when `WORKER_QUEUE_GROUP=mail` |
|
|
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
|
|
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
|
|
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
|
|
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
|
|
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
|
|
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
|
|
| `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `false`; managed Coolify `worker-mail` uses `true` | Mail-group only. Set to `true` in staging/production so `worker-mail` exits if the `outbox.mail` bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed `worker-mail` apps. |
|
|
|
|
## Health endpoints
|
|
|
|
Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
|
|
|
|
- `GET /healthz` — liveness. Returns 200 as long as the event loop is
|
|
responsive. No external dependency calls; safe to hammer.
|
|
- `GET /readyz` — readiness. Runs a Redis ping and a Prisma `SELECT 1`
|
|
in parallel with a 2-second timeout each. Returns 200 only when both
|
|
succeed; 503 otherwise. Queue-group subscriber probes are also included
|
|
for required/expected runtime bridges such as `outbox.mail` on
|
|
`WORKER_QUEUE_GROUP=mail`.
|
|
|
|
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
|
|
as the liveness probe.
|
|
|
|
Worker health is primarily a queue-runtime check. For managed `worker-mail`
|
|
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
|
|
fail loud when the required outbound provider pair `MAILJET_API_KEY` and
|
|
`MAILJET_FROM_EMAIL` is absent, using key names only. A healthy
|
|
`worker-mail` therefore proves Redis/Postgres reachability, the mail queue
|
|
process, the transactional outbox bridge, and required Mailjet env presence;
|
|
it still does not prove sender-domain verification or live provider delivery.
|
|
Managed Coolify `worker-mail` apps must keep
|
|
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
|
|
`outbox.mail` cannot subscribe; without that bridge, invite and other
|
|
`mail.transactional.*` outbox events can remain pending even if the direct
|
|
`toolctrl.email` worker is alive. The Coolify worker-group provisioner writes
|
|
and drift-checks this env value for managed mail workers.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
curl -fsS http://worker-mail:9102/healthz
|
|
curl -fsS http://worker-mail:9102/readyz
|
|
```
|
|
|
|
## Connection naming
|
|
|
|
Each worker tags its ioredis connection with `worker-<group>-<pid>`
|
|
(via `apps/worker/src/connection.ts`). Run `CLIENT LIST` against Redis
|
|
to attribute traffic to a specific worker group:
|
|
|
|
```redis
|
|
CLIENT LIST
|
|
# id=42 name=worker-mail-218 ...
|
|
# id=43 name=worker-webhook-219 ...
|
|
```
|
|
|
|
## Scaling cookbook
|
|
|
|
| Symptom | Action |
|
|
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
|
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
|
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
|
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
|
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
|
|
|
## Operator commands
|
|
|
|
```bash
|
|
# Inspect all worker-group logs
|
|
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin
|
|
|
|
# Restart a single group
|
|
docker compose restart worker-import
|
|
|
|
# Smoke-test all five readiness endpoints
|
|
for port in 9101 9102 9103 9104 9105; do
|
|
curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
|
|
done
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
**Wrong or missing worker group.** The worker Docker image defaults
|
|
`WORKER_QUEUE_GROUP` to `general` for raw image users. Coolify group apps
|
|
must still set the explicit group marker (`general`, `mail`, `import`,
|
|
`webhook`, or `plugin`) so each app drains the intended queues. A missing
|
|
Coolify override can therefore start as another `general` worker instead of a
|
|
mail/import/webhook/plugin worker; `scripts/coolify-ensure-worker-groups.mjs`
|
|
treats that as env drift. An unknown group still fails worker startup when the
|
|
worker is enabled. Disabled workers (`WORKER_ENABLED=false`) skip this check so
|
|
dev environments without Redis still boot.
|
|
|
|
**`/readyz` returns 503 with `checks.redis="error"`** — BullMQ couldn't
|
|
reach Redis. Confirm `BULLMQ_REDIS_URL` and that the redis container is
|
|
healthy. The error is intentionally short (no host names) so the probe
|
|
response never leaks credentials.
|
|
|
|
**`/readyz` returns 503 with `checks.database="error"`** — Prisma
|
|
couldn't run `SELECT 1`. Check `DATABASE_URL`, PgBouncer health, and
|
|
whether the pool is saturated.
|
|
|
|
**A queue is silent on every worker** — verify the queue is mapped to a
|
|
group in `packages/queue/src/groups.ts`. If you just added a new queue,
|
|
the `groups.test.ts` exhaustiveness check should have failed CI; if it
|
|
didn't, the queue is missing from `names.ts`.
|
|
|
|
**Outbox dispatcher missing.** The outbox dispatcher runs on the
|
|
`general` group only (see `apps/worker/src/index.ts`). If your stack
|
|
runs no `general` replica, outbox events will not dispatch — keep at
|
|
least one `general` replica.
|
|
|
|
## Rollback
|
|
|
|
Current production worker topology requires an explicit `WORKER_QUEUE_GROUP`
|
|
on each Coolify group app whenever `WORKER_ENABLED=true`; there is no
|
|
supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
|
|
standalone default is `general`, so removing the marker during rollback can
|
|
silently turn a non-general app into another general worker, while an unknown
|
|
group fails startup by design. For an incident rollback, redeploy the previous
|
|
known-good image/tag or compose/deploy definition, or keep the five group apps
|
|
and roll the affected group back independently. In-flight BullMQ job state
|
|
lives in Redis, so jobs are not lost while workers are rolled.
|