Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,381 @@
|
||||
# Worker queue groups — operations guide
|
||||
|
||||
The standalone `apps/worker` runtime is split into five queue groups. Each
|
||||
group is the same image and the same code; the only difference is the
|
||||
`WORKER_QUEUE_GROUP` env variable that decides which BullMQ queues the
|
||||
process subscribes to.
|
||||
|
||||
This page is the canonical worker topology reference. Shorter worker tables
|
||||
in deployment, environment, or feature docs are summaries only and should link
|
||||
back here instead of redefining queue ownership.
|
||||
|
||||
## Why five groups
|
||||
|
||||
A single worker process processed every queue. That had three operational
|
||||
problems:
|
||||
|
||||
1. **Head-of-line blocking.** A long-running import (minutes) stalled
|
||||
transactional mail and notifications that ran on different queues but
|
||||
the same event loop.
|
||||
2. **Wrong concurrency for the workload.** Webhook delivery needs high
|
||||
parallelism (I/O-bound HTTP calls); imports need low parallelism
|
||||
(CPU/memory-bound). One `WORKER_CONCURRENCY` value cannot suit both.
|
||||
3. **No per-workload scaling.** When mail queues backlog we couldn't scale
|
||||
mail capacity without also duplicating the import worker (waste of RAM
|
||||
and Redis connections).
|
||||
|
||||
The split fixes all three by giving each workload its own process and its
|
||||
own concurrency. Add replicas to only the group that needs them.
|
||||
|
||||
## Production worker matrix
|
||||
|
||||
Production-like Coolify environments must run one persistent worker service per
|
||||
queue group. These are long-lived internal services from the same worker image,
|
||||
not manual emergency processes and not public web apps. `worker-mail` with
|
||||
`WORKER_QUEUE_GROUP=mail` is part of the baseline topology because
|
||||
transactional mail, mail outbox, form-submission webhook delivery, and
|
||||
email-sequence worker contracts are intentionally isolated from general outbox
|
||||
fanout.
|
||||
|
||||
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
||||
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
||||
|
||||
The detailed queue mapping below is the source of truth for package queues,
|
||||
subscriber queues, concurrency, and workload profile. Update this page first
|
||||
when queue ownership changes; deployment-specific docs should keep only the
|
||||
short app/group/port summary and link back here.
|
||||
|
||||
`worker-plugin` is required as a persistent Coolify service so the deploy
|
||||
topology is stable, but production plugin execution is still gated. With
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
|
||||
as an idle topology placeholder and must not consume `toolctrl.plugin-invoke`.
|
||||
Only after the external WASM sandbox signoff in
|
||||
`docs/security/plugin-runtime-signoff.md` may operators set
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
|
||||
accept plugin invoke consumption as production-ready. The final live proof is
|
||||
operator-run, inside both `worker-general` and `worker-plugin`:
|
||||
`pnpm run verify:plugin-runtime-live`.
|
||||
|
||||
`worker-import` follows the same placeholder-first principle. Provision the
|
||||
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
|
||||
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
|
||||
export, and scan services are wired into the worker bootstrap. In that idle
|
||||
state `/readyz` may be healthy with an empty queue list, which proves topology
|
||||
without consuming `toolctrl.imports` or `toolctrl.exports`. When both runtime
|
||||
gates are set, startup fails closed unless the worker has real import, export,
|
||||
and scan service dependencies.
|
||||
|
||||
## Group → queue mapping
|
||||
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `toolctrl.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | `toolctrl.imports`, `toolctrl.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
||||
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `toolctrl.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
|
||||
Source of truth: `packages/queue/src/groups.ts`. A test
|
||||
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
||||
group — adding a new package queue forces a deliberate group assignment.
|
||||
Outbox and auxiliary subscriber queues are wired by `apps/worker/src/index.ts`
|
||||
through their dedicated bootstrap modules; update this table when adding a
|
||||
new subscriber that is not represented in `packages/queue`.
|
||||
|
||||
`outbox.workflow` runs on the `general` group. The worker performs
|
||||
live-workflow lookup and preflight checks in `apps/worker/src/jobs`, then
|
||||
starts matching workflows through the selected executor: signed internal
|
||||
workflow-run API by default, or hybrid/native execution when explicitly
|
||||
configured. The native executor is a credential-free safe-core path: it must
|
||||
not hydrate stored workflow credentials, and any `$credentials.*`,
|
||||
`lookup $credentials`, or `#with $credentials` reference is a preflight gap.
|
||||
That bridge avoids importing `apps/web` server modules into the worker
|
||||
while keeping the existing workflow execution semantics centralized. The
|
||||
same receiver also drains `workflow.execution.requested` rows produced by
|
||||
manual workflow runs and tenant API workflow runs: the web action creates the
|
||||
`WorkflowExecution` and outbox row in one transaction, then worker-general
|
||||
starts that existing execution through the signed internal execution-start
|
||||
bridge or the native safe-core executor. Waiting execution resumes publish
|
||||
`workflow.execution.continue.requested`; worker-general then resumes the
|
||||
existing execution through the signed internal continuation bridge or, in
|
||||
native/hybrid mode, through the worker-native safe-core continuation path when
|
||||
the workflow's only native preflight gap is the current `waitForInput` boundary.
|
||||
Unsupported continuation graphs still fall back to the HTTP bridge in `hybrid`
|
||||
mode and fail loud in `native` mode. Stream runs remain separate until their
|
||||
progress contract moves to a distributed execution path. The
|
||||
subscriber/bootstrap path is active from `apps/worker/src/index.ts` when the
|
||||
`general` worker is configured with the workflow execution settings.
|
||||
Set `OUTBOX_DISPATCHER_ENABLED=true`, `OUTBOX_DISPATCHER_REQUIRED=true`, and
|
||||
`WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` on staging/production `worker-general`
|
||||
once Redis plus the bridge URL/token are provisioned; in that mode a missing
|
||||
dispatcher or bridge dependency fails startup instead of leaving outbox rows or
|
||||
`outbox.workflow` silently undrained behind a green process.
|
||||
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
|
||||
call-sites stay absent and that only the signed internal trigger bridge may
|
||||
start `type: "trigger"` executions directly. The `row.created`, `row.updated`,
|
||||
and `row.deleted` route-backed and workflow-context producers now write the row
|
||||
mutation, application event, and `workflow.trigger.event` outbox row in the same
|
||||
transaction when a request or workflow event context is passed. Other internal
|
||||
requestless mutations remain eventless by default to avoid worker feedback loops.
|
||||
|
||||
`worker-general` also registers the
|
||||
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
|
||||
old manual/API `WorkflowExecution` rows that are still `running`, have no block
|
||||
runs, have no waiting block, and whose matching
|
||||
`workflow.execution.requested` outbox event with dispatch key
|
||||
`workflow-execution:<executionId>` is already `FAILED`. It does not infer
|
||||
failure from `DONE` outbox rows because `DONE` only proves dispatch into BullMQ,
|
||||
not workflow completion.
|
||||
|
||||
Use `pnpm run verify:workflow-outbox-native` as the canonical non-browser gate
|
||||
before changing workflow outbox/native execution readiness. The gate composes
|
||||
existing focused tests for `outbox.workflow` routing, subscriber bootstrap,
|
||||
native/hybrid safe-core execution, worker health required-mode checks, and
|
||||
queued manual/API/continuation producer inventory. It does not prove a live
|
||||
environment is draining Redis jobs; pair it with the opt-in integration test or
|
||||
operator-gated E2E drain probe for live worker readiness.
|
||||
|
||||
Form submissions use a two-hop worker path. The public submit route stores
|
||||
the `FormSubmission` row and emits `form.submission.process`, which the
|
||||
outbox dispatcher routes to `outbox.form-submission-route` on
|
||||
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
|
||||
marks the submission, and publishes follow-up events. Webhook follow-ups use
|
||||
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
|
||||
`worker-mail`, where the bridge enqueues the existing `toolctrl.email`
|
||||
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
|
||||
throughput group and leaves the general group responsible only for the
|
||||
database-backed routing decision.
|
||||
|
||||
## Env variables per group
|
||||
|
||||
Required (per worker container):
|
||||
|
||||
| Variable | Purpose |
|
||||
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKER_QUEUE_GROUP` | `general` / `mail` / `import` / `webhook` / `plugin` |
|
||||
| `WORKER_HEALTH_PORT` | TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105 |
|
||||
| `BULLMQ_REDIS_URL` | Redis target |
|
||||
| `DATABASE_URL` | Postgres (via PgBouncer in staging/production) |
|
||||
|
||||
Required only on `worker-general`. The default execution adapter still bridges
|
||||
through the web runtime; `hybrid` canaries credential-free safe core
|
||||
event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped
|
||||
rowGet/email blocks, and single-boundary `waitForInput` continuation resumes,
|
||||
then falls back to the signed HTTP bridge for unsupported blocks or credential
|
||||
references; `native` is reserved for credential-free safe-core-only deployments
|
||||
and fails loud for unsupported blocks or credential references.
|
||||
The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless
|
||||
platform workflow trigger execution is intentionally unsupported in HTTP mode
|
||||
and should use native mode once the workflow is within the supported block
|
||||
contract.
|
||||
`scripts/coolify-ensure-worker-groups.mjs` verifies these settings on the
|
||||
Coolify worker apps before deploy: every managed worker group must have
|
||||
`DATABASE_URL` and `BULLMQ_REDIS_URL` configured, and `worker-general` must
|
||||
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
|
||||
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
|
||||
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
||||
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
||||
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
||||
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
||||
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
||||
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
||||
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
||||
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
||||
|
||||
## Coolify resources
|
||||
|
||||
Coolify must run queue groups as separate persistent applications that use
|
||||
the same `worker` image with different `WORKER_QUEUE_GROUP` values. The
|
||||
deploy workflow requires `worker-general`, `worker-mail`,
|
||||
`worker-import`, `worker-webhook`, and `worker-plugin` resources in the staging and
|
||||
production resource allow-lists. `worker-mail` is a permanent service, not a
|
||||
manual workaround: without it, queued transactional mail and mail-backed
|
||||
outbox/webhook work is not drained after deploy. Without the other group
|
||||
resources, production traffic falls back into manual starts.
|
||||
This is an operator-owned runtime prerequisite: the repository and static
|
||||
verifiers document the required topology, but they do not by themselves prove
|
||||
that a target Coolify environment currently has every worker group deployed and
|
||||
healthy.
|
||||
The same operator/code split applies to plugin runtime status: docs and code
|
||||
require the `worker-plugin` service, while tenant-facing production execution
|
||||
remains blocked on external signoff and the explicit
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` flip. A live rollout must additionally
|
||||
pass `pnpm run verify:plugin-runtime-live` in both `worker-general` and
|
||||
`worker-plugin` so the general fanout/audit path and isolated invoke worker are
|
||||
checked separately.
|
||||
|
||||
Name the resources with the group marker (`worker-general`,
|
||||
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) or expose the
|
||||
`WORKER_QUEUE_GROUP=<group>` marker in the resource metadata returned by
|
||||
Coolify. Keep `WORKER_ENABLED=true` on each worker resource and assign
|
||||
distinct health ports (`9101` general, `9102` mail, `9103` import,
|
||||
`9104` webhook, `9105` plugin). When metrics/admin endpoints are enabled, keep them on
|
||||
separate ports (`9201` general, `9202` mail, `9203` import, `9204`
|
||||
webhook, `9205` plugin) so the admin server does not collide with the health server.
|
||||
|
||||
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
|
||||
applications during the deploy workflow when provisioning is explicitly
|
||||
enabled. Staging uses:
|
||||
|
||||
```bash
|
||||
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
|
||||
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
|
||||
COOLIFY_STAGING_SERVER_UUID=<server uuid>
|
||||
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
|
||||
COOLIFY_STAGING_ENVIRONMENT_NAME=staging
|
||||
```
|
||||
|
||||
Production uses the same keys without `STAGING_`:
|
||||
`COOLIFY_PROVISION_WORKER_GROUPS`, `COOLIFY_PROJECT_UUID`,
|
||||
`COOLIFY_SERVER_UUID`, `COOLIFY_DESTINATION_UUID`, and
|
||||
`COOLIFY_ENVIRONMENT_NAME`. When it creates a new group app, the script
|
||||
copies the existing worker app env from an allowed non-group worker
|
||||
resource or from `COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID` /
|
||||
`COOLIFY_WORKER_ENV_SOURCE_UUID`, then overwrites only the group-specific
|
||||
settings (`WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`,
|
||||
`WORKER_ADMIN_PORT`, and group concurrency). Keep that source worker app
|
||||
available until every group resource has its own runtime env.
|
||||
|
||||
Optional overrides:
|
||||
|
||||
| Variable | Default | Notes |
|
||||
| ------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKER_CONCURRENCY_GENERAL` | 10 | Only read when `WORKER_QUEUE_GROUP=general` |
|
||||
| `WORKER_CONCURRENCY_MAIL` | 5 | Only read when `WORKER_QUEUE_GROUP=mail` |
|
||||
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
|
||||
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
|
||||
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
|
||||
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
|
||||
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
|
||||
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
|
||||
| `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `false`; managed Coolify `worker-mail` uses `true` | Mail-group only. Set to `true` in staging/production so `worker-mail` exits if the `outbox.mail` bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed `worker-mail` apps. |
|
||||
|
||||
## Health endpoints
|
||||
|
||||
Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
|
||||
|
||||
- `GET /healthz` — liveness. Returns 200 as long as the event loop is
|
||||
responsive. No external dependency calls; safe to hammer.
|
||||
- `GET /readyz` — readiness. Runs a Redis ping and a Prisma `SELECT 1`
|
||||
in parallel with a 2-second timeout each. Returns 200 only when both
|
||||
succeed; 503 otherwise. Queue-group subscriber probes are also included
|
||||
for required/expected runtime bridges such as `outbox.mail` on
|
||||
`WORKER_QUEUE_GROUP=mail`.
|
||||
|
||||
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
|
||||
as the liveness probe.
|
||||
|
||||
Worker health is primarily a queue-runtime check. For managed `worker-mail`
|
||||
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
|
||||
fail loud when the required outbound provider pair `MAILJET_API_KEY` and
|
||||
`MAILJET_FROM_EMAIL` is absent, using key names only. A healthy
|
||||
`worker-mail` therefore proves Redis/Postgres reachability, the mail queue
|
||||
process, the transactional outbox bridge, and required Mailjet env presence;
|
||||
it still does not prove sender-domain verification or live provider delivery.
|
||||
Managed Coolify `worker-mail` apps must keep
|
||||
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
|
||||
`outbox.mail` cannot subscribe; without that bridge, invite and other
|
||||
`mail.transactional.*` outbox events can remain pending even if the direct
|
||||
`toolctrl.email` worker is alive. The Coolify worker-group provisioner writes
|
||||
and drift-checks this env value for managed mail workers.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
curl -fsS http://worker-mail:9102/healthz
|
||||
curl -fsS http://worker-mail:9102/readyz
|
||||
```
|
||||
|
||||
## Connection naming
|
||||
|
||||
Each worker tags its ioredis connection with `worker-<group>-<pid>`
|
||||
(via `apps/worker/src/connection.ts`). Run `CLIENT LIST` against Redis
|
||||
to attribute traffic to a specific worker group:
|
||||
|
||||
```redis
|
||||
CLIENT LIST
|
||||
# id=42 name=worker-mail-218 ...
|
||||
# id=43 name=worker-webhook-219 ...
|
||||
```
|
||||
|
||||
## Scaling cookbook
|
||||
|
||||
| Symptom | Action |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
||||
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
||||
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
||||
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
||||
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
||||
|
||||
## Operator commands
|
||||
|
||||
```bash
|
||||
# Inspect all worker-group logs
|
||||
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin
|
||||
|
||||
# Restart a single group
|
||||
docker compose restart worker-import
|
||||
|
||||
# Smoke-test all five readiness endpoints
|
||||
for port in 9101 9102 9103 9104 9105; do
|
||||
curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
|
||||
done
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Wrong or missing worker group.** The worker Docker image defaults
|
||||
`WORKER_QUEUE_GROUP` to `general` for raw image users. Coolify group apps
|
||||
must still set the explicit group marker (`general`, `mail`, `import`,
|
||||
`webhook`, or `plugin`) so each app drains the intended queues. A missing
|
||||
Coolify override can therefore start as another `general` worker instead of a
|
||||
mail/import/webhook/plugin worker; `scripts/coolify-ensure-worker-groups.mjs`
|
||||
treats that as env drift. An unknown group still fails worker startup when the
|
||||
worker is enabled. Disabled workers (`WORKER_ENABLED=false`) skip this check so
|
||||
dev environments without Redis still boot.
|
||||
|
||||
**`/readyz` returns 503 with `checks.redis="error"`** — BullMQ couldn't
|
||||
reach Redis. Confirm `BULLMQ_REDIS_URL` and that the redis container is
|
||||
healthy. The error is intentionally short (no host names) so the probe
|
||||
response never leaks credentials.
|
||||
|
||||
**`/readyz` returns 503 with `checks.database="error"`** — Prisma
|
||||
couldn't run `SELECT 1`. Check `DATABASE_URL`, PgBouncer health, and
|
||||
whether the pool is saturated.
|
||||
|
||||
**A queue is silent on every worker** — verify the queue is mapped to a
|
||||
group in `packages/queue/src/groups.ts`. If you just added a new queue,
|
||||
the `groups.test.ts` exhaustiveness check should have failed CI; if it
|
||||
didn't, the queue is missing from `names.ts`.
|
||||
|
||||
**Outbox dispatcher missing.** The outbox dispatcher runs on the
|
||||
`general` group only (see `apps/worker/src/index.ts`). If your stack
|
||||
runs no `general` replica, outbox events will not dispatch — keep at
|
||||
least one `general` replica.
|
||||
|
||||
## Rollback
|
||||
|
||||
Current production worker topology requires an explicit `WORKER_QUEUE_GROUP`
|
||||
on each Coolify group app whenever `WORKER_ENABLED=true`; there is no
|
||||
supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
|
||||
standalone default is `general`, so removing the marker during rollback can
|
||||
silently turn a non-general app into another general worker, while an unknown
|
||||
group fails startup by design. For an incident rollback, redeploy the previous
|
||||
known-good image/tag or compose/deploy definition, or keep the five group apps
|
||||
and roll the affected group back independently. In-flight BullMQ job state
|
||||
lives in Redis, so jobs are not lost while workers are rolled.
|
||||
Reference in new issue
Block a user