242 lines
23 KiB
Markdown
242 lines
23 KiB
Markdown
# Worker Operations
|
|
|
|
## Purpose
|
|
|
|
This guide describes the production worker runtime and queue operations boundary. It is intentionally redacted: use environment key names, queue names, job status, retry counters, health codes, timestamps, and dependency diagnostic codes only. Do not paste Redis URLs, database URLs, mail tokens, browser runtime credentials, job payload secrets, tenant identifiers, customer data, screenshots with secrets, or raw secret values into docs, logs, CI output, tickets, or support notes.
|
|
|
|
For the worker-backed feature contract, see `docs/agent-playbooks/worker-backed-features.md`. For runtime configuration ownership and redacted environment validation, see `docs/configuration.md`. For Docker target, healthcheck, and manual deployment inspection boundaries, see `docs/deployment.md`.
|
|
|
|
## When a worker is required
|
|
|
|
The normal web and API development loop does not require a worker. Keep the worker disabled unless the feature under test or deployment intentionally needs asynchronous, retryable, external, scheduled, browser-backed, or resource-heavy work.
|
|
|
|
A worker is required when a feature:
|
|
|
|
- may exceed normal request time;
|
|
- needs queue-backed retries or dead-letter handling;
|
|
- calls unreliable external providers such as transactional mail services;
|
|
- processes imports, exports, files, media, cleanup, usage metering, or scheduled work;
|
|
- must continue after the user navigates away;
|
|
- performs browser-backed automation through an approved browser runtime boundary.
|
|
|
|
Feature manifests for worker-backed work must name the queue and job contracts, note retry/idempotency behavior, identify the API route or UI flow that enqueues the job, and document user/admin-visible success and failure states.
|
|
|
|
## Environment keys
|
|
|
|
Worker configuration is owned by the worker runtime entrypoints. Those entrypoints read `process.env`, validate and normalize values, then pass explicit options into queue, mail, database, browser runtime, and handler code. Package internals should not discover hidden environment state.
|
|
|
|
| Key | Required when | Safe diagnostic surface |
|
|
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
|
|
| `WORKER_ENABLED` | Queue processors should start for the deployment. | Boolean enablement status and a redacted diagnostic code. |
|
|
| `BULLMQ_REDIS_URL` | `WORKER_ENABLED=true` and BullMQ queue processing is enabled. | Key name, present/missing/parse status, queue readiness code. |
|
|
| `REDIS_URL` | Fallback only when `BULLMQ_REDIS_URL` is absent and the deployment intentionally permits the generic Redis key. | Fallback key name and readiness code only. |
|
|
| `DATABASE_URL` | Jobs need database reads/writes or worker health reports DB readiness. | Key name, DB readiness status, migration/status code. |
|
|
| `WORKER_QUEUE_GROUP` | A persistent worker app should own exactly one queue group. | Group name only: `general`, `mail`, `import`, `webhook`, or `plugin`. |
|
|
| `WORKER_HEALTH_PORT` | A worker exposes `/healthz` and `/readyz`. | Port assignment only; do not expose internals publicly. |
|
|
| `WORKER_ADMIN_PORT` | Worker admin/metrics endpoints are enabled. | Port assignment only; keep separate from health probes. |
|
|
| `WORKFLOW_TRIGGER_EXECUTOR_MODE` | Selects the `outbox.workflow` executor adapter. Use `http` by default, `hybrid` for native safe-core canaries with HTTP fallback, and `native` only for safe-core worker runners that should fail loud for unsupported graph semantics. | Key name and selected mode only. |
|
|
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | `outbox.workflow` bridge starts workflow executions through the web runtime. | Key name, configured/missing status, and redacted endpoint readiness code only. |
|
|
| `API_ACCESS_TOKEN` | `outbox.workflow` bridge authenticates to the signed internal workflow execution endpoint. | Key name and configured/missing status only. |
|
|
| `OUTBOX_DISPATCHER_ENABLED` | `worker-general` should drain outbox events in staging/production. | Boolean enablement status and redacted dependency code only. |
|
|
| `OUTBOX_DISPATCHER_REQUIRED` | `worker-general` startup must fail if the outbox dispatcher cannot start after it is expected to be live. | Boolean enablement status and redacted missing-dependency code only. |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED` | `worker-general` should subscribe to `outbox.workflow` events. | Boolean enablement status and redacted dependency code only. |
|
|
| `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `worker-mail` should fail startup/readiness when the `outbox.mail` bridge cannot subscribe. | Boolean enablement status and redacted missing-dependency code only. |
|
|
| `MARKETING_SEQUENCES_ENABLED` | Email-sequence trigger/scheduler subscribers and cron producers should run. Keep unset/false until production sequence delivery is approved. | Boolean enablement status only. |
|
|
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED` | General worker startup fails if `outbox.workflow` cannot subscribe after the bridge is expected to be live. | Boolean enablement status and redacted missing-dependency code only. |
|
|
| `PLUGIN_RUNTIME_PRODUCTION_ENABLED` | Production plugin runtime should be allowed after external runtime signoff. | Boolean gate status only; no plugin secrets or binary paths. |
|
|
| `PLUGIN_TRUSTED_PUBLIC_KEYS` | Plugin production runtime is enabled and signature trust roots are required. | Key name and configured/missing status only. |
|
|
| `PLUGIN_HOOK_FANOUT_REQUIRED` | Staging/canary should fail closed if `worker-general` cannot bootstrap plugin hook fanout. | Boolean enablement status and redacted missing-dependency code only. |
|
|
| `PLUGIN_HOOK_FANOUT_ENABLED` | `worker-general` plugin hook fanout is intentionally disabled for maintenance. | Boolean enablement status only; `false` is a readiness error when production plugin runtime is enabled. |
|
|
| `PLUGIN_CAPABILITY_USE_ENABLED` | `worker-general` plugin capability-use persistence is intentionally disabled for maintenance. | Boolean enablement status only; `false` is a readiness error when production plugin runtime is enabled. |
|
|
| `PLUGIN_BINARY_BASE_PATH` | `worker-plugin` loads verified plugin binaries from disk. | Key name and configured/default status only; do not log filesystem listings or bundle contents. |
|
|
| `MAILJET_API_KEY` | Worker sends outbound Mailjet jobs. | Key name and redacted validator code only. |
|
|
| `MAILJET_FROM_EMAIL` | Worker sends outbound Mailjet jobs. | Key name and sender-domain validation status only. |
|
|
| `MAILJET_FROM_NAME` | Optional outbound sender display metadata. | Key name and configured/unconfigured status. |
|
|
| `BROWSER_RUNTIME_URL` | Browser-backed worker integration is enabled. | Key name, endpoint configured/disabled status, dependency code. |
|
|
| `BROWSER_RUNTIME_API_KEY` | Browser-backed worker integration is enabled. | Key name and redacted validator code only. |
|
|
| `BROWSER_SESSION_TIMEOUT_MS` | Browser-backed worker integration configures session timeout. | Key name and parse status only. |
|
|
|
|
Do not log or document the values behind these keys. If a value is missing or invalid, report messages such as `worker disabled by WORKER_ENABLED`, `queue config missing BULLMQ_REDIS_URL`, `redis_parse_failed`, or `browser_runtime_timeout_invalid` without interpolating rejected input.
|
|
|
|
The Coolify worker-group verifier checks `worker-general` for Redis and the
|
|
workflow trigger dependency path before accepting a rollout. In staging and
|
|
production, `worker-general` must set `OUTBOX_DISPATCHER_ENABLED=true`,
|
|
`OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true`, and
|
|
`WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` so outbox and workflow fanout fail loud
|
|
instead of silently leaving jobs undrained. HTTP and hybrid modes require a
|
|
configured bridge origin plus service token; native mode is allowed only when
|
|
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` is explicit. The worker health CLI
|
|
reports missing required-mode pairs and missing workflow bridge prerequisites as
|
|
redacted key-only diagnostics, so operators can catch a partial `worker-general`
|
|
configuration before the service is deployed.
|
|
|
|
When `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, worker readiness also checks the
|
|
plugin runtime gate on `worker-general` and `worker-plugin`. Both groups require
|
|
`PLUGIN_TRUSTED_PUBLIC_KEYS`; `worker-general` additionally fails readiness when
|
|
the outbox dispatcher is not enabled or plugin hook fanout/capability-use
|
|
subscribers are explicitly disabled. With the production gate unset or `false`,
|
|
the production `worker-plugin` service may still be deployed as an idle topology
|
|
placeholder, but it must not consume `toolctrl.plugin-invoke`.
|
|
|
|
For the first tenant-facing plugin rollout, run the explicit live gate inside
|
|
both relevant worker containers after Coolify reports them healthy:
|
|
|
|
```sh
|
|
pnpm run verify:plugin-runtime-live
|
|
```
|
|
|
|
The command must pass once in `worker-general` and once in `worker-plugin`.
|
|
It intentionally fails when only the `worker-plugin` topology exists without
|
|
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, valid `PLUGIN_TRUSTED_PUBLIC_KEYS`,
|
|
Redis, database access, and the general-worker fanout/audit subscribers.
|
|
|
|
## Redis and BullMQ configuration
|
|
|
|
`BULLMQ_REDIS_URL` is the preferred Redis connection setting for BullMQ queues. `REDIS_URL` is accepted only as an intentional fallback when `BULLMQ_REDIS_URL` is absent. Treat both keys as secret material even when a local Redis instance has no production credentials.
|
|
|
|
Operational Redis/BullMQ checks may record:
|
|
|
|
- whether the worker is enabled or disabled;
|
|
- which key name was selected, not the value;
|
|
- Redis URL parse status;
|
|
- queue readiness status;
|
|
- queue names from the queue contract;
|
|
- job counts by status, such as waiting, active, completed, failed, delayed, or paused;
|
|
- dependency diagnostic codes and timestamps.
|
|
|
|
They must not record Redis hosts, usernames, passwords, connection strings, database numbers, TLS material, queue payload secrets, tenant data, or customer data.
|
|
|
|
## Local opt-in workflow
|
|
|
|
Local web development can run with the worker disabled. Leave `WORKER_ENABLED` unset or set to a disabled value when testing normal request/response flows that do not enqueue background jobs.
|
|
|
|
For a local worker-backed feature check:
|
|
|
|
1. Confirm the feature actually requires a queue-backed worker path.
|
|
2. Provide local placeholder configuration through the local environment mechanism described in `docs/configuration.md` and `docs/env-vars.md`.
|
|
3. Set `WORKER_ENABLED=true` only for the terminal/session that should start queue processors.
|
|
4. Provide `BULLMQ_REDIS_URL` or the intentionally allowed `REDIS_URL` fallback from a local secret source, not from copied documentation.
|
|
5. Start the worker runtime separately from the web/API runtime.
|
|
6. Inspect worker health and queue status using redacted output only.
|
|
7. Disable the worker again when returning to normal web development.
|
|
|
|
This guide does not require live Redis, Docker, browser automation, Playwright, Cypress, or E2E execution. Static/package verification and redacted health output are the Auto-Mode-safe surfaces.
|
|
|
|
## Production deployment notes
|
|
|
|
The worker production container starts only the compiled worker runtime. The deployment contract in `docs/deployment.md` describes the Docker `worker` target, production `start` command, and health command. The canonical queue-group topology, queue ownership, default concurrency, ports, and scaling guidance live in [Worker queue groups](ops/worker-queues.md); the table below is only a deployment summary. Production deploys should inject environment values through the deployment secret store or CI secret store; documentation should name only the required keys.
|
|
|
|
The Dockerfile standalone worker image defaults to `WORKER_HEALTH_PORT=9100` and `WORKER_ADMIN_PORT=9101`; Coolify queue-group apps must override those defaults with the per-group ports below.
|
|
|
|
Coolify deployments must run five separate persistent worker applications from the same worker image. `worker-mail` is a permanent production-like service, not a manual fallback, because mail and mail-backed outbox/webhook work must not depend on an ad hoc worker start.
|
|
`worker-plugin` is also a permanent topology service, but production plugin
|
|
execution is not automatically enabled by deploying it. Until the external WASM
|
|
sandbox signoff is complete and `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is
|
|
set, `worker-plugin` is expected to remain idle and must not consume
|
|
`toolctrl.plugin-invoke`.
|
|
`worker-import` is provisioned the same way: keep it as a separate internal app,
|
|
but leave `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until
|
|
the import, export, and scan services are wired. While idle, it must not consume
|
|
`toolctrl.imports` or `toolctrl.exports`; once both gates are enabled, startup
|
|
fails closed unless those services are present.
|
|
|
|
| Coolify app | Required env | Public FQDN | Health port | Admin/metrics port |
|
|
| ---------------- | ---------------------------- | ----------- | ----------- | ------------------ |
|
|
| `worker-general` | `WORKER_QUEUE_GROUP=general` | none | `9101` | `9201` |
|
|
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | none | `9102` | `9202` |
|
|
| `worker-import` | `WORKER_QUEUE_GROUP=import` | none | `9103` | `9203` |
|
|
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | none | `9104` | `9204` |
|
|
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | none | `9105` | `9205` |
|
|
|
|
These worker apps are internal services, not public web resources. Do not assign public FQDNs to them. Coolify readiness should probe `/readyz` on each app's `WORKER_HEALTH_PORT`; `/healthz` remains the lightweight liveness/manual probe. Metrics/admin endpoints, when enabled, must use the separate `WORKER_ADMIN_PORT` so they do not collide with health probes.
|
|
|
|
Before enabling production worker processing:
|
|
|
|
- set `WORKER_ENABLED=true` for worker instances only;
|
|
- configure `BULLMQ_REDIS_URL` or an approved `REDIS_URL` fallback in the runtime secret store;
|
|
- provision all five persistent Coolify worker apps (`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) with distinct `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`, and `WORKER_ADMIN_PORT` values;
|
|
- ensure database, mail, and browser runtime keys are present only for integrations used by that deployment;
|
|
- keep `worker-import` idle until `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` are backed by wired import, export, and scan services;
|
|
- for production plugin runtime, set `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` only after external signoff, provide `PLUGIN_TRUSTED_PUBLIC_KEYS`, keep `PLUGIN_HOOK_FANOUT_ENABLED` and `PLUGIN_CAPABILITY_USE_ENABLED` unset or enabled on `worker-general`, and run `pnpm run verify:plugin-runtime-live` in both `worker-general` and `worker-plugin` before tenant enablement;
|
|
- deploy enough worker capacity for expected queue volume and retry behavior;
|
|
- confirm queue names and job names match the feature manifest and `docs/agent-playbooks/worker-backed-features.md`;
|
|
- keep web/API instances from accidentally starting queue processors unless explicitly intended.
|
|
|
|
Deployment logs may show package names, command names, health status, and diagnostic codes. They must not echo Redis URLs, database URLs, API keys, tokens, passwords, or secret-shaped examples.
|
|
|
|
## Health and diagnostics
|
|
|
|
Use worker health output and queue inspection as the primary safe diagnostics. Expected health summaries may include service name, revision, worker enablement, Redis parse/readiness state, database readiness state, configured queue names, dependency codes, and timestamps. The Gitea/Coolify deploy workflow proves live worker revision through Coolify execute-based health JSON and fails closed when execute is unavailable, empty, or non-JSON. Manual queue inspection can help diagnose the failure, but it must not replace the automated worker execute-health proof for accepting a staged or promoted rollout.
|
|
|
|
Useful redacted troubleshooting facts include:
|
|
|
|
- `WORKER_ENABLED` enabled/disabled status;
|
|
- selected Redis key name, such as `BULLMQ_REDIS_URL` or fallback `REDIS_URL`;
|
|
- queue health state and queue name;
|
|
- job status counts by queue;
|
|
- retry count and maximum attempts;
|
|
- last error class or redacted diagnostic code;
|
|
- last attempted, completed, failed, delayed, or stalled timestamp;
|
|
- correlation id or request id when already safe and non-secret.
|
|
|
|
Unsafe troubleshooting facts include raw job payloads, message bodies, uploaded file contents, tenant ids, customer identifiers, Redis/database/mail/browser secret values, stack traces with connection strings, and copied provider responses containing secrets.
|
|
|
|
## Retry, failure, and DLQ handling
|
|
|
|
Worker jobs should be retryable, idempotent, or protected against duplicate side effects. When a job fails, the user/admin-facing surface should show a clear redacted status rather than raw exception details.
|
|
|
|
Record and inspect:
|
|
|
|
- queue name and job type;
|
|
- status such as waiting, active, delayed, retrying, failed, completed, or dead-lettered;
|
|
- attempt number and configured retry limit;
|
|
- last error class or redacted code;
|
|
- next retry timestamp or final failure timestamp;
|
|
- cleanup status for temporary artifacts;
|
|
- remediation owner, such as configuration, provider, payload validation, or dependency availability.
|
|
|
|
A dead-letter queue or equivalent failed-job view should be treated as sensitive operational data. It may be used to identify failed job classes and retry decisions, but it must not become a shared dump of payloads, secrets, customer data, or browser artifacts.
|
|
|
|
## Queue-backed feature notes
|
|
|
|
### Mail
|
|
|
|
Queue-backed transactional mail should enqueue normalized metadata, then let the worker call Mailjet with `MAILJET_API_KEY`, `MAILJET_FROM_EMAIL`, and optional `MAILJET_FROM_NAME`. Safe diagnostics are queue status, retry count, provider status class, and correlation id. Do not log message contents, Mailjet tokens, webhook secrets, tenant ids, or real attempt ids.
|
|
|
|
### Imports, exports, and cleanup
|
|
|
|
Imports and exports belong on workers when they may exceed request time, need retries, or process files. Persist enough status for the initiating user or admin to see progress, validation failure, retry, completion, or cleanup state. Do not expose file contents, customer data, or raw database records in queue diagnostics.
|
|
|
|
### Browser-backed features
|
|
|
|
Browser-backed work must use the approved browser runtime boundary and feature manifest constraints. Required notes include domain allowlist/denylist, timeout policy, screenshot/artifact storage policy, credential handling rules, cleanup jobs, and audit events for start, completion, and failure. Browser runtime diagnostics may include key names, timeout parse status, queue state, and redacted dependency codes only.
|
|
|
|
## Manual E2E boundary
|
|
|
|
Outside the configured Gitea CI/deploy workflows, live Docker builds, live Redis checks, browser flows, Playwright runs, Cypress runs, and E2E checks remain manual runtime verification unless a human explicitly approves the exact focused run in the current session. Auto-Mode documentation and CI-safe verification must not recommend automatic browser/E2E commands.
|
|
|
|
If a human performs an approved runtime smoke, record only non-secret facts: command purpose, queue name, health status, dependency diagnostic code, timestamp, and pass/fail result. Do not paste command output containing secrets, connection strings, screenshots with credentials, job payloads, or customer data.
|
|
|
|
## Redaction constraints
|
|
|
|
Allowed public surfaces:
|
|
|
|
- environment key names;
|
|
- queue names and job type names;
|
|
- enabled/disabled, present/missing, parsed/invalid, ready/unready, and status values;
|
|
- retry counts and timestamps;
|
|
- redacted diagnostic codes;
|
|
- CI job names and non-secret verifier command names;
|
|
- high-level remediation steps.
|
|
|
|
Forbidden public surfaces:
|
|
|
|
- Redis URLs or connection strings;
|
|
- database URLs, query output with sensitive records, usernames, passwords, or hosts;
|
|
- Mailjet API keys, webhook signing secrets, provider payloads, or message bodies;
|
|
- browser runtime API keys, session credentials, screenshots with secrets, or artifact contents;
|
|
- tenant ids, customer identifiers, file contents, and raw job payloads;
|
|
- unfinished placeholder markers or secret-shaped examples.
|
|
|
|
If an unsafe value appears in a shared artifact, remove the artifact from the public surface, rotate the affected secret when applicable, and replace the artifact with a redacted summary that names only keys, codes, and statuses.
|