HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/worker.md

Worker Operations

Purpose

This guide describes the production worker runtime and queue operations boundary. It is intentionally redacted: use environment key names, queue names, job status, retry counters, health codes, timestamps, and dependency diagnostic codes only. Do not paste Redis URLs, database URLs, mail tokens, browser runtime credentials, job payload secrets, tenant identifiers, customer data, screenshots with secrets, or raw secret values into docs, logs, CI output, tickets, or support notes.

For the worker-backed feature contract, see docs/agent-playbooks/worker-backed-features.md. For runtime configuration ownership and redacted environment validation, see docs/configuration.md. For Docker target, healthcheck, and manual deployment inspection boundaries, see docs/deployment.md.

When a worker is required

The normal web and API development loop does not require a worker. Keep the worker disabled unless the feature under test or deployment intentionally needs asynchronous, retryable, external, scheduled, browser-backed, or resource-heavy work.

A worker is required when a feature:

• may exceed normal request time;
• needs queue-backed retries or dead-letter handling;
• calls unreliable external providers such as transactional mail services;
• processes imports, exports, files, media, cleanup, usage metering, or scheduled work;
• must continue after the user navigates away;
• performs browser-backed automation through an approved browser runtime boundary.

Feature manifests for worker-backed work must name the queue and job contracts, note retry/idempotency behavior, identify the API route or UI flow that enqueues the job, and document user/admin-visible success and failure states.

Environment keys

Worker configuration is owned by the worker runtime entrypoints. Those entrypoints read process.env, validate and normalize values, then pass explicit options into queue, mail, database, browser runtime, and handler code. Package internals should not discover hidden environment state.

Key	Required when	Safe diagnostic surface
WORKER_ENABLED	Queue processors should start for the deployment.	Boolean enablement status and a redacted diagnostic code.
BULLMQ_REDIS_URL	WORKER_ENABLED=true and BullMQ queue processing is enabled.	Key name, present/missing/parse status, queue readiness code.
REDIS_URL	Fallback only when BULLMQ_REDIS_URL is absent and the deployment intentionally permits the generic Redis key.	Fallback key name and readiness code only.
DATABASE_URL	Jobs need database reads/writes or worker health reports DB readiness.	Key name, DB readiness status, migration/status code.
WORKER_QUEUE_GROUP	A persistent worker app should own exactly one queue group.	Group name only: general, mail, import, webhook, or plugin.
WORKER_HEALTH_PORT	A worker exposes /healthz and /readyz.	Port assignment only; do not expose internals publicly.
WORKER_ADMIN_PORT	Worker admin/metrics endpoints are enabled.	Port assignment only; keep separate from health probes.
WORKFLOW_TRIGGER_EXECUTOR_MODE	Selects the outbox.workflow executor adapter. Use http by default, hybrid for native safe-core canaries with HTTP fallback, and native only for safe-core worker runners that should fail loud for unsupported graph semantics.	Key name and selected mode only.
APP_URL, WEB_ORIGIN, or WORKFLOW_TRIGGER_EXECUTOR_URL	outbox.workflow bridge starts workflow executions through the web runtime.	Key name, configured/missing status, and redacted endpoint readiness code only.
API_ACCESS_TOKEN	outbox.workflow bridge authenticates to the signed internal workflow execution endpoint.	Key name and configured/missing status only.
OUTBOX_DISPATCHER_ENABLED	worker-general should drain outbox events in staging/production.	Boolean enablement status and redacted dependency code only.
OUTBOX_DISPATCHER_REQUIRED	worker-general startup must fail if the outbox dispatcher cannot start after it is expected to be live.	Boolean enablement status and redacted missing-dependency code only.
WORKFLOW_TRIGGER_OUTBOX_ENABLED	worker-general should subscribe to outbox.workflow events.	Boolean enablement status and redacted dependency code only.
TRANSACTIONAL_MAIL_OUTBOX_REQUIRED	worker-mail should fail startup/readiness when the outbox.mail bridge cannot subscribe.	Boolean enablement status and redacted missing-dependency code only.
MARKETING_SEQUENCES_ENABLED	Email-sequence trigger/scheduler subscribers and cron producers should run. Keep unset/false until production sequence delivery is approved.	Boolean enablement status only.
WORKFLOW_TRIGGER_OUTBOX_REQUIRED	General worker startup fails if outbox.workflow cannot subscribe after the bridge is expected to be live.	Boolean enablement status and redacted missing-dependency code only.
PLUGIN_RUNTIME_PRODUCTION_ENABLED	Production plugin runtime should be allowed after external runtime signoff.	Boolean gate status only; no plugin secrets or binary paths.
PLUGIN_TRUSTED_PUBLIC_KEYS	Plugin production runtime is enabled and signature trust roots are required.	Key name and configured/missing status only.
PLUGIN_HOOK_FANOUT_REQUIRED	Staging/canary should fail closed if worker-general cannot bootstrap plugin hook fanout.	Boolean enablement status and redacted missing-dependency code only.
PLUGIN_HOOK_FANOUT_ENABLED	worker-general plugin hook fanout is intentionally disabled for maintenance.	Boolean enablement status only; false is a readiness error when production plugin runtime is enabled.
PLUGIN_CAPABILITY_USE_ENABLED	worker-general plugin capability-use persistence is intentionally disabled for maintenance.	Boolean enablement status only; false is a readiness error when production plugin runtime is enabled.
PLUGIN_BINARY_BASE_PATH	worker-plugin loads verified plugin binaries from disk.	Key name and configured/default status only; do not log filesystem listings or bundle contents.
MAILJET_API_KEY	Worker sends outbound Mailjet jobs.	Key name and redacted validator code only.
MAILJET_FROM_EMAIL	Worker sends outbound Mailjet jobs.	Key name and sender-domain validation status only.
MAILJET_FROM_NAME	Optional outbound sender display metadata.	Key name and configured/unconfigured status.
BROWSER_RUNTIME_URL	Browser-backed worker integration is enabled.	Key name, endpoint configured/disabled status, dependency code.
BROWSER_RUNTIME_API_KEY	Browser-backed worker integration is enabled.	Key name and redacted validator code only.
BROWSER_SESSION_TIMEOUT_MS	Browser-backed worker integration configures session timeout.	Key name and parse status only.

Do not log or document the values behind these keys. If a value is missing or invalid, report messages such as worker disabled by WORKER_ENABLED, queue config missing BULLMQ_REDIS_URL, redis_parse_failed, or browser_runtime_timeout_invalid without interpolating rejected input.

The Coolify worker-group verifier checks worker-general for Redis and the workflow trigger dependency path before accepting a rollout. In staging and production, worker-general must set OUTBOX_DISPATCHER_ENABLED=true, OUTBOX_DISPATCHER_REQUIRED=true, WORKFLOW_TRIGGER_OUTBOX_ENABLED=true, and WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true so outbox and workflow fanout fail loud instead of silently leaving jobs undrained. HTTP and hybrid modes require a configured bridge origin plus service token; native mode is allowed only when WORKFLOW_TRIGGER_EXECUTOR_MODE=native is explicit. The worker health CLI reports missing required-mode pairs and missing workflow bridge prerequisites as redacted key-only diagnostics, so operators can catch a partial worker-general configuration before the service is deployed.

When PLUGIN_RUNTIME_PRODUCTION_ENABLED=true, worker readiness also checks the plugin runtime gate on worker-general and worker-plugin. Both groups require PLUGIN_TRUSTED_PUBLIC_KEYS; worker-general additionally fails readiness when the outbox dispatcher is not enabled or plugin hook fanout/capability-use subscribers are explicitly disabled. With the production gate unset or false, the production worker-plugin service may still be deployed as an idle topology placeholder, but it must not consume hivegrid.plugin-invoke.

For the first tenant-facing plugin rollout, run the explicit live gate inside both relevant worker containers after Coolify reports them healthy:

pnpm run verify:plugin-runtime-live

The command must pass once in worker-general and once in worker-plugin. It intentionally fails when only the worker-plugin topology exists without PLUGIN_RUNTIME_PRODUCTION_ENABLED=true, valid PLUGIN_TRUSTED_PUBLIC_KEYS, Redis, database access, and the general-worker fanout/audit subscribers.

Redis and BullMQ configuration

BULLMQ_REDIS_URL is the preferred Redis connection setting for BullMQ queues. REDIS_URL is accepted only as an intentional fallback when BULLMQ_REDIS_URL is absent. Treat both keys as secret material even when a local Redis instance has no production credentials.

Operational Redis/BullMQ checks may record:

• whether the worker is enabled or disabled;
• which key name was selected, not the value;
• Redis URL parse status;
• queue readiness status;
• queue names from the queue contract;
• job counts by status, such as waiting, active, completed, failed, delayed, or paused;
• dependency diagnostic codes and timestamps.

They must not record Redis hosts, usernames, passwords, connection strings, database numbers, TLS material, queue payload secrets, tenant data, or customer data.

Local opt-in workflow

Local web development can run with the worker disabled. Leave WORKER_ENABLED unset or set to a disabled value when testing normal request/response flows that do not enqueue background jobs.

For a local worker-backed feature check:

1. Confirm the feature actually requires a queue-backed worker path.
2. Provide local placeholder configuration through the local environment mechanism described in docs/configuration.md and docs/env-vars.md.
3. Set WORKER_ENABLED=true only for the terminal/session that should start queue processors.
4. Provide BULLMQ_REDIS_URL or the intentionally allowed REDIS_URL fallback from a local secret source, not from copied documentation.
5. Start the worker runtime separately from the web/API runtime.
6. Inspect worker health and queue status using redacted output only.
7. Disable the worker again when returning to normal web development.

This guide does not require live Redis, Docker, browser automation, Playwright, Cypress, or E2E execution. Static/package verification and redacted health output are the Auto-Mode-safe surfaces.

Production deployment notes

The worker production container starts only the compiled worker runtime. The deployment contract in docs/deployment.md describes the Docker worker target, production start command, and health command. The canonical queue-group topology, queue ownership, default concurrency, ports, and scaling guidance live in Worker queue groups; the table below is only a deployment summary. Production deploys should inject environment values through the deployment secret store or CI secret store; documentation should name only the required keys.

The Dockerfile standalone worker image defaults to WORKER_HEALTH_PORT=9100 and WORKER_ADMIN_PORT=9101; Coolify queue-group apps must override those defaults with the per-group ports below.

Coolify deployments must run five separate persistent worker applications from the same worker image. worker-mail is a permanent production-like service, not a manual fallback, because mail and mail-backed outbox/webhook work must not depend on an ad hoc worker start. worker-plugin is also a permanent topology service, but production plugin execution is not automatically enabled by deploying it. Until the external WASM sandbox signoff is complete and PLUGIN_RUNTIME_PRODUCTION_ENABLED=true is set, worker-plugin is expected to remain idle and must not consume hivegrid.plugin-invoke. worker-import is provisioned the same way: keep it as a separate internal app, but leave IMPORT_QUEUE_ENABLED and IMPORT_QUEUE_RUNTIME_READY unset until the generic import, export, and scan services are wired. It always consumes the isolated hivegrid.kwp-imports provider queue; while the generic gates are off, it must not consume hivegrid.imports or hivegrid.exports. Once both gates are enabled, startup fails closed unless those generic services are present.

For the internal customer-portal closeout, worker-import also supports a strictly narrower, mutually exclusive mode: set PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true while both IMPORT_QUEUE_* gates remain unset or false. That mode adds hivegrid.exports beside the always-active KWP queue, bootstraps only the portal and privacy GDPR export services, and requires the complete STORAGE_* S3 contract. A wrong worker group, mixed gate state, missing storage, or either missing GDPR service keeps readiness/startup fail-closed. It does not enable tenant import submission or the generic import/scan/export runtime.

Coolify app	Required env	Public FQDN	Health port	Admin/metrics port
worker-general	WORKER_QUEUE_GROUP=general	none	9101	9201
worker-mail	WORKER_QUEUE_GROUP=mail	none	9102	9202
worker-import	WORKER_QUEUE_GROUP=import	none	9103	9203
worker-webhook	WORKER_QUEUE_GROUP=webhook	none	9104	9204
worker-plugin	WORKER_QUEUE_GROUP=plugin	none	9105	9205

These worker apps are internal services, not public web resources. Do not assign public FQDNs to them. Managed Coolify apps must use an enabled cmd healthcheck with node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>, where both arguments identify the immutable image and the exact general, mail, import, webhook, or plugin resource expected for that rollout. The command compares the requested revision with the root-owned image-baked revision and APP_REVISION, requires WORKER_ENABLED=true plus the expected WORKER_QUEUE_GROUP, and then probes the running worker process over loopback /readyz. /healthz remains the lightweight liveness/manual probe. Metrics/admin endpoints, when enabled, must use the separate WORKER_ADMIN_PORT so they do not collide with health probes.

Before enabling production worker processing:

• set WORKER_ENABLED=true for worker instances only;
• configure BULLMQ_REDIS_URL or an approved REDIS_URL fallback in the runtime secret store;
• provision all five persistent Coolify worker apps (worker-general, worker-mail, worker-import, worker-webhook, worker-plugin) with distinct WORKER_QUEUE_GROUP, WORKER_HEALTH_PORT, and WORKER_ADMIN_PORT values;
• ensure database, mail, and browser runtime keys are present only for integrations used by that deployment;
• keep worker-import idle until either the full import gates are backed by wired import/export/scan services or the mutually exclusive PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true mode is backed by both GDPR export services and complete STORAGE_* configuration;
• for production plugin runtime, set PLUGIN_RUNTIME_PRODUCTION_ENABLED=true only after external signoff, provide PLUGIN_TRUSTED_PUBLIC_KEYS, keep PLUGIN_HOOK_FANOUT_ENABLED and PLUGIN_CAPABILITY_USE_ENABLED unset or enabled on worker-general, and run pnpm run verify:plugin-runtime-live in both worker-general and worker-plugin before tenant enablement;
• deploy enough worker capacity for expected queue volume and retry behavior;
• confirm queue names and job names match the owning ticket doc and docs/agent-playbooks/worker-backed-features.md;
• keep web/API instances from accidentally starting queue processors unless explicitly intended.

Deployment logs may show package names, command names, health status, and diagnostic codes. They must not echo Redis URLs, database URLs, API keys, tokens, passwords, or secret-shaped examples.

Health and diagnostics

Use worker health output and queue inspection as the primary safe diagnostics. Expected health summaries may include service name, revision, worker enablement, Redis parse/readiness state, database readiness state, configured queue names, dependency codes, and timestamps. The Gitea/Coolify deploy workflow proves the live worker revision through Coolify's enabled, revision-and-group-bound cmd healthcheck. It accepts a rollout only when Coolify reports the deployment healthy and the configured command exactly matches node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group> for the deployed immutable revision and resource. The command fails before its network probe when the root-owned image revision, APP_REVISION, enablement, or queue group differs. It then calls the running worker's loopback /readyz, which proves the PID1 event loop, Redis, database, and required runtime subscribers instead of starting a second Prisma/BullMQ health process. An image alias or Coolify application status without this exact command is not live-revision proof. Manual queue inspection can help diagnose a failure, but it must not replace the automated command-health proof for accepting a staged or promoted rollout.

Useful redacted troubleshooting facts include:

• WORKER_ENABLED enabled/disabled status;
• selected Redis key name, such as BULLMQ_REDIS_URL or fallback REDIS_URL;
• queue health state and queue name;
• job status counts by queue;
• retry count and maximum attempts;
• last error class or redacted diagnostic code;
• last attempted, completed, failed, delayed, or stalled timestamp;
• correlation id or request id when already safe and non-secret.

Unsafe troubleshooting facts include raw job payloads, message bodies, uploaded file contents, tenant ids, customer identifiers, Redis/database/mail/browser secret values, stack traces with connection strings, and copied provider responses containing secrets.

Retry, failure, and DLQ handling

Worker jobs should be retryable, idempotent, or protected against duplicate side effects. When a job fails, the user/admin-facing surface should show a clear redacted status rather than raw exception details.

Record and inspect:

• queue name and job type;
• status such as waiting, active, delayed, retrying, failed, completed, or dead-lettered;
• attempt number and configured retry limit;
• last error class or redacted code;
• next retry timestamp or final failure timestamp;
• cleanup status for temporary artifacts;
• remediation owner, such as configuration, provider, payload validation, or dependency availability.

A dead-letter queue or equivalent failed-job view should be treated as sensitive operational data. It may be used to identify failed job classes and retry decisions, but it must not become a shared dump of payloads, secrets, customer data, or browser artifacts.

Queue-backed feature notes

Mail

Queue-backed transactional mail should enqueue normalized metadata, then let the worker call Mailjet with MAILJET_API_KEY, MAILJET_FROM_EMAIL, and optional MAILJET_FROM_NAME. Safe diagnostics are queue status, retry count, provider status class, and correlation id. Do not log message contents, Mailjet tokens, webhook secrets, tenant ids, or real attempt ids.

Imports, exports, and cleanup

Imports and exports belong on workers when they may exceed request time, need retries, or process files. Persist enough status for the initiating user or admin to see progress, validation failure, retry, completion, or cleanup state. Do not expose file contents, customer data, or raw database records in queue diagnostics.

Browser-backed features

Browser-backed work must use the approved browser runtime and ticket constraints. Required notes include domain allowlist/denylist, timeout policy, screenshot/artifact storage policy, credential handling, cleanup jobs and audit events. Runtime diagnostics may include only key names, timeout parse status, queue state and redacted dependency codes.

Manual E2E boundary

Outside the configured Gitea CI/deploy workflows, live Docker builds, live Redis checks, browser flows, Playwright runs, Cypress runs, and E2E checks remain manual runtime verification unless a human explicitly approves the exact focused run in the current session. Auto-Mode documentation and CI-safe verification must not recommend automatic browser/E2E commands.

If a human performs an approved runtime smoke, record only non-secret facts: command purpose, queue name, health status, dependency diagnostic code, timestamp, and pass/fail result. Do not paste command output containing secrets, connection strings, screenshots with credentials, job payloads, or customer data.

Redaction constraints

Allowed public surfaces:

• environment key names;
• queue names and job type names;
• enabled/disabled, present/missing, parsed/invalid, ready/unready, and status values;
• retry counts and timestamps;
• redacted diagnostic codes;
• CI job names and non-secret verifier command names;
• high-level remediation steps.

Forbidden public surfaces:

• Redis URLs or connection strings;
• database URLs, query output with sensitive records, usernames, passwords, or hosts;
• Mailjet API keys, webhook signing secrets, provider payloads, or message bodies;
• browser runtime API keys, session credentials, screenshots with secrets, or artifact contents;
• tenant ids, customer identifiers, file contents, and raw job payloads;
• unfinished placeholder markers or secret-shaped examples.

If an unsafe value appears in a shared artifact, remove the artifact from the public surface, rotate the affected secret when applicable, and replace the artifact with a redacted summary that names only keys, codes, and statuses.