HiveGrid-Wireframes/hivegrid-cloud/docs/architecture/outbox-pattern.md

Transactional Outbox Pattern

The outbox pattern keeps side-effects (mail sends, webhooks, audit logs, workflow triggers) transactionally consistent with the database writes that triggered them. Without it, a crash between DB COMMIT and "call mailer" leaves the DB updated but the side-effect lost.

This document describes how to publish and consume outbox events in this codebase. See docs/improvement-plan/03-scaling-infrastructure/outbox-pattern.md for the broader design rationale.

When to publish

Publish inside a Prisma $transaction block whenever a successful DB write should fan out an asynchronous side-effect.

import { publishOutboxEvent } from "@toolctrl/db/outbox";

await prisma.$transaction(async (tx) => {
  const subscription = await tx.subscription.update({
    where: { id },
    data: { status: "active" },
  });

  await publishOutboxEvent(tx, {
    tenantId: subscription.tenantId,
    type: "billing.subscription.updated",
    payload: { subscriptionId: subscription.id, status: "active" },
    dispatchKey: `stripe:${stripeEvent.id}`, // optional idempotency key
  });
});

publishOutboxEvent is type-constrained to require a Prisma.TransactionClient — calling it with a top-level PrismaClient fails to type-check.

Event-type naming convention

Events follow <domain>.<entity>.<verb>, lowercase, dot-separated. The dispatcher routes by the first segment by default:

Prefix	Target BullMQ queue
tenant.subscription.	outbox.notification
mail.	outbox.mail
webhook.	outbox.webhook
notification.	outbox.notification
audit.	outbox.audit
workflow.	outbox.workflow
search.index.	outbox.search-index
rag.index.	outbox.rag-index
ai.cost.	outbox.ai-cost
marketing.lead.	outbox.marketing
form.submission.	outbox.form-submission-route
billing.	outbox.notification
plugin.hook.	outbox.plugin-hook
plugin.capability-use.	outbox.plugin-capability-use

Add a routing rule before introducing a new prefix — events with no matching rule go straight to FAILED with a Sentry alert. See apps/worker/src/outbox/event-router.ts.

Workflow routing has an active receiver/subscriber path when the worker is configured for the general queue group with the workflow execution settings. apps/worker/src/jobs/workflow-trigger.ts parses workflow.* outbox jobs, applies receiver-side idempotency, fans out to matching live event-trigger workflows, starts queued manual workflow.execution.requested executions, and resumes queued workflow.execution.continue.requested continuations through the selected executor. HTTP mode uses the signed internal bridge; hybrid/native mode can resume worker-native safe-core continuations when the only native preflight gap is the current waitForInput boundary. Candidates that fail preflight are skipped or recorded as rejected. apps/worker/src/jobs/workflow-trigger-store.ts provides the worker-native Prisma lookup/preflight port. Application events publish workflow.trigger.event through EventsService. The route-backed row mutation producers for row.created, row.updated, and row.deleted record the row write, application event, and workflow outbox event in the same transaction before post-commit notifications/webhook callouts run. Internal mutations without a request context remain intentionally eventless unless the caller opts into an event context. Workflow row blocks pass a workflow event context so requestless rowCreate, rowUpdate, and rowDelete blocks still publish row events without enabling external webhook callouts. A source-inventory test guards the current execution boundary: manual and tenant API runs must publish workflow.execution.requested instead of executing inline, non-stream continuations must publish workflow.execution.continue.requested, only the internal workflow bridge may execute type: "trigger" directly, and the legacy WorkflowEngine.trigger(...) facade must stay absent.

The worker-native block contract is intentionally explicit. email is the first-class mail block, httpRequest is the outbound/webhook-style block, and conditions are represented by if plus switch. There is no separate mail, webhook, notification, audit, delay, or schedule block id in the registry today; adding any of those requires updating packages/foundation/src/workflows.ts, its capability matrix tests, workflow UI i18n, and worker-native execution/preflight behavior together.

Operationally, the outbox dispatcher and workflow subscriber are separate from mail delivery workers. Production-like Coolify stacks need a persistent worker-general (WORKER_QUEUE_GROUP=general) for the dispatcher and outbox.workflow, plus a persistent worker-mail (WORKER_QUEUE_GROUP=mail) for outbox.mail, outbox.webhook, outbox.marketing, and toolctrl.email work. A green outbox row does not prove that both services exist or that the downstream BullMQ job finished.

Non-durable exception: stream runs

Workflow stream starts remain an explicit exception to the worker-durable execution boundary. The stream route starts through WorkflowsExecutionsService.stream(...), creates a running execution row, and then calls execute(...) in the web process so the existing SSE/emitter progress contract can deliver local run updates to the browser. Web restarts can interrupt an in-flight stream run before the worker can resume it. Stream continuations are already queued through workflow.execution.continue.requested; moving stream starts to the worker requires a distributed progress channel or SSE broker rather than a partial outbox migration.

The worker bridge has an opt-in real-Postgres regression test at apps/worker/src/jobs/workflow-trigger-e2e.test.ts. It skips without a real database URL and runs with WORKFLOW_TRIGGER_E2E_DATABASE_URL, falling back to OUTBOX_E2E_DATABASE_URL or any non-stub DATABASE_URL. The test publishes a workflow trigger outbox event, claims and routes it through the production dispatcher seam, then executes a live safe-core workflow through the native worker runner and verifies idempotent re-delivery. It also verifies the workflow.executed plugin hook path by routing plugin.hook.workflow.executed through outbox.plugin-hook fanout into toolctrl.plugin-invoke. A paired failure-path fixture forces an unsupported native block, verifies terminal status: "error" persistence, then routes plugin.hook.workflow.failed through the same fanout boundary into toolctrl.plugin-invoke.

The canonical non-browser readiness gate for this path is pnpm run verify:workflow-outbox-native. It composes the existing focused worker, foundation, and source-inventory tests for outbox.workflow routing, workflow-trigger subscriber bootstrap, native/hybrid executor selection, credential-free native execution boundaries, worker health fail-loud settings, and queued manual/API/continuation producers. It is intentionally not a live drain probe; use the opt-in real-Postgres regression or the operator-gated E2E drain readiness check when an environment-level worker/Redis/Postgres proof is needed.

Dispatcher

apps/worker/src/outbox/dispatcher.ts polls every 1 second (configurable via OUTBOX_DISPATCH_INTERVAL_MS). Each tick:

1. Atomic claim: UPDATE … SELECT … FOR UPDATE SKIP LOCKED RETURNING * moves up to OUTBOX_DISPATCH_BATCH_SIZE (default 100) PENDING rows to PROCESSING with lockedUntil = now() + 5 min. Two dispatchers can run concurrently without ever claiming the same event.
2. Crash recovery: rows stuck in PROCESSING past lockedUntil are re-claimed automatically.
3. Routing: each event's eventType is mapped to a BullMQ queue via the event router.
4. Enqueue + status update: claimed rows are processed with bounded concurrency (OUTBOX_DISPATCH_CONCURRENCY, default 10) so a large batch does not serialize every Redis enqueue and status update. The default BullMQ enqueue path adds receiver-level retries with exponential backoff before marking the outbox row DONE; transient subscriber failures are retried by the target queue instead of being lost after dispatch. It also uses a deterministic BullMQ jobId of outbox:<eventId> so a dispatcher crash after enqueue but before the Postgres DONE update does not create a second receiver job when the same outbox row is re-claimed.
   • success → status = DONE, processedAt = now()
   • failure → increment retries, scheduledAt = now() + backoff(retries) with backoff [1, 2, 4, 8, 16, 32] minutes.
   • retries > OUTBOX_MAX_RETRIES (default 6) → status = FAILED (dead- letter).

DONE means the dispatcher successfully handed the event to the target BullMQ queue and recorded that handoff in Postgres. It does not mean the subscriber processed the job, the provider call succeeded, or a domain state machine reached a terminal success state. For workflow events, operators must also watch the outbox.workflow BullMQ job state and the associated WorkflowExecution status. For mail/webhook events, also watch the worker-mail subscriber queues and the toolctrl.email job state.

The dispatcher is enabled by OUTBOX_DISPATCHER_ENABLED=true. Defaults to off in dev so a missing Redis URL does not crash the worker.

Retention

worker-general registers a daily cleanup.run maintenance job named outbox-done-retention. It deletes only rows with status = DONE and a non-null processedAt older than OUTBOX_DONE_RETENTION_HOURS (default 720 hours, 30 days). The deletion is bounded by OUTBOX_DONE_RETENTION_BATCH_SIZE (default 1000, clamped to 1..10000) and uses FOR UPDATE SKIP LOCKED, so large first runs drain gradually without blocking the dispatcher.

FAILED and CANCELLED rows are intentionally retained for audit and incident review. Deleting DONE rows releases their dispatchKey uniqueness after the retention window; producers that require longer idempotency windows must use a domain-level idempotency key as well.

Receiver-side idempotency

BullMQ delivery is at-least-once: a receiver crash between "job processed" and "ack" causes a redelivery. Wrap your handler in withOutboxIdempotency so duplicate deliveries are a no-op.

import { withOutboxIdempotency } from "@toolctrl/db/outbox-receiver";
import { createIdempotencyStore } from "@toolctrl/shared-store";

const idempotencyStore = createIdempotencyStore({
  store: sharedStore,
  prefix: "outbox",
  ttlSeconds: 60 * 60 * 24, // 24h — longer than the longest retry window
});

await withOutboxIdempotency(idempotencyStore, job.data.outboxEventId, async () => {
  await sendMail(job.data.payload);
});

claim() does an atomic SET NX EX. If the claim fails (another worker already processed the event), the handler is skipped and the cached result returned.

Operational

Pending count alert

SELECT count(*) FROM "OutboxEvent" WHERE status = 'PENDING' AND scheduledAt < now() - interval '60 seconds'

Alert when > 1000 for > 5 minutes — that means the dispatcher is behind.

Downstream queue alert

Outbox DONE counts are not sufficient for worker health. Alert separately on target BullMQ queue depth, failed jobs, and stalled jobs for outbox.workflow, outbox.mail, outbox.webhook, outbox.marketing, and toolctrl.email. Workflow incidents must include the WorkflowExecution terminal status because workflow.execution.requested can be handed to BullMQ successfully while the execution later fails, stalls, or remains queued behind a missing bridge.

Re-queueing failed events

Use toolctrl-cli outbox requeue after fixing the underlying receiver. The CLI supports single-row and filtered bulk requeues, --dry-run, JSON output, and guardrails that avoid replaying PENDING / PROCESSING rows. Prefer it over raw SQL so processedAt, lastError, retry state, and replay warnings stay consistent.

Raw SQL is an emergency-only fallback for a DBA-led incident when the CLI cannot run; record the exact statement and reason in the incident log.

Environment

Variable	Default	Purpose
OUTBOX_DISPATCHER_ENABLED	false	Master switch for the dispatcher worker
OUTBOX_DISPATCH_INTERVAL_MS	1000	Polling cadence (ms)
OUTBOX_DISPATCH_BATCH_SIZE	100	Max events claimed per tick
OUTBOX_DISPATCH_CONCURRENCY	10	Max claimed events processed in parallel per tick
OUTBOX_MAX_RETRIES	6	Retries before dead-letter
OUTBOX_DONE_RETENTION_HOURS	720	DONE-row replay/debug retention window
OUTBOX_DONE_RETENTION_BATCH_SIZE	1000	Max DONE rows purged per maintenance run

Trade-offs

• Latency: ~1s polling tick. A LISTEN/NOTIFY variant could cut that to <10ms; deferred until needed.
• At-least-once delivery: receivers MUST be idempotent (withOutboxIdempotency is the cheap path).
• Storage: each successful event leaves a DONE row until the outbox-done-retention job purges DONE rows older than OUTBOX_DONE_RETENTION_HOURS. FAILED rows are kept for operator triage and compliance review.