271 lines
14 KiB
Markdown
271 lines
14 KiB
Markdown
# Transactional Outbox Pattern
|
|
|
|
The outbox pattern keeps side-effects (mail sends, webhooks, audit logs,
|
|
workflow triggers) **transactionally consistent** with the database writes
|
|
that triggered them. Without it, a crash between `DB COMMIT` and "call
|
|
mailer" leaves the DB updated but the side-effect lost.
|
|
|
|
This document describes how to publish and consume outbox events in this
|
|
codebase. See `docs/improvement-plan/03-scaling-infrastructure/outbox-pattern.md`
|
|
for the broader design rationale.
|
|
|
|
## When to publish
|
|
|
|
Publish **inside** a Prisma `$transaction` block whenever a successful DB
|
|
write should fan out an asynchronous side-effect.
|
|
|
|
```ts
|
|
import { publishOutboxEvent } from "@toolctrl/db/outbox";
|
|
|
|
await prisma.$transaction(async (tx) => {
|
|
const subscription = await tx.subscription.update({
|
|
where: { id },
|
|
data: { status: "active" },
|
|
});
|
|
|
|
await publishOutboxEvent(tx, {
|
|
tenantId: subscription.tenantId,
|
|
type: "billing.subscription.updated",
|
|
payload: { subscriptionId: subscription.id, status: "active" },
|
|
dispatchKey: `stripe:${stripeEvent.id}`, // optional idempotency key
|
|
});
|
|
});
|
|
```
|
|
|
|
`publishOutboxEvent` is type-constrained to require a
|
|
`Prisma.TransactionClient` — calling it with a top-level `PrismaClient`
|
|
fails to type-check.
|
|
|
|
## Event-type naming convention
|
|
|
|
Events follow `<domain>.<entity>.<verb>`, lowercase, dot-separated. The
|
|
dispatcher routes by the first segment by default:
|
|
|
|
| Prefix | Target BullMQ queue |
|
|
| ------------------------ | ------------------------------ |
|
|
| `tenant.subscription.` | `outbox.notification` |
|
|
| `mail.` | `outbox.mail` |
|
|
| `webhook.` | `outbox.webhook` |
|
|
| `notification.` | `outbox.notification` |
|
|
| `audit.` | `outbox.audit` |
|
|
| `workflow.` | `outbox.workflow` |
|
|
| `search.index.` | `outbox.search-index` |
|
|
| `rag.index.` | `outbox.rag-index` |
|
|
| `ai.cost.` | `outbox.ai-cost` |
|
|
| `marketing.lead.` | `outbox.marketing` |
|
|
| `form.submission.` | `outbox.form-submission-route` |
|
|
| `billing.` | `outbox.notification` |
|
|
| `plugin.hook.` | `outbox.plugin-hook` |
|
|
| `plugin.capability-use.` | `outbox.plugin-capability-use` |
|
|
|
|
Add a routing rule before introducing a new prefix — events with no
|
|
matching rule go straight to FAILED with a Sentry alert. See
|
|
`apps/worker/src/outbox/event-router.ts`.
|
|
|
|
Workflow routing has an active receiver/subscriber path when the worker is
|
|
configured for the `general` queue group with the workflow execution settings.
|
|
`apps/worker/src/jobs/workflow-trigger.ts` parses `workflow.*` outbox jobs,
|
|
applies receiver-side idempotency, fans out to matching live event-trigger
|
|
workflows, starts queued manual `workflow.execution.requested` executions, and
|
|
resumes queued `workflow.execution.continue.requested` continuations through the
|
|
selected executor. HTTP mode uses the signed internal bridge; hybrid/native mode
|
|
can resume worker-native safe-core continuations when the only native preflight
|
|
gap is the current `waitForInput` boundary. Candidates that fail preflight are
|
|
skipped or recorded as rejected.
|
|
`apps/worker/src/jobs/workflow-trigger-store.ts` provides the worker-native
|
|
Prisma lookup/preflight port. Application events publish
|
|
`workflow.trigger.event` through `EventsService`. The route-backed row mutation
|
|
producers for `row.created`, `row.updated`, and `row.deleted` record the row
|
|
write, application event, and workflow outbox event in the same transaction
|
|
before post-commit notifications/webhook callouts run. Internal mutations
|
|
without a request context remain intentionally eventless unless the caller opts
|
|
into an event context. Workflow row blocks pass a `workflow` event context so
|
|
requestless `rowCreate`, `rowUpdate`, and `rowDelete` blocks still publish row
|
|
events without enabling external webhook callouts. A source-inventory test guards the current execution
|
|
boundary: manual and tenant API runs must publish
|
|
`workflow.execution.requested` instead of executing inline, non-stream
|
|
continuations must publish `workflow.execution.continue.requested`, only the
|
|
internal workflow bridge may execute
|
|
`type: "trigger"` directly, and the legacy `WorkflowEngine.trigger(...)`
|
|
facade must stay absent.
|
|
|
|
The worker-native block contract is intentionally explicit. `email` is the
|
|
first-class mail block, `httpRequest` is the outbound/webhook-style block, and
|
|
conditions are represented by `if` plus `switch`. There is no separate
|
|
`mail`, `webhook`, `notification`, `audit`, `delay`, or `schedule` block id in
|
|
the registry today; adding any of those requires updating
|
|
`packages/foundation/src/workflows.ts`, its capability matrix tests, workflow
|
|
UI i18n, and worker-native execution/preflight behavior together.
|
|
|
|
Operationally, the outbox dispatcher and workflow subscriber are separate from
|
|
mail delivery workers. Production-like Coolify stacks need a persistent
|
|
`worker-general` (`WORKER_QUEUE_GROUP=general`) for the dispatcher and
|
|
`outbox.workflow`, plus a persistent `worker-mail`
|
|
(`WORKER_QUEUE_GROUP=mail`) for `outbox.mail`, `outbox.webhook`,
|
|
`outbox.marketing`, and `toolctrl.email` work. A green outbox row does not prove
|
|
that both services exist or that the downstream BullMQ job finished.
|
|
|
|
### Non-durable exception: stream runs
|
|
|
|
Workflow stream starts remain an explicit exception to the worker-durable
|
|
execution boundary. The stream route starts through
|
|
`WorkflowsExecutionsService.stream(...)`, creates a running execution row, and
|
|
then calls `execute(...)` in the web process so the existing SSE/emitter
|
|
progress contract can deliver local run updates to the browser. Web restarts can
|
|
interrupt an in-flight stream run before the worker can resume it. Stream
|
|
continuations are already queued through `workflow.execution.continue.requested`;
|
|
moving stream starts to the worker requires a distributed progress channel or
|
|
SSE broker rather than a partial outbox migration.
|
|
|
|
The worker bridge has an opt-in real-Postgres regression test at
|
|
`apps/worker/src/jobs/workflow-trigger-e2e.test.ts`. It skips without a real
|
|
database URL and runs with `WORKFLOW_TRIGGER_E2E_DATABASE_URL`, falling back to
|
|
`OUTBOX_E2E_DATABASE_URL` or any non-stub `DATABASE_URL`. The test publishes a
|
|
workflow trigger outbox event, claims and routes it through the production
|
|
dispatcher seam, then executes a live safe-core workflow through the native
|
|
worker runner and verifies idempotent re-delivery. It also verifies the
|
|
`workflow.executed` plugin hook path by routing `plugin.hook.workflow.executed`
|
|
through `outbox.plugin-hook` fanout into `toolctrl.plugin-invoke`. A paired
|
|
failure-path fixture forces an unsupported native block, verifies terminal
|
|
`status: "error"` persistence, then routes `plugin.hook.workflow.failed`
|
|
through the same fanout boundary into `toolctrl.plugin-invoke`.
|
|
|
|
The canonical non-browser readiness gate for this path is
|
|
`pnpm run verify:workflow-outbox-native`. It composes the existing focused
|
|
worker, foundation, and source-inventory tests for `outbox.workflow` routing,
|
|
workflow-trigger subscriber bootstrap, native/hybrid executor selection,
|
|
credential-free native execution boundaries, worker health fail-loud settings,
|
|
and queued manual/API/continuation producers. It is intentionally not a live
|
|
drain probe; use the opt-in real-Postgres regression or the operator-gated E2E
|
|
drain readiness check when an environment-level worker/Redis/Postgres proof is
|
|
needed.
|
|
|
|
## Dispatcher
|
|
|
|
`apps/worker/src/outbox/dispatcher.ts` polls every 1 second (configurable
|
|
via `OUTBOX_DISPATCH_INTERVAL_MS`). Each tick:
|
|
|
|
1. **Atomic claim**: `UPDATE … SELECT … FOR UPDATE SKIP LOCKED RETURNING *`
|
|
moves up to `OUTBOX_DISPATCH_BATCH_SIZE` (default 100) PENDING rows to
|
|
PROCESSING with `lockedUntil = now() + 5 min`. Two dispatchers can run
|
|
concurrently without ever claiming the same event.
|
|
2. **Crash recovery**: rows stuck in PROCESSING past `lockedUntil` are
|
|
re-claimed automatically.
|
|
3. **Routing**: each event's `eventType` is mapped to a BullMQ queue via
|
|
the event router.
|
|
4. **Enqueue + status update**: claimed rows are processed with bounded
|
|
concurrency (`OUTBOX_DISPATCH_CONCURRENCY`, default 10) so a large
|
|
batch does not serialize every Redis enqueue and status update.
|
|
The default BullMQ enqueue path adds receiver-level retries with exponential
|
|
backoff before marking the outbox row `DONE`; transient subscriber failures
|
|
are retried by the target queue instead of being lost after dispatch.
|
|
It also uses a deterministic BullMQ `jobId` of `outbox:<eventId>` so a
|
|
dispatcher crash after enqueue but before the Postgres `DONE` update does
|
|
not create a second receiver job when the same outbox row is re-claimed.
|
|
- success → `status = DONE`, `processedAt = now()`
|
|
- failure → increment `retries`, `scheduledAt = now() + backoff(retries)`
|
|
with backoff `[1, 2, 4, 8, 16, 32]` minutes.
|
|
- retries > `OUTBOX_MAX_RETRIES` (default 6) → `status = FAILED` (dead-
|
|
letter).
|
|
|
|
`DONE` means the dispatcher successfully handed the event to the target BullMQ
|
|
queue and recorded that handoff in Postgres. It does not mean the subscriber
|
|
processed the job, the provider call succeeded, or a domain state machine
|
|
reached a terminal success state. For workflow events, operators must also
|
|
watch the `outbox.workflow` BullMQ job state and the associated
|
|
`WorkflowExecution` status. For mail/webhook events, also watch the
|
|
`worker-mail` subscriber queues and the `toolctrl.email` job state.
|
|
|
|
The dispatcher is enabled by `OUTBOX_DISPATCHER_ENABLED=true`. Defaults to
|
|
**off** in dev so a missing Redis URL does not crash the worker.
|
|
|
|
## Retention
|
|
|
|
`worker-general` registers a daily `cleanup.run` maintenance job named
|
|
`outbox-done-retention`. It deletes only rows with `status = DONE` and a
|
|
non-null `processedAt` older than `OUTBOX_DONE_RETENTION_HOURS` (default 720
|
|
hours, 30 days). The deletion is bounded by `OUTBOX_DONE_RETENTION_BATCH_SIZE`
|
|
(default 1000, clamped to 1..10000) and uses `FOR UPDATE SKIP LOCKED`, so large
|
|
first runs drain gradually without blocking the dispatcher.
|
|
|
|
FAILED and CANCELLED rows are intentionally retained for audit and incident
|
|
review. Deleting DONE rows releases their `dispatchKey` uniqueness after the
|
|
retention window; producers that require longer idempotency windows must use a
|
|
domain-level idempotency key as well.
|
|
|
|
## Receiver-side idempotency
|
|
|
|
BullMQ delivery is at-least-once: a receiver crash between
|
|
"job processed" and "ack" causes a redelivery. Wrap your handler in
|
|
`withOutboxIdempotency` so duplicate deliveries are a no-op.
|
|
|
|
```ts
|
|
import { withOutboxIdempotency } from "@toolctrl/db/outbox-receiver";
|
|
import { createIdempotencyStore } from "@toolctrl/shared-store";
|
|
|
|
const idempotencyStore = createIdempotencyStore({
|
|
store: sharedStore,
|
|
prefix: "outbox",
|
|
ttlSeconds: 60 * 60 * 24, // 24h — longer than the longest retry window
|
|
});
|
|
|
|
await withOutboxIdempotency(idempotencyStore, job.data.outboxEventId, async () => {
|
|
await sendMail(job.data.payload);
|
|
});
|
|
```
|
|
|
|
`claim()` does an atomic `SET NX EX`. If the claim fails (another worker
|
|
already processed the event), the handler is skipped and the cached
|
|
result returned.
|
|
|
|
## Operational
|
|
|
|
### Pending count alert
|
|
|
|
`SELECT count(*) FROM "OutboxEvent" WHERE status = 'PENDING' AND scheduledAt < now() - interval '60 seconds'`
|
|
|
|
Alert when > 1000 for > 5 minutes — that means the dispatcher is behind.
|
|
|
|
### Downstream queue alert
|
|
|
|
Outbox `DONE` counts are not sufficient for worker health. Alert separately on
|
|
target BullMQ queue depth, failed jobs, and stalled jobs for `outbox.workflow`,
|
|
`outbox.mail`, `outbox.webhook`, `outbox.marketing`, and `toolctrl.email`.
|
|
Workflow incidents must include the `WorkflowExecution` terminal status because
|
|
`workflow.execution.requested` can be handed to BullMQ successfully while the
|
|
execution later fails, stalls, or remains queued behind a missing bridge.
|
|
|
|
### Re-queueing failed events
|
|
|
|
Use `toolctrl-cli outbox requeue` after fixing the underlying receiver.
|
|
The CLI supports single-row and filtered bulk requeues, `--dry-run`, JSON
|
|
output, and guardrails that avoid replaying `PENDING` / `PROCESSING`
|
|
rows. Prefer it over raw SQL so `processedAt`, `lastError`, retry state,
|
|
and replay warnings stay consistent.
|
|
|
|
Raw SQL is an emergency-only fallback for a DBA-led incident when the CLI
|
|
cannot run; record the exact statement and reason in the incident log.
|
|
|
|
### Environment
|
|
|
|
| Variable | Default | Purpose |
|
|
| ----------------------------- | ------- | ------------------------------------------------- |
|
|
| `OUTBOX_DISPATCHER_ENABLED` | `false` | Master switch for the dispatcher worker |
|
|
| `OUTBOX_DISPATCH_INTERVAL_MS` | `1000` | Polling cadence (ms) |
|
|
| `OUTBOX_DISPATCH_BATCH_SIZE` | `100` | Max events claimed per tick |
|
|
| `OUTBOX_DISPATCH_CONCURRENCY` | `10` | Max claimed events processed in parallel per tick |
|
|
| `OUTBOX_MAX_RETRIES` | `6` | Retries before dead-letter |
|
|
| `OUTBOX_DONE_RETENTION_HOURS` | `720` | DONE-row replay/debug retention window |
|
|
| `OUTBOX_DONE_RETENTION_BATCH_SIZE` | `1000` | Max DONE rows purged per maintenance run |
|
|
|
|
## Trade-offs
|
|
|
|
- **Latency**: ~1s polling tick. A LISTEN/NOTIFY variant could cut that
|
|
to <10ms; deferred until needed.
|
|
- **At-least-once delivery**: receivers MUST be idempotent
|
|
(`withOutboxIdempotency` is the cheap path).
|
|
- **Storage**: each successful event leaves a DONE row until the
|
|
`outbox-done-retention` job purges DONE rows older than
|
|
`OUTBOX_DONE_RETENTION_HOURS`. FAILED rows are kept for operator triage and
|
|
compliance review.
|