# Transactional Outbox Pattern

The outbox pattern keeps side-effects (mail sends, webhooks, audit logs,
workflow triggers) **transactionally consistent** with the database writes
that triggered them. Without it, a crash between `DB COMMIT` and "call
mailer" leaves the DB updated but the side-effect lost.

This document describes how to publish and consume outbox events in this
codebase. See `docs/improvement-plan/03-scaling-infrastructure/outbox-pattern.md`
for the broader design rationale.

## When to publish

Publish **inside** a Prisma `$transaction` block whenever a successful DB
write should fan out an asynchronous side-effect.

```ts
import { publishOutboxEvent } from "@toolctrl/db/outbox";

await prisma.$transaction(async (tx) => {
  const subscription = await tx.subscription.update({
    where: { id },
    data: { status: "active" },
  });

  await publishOutboxEvent(tx, {
    tenantId: subscription.tenantId,
    type: "billing.subscription.updated",
    payload: { subscriptionId: subscription.id, status: "active" },
    dispatchKey: `stripe:${stripeEvent.id}`, // optional idempotency key
  });
});
```

`publishOutboxEvent` is type-constrained to require a
`Prisma.TransactionClient` — calling it with a top-level `PrismaClient`
fails to type-check.

## Event-type naming convention

Events follow `<domain>.<entity>.<verb>`, lowercase, dot-separated. The
dispatcher routes by the first segment by default:

| Prefix                   | Target BullMQ queue            |
| ------------------------ | ------------------------------ |
| `tenant.subscription.`   | `outbox.notification`          |
| `mail.`                  | `outbox.mail`                  |
| `webhook.`               | `outbox.webhook`               |
| `notification.`          | `outbox.notification`          |
| `audit.`                 | `outbox.audit`                 |
| `workflow.`              | `outbox.workflow`              |
| `search.index.`          | `outbox.search-index`          |
| `rag.index.`             | `outbox.rag-index`             |
| `ai.cost.`               | `outbox.ai-cost`               |
| `marketing.lead.`        | `outbox.marketing`             |
| `form.submission.`       | `outbox.form-submission-route` |
| `billing.`               | `outbox.notification`          |
| `plugin.hook.`           | `outbox.plugin-hook`           |
| `plugin.capability-use.` | `outbox.plugin-capability-use` |

Add a routing rule before introducing a new prefix — events with no
matching rule go straight to FAILED with a Sentry alert. See
`apps/worker/src/outbox/event-router.ts`.

Workflow routing has an active receiver/subscriber path when the worker is
configured for the `general` queue group with the workflow execution settings.
`apps/worker/src/jobs/workflow-trigger.ts` parses `workflow.*` outbox jobs,
applies receiver-side idempotency, fans out to matching live event-trigger
workflows, starts queued manual `workflow.execution.requested` executions, and
resumes queued `workflow.execution.continue.requested` continuations through the
selected executor. HTTP mode uses the signed internal bridge; hybrid/native mode
can resume worker-native safe-core continuations when the only native preflight
gap is the current `waitForInput` boundary. Candidates that fail preflight are
skipped or recorded as rejected.
`apps/worker/src/jobs/workflow-trigger-store.ts` provides the worker-native
Prisma lookup/preflight port. Application events publish
`workflow.trigger.event` through `EventsService`. The route-backed row mutation
producers for `row.created`, `row.updated`, and `row.deleted` record the row
write, application event, and workflow outbox event in the same transaction
before post-commit notifications/webhook callouts run. Internal mutations
without a request context remain intentionally eventless unless the caller opts
into an event context. Workflow row blocks pass a `workflow` event context so
requestless `rowCreate`, `rowUpdate`, and `rowDelete` blocks still publish row
events without enabling external webhook callouts. A source-inventory test guards the current execution
boundary: manual and tenant API runs must publish
`workflow.execution.requested` instead of executing inline, non-stream
continuations must publish `workflow.execution.continue.requested`, only the
internal workflow bridge may execute
`type: "trigger"` directly, and the legacy `WorkflowEngine.trigger(...)`
facade must stay absent.

The worker-native block contract is intentionally explicit. `email` is the
first-class mail block, `httpRequest` is the outbound/webhook-style block, and
conditions are represented by `if` plus `switch`. There is no separate
`mail`, `webhook`, `notification`, `audit`, `delay`, or `schedule` block id in
the registry today; adding any of those requires updating
`packages/foundation/src/workflows.ts`, its capability matrix tests, workflow
UI i18n, and worker-native execution/preflight behavior together.

Operationally, the outbox dispatcher and workflow subscriber are separate from
mail delivery workers. Production-like Coolify stacks need a persistent
`worker-general` (`WORKER_QUEUE_GROUP=general`) for the dispatcher and
`outbox.workflow`, plus a persistent `worker-mail`
(`WORKER_QUEUE_GROUP=mail`) for `outbox.mail`, `outbox.webhook`,
`outbox.marketing`, and `toolctrl.email` work. A green outbox row does not prove
that both services exist or that the downstream BullMQ job finished.

### Non-durable exception: stream runs

Workflow stream starts remain an explicit exception to the worker-durable
execution boundary. The stream route starts through
`WorkflowsExecutionsService.stream(...)`, creates a running execution row, and
then calls `execute(...)` in the web process so the existing SSE/emitter
progress contract can deliver local run updates to the browser. Web restarts can
interrupt an in-flight stream run before the worker can resume it. Stream
continuations are already queued through `workflow.execution.continue.requested`;
moving stream starts to the worker requires a distributed progress channel or
SSE broker rather than a partial outbox migration.

The worker bridge has an opt-in real-Postgres regression test at
`apps/worker/src/jobs/workflow-trigger-e2e.test.ts`. It skips without a real
database URL and runs with `WORKFLOW_TRIGGER_E2E_DATABASE_URL`, falling back to
`OUTBOX_E2E_DATABASE_URL` or any non-stub `DATABASE_URL`. The test publishes a
workflow trigger outbox event, claims and routes it through the production
dispatcher seam, then executes a live safe-core workflow through the native
worker runner and verifies idempotent re-delivery. It also verifies the
`workflow.executed` plugin hook path by routing `plugin.hook.workflow.executed`
through `outbox.plugin-hook` fanout into `toolctrl.plugin-invoke`. A paired
failure-path fixture forces an unsupported native block, verifies terminal
`status: "error"` persistence, then routes `plugin.hook.workflow.failed`
through the same fanout boundary into `toolctrl.plugin-invoke`.

The canonical non-browser readiness gate for this path is
`pnpm run verify:workflow-outbox-native`. It composes the existing focused
worker, foundation, and source-inventory tests for `outbox.workflow` routing,
workflow-trigger subscriber bootstrap, native/hybrid executor selection,
credential-free native execution boundaries, worker health fail-loud settings,
and queued manual/API/continuation producers. It is intentionally not a live
drain probe; use the opt-in real-Postgres regression or the operator-gated E2E
drain readiness check when an environment-level worker/Redis/Postgres proof is
needed.

## Dispatcher

`apps/worker/src/outbox/dispatcher.ts` polls every 1 second (configurable
via `OUTBOX_DISPATCH_INTERVAL_MS`). Each tick:

1. **Atomic claim**: `UPDATE … SELECT … FOR UPDATE SKIP LOCKED RETURNING *`
   moves up to `OUTBOX_DISPATCH_BATCH_SIZE` (default 100) PENDING rows to
   PROCESSING with `lockedUntil = now() + 5 min`. Two dispatchers can run
   concurrently without ever claiming the same event.
2. **Crash recovery**: rows stuck in PROCESSING past `lockedUntil` are
   re-claimed automatically.
3. **Routing**: each event's `eventType` is mapped to a BullMQ queue via
   the event router.
4. **Enqueue + status update**: claimed rows are processed with bounded
   concurrency (`OUTBOX_DISPATCH_CONCURRENCY`, default 10) so a large
   batch does not serialize every Redis enqueue and status update.
   The default BullMQ enqueue path adds receiver-level retries with exponential
   backoff before marking the outbox row `DONE`; transient subscriber failures
   are retried by the target queue instead of being lost after dispatch.
   It also uses a deterministic BullMQ `jobId` of `outbox:<eventId>` so a
   dispatcher crash after enqueue but before the Postgres `DONE` update does
   not create a second receiver job when the same outbox row is re-claimed.
   - success → `status = DONE`, `processedAt = now()`
   - failure → increment `retries`, `scheduledAt = now() + backoff(retries)`
     with backoff `[1, 2, 4, 8, 16, 32]` minutes.
   - retries > `OUTBOX_MAX_RETRIES` (default 6) → `status = FAILED` (dead-
     letter).

`DONE` means the dispatcher successfully handed the event to the target BullMQ
queue and recorded that handoff in Postgres. It does not mean the subscriber
processed the job, the provider call succeeded, or a domain state machine
reached a terminal success state. For workflow events, operators must also
watch the `outbox.workflow` BullMQ job state and the associated
`WorkflowExecution` status. For mail/webhook events, also watch the
`worker-mail` subscriber queues and the `toolctrl.email` job state.

The dispatcher is enabled by `OUTBOX_DISPATCHER_ENABLED=true`. Defaults to
**off** in dev so a missing Redis URL does not crash the worker.

## Retention

`worker-general` registers a daily `cleanup.run` maintenance job named
`outbox-done-retention`. It deletes only rows with `status = DONE` and a
non-null `processedAt` older than `OUTBOX_DONE_RETENTION_HOURS` (default 720
hours, 30 days). The deletion is bounded by `OUTBOX_DONE_RETENTION_BATCH_SIZE`
(default 1000, clamped to 1..10000) and uses `FOR UPDATE SKIP LOCKED`, so large
first runs drain gradually without blocking the dispatcher.

FAILED and CANCELLED rows are intentionally retained for audit and incident
review. Deleting DONE rows releases their `dispatchKey` uniqueness after the
retention window; producers that require longer idempotency windows must use a
domain-level idempotency key as well.

## Receiver-side idempotency

BullMQ delivery is at-least-once: a receiver crash between
"job processed" and "ack" causes a redelivery. Wrap your handler in
`withOutboxIdempotency` so duplicate deliveries are a no-op.

```ts
import { withOutboxIdempotency } from "@toolctrl/db/outbox-receiver";
import { createIdempotencyStore } from "@toolctrl/shared-store";

const idempotencyStore = createIdempotencyStore({
  store: sharedStore,
  prefix: "outbox",
  ttlSeconds: 60 * 60 * 24, // 24h — longer than the longest retry window
});

await withOutboxIdempotency(idempotencyStore, job.data.outboxEventId, async () => {
  await sendMail(job.data.payload);
});
```

`claim()` does an atomic `SET NX EX`. If the claim fails (another worker
already processed the event), the handler is skipped and the cached
result returned.

## Operational

### Pending count alert

`SELECT count(*) FROM "OutboxEvent" WHERE status = 'PENDING' AND scheduledAt < now() - interval '60 seconds'`

Alert when > 1000 for > 5 minutes — that means the dispatcher is behind.

### Downstream queue alert

Outbox `DONE` counts are not sufficient for worker health. Alert separately on
target BullMQ queue depth, failed jobs, and stalled jobs for `outbox.workflow`,
`outbox.mail`, `outbox.webhook`, `outbox.marketing`, and `toolctrl.email`.
Workflow incidents must include the `WorkflowExecution` terminal status because
`workflow.execution.requested` can be handed to BullMQ successfully while the
execution later fails, stalls, or remains queued behind a missing bridge.

### Re-queueing failed events

Use `toolctrl-cli outbox requeue` after fixing the underlying receiver.
The CLI supports single-row and filtered bulk requeues, `--dry-run`, JSON
output, and guardrails that avoid replaying `PENDING` / `PROCESSING`
rows. Prefer it over raw SQL so `processedAt`, `lastError`, retry state,
and replay warnings stay consistent.

Raw SQL is an emergency-only fallback for a DBA-led incident when the CLI
cannot run; record the exact statement and reason in the incident log.

### Environment

| Variable                      | Default | Purpose                                           |
| ----------------------------- | ------- | ------------------------------------------------- |
| `OUTBOX_DISPATCHER_ENABLED`   | `false` | Master switch for the dispatcher worker           |
| `OUTBOX_DISPATCH_INTERVAL_MS` | `1000`  | Polling cadence (ms)                              |
| `OUTBOX_DISPATCH_BATCH_SIZE`  | `100`   | Max events claimed per tick                       |
| `OUTBOX_DISPATCH_CONCURRENCY` | `10`    | Max claimed events processed in parallel per tick |
| `OUTBOX_MAX_RETRIES`          | `6`     | Retries before dead-letter                        |
| `OUTBOX_DONE_RETENTION_HOURS` | `720`   | DONE-row replay/debug retention window            |
| `OUTBOX_DONE_RETENTION_BATCH_SIZE` | `1000` | Max DONE rows purged per maintenance run       |

## Trade-offs

- **Latency**: ~1s polling tick. A LISTEN/NOTIFY variant could cut that
  to <10ms; deferred until needed.
- **At-least-once delivery**: receivers MUST be idempotent
  (`withOutboxIdempotency` is the cheap path).
- **Storage**: each successful event leaves a DONE row until the
  `outbox-done-retention` job purges DONE rows older than
  `OUTBOX_DONE_RETENTION_HOURS`. FAILED rows are kept for operator triage and
  compliance review.