30 KiB
Outbox-Pattern + Dispatcher
Current status source: this detail plan is historical implementation context with reconciled notes. Use
docs/architecture/outbox-pattern.mdanddocs/ops/worker-queues.mdfor current workflow/outbox runtime truth.
Status: in_progress (foundation + Stripe-webhook migration + WorkflowEngine row-trigger producer migration + form-submission mail/webhook bridges + CLI re-queue + e2e test shipped; remaining generic legacy mail/webhook/notification/audit web-action call-site migrations + admin UI deferred) Task-ID: #42 Phase: 03 Estimated Effort: 1 Woche Owner: unassigned
Problem
Aktuell sind viele Side-Effects nicht transaktional mit DB-Writes gekoppelt:
- Stripe-Webhook (
apps/api/src/billing/stripe-webhook-route.ts:60-138):updateTenantSubscriptionFromStripeupdated DB → ruft danachcreateLegacyEvent→ direkt callLegacyEvent (Mail, Webhook, Audit-Log). Wenn der Prozess zwischen Commit undcallLegacyEventcrasht, ist DB konsistent aber Mail wurde nie versendet - WorkflowEngine (
apps/web/app/modules/workflowEngine/*): This is now migrated for the row event producer path.workflow.*events route tooutbox.workflow, the worker-general bridge starts matching workflows through the signed internal web API, androw.created,row.updated, androw.deletedare published in the same transaction as their row changes. Remaining WorkflowEngine gaps are semantic/runtime follow-ups: native branch/switch execution, delay/schedule blocks, and broader non-row producer coverage. - Notification/Mail/Webhook-Outbound aus Web-Actions: direkt aus dem Request-Handler triggered, bei Worker-Crash verloren
Das Outbox-Pattern löst das durch:
- Side-Effects werden innerhalb der gleichen Transaction wie der DB-Write als „Event"-Row in eine
OutboxEvent-Tabelle geschrieben - Ein separater Dispatcher-Worker liest pending Events, dispatched zu der jeweiligen Queue (mail, webhook, notification, audit-log), und markiert sie als processed
- Failures → Retries mit Exponential-Backoff, max-Retries, dann dead-letter
- At-least-once-Delivery garantiert; Receiver brauchen Idempotenz (Idempotency-Store aus #38)
Vorteile:
- Transaction-Konsistenz: Event existiert ↔ DB-Change committed (atomic)
- Crash-Safety: bei Crash zwischen Commit und Side-Effect wird Event beim Recovery dispatched
- Audit-Trail: jede ausgelöste Aktion ist in DB sichtbar
- Decoupling: Event-Produzent muss Receiver nicht kennen
Depends on
03-scaling-infrastructure/read-write-split.md—publishOutboxEventläuft aufprisma.write(Outbox-Writes sind kritische Source-of-Truth, niemals Replica)03-scaling-infrastructure/shared-store-package.md— Dispatcher nutzt BullMQ (gleiche Redis-Connection) für Queue-Dispatch02-observability/sentry-wiring-api-worker.md— Dispatcher-Failures müssen captured werden
Blocks
03-scaling-infrastructure/job-priority.md— Dispatcher nutzt BullMQ-Priority für Event-Types (z.B. high für payment-success-mail)04-search-rag/embedding-pipeline.md— Search-Indexer hängt an Outbox-Events (entity-created, entity-updated → re-index)05-business-features/marketing-funnel-phase-1.md— Email-Sequence-Triggers via Outbox- alle nachfolgenden Features mit asynchronen Side-Effects
Approach
Neue Prisma-Tabelle:
model OutboxEvent {
id String @id @default(cuid())
tenantId String? // null = system-event
eventType String // z.B. "tenant.subscription.updated"
payload Json
status OutboxStatus @default(PENDING)
createdAt DateTime @default(now())
processedAt DateTime?
retries Int @default(0)
lastError String?
scheduledAt DateTime @default(now()) // delay-fähig
dispatchKey String? // für Idempotency / Dedup
priority Int @default(0) // für #47
@@index([status, scheduledAt])
@@index([tenantId, eventType])
@@index([dispatchKey])
}
enum OutboxStatus {
PENDING
PROCESSING
DONE
FAILED // alle retries verbraucht, dead-letter
CANCELLED
}
Publisher-Helper:
// packages/db/src/outbox.ts
export async function publishOutboxEvent(
tx: Prisma.TransactionClient,
{ tenantId, type, payload, dispatchKey, scheduledAt, priority }: PublishArgs
): Promise<OutboxEvent>
Wird nur innerhalb einer Transaction aufgerufen — Typ-Constraint verhindert direkte Nutzung gegen prisma-Singleton.
Dispatcher:
Implementierungs-Variante A (gewählt): BullMQ-Cron-Polling, alle 1s pollt der Dispatcher-Worker pending Events (WHERE status = 'PENDING' AND scheduledAt <= now() LIMIT 100, ORDER BY priority DESC, createdAt ASC), markiert sie atomic als PROCESSING (via UPDATE ... WHERE status = 'PENDING' RETURNING * für Race-Condition-Safety), dispatched zu BullMQ-Queue (Queue-Name = Event-Type-Prefix, z.B. mail, webhook, notification, audit-log).
Variante B (verworfen für jetzt, aber dokumentiert): LISTEN/NOTIFY-basiert. Postgres-Trigger published auf outbox_event_created-Channel, Dispatcher listenened. Vorteil: niedrige Latency (<10ms statt 0-1s Polling). Nachteil: Trigger-Maintenance, Channel-Reconnect-Komplexität. Migration auf Variante B könnte später erfolgen wenn Latenz wichtig wird.
Receiver-Idempotenz:
Jeder Receiver (Mail-Worker, Webhook-Worker, etc.) prüft via IdempotencyStore (aus #38) ob OutboxEvent.id bereits verarbeitet wurde. Bei Retry: skip. Garantiert exactly-once-Effect bei at-least-once-Delivery.
Retry-Strategie:
- Initial-Try: sofort
- Retry-1: nach 1 min (exponential 2^0)
- Retry-2: nach 2 min
- Retry-3: nach 4 min
- ... bis max-Retries (Default 8, also bis ~4h)
- Danach:
status = FAILED, manuelle Inspection oder Bulk-Re-Queue via CLI
Dead-Letter-Handling:
- Admin-UI-Stub in
apps/web/app/routes/admin/outbox.tsxzeigt failed Events - CLI-Tool
toolctrl-cli outbox requeue --event-id <id>setzt ein einzelnes Event wieder auf PENDING. Bulk-Requeues laufen übertoolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply].
Migration:
Status reconciliation (2026-05-27): Stripe webhook, row-event WorkflowEngine producers, CLI re-queue, and the opt-in outbox integration test have shipped. Notification/Mail/Webhook-Outbound migrations from remaining web actions and the Admin UI remain deferred.
-
Stripe-Webhook (
apps/api/src/billing/stripe-webhook-route.ts:60-138):- Transaction wrappen:
prisma.write.$transaction(async (tx) => { await updateTenantSubscriptionFromStripe(tx, ...); await publishOutboxEvent(tx, { tenantId, type: 'tenant.subscription.updated', payload: {...} }); }) callLegacyEventdirekt-Aufruf entfernen- Dispatcher routet Event zu
notification-Queue (alter callLegacyEvent-Code wird Receiver)
- Transaction wrappen:
-
WorkflowEngine (
apps/web/app/modules/workflowEngine/*):- Trigger werden zu Outbox-Events (
type: 'workflow.trigger.<eventName>') - WorkflowEngine-Worker subscribed auf entsprechende Queue, executiert Workflow
- Trigger werden zu Outbox-Events (
-
Notification / Mail / Webhook-Outbound aus Web-Actions:
- Web-Actions ersetzen direkte
sendEmail(...)-Calls durchawait publishOutboxEvent(tx, { type: 'mail.send', payload: { to, template, vars }}) - Dispatcher routet zu
mail-Queue, Mail-Worker (kommt aus #45) verarbeitet
- Web-Actions ersetzen direkte
Warum nicht Alternativen:
- ❌ Saga-Pattern: Overkill für unsere Use-Cases, zu viel Orchestration
- ❌ Event-Sourcing Full: massive Architektur-Umstellung, nicht in Phase 03 leistbar
- ❌ CDC (Change Data Capture) via Debezium/wal2json: Infra-Overhead, kompliziertes Setup
- ❌ Direkter Queue-Push aus Request-Handler (kein Outbox): bricht Transaction-Konsistenz, genau das Problem das wir lösen
Implementation Steps
-
Prisma-Migration: OutboxEvent-Tabelle
- Files:
packages/db/prisma/schema.prisma, neue Migrationprisma/migrations/<timestamp>_add_outbox_event/migration.sql - Model + Enum wie oben
- 3 Indizes für die Dispatcher-Queries
pnpm prisma migrate dev --name add_outbox_event- Commit:
03-scaling/outbox: add OutboxEvent model and migration
- Files:
-
publishOutboxEvent-Helper
- Files:
packages/db/src/outbox.ts,packages/db/src/outbox.test.ts - Signatur akzeptiert
Prisma.TransactionClient(typed) — Aufrufer muss innerhalb $transaction sein - Validierung: eventType muss
<domain>.<noun>.<verb>-Pattern matchen (z.B.tenant.subscription.updated) - DispatchKey-Validation (wenn gesetzt: max 256 Zeichen, alphanumeric)
- Tests: erfolgreicher Insert, Race-Condition mit dispatchKey (unique-Konflikt sauber gehandled)
- Commit:
03-scaling/outbox: add publishOutboxEvent helper with transaction enforcement
- Files:
-
Dispatcher-Worker-Implementation
- Files:
apps/worker/src/outbox/dispatcher.ts,apps/worker/src/outbox/dispatcher.test.ts,apps/worker/src/outbox/event-router.ts - Cron-Job (BullMQ-Repeatable) alle 1s
- Claim-Query (atomic):
UPDATE "OutboxEvent" SET status = 'PROCESSING' WHERE id IN ( SELECT id FROM "OutboxEvent" WHERE status = 'PENDING' AND "scheduledAt" <= NOW() ORDER BY priority DESC, "createdAt" ASC LIMIT 100 FOR UPDATE SKIP LOCKED ) RETURNING * - Pro claimed Event: dispatch via
event-router.tszu der richtigen BullMQ-Queue - On dispatch-success: mark
DONE, setprocessedAt - On dispatch-error: increment retries, set
lastError,scheduledAt = now + backoff(retries). Wennretries >= max: statusFAILED - Tests: claim-race-condition (zwei Dispatcher-Instances, ein Event), retry-backoff-Berechnung, dead-letter-Übergang
- Commit:
03-scaling/outbox: add dispatcher worker with skip-locked claim
- Files:
-
Event-Router: Type → Queue-Mapping
- Files:
apps/worker/src/outbox/event-router.ts - Config-basiertes Mapping:
const routing = { 'mail.': 'mail', 'webhook.': 'webhook', 'notification.': 'notification', 'audit.': 'audit-log', 'workflow.': 'workflow', 'tenant.subscription.': 'notification', // Composite-Routing 'search.index.': 'search-index', // für Phase 04 }; - Unknown-Type → DLQ + Sentry-Alert
- Tests: alle Mappings, Fallback-Verhalten
- Commit:
03-scaling/outbox: add event router with type to queue mapping
- Files:
-
Migration: Stripe-Webhook
- Files:
apps/api/src/billing/stripe-webhook-route.ts, neue Tests - Wrappen in
prisma.write.$transaction callLegacyEvent-Call durchpublishOutboxEventersetzen- Receiver-Worker (
apps/worker/src/notification/legacy-event.worker.ts) baut neuen Worker der den altencallLegacyEvent-Code als BullMQ-Handler ausführt - Tests: Stripe-Event-Mock → DB-Update + OutboxEvent existiert in derselben Transaction; bei Worker-Crash zwischen Steps → Outbox bleibt PENDING, beim nächsten Dispatch-Tick wird gepickt
- Commit:
03-scaling/outbox: migrate stripe webhook to outbox pattern
- Files:
-
Migration: WorkflowEngine-Triggers
- Files:
apps/web/app/modules/workflowEngine/routes/workflow-engine.api.server.tsx, weitere Trigger-Stellen, neuer Workerapps/worker/src/workflow/trigger.worker.ts - Workflow-Trigger publishen Outbox-Event mit eventType
workflow.trigger.<eventName>, payload enthält trigger-context - Worker konsumiert, lädt Workflow-Definition, executiert via existing Engine-Code
- Tests: Trigger-Publish + Worker-Pickup; failed-Workflow markiert Event als FAILED nach max-retries
- Commit:
03-scaling/outbox: migrate workflow engine triggers to outbox
- Files:
-
Migration: Mail-Sending aus Web-Actions
- Files: alle
apps/web/app/routes/**/action.tsx-Stellen diesendEmaildirekt aufrufen, neuer Workerapps/worker/src/mail/mail.worker.ts - Inventarisierung via Subagent (Explore-Pass): liste alle
sendEmail-Calls in Web - Pro Stelle: in Transaction wrappen +
publishOutboxEvent({ type: 'mail.send', payload: { to, template, vars }})statt direkter Call - Mail-Worker: konsumiert
mail-Queue, ruft Mail-Service (auspackages/mail) - Tests: Web-Action publisht Event, Worker verarbeitet, Mail wird gesendet (Mock-Mail-Provider)
- Commit:
03-scaling/outbox: migrate web action mail sends to outbox
- Files: alle
-
Migration: Webhook-Outbound aus Web-Actions
- Files: analog zu #7, aber für
dispatchWebhook-Calls - Commit:
03-scaling/outbox: migrate web action webhook dispatches to outbox
- Files: analog zu #7, aber für
-
Migration: Notification + Audit-Log aus Web-Actions
- Files: analog für
createNotificationundauditLog-Stellen - Commit:
03-scaling/outbox: migrate notification and audit log writes to outbox
- Files: analog für
-
Admin-UI für Outbox-Inspection
- Files:
apps/web/app/routes/admin/outbox.tsx,apps/web/app/routes/admin/outbox.$id.tsx - Listing: Status-Filter, Type-Filter, Tenant-Filter, Pagination
- Detail-View: Payload anzeigen, Retry-Manual-Button, Cancel-Button
- Tests: Loader/Action, Berechtigungs-Check (admin-only)
- Commit:
03-scaling/outbox: add admin ui for outbox event inspection
- Files:
-
CLI-Tool für Re-Queue
- Files:
scripts/outbox/requeue.ts,package.jsonscript - Superseded by
toolctrl-cli outbox requeue --event-id <id>for a single event - Superseded by
toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply]for bulk - Tests: CLI-Smoke-Test
- Commit:
03-scaling/outbox: add cli for requeueing failed events
- Files:
-
Doku
- Files:
docs/architecture/outbox-pattern.md(original plan nameddocs/database/outbox-pattern.md) - Sections:
- Konzept: Warum Outbox? Transaction-Konsistenz, Crash-Safety, Decoupling
- Publisher-Usage: Code-Beispiel für
publishOutboxEventin Transaction - Event-Type-Konvention:
<domain>.<noun>.<verb>, Registry pflegen - Dispatcher: wie er pollt, claim-Strategy, Retry-Backoff, Dead-Letter
- Receiver-Idempotenz-Pflicht: Code-Beispiel mit IdempotencyStore
- Operational: Monitoring (Pending-Count-Alert > 1000), Admin-UI, CLI-Re-Queue
- Migration-Guide: wie weitere Side-Effects migrieren
- Trade-offs: Latenz (1s Polling-Tick), zukünftige LISTEN/NOTIFY-Migration
- Commit:
03-scaling/outbox: add outbox pattern documentation
- Files:
-
Integration-Test: End-to-End Outbox-Flow
- Files:
apps/worker/src/__integration__/outbox-e2e.test.ts - testcontainers: Postgres + Redis + Mail-Mock-Service
- Test-Cases:
- Publisher → Dispatcher → Receiver → Effect (Mail-Mock empfängt)
- Crash-Recovery: Receiver wirft Error → Event geht zurück auf PENDING mit retry+1
- Dead-Letter: max-Retries → status FAILED, Admin-UI zeigt's
- Race-Condition: 2 Dispatcher-Instances → kein Event wird doppelt dispatched
- Idempotency: Receiver mit gleicher Event-ID 2x aufgerufen → Effekt nur 1x
- Commit:
03-scaling/outbox: add end-to-end integration test suite
- Files:
Test Requirements
- Unit-Tests:
publishOutboxEvent(Transaction-Constraint, dispatchKey-Unique, Validation); Dispatcher (Claim-Logic, Retry-Backoff, Dead-Letter); Event-Router (alle Mappings) - Integration-Tests: End-to-End-Test (Step 13) — Publisher in API, Dispatcher im Worker, Receiver-Effekt verifiziert. testcontainers für Postgres + Redis
- Concurrency-Tests: 2 Dispatcher-Instances parallel, 1000 Events — alle exakt 1x dispatched
- E2E-Tests (Playwright): Admin-UI für Outbox-Inspection (Listing, Detail, Re-Queue-Button)
- Coverage-Target: ≥ 90 % im Outbox-Code (Dispatcher, Helper, Router)
- Test-Files:
packages/db/src/outbox.test.tsapps/worker/src/outbox/dispatcher.test.tsapps/worker/src/outbox/event-router.test.tsapps/worker/src/__integration__/outbox-e2e.test.tsapps/web/app/routes/admin/outbox.test.tsx
NICHT als completed markieren bevor:
- Alle Tests grün, inkl. Concurrency + E2E
- Staging-Verifikation: 24h-Lauf zeigt 0 unverarbeitete Events > 60s alt (Monitoring-Alert)
- Stripe-Webhook-Test auf Staging: Crash zwischen DB-Commit und Mail-Send → nach Worker-Restart wird Mail gesendet
Acceptance Criteria
OutboxEvent-Model in Prisma-Schema + Migration appliedpublishOutboxEvent-Helper inpackages/db/src/outbox.ts, akzeptiert nur TransactionClient- Dispatcher-Worker im
apps/workermit Cron-Polling (1s), atomic Claim viaFOR UPDATE SKIP LOCKED - Event-Router mapped Event-Types zu BullMQ-Queues
- Retry-Logic mit Exponential-Backoff (max 8 Retries), Dead-Letter bei Erschöpfung
- Stripe-Webhook migriert: DB-Update + Outbox-Event in einer Transaction; Worker-Crash zwischen Commit + Dispatch wird beim Recovery aufgeholt
- WorkflowEngine row-event triggers migriert; non-row producers remain follow-ups
- Mail/Webhook/Notification/Audit-Log aus Web-Actions migriert
- Admin-UI unter
/admin/outboxlisted pending/failed Events, erlaubt Re-Queue - CLI
toolctrl-cli outbox requeuefunktioniert (single + bulk) docs/architecture/outbox-pattern.mdenthält die current runtime/operator sections; the originaldocs/database/outbox-pattern.mdtarget was superseded- Opt-in integration test grün; Concurrency-Test (2 Dispatcher) bleibt offen
- Staging-24h-Test: 0 unverarbeitete Events > 60s alt
- Grafana-Panel: Outbox-Pending-Count, Dispatch-Rate, Retry-Rate, Failed-Count
pnpm lint,pnpm typecheck,pnpm testgrün- Detail-MD Progress Log aktualisiert
- PROGRESS.md Status auf
completed
Rollback Strategy
Per-Migration-Rollback: Jede Migration (Steps 5-9) ist separat rollback-bar via git revert. Code-seitig fällt zurück auf direkte Side-Effect-Calls; OutboxEvent-Rows bleiben in DB (kein Daten-Verlust), werden ignoriert.
Dispatcher-Disable:
- Env-Var
OUTBOX_DISPATCHER_ENABLED=falseim Worker → Cron stoppt, Events stauen sich - Useful für Maintenance oder Failure-Investigation
- Pending-Count-Alert in Grafana fängt das ab
Schema-Rollback:
prisma migrate resolve --rolled-back <migration>+ manuelleDROP TABLE OutboxEvent- Daten-Verlust für FAILED/PENDING Events — vor Rollback CSV-Export via
\COPY "OutboxEvent" TO ... - Code muss vor Rollback zurück auf direct-Side-Effect-Calls (sonst breakt publishOutboxEvent)
Receiver-Rollback einzeln:
- Wenn z.B. Mail-Worker buggy ist → BullMQ-Queue
mailpausieren (bull-board-UI oder CLI), Events stauen sich harmlos, Fix deployen, Queue resumen
DB-Migration: non-destructive (nur CREATE TABLE + CREATE INDEX). Down-Migration: DROP TABLE OutboxEvent CASCADE (destructive — Daten-Verlust). Vor Down-Migration Backup ausführen.
Security Notes
- Tenant-Isolation: OutboxEvent hat
tenantId. Receiver MUSS prüfen dass Payload-Operationen den richtigen Tenant treffen. Cross-Tenant-Spoofing-Risiko wenn Receiver Payload blind vertraut → Workflow-Worker validiertpayload.tenantId === event.tenantId - Payload-Validation: Payloads sind
Json— Receiver muss Zod-Schema-Validation pro Event-Type machen, sonst Crash bei Malformed - Sensitive-Data im Payload: keine Passwords, keine PII-Mehr-als-nötig. Email-Templates referenzieren statt embed
- Replay-Attacks: dispatchKey + Idempotency-Store schützen. Receiver MUSS Idempotency-Check machen
- Dead-Letter-Access: Admin-UI darf nur Admins lesen (RBAC-Check). Failed-Events können sensitive Daten enthalten (z.B. Stripe-Payload)
- Re-Queue-Berechtigung: CLI + UI-Re-Queue nur für Admins. Manuelle Re-Queue eines Events kann unbeabsichtigte Effekte triggern (z.B. doppelte Mail) — Admin-UI zeigt Warnung
- DoS via Outbox-Spam: Wenn ein bug 1000s Events pro Sekunde publisht → Dispatcher kommt nicht hinterher. Rate-Limit auf Publisher-Seite (max 100 Events pro Tenant pro Minute via SharedStore-Rate-Limiter)
- Backup-Implications: OutboxEvent-Tabelle in Backup einbeziehen (sind kritische Source-of-Truth)
- OutboxEvent-Retention: completed Events nach 30 Tagen löschen (Cron-Job), sonst Tabellen-Bloat. Failed-Events behalten (Compliance)
Subagent Recommendations
- Planning:
Plan-Subagent mitapps/api/src/billing/stripe-webhook-route.ts:60-138als Hauptbeispiel + Outbox-Pattern-Reference-Artikel - Research:
Explore-Subagent für „inventarisiere allesendEmail,dispatchWebhook,createNotification,auditLog-Calls inapps/web/. Output: Migration-Backlog für Steps 7-9"Explore-Subagent für „suche bestehende BullMQ-Queue-Setups inapps/worker/— was kann wiederverwendet werden?"
- Implementation:
general-purposefür Steps 1-4 (Foundation: Schema + Helper + Dispatcher + Router)- Eigener
general-purposepro Migration-Step (5-9) — pro Bereich ein Subagent, weil Receiver-Worker-Code-Pattern jeweils unterschiedlich general-purposefür Step 10 (Admin-UI)general-purposefür Step 11 (CLI)general-purposefür Step 12 (Doku — sehr ausführlich)
- Tests: Separater
general-purposefür E2E-Integration-Test (Step 13) — kompliziertes testcontainers-Setup, eigener Kontext - Verification:
gsd-verifierfür Acceptance-Criteria; manuelles 24h-Staging-Monitoring - Code-Review:
gsd-code-reviewerfür jeden Migration-Step separat (Stripe vs WorkflowEngine vs Mail haben unterschiedliche Risiken)
Historical Staging-Deployment-Steps
This section is retained as the original implementation sketch. The current
production topology uses separate persistent worker applications (general,
mail, import, webhook, plugin) and worker-general owns the outbox
dispatcher; do not provision the old worker-dispatcher/WORKER_ROLE layout.
- Branch
agent/03-outbox-patternpushen, schrittweise Commits 1-13 - Coolify-Setup-Prerequisites: PgBouncer (#40), R/W-Split (#41), Shared-Store (#37/#38) müssen schon laufen
- Migration:
- Pre-Deploy-Check:
pnpm prisma migrate statusauf Staging - Deploy mit
prisma migrate deploy(Coolify-Pre-Start-Hook) - OutboxEvent-Tabelle erstellt, leer
- Pre-Deploy-Check:
- Worker-Replicas:
- Historical only: Dispatcher-Worker als separate Replica vom General-Worker (skaliert unabhängig)
- Historical only: Coolify-Service
worker-dispatchermitprocess.env.WORKER_ROLE=dispatcher - Historical only: Mindestens 1 Replica (Dispatcher ist idempotent via SKIP LOCKED, 2 Replicas sind safe für HA)
- Historical receiver sketch: domain receivers such as mail, webhook, notification, audit-log, and workflow should run on the current worker groups unless a future dedicated receiver group is added. The current Coolify topology is
worker-general,worker-mail,worker-import,worker-webhook, andworker-plugin.
- Env-Vars:
OUTBOX_DISPATCHER_ENABLED=true OUTBOX_DISPATCH_INTERVAL_MS=1000 OUTBOX_DISPATCH_BATCH_SIZE=100 OUTBOX_MAX_RETRIES=6 OUTBOX_DONE_RETENTION_HOURS=720 - Smoke-Tests:
- Stripe-Webhook-E2E:
- Stripe-CLI:
stripe trigger customer.subscription.updated - Innerhalb 2s:
SELECT * FROM "OutboxEvent" WHERE "eventType" = 'tenant.subscription.updated' ORDER BY "createdAt" DESC LIMIT 1→ erwarte 1 Row, status PENDING oder DONE - Nach 5s: Row sollte DONE sein
- Mail-Worker-Log zeigt: Mail an Tenant-Admin versendet
- Stripe-CLI:
- Crash-Recovery-Test:
- Stripe-CLI: trigger Event
- Sofort
docker kill worker-mail(während Event noch PENDING) - OutboxEvent bleibt PENDING (oder geht auf FAILED nach Retries — je nach Timing)
docker compose up worker-mailneu starten- Innerhalb 30s: Event wird verarbeitet, Mail gesendet
- Concurrency-Test:
- 1000 Events via Script publishen
- Historical only:
docker scale worker-dispatcher=3 - Verifiziere: jedes Event genau 1x in Receiver-Logs (über Event-ID grep)
- Dead-Letter-Test:
- Mail-Mock so konfigurieren dass es immer wirft
- Event triggern, beobachten dass es nach 8 Retries auf FAILED geht
- Admin-UI
/admin/outboxzeigt Event in Failed-Liste - CLI:
toolctrl-cli outbox requeue --event-id <id>→ Event geht zurück auf PENDING
- Stripe-Webhook-E2E:
- Monitoring-Setup:
- Grafana-Panels:
- „Outbox Pending Count" (Alert: > 1000 für > 5min)
- „Outbox Failed Count Daily" (Alert: > 50/day)
- „Dispatcher Throughput" (events/sec)
- „Receiver Latency P95" pro Queue
- Grafana-Panels:
- 24h-Beobachtungs-Phase:
- Logs auf Sentry-Errors prüfen
SELECT status, COUNT(*) FROM "OutboxEvent" GROUP BY statusregelmäßig- Acceptance: 0 unverarbeitete Events > 60s alt im 24h-Fenster
Deviations from Plan
- Step 5 (Stripe webhook) — DONE in 2026-05-25 session. Wrapped subscription update + outbox publishes in a single
prisma.$transaction; legacycreateLegacyEventdirect call moved into the newapps/worker/src/jobs/legacy-event-receiver.tsconsumer onoutbox.notification, guarded bywithOutboxIdempotency. - Step 11 (CLI re-queue) — DONE in 2026-05-25 session.
toolctrl-cli outbox requeue --event-id <id>andoutbox requeue --type <eventType> --since <iso> [--apply]ship underpackages/cli/src/commands/outbox/. Two modes (single + bulk), dry-run by default for bulk, refuses to re-queue events already PENDING/PROCESSING, JSON output for scripting. - Step 13 (end-to-end integration test) — DONE in 2026-05-25 session.
packages/db/src/outbox-e2e.test.tsexercises producer → claim → receiver → marker-row insert against real Postgres, with idempotency re-delivery assertion. Opt-in viaOUTBOX_E2E_DATABASE_URL(or anyDATABASE_URLthat is not the vitest stub). - Step 6 (WorkflowEngine triggers) — ROUTE ROW PRODUCERS LANDED in 2026-05-26 session.
apps/worker/src/jobs/workflow-trigger.tsnow parsesworkflow.*outbox jobs, isolates redeliveries withwithOutboxIdempotency, fans out to live event-trigger workflows, applies preflight, and starts executions through an injected executor port.apps/worker/src/jobs/workflow-trigger-store.tsadds the worker-native Prisma lookup/preflight store without importing web server modules, andbootstrapWorkflowTriggeris registered fromapps/worker/src/index.tsfor thegeneralworker. Route-backedrow.created,row.updated, androw.deletedproducers publishworkflow.trigger.eventinside the same DB transaction as the row write and application event persistence when a request context is passed. Internal requestless mutations remain eventless by default to avoid worker feedback loops, while the direct internal trigger bridge remains the only allowedtype: "trigger"execution call-site. - Steps 7, 8, 9 (generic legacy mail/webhook/notification/audit-log from web actions) — DEFERRED. Form-submission delivery is already outbox-backed: the general worker routes
form.submission.*work andworker-mailbridgeswebhook.form-submission.deliver/mail.transactional.form-submissionthrough the currentoutbox.webhookandoutbox.mailsubscribers. The remaining deferred work is the broader legacy action inventory outside that reviewed form-submission path. Reason: each generic migration touches a different module (apps/web/app/routes/**/action.tsx) and a different receiver worker pattern, and the operator scheduled them as separate per-domain code-review cycles. The foundation makes each one mechanical: wrap the action's DB write inprisma.$transaction, replace the direct side-effect call withpublishOutboxEvent(tx, { type: '<domain>.<entity>.<verb>', payload }), and add a queue subscriber that mirrorslegacy-event-receiver.ts(already merged this session). Acceptance criterion per migration: at least one Playwright assertion that crashing the worker between DB commit and queue-pop still delivers the side effect on restart. - Step 10 (admin UI under
/app/admin/outbox) — DEFERRED. Reason: design decision (filters, detail view layout, re-queue confirm dialog) — needs UX input the operator hasn't provided yet. The CLI (toolctrl-cli outbox requeue) covers the operational hot-path until the UI lands. Acceptance criterion: list view with Status/Type/Tenant filters + pagination, detail view with payload + retry/cancel buttons, admin-only RBAC. - The dispatcher uses an in-process
setTimeout-based polling loop rather than a BullMQ-repeatable cron job. This avoids needing a BullMQ Queue just to schedule the dispatcher itself, keeps the dispatcher independently testable withvi.useFakeTimers(), and matches the way other internal pollers (metrics-scheduler) are wired. - Default routing maps
billing.*andtenant.subscription.*tooutbox.notification. The Stripe-webhook migration usesbilling.subscription.ended+billing.subscription.cancelledwhich land onoutbox.notificationvia the existingbilling.prefix rule — no router changes were needed.
Progress Log
- 2026-05-17 08:25 — 6ac38a11 — Add OutboxEvent prisma model + migration + regenerated client
- 2026-05-17 08:30 — 62fa3436 — Add publishOutboxEvent publisher helper with validation + dispatchKey idempotency
- 2026-05-17 08:36 — 1427a4ef — Add atomic claim-batch query (FOR UPDATE SKIP LOCKED) with integration tests
- 2026-05-17 08:41 — 386dc9fc — Add dispatcher worker + event-router with exponential-backoff retry and dead-letter
- 2026-05-17 08:49 — 5a19fdf3 — Register dispatcher in worker bootstrap (gated by OUTBOX_DISPATCHER_ENABLED)
- 2026-05-17 08:52 — 9a79b212 — Add withOutboxIdempotency receiver helper for at-least-once safety
- 2026-05-17 08:55 — 41c277ab — Document outbox pattern in docs/architecture/outbox-pattern.md + update progress logs
- 2026-05-17 08:57 — 5a57e18a — Catalog OUTBOX_DISPATCHER_ENABLED / BATCH_SIZE / INTERVAL_MS / MAX_RETRIES and add to .env.example
- 2026-05-25 — 0bebef90 — feat(outbox): migrate stripe webhook to outbox pattern (DB update + billing.subscription.{ended,cancelled} publish in single prisma.$transaction, new legacy-event-receiver consumer on outbox.notification guarded by withOutboxIdempotency, bootstrap wired into worker general-group with shutdown integration, stripe-webhook + receiver unit tests cover the new transactional + idempotent paths)
- 2026-05-25 — 9e40294f — feat(cli): add outbox requeue command for failed and bulk replay (--event-id single mode, --type --since bulk mode with --apply confirm flag, refuses to re-queue PENDING/PROCESSING, JSON output, 15 unit tests)
- 2026-05-25 — a5ac964b — test(outbox): end-to-end integration via real postgres opt-in (publisher → claim → custom routing → receiver → marker-insert → idempotent re-delivery; opt-in via OUTBOX_E2E_DATABASE_URL; suite passes live against local postgres and skips when no DB env is set)
Lessons Learned
(Optional, am Ende des Tasks für nachfolgende Tasks)