Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/03-scaling-infrastructure/outbox-pattern.md
T

30 KiB

Outbox-Pattern + Dispatcher

Current status source: this detail plan is historical implementation context with reconciled notes. Use docs/architecture/outbox-pattern.md and docs/ops/worker-queues.md for current workflow/outbox runtime truth.

Status: in_progress (foundation + Stripe-webhook migration + WorkflowEngine row-trigger producer migration + form-submission mail/webhook bridges + CLI re-queue + e2e test shipped; remaining generic legacy mail/webhook/notification/audit web-action call-site migrations + admin UI deferred) Task-ID: #42 Phase: 03 Estimated Effort: 1 Woche Owner: unassigned

Problem

Aktuell sind viele Side-Effects nicht transaktional mit DB-Writes gekoppelt:

  • Stripe-Webhook (apps/api/src/billing/stripe-webhook-route.ts:60-138): updateTenantSubscriptionFromStripe updated DB → ruft danach createLegacyEvent → direkt callLegacyEvent (Mail, Webhook, Audit-Log). Wenn der Prozess zwischen Commit und callLegacyEvent crasht, ist DB konsistent aber Mail wurde nie versendet
  • WorkflowEngine (apps/web/app/modules/workflowEngine/*): This is now migrated for the row event producer path. workflow.* events route to outbox.workflow, the worker-general bridge starts matching workflows through the signed internal web API, and row.created, row.updated, and row.deleted are published in the same transaction as their row changes. Remaining WorkflowEngine gaps are semantic/runtime follow-ups: native branch/switch execution, delay/schedule blocks, and broader non-row producer coverage.
  • Notification/Mail/Webhook-Outbound aus Web-Actions: direkt aus dem Request-Handler triggered, bei Worker-Crash verloren

Das Outbox-Pattern löst das durch:

  1. Side-Effects werden innerhalb der gleichen Transaction wie der DB-Write als „Event"-Row in eine OutboxEvent-Tabelle geschrieben
  2. Ein separater Dispatcher-Worker liest pending Events, dispatched zu der jeweiligen Queue (mail, webhook, notification, audit-log), und markiert sie als processed
  3. Failures → Retries mit Exponential-Backoff, max-Retries, dann dead-letter
  4. At-least-once-Delivery garantiert; Receiver brauchen Idempotenz (Idempotency-Store aus #38)

Vorteile:

  • Transaction-Konsistenz: Event existiert ↔ DB-Change committed (atomic)
  • Crash-Safety: bei Crash zwischen Commit und Side-Effect wird Event beim Recovery dispatched
  • Audit-Trail: jede ausgelöste Aktion ist in DB sichtbar
  • Decoupling: Event-Produzent muss Receiver nicht kennen

Depends on

  • 03-scaling-infrastructure/read-write-split.md — publishOutboxEvent läuft auf prisma.write (Outbox-Writes sind kritische Source-of-Truth, niemals Replica)
  • 03-scaling-infrastructure/shared-store-package.md — Dispatcher nutzt BullMQ (gleiche Redis-Connection) für Queue-Dispatch
  • 02-observability/sentry-wiring-api-worker.md — Dispatcher-Failures müssen captured werden

Blocks

  • 03-scaling-infrastructure/job-priority.md — Dispatcher nutzt BullMQ-Priority für Event-Types (z.B. high für payment-success-mail)
  • 04-search-rag/embedding-pipeline.md — Search-Indexer hängt an Outbox-Events (entity-created, entity-updated → re-index)
  • 05-business-features/marketing-funnel-phase-1.md — Email-Sequence-Triggers via Outbox
  • alle nachfolgenden Features mit asynchronen Side-Effects

Approach

Neue Prisma-Tabelle:

model OutboxEvent {
  id          String         @id @default(cuid())
  tenantId    String?        // null = system-event
  eventType   String         // z.B. "tenant.subscription.updated"
  payload     Json
  status      OutboxStatus   @default(PENDING)
  createdAt   DateTime       @default(now())
  processedAt DateTime?
  retries     Int            @default(0)
  lastError   String?
  scheduledAt DateTime       @default(now())  // delay-fähig
  dispatchKey String?        // für Idempotency / Dedup
  priority    Int            @default(0)      // für #47

  @@index([status, scheduledAt])
  @@index([tenantId, eventType])
  @@index([dispatchKey])
}

enum OutboxStatus {
  PENDING
  PROCESSING
  DONE
  FAILED          // alle retries verbraucht, dead-letter
  CANCELLED
}

Publisher-Helper:

// packages/db/src/outbox.ts
export async function publishOutboxEvent(
  tx: Prisma.TransactionClient,
  { tenantId, type, payload, dispatchKey, scheduledAt, priority }: PublishArgs
): Promise<OutboxEvent>

Wird nur innerhalb einer Transaction aufgerufen — Typ-Constraint verhindert direkte Nutzung gegen prisma-Singleton.

Dispatcher:

Implementierungs-Variante A (gewählt): BullMQ-Cron-Polling, alle 1s pollt der Dispatcher-Worker pending Events (WHERE status = 'PENDING' AND scheduledAt <= now() LIMIT 100, ORDER BY priority DESC, createdAt ASC), markiert sie atomic als PROCESSING (via UPDATE ... WHERE status = 'PENDING' RETURNING * für Race-Condition-Safety), dispatched zu BullMQ-Queue (Queue-Name = Event-Type-Prefix, z.B. mail, webhook, notification, audit-log).

Variante B (verworfen für jetzt, aber dokumentiert): LISTEN/NOTIFY-basiert. Postgres-Trigger published auf outbox_event_created-Channel, Dispatcher listenened. Vorteil: niedrige Latency (<10ms statt 0-1s Polling). Nachteil: Trigger-Maintenance, Channel-Reconnect-Komplexität. Migration auf Variante B könnte später erfolgen wenn Latenz wichtig wird.

Receiver-Idempotenz:

Jeder Receiver (Mail-Worker, Webhook-Worker, etc.) prüft via IdempotencyStore (aus #38) ob OutboxEvent.id bereits verarbeitet wurde. Bei Retry: skip. Garantiert exactly-once-Effect bei at-least-once-Delivery.

Retry-Strategie:

  • Initial-Try: sofort
  • Retry-1: nach 1 min (exponential 2^0)
  • Retry-2: nach 2 min
  • Retry-3: nach 4 min
  • ... bis max-Retries (Default 8, also bis ~4h)
  • Danach: status = FAILED, manuelle Inspection oder Bulk-Re-Queue via CLI

Dead-Letter-Handling:

  • Admin-UI-Stub in apps/web/app/routes/admin/outbox.tsx zeigt failed Events
  • CLI-Tool toolctrl-cli outbox requeue --event-id <id> setzt ein einzelnes Event wieder auf PENDING. Bulk-Requeues laufen über toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply].

Migration:

Status reconciliation (2026-05-27): Stripe webhook, row-event WorkflowEngine producers, CLI re-queue, and the opt-in outbox integration test have shipped. Notification/Mail/Webhook-Outbound migrations from remaining web actions and the Admin UI remain deferred.

  1. Stripe-Webhook (apps/api/src/billing/stripe-webhook-route.ts:60-138):

    • Transaction wrappen: prisma.write.$transaction(async (tx) => { await updateTenantSubscriptionFromStripe(tx, ...); await publishOutboxEvent(tx, { tenantId, type: 'tenant.subscription.updated', payload: {...} }); })
    • callLegacyEvent direkt-Aufruf entfernen
    • Dispatcher routet Event zu notification-Queue (alter callLegacyEvent-Code wird Receiver)
  2. WorkflowEngine (apps/web/app/modules/workflowEngine/*):

    • Trigger werden zu Outbox-Events (type: 'workflow.trigger.<eventName>')
    • WorkflowEngine-Worker subscribed auf entsprechende Queue, executiert Workflow
  3. Notification / Mail / Webhook-Outbound aus Web-Actions:

    • Web-Actions ersetzen direkte sendEmail(...)-Calls durch await publishOutboxEvent(tx, { type: 'mail.send', payload: { to, template, vars }})
    • Dispatcher routet zu mail-Queue, Mail-Worker (kommt aus #45) verarbeitet

Warum nicht Alternativen:

  • ❌ Saga-Pattern: Overkill für unsere Use-Cases, zu viel Orchestration
  • ❌ Event-Sourcing Full: massive Architektur-Umstellung, nicht in Phase 03 leistbar
  • ❌ CDC (Change Data Capture) via Debezium/wal2json: Infra-Overhead, kompliziertes Setup
  • ❌ Direkter Queue-Push aus Request-Handler (kein Outbox): bricht Transaction-Konsistenz, genau das Problem das wir lösen

Implementation Steps

  1. Prisma-Migration: OutboxEvent-Tabelle

    • Files: packages/db/prisma/schema.prisma, neue Migration prisma/migrations/<timestamp>_add_outbox_event/migration.sql
    • Model + Enum wie oben
    • 3 Indizes für die Dispatcher-Queries
    • pnpm prisma migrate dev --name add_outbox_event
    • Commit: 03-scaling/outbox: add OutboxEvent model and migration
  2. publishOutboxEvent-Helper

    • Files: packages/db/src/outbox.ts, packages/db/src/outbox.test.ts
    • Signatur akzeptiert Prisma.TransactionClient (typed) — Aufrufer muss innerhalb $transaction sein
    • Validierung: eventType muss <domain>.<noun>.<verb>-Pattern matchen (z.B. tenant.subscription.updated)
    • DispatchKey-Validation (wenn gesetzt: max 256 Zeichen, alphanumeric)
    • Tests: erfolgreicher Insert, Race-Condition mit dispatchKey (unique-Konflikt sauber gehandled)
    • Commit: 03-scaling/outbox: add publishOutboxEvent helper with transaction enforcement
  3. Dispatcher-Worker-Implementation

    • Files: apps/worker/src/outbox/dispatcher.ts, apps/worker/src/outbox/dispatcher.test.ts, apps/worker/src/outbox/event-router.ts
    • Cron-Job (BullMQ-Repeatable) alle 1s
    • Claim-Query (atomic):
      UPDATE "OutboxEvent"
      SET status = 'PROCESSING'
      WHERE id IN (
        SELECT id FROM "OutboxEvent"
        WHERE status = 'PENDING' AND "scheduledAt" <= NOW()
        ORDER BY priority DESC, "createdAt" ASC
        LIMIT 100
        FOR UPDATE SKIP LOCKED
      )
      RETURNING *
      
    • Pro claimed Event: dispatch via event-router.ts zu der richtigen BullMQ-Queue
    • On dispatch-success: mark DONE, set processedAt
    • On dispatch-error: increment retries, set lastError, scheduledAt = now + backoff(retries). Wenn retries >= max: status FAILED
    • Tests: claim-race-condition (zwei Dispatcher-Instances, ein Event), retry-backoff-Berechnung, dead-letter-Übergang
    • Commit: 03-scaling/outbox: add dispatcher worker with skip-locked claim
  4. Event-Router: Type → Queue-Mapping

    • Files: apps/worker/src/outbox/event-router.ts
    • Config-basiertes Mapping:
      const routing = {
        'mail.': 'mail',
        'webhook.': 'webhook',
        'notification.': 'notification',
        'audit.': 'audit-log',
        'workflow.': 'workflow',
        'tenant.subscription.': 'notification',  // Composite-Routing
        'search.index.': 'search-index',  // für Phase 04
      };
      
    • Unknown-Type → DLQ + Sentry-Alert
    • Tests: alle Mappings, Fallback-Verhalten
    • Commit: 03-scaling/outbox: add event router with type to queue mapping
  5. Migration: Stripe-Webhook

    • Files: apps/api/src/billing/stripe-webhook-route.ts, neue Tests
    • Wrappen in prisma.write.$transaction
    • callLegacyEvent-Call durch publishOutboxEvent ersetzen
    • Receiver-Worker (apps/worker/src/notification/legacy-event.worker.ts) baut neuen Worker der den alten callLegacyEvent-Code als BullMQ-Handler ausführt
    • Tests: Stripe-Event-Mock → DB-Update + OutboxEvent existiert in derselben Transaction; bei Worker-Crash zwischen Steps → Outbox bleibt PENDING, beim nächsten Dispatch-Tick wird gepickt
    • Commit: 03-scaling/outbox: migrate stripe webhook to outbox pattern
  6. Migration: WorkflowEngine-Triggers

    • Files: apps/web/app/modules/workflowEngine/routes/workflow-engine.api.server.tsx, weitere Trigger-Stellen, neuer Worker apps/worker/src/workflow/trigger.worker.ts
    • Workflow-Trigger publishen Outbox-Event mit eventType workflow.trigger.<eventName>, payload enthält trigger-context
    • Worker konsumiert, lädt Workflow-Definition, executiert via existing Engine-Code
    • Tests: Trigger-Publish + Worker-Pickup; failed-Workflow markiert Event als FAILED nach max-retries
    • Commit: 03-scaling/outbox: migrate workflow engine triggers to outbox
  7. Migration: Mail-Sending aus Web-Actions

    • Files: alle apps/web/app/routes/**/action.tsx-Stellen die sendEmail direkt aufrufen, neuer Worker apps/worker/src/mail/mail.worker.ts
    • Inventarisierung via Subagent (Explore-Pass): liste alle sendEmail-Calls in Web
    • Pro Stelle: in Transaction wrappen + publishOutboxEvent({ type: 'mail.send', payload: { to, template, vars }}) statt direkter Call
    • Mail-Worker: konsumiert mail-Queue, ruft Mail-Service (aus packages/mail)
    • Tests: Web-Action publisht Event, Worker verarbeitet, Mail wird gesendet (Mock-Mail-Provider)
    • Commit: 03-scaling/outbox: migrate web action mail sends to outbox
  8. Migration: Webhook-Outbound aus Web-Actions

    • Files: analog zu #7, aber für dispatchWebhook-Calls
    • Commit: 03-scaling/outbox: migrate web action webhook dispatches to outbox
  9. Migration: Notification + Audit-Log aus Web-Actions

    • Files: analog für createNotification und auditLog-Stellen
    • Commit: 03-scaling/outbox: migrate notification and audit log writes to outbox
  10. Admin-UI für Outbox-Inspection

    • Files: apps/web/app/routes/admin/outbox.tsx, apps/web/app/routes/admin/outbox.$id.tsx
    • Listing: Status-Filter, Type-Filter, Tenant-Filter, Pagination
    • Detail-View: Payload anzeigen, Retry-Manual-Button, Cancel-Button
    • Tests: Loader/Action, Berechtigungs-Check (admin-only)
    • Commit: 03-scaling/outbox: add admin ui for outbox event inspection
  11. CLI-Tool für Re-Queue

    • Files: scripts/outbox/requeue.ts, package.json script
    • Superseded by toolctrl-cli outbox requeue --event-id <id> for a single event
    • Superseded by toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply] for bulk
    • Tests: CLI-Smoke-Test
    • Commit: 03-scaling/outbox: add cli for requeueing failed events
  12. Doku

    • Files: docs/architecture/outbox-pattern.md (original plan named docs/database/outbox-pattern.md)
    • Sections:
      • Konzept: Warum Outbox? Transaction-Konsistenz, Crash-Safety, Decoupling
      • Publisher-Usage: Code-Beispiel für publishOutboxEvent in Transaction
      • Event-Type-Konvention: <domain>.<noun>.<verb>, Registry pflegen
      • Dispatcher: wie er pollt, claim-Strategy, Retry-Backoff, Dead-Letter
      • Receiver-Idempotenz-Pflicht: Code-Beispiel mit IdempotencyStore
      • Operational: Monitoring (Pending-Count-Alert > 1000), Admin-UI, CLI-Re-Queue
      • Migration-Guide: wie weitere Side-Effects migrieren
      • Trade-offs: Latenz (1s Polling-Tick), zukünftige LISTEN/NOTIFY-Migration
    • Commit: 03-scaling/outbox: add outbox pattern documentation
  13. Integration-Test: End-to-End Outbox-Flow

    • Files: apps/worker/src/__integration__/outbox-e2e.test.ts
    • testcontainers: Postgres + Redis + Mail-Mock-Service
    • Test-Cases:
      • Publisher → Dispatcher → Receiver → Effect (Mail-Mock empfängt)
      • Crash-Recovery: Receiver wirft Error → Event geht zurück auf PENDING mit retry+1
      • Dead-Letter: max-Retries → status FAILED, Admin-UI zeigt's
      • Race-Condition: 2 Dispatcher-Instances → kein Event wird doppelt dispatched
      • Idempotency: Receiver mit gleicher Event-ID 2x aufgerufen → Effekt nur 1x
    • Commit: 03-scaling/outbox: add end-to-end integration test suite

Test Requirements

  • Unit-Tests: publishOutboxEvent (Transaction-Constraint, dispatchKey-Unique, Validation); Dispatcher (Claim-Logic, Retry-Backoff, Dead-Letter); Event-Router (alle Mappings)
  • Integration-Tests: End-to-End-Test (Step 13) — Publisher in API, Dispatcher im Worker, Receiver-Effekt verifiziert. testcontainers für Postgres + Redis
  • Concurrency-Tests: 2 Dispatcher-Instances parallel, 1000 Events — alle exakt 1x dispatched
  • E2E-Tests (Playwright): Admin-UI für Outbox-Inspection (Listing, Detail, Re-Queue-Button)
  • Coverage-Target: ≥ 90 % im Outbox-Code (Dispatcher, Helper, Router)
  • Test-Files:
    • packages/db/src/outbox.test.ts
    • apps/worker/src/outbox/dispatcher.test.ts
    • apps/worker/src/outbox/event-router.test.ts
    • apps/worker/src/__integration__/outbox-e2e.test.ts
    • apps/web/app/routes/admin/outbox.test.tsx

NICHT als completed markieren bevor:

  • Alle Tests grün, inkl. Concurrency + E2E
  • Staging-Verifikation: 24h-Lauf zeigt 0 unverarbeitete Events > 60s alt (Monitoring-Alert)
  • Stripe-Webhook-Test auf Staging: Crash zwischen DB-Commit und Mail-Send → nach Worker-Restart wird Mail gesendet

Acceptance Criteria

  • OutboxEvent-Model in Prisma-Schema + Migration applied
  • publishOutboxEvent-Helper in packages/db/src/outbox.ts, akzeptiert nur TransactionClient
  • Dispatcher-Worker im apps/worker mit Cron-Polling (1s), atomic Claim via FOR UPDATE SKIP LOCKED
  • Event-Router mapped Event-Types zu BullMQ-Queues
  • Retry-Logic mit Exponential-Backoff (max 8 Retries), Dead-Letter bei Erschöpfung
  • Stripe-Webhook migriert: DB-Update + Outbox-Event in einer Transaction; Worker-Crash zwischen Commit + Dispatch wird beim Recovery aufgeholt
  • WorkflowEngine row-event triggers migriert; non-row producers remain follow-ups
  • Mail/Webhook/Notification/Audit-Log aus Web-Actions migriert
  • Admin-UI unter /admin/outbox listed pending/failed Events, erlaubt Re-Queue
  • CLI toolctrl-cli outbox requeue funktioniert (single + bulk)
  • docs/architecture/outbox-pattern.md enthält die current runtime/operator sections; the original docs/database/outbox-pattern.md target was superseded
  • Opt-in integration test grün; Concurrency-Test (2 Dispatcher) bleibt offen
  • Staging-24h-Test: 0 unverarbeitete Events > 60s alt
  • Grafana-Panel: Outbox-Pending-Count, Dispatch-Rate, Retry-Rate, Failed-Count
  • pnpm lint, pnpm typecheck, pnpm test grün
  • Detail-MD Progress Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

Per-Migration-Rollback: Jede Migration (Steps 5-9) ist separat rollback-bar via git revert. Code-seitig fällt zurück auf direkte Side-Effect-Calls; OutboxEvent-Rows bleiben in DB (kein Daten-Verlust), werden ignoriert.

Dispatcher-Disable:

  • Env-Var OUTBOX_DISPATCHER_ENABLED=false im Worker → Cron stoppt, Events stauen sich
  • Useful für Maintenance oder Failure-Investigation
  • Pending-Count-Alert in Grafana fängt das ab

Schema-Rollback:

  • prisma migrate resolve --rolled-back <migration> + manuelle DROP TABLE OutboxEvent
  • Daten-Verlust für FAILED/PENDING Events — vor Rollback CSV-Export via \COPY "OutboxEvent" TO ...
  • Code muss vor Rollback zurück auf direct-Side-Effect-Calls (sonst breakt publishOutboxEvent)

Receiver-Rollback einzeln:

  • Wenn z.B. Mail-Worker buggy ist → BullMQ-Queue mail pausieren (bull-board-UI oder CLI), Events stauen sich harmlos, Fix deployen, Queue resumen

DB-Migration: non-destructive (nur CREATE TABLE + CREATE INDEX). Down-Migration: DROP TABLE OutboxEvent CASCADE (destructive — Daten-Verlust). Vor Down-Migration Backup ausführen.

Security Notes

  • Tenant-Isolation: OutboxEvent hat tenantId. Receiver MUSS prüfen dass Payload-Operationen den richtigen Tenant treffen. Cross-Tenant-Spoofing-Risiko wenn Receiver Payload blind vertraut → Workflow-Worker validiert payload.tenantId === event.tenantId
  • Payload-Validation: Payloads sind Json — Receiver muss Zod-Schema-Validation pro Event-Type machen, sonst Crash bei Malformed
  • Sensitive-Data im Payload: keine Passwords, keine PII-Mehr-als-nötig. Email-Templates referenzieren statt embed
  • Replay-Attacks: dispatchKey + Idempotency-Store schützen. Receiver MUSS Idempotency-Check machen
  • Dead-Letter-Access: Admin-UI darf nur Admins lesen (RBAC-Check). Failed-Events können sensitive Daten enthalten (z.B. Stripe-Payload)
  • Re-Queue-Berechtigung: CLI + UI-Re-Queue nur für Admins. Manuelle Re-Queue eines Events kann unbeabsichtigte Effekte triggern (z.B. doppelte Mail) — Admin-UI zeigt Warnung
  • DoS via Outbox-Spam: Wenn ein bug 1000s Events pro Sekunde publisht → Dispatcher kommt nicht hinterher. Rate-Limit auf Publisher-Seite (max 100 Events pro Tenant pro Minute via SharedStore-Rate-Limiter)
  • Backup-Implications: OutboxEvent-Tabelle in Backup einbeziehen (sind kritische Source-of-Truth)
  • OutboxEvent-Retention: completed Events nach 30 Tagen löschen (Cron-Job), sonst Tabellen-Bloat. Failed-Events behalten (Compliance)

Subagent Recommendations

  • Planning: Plan-Subagent mit apps/api/src/billing/stripe-webhook-route.ts:60-138 als Hauptbeispiel + Outbox-Pattern-Reference-Artikel
  • Research:
    • Explore-Subagent für „inventarisiere alle sendEmail, dispatchWebhook, createNotification, auditLog-Calls in apps/web/. Output: Migration-Backlog für Steps 7-9"
    • Explore-Subagent für „suche bestehende BullMQ-Queue-Setups in apps/worker/ — was kann wiederverwendet werden?"
  • Implementation:
    • general-purpose für Steps 1-4 (Foundation: Schema + Helper + Dispatcher + Router)
    • Eigener general-purpose pro Migration-Step (5-9) — pro Bereich ein Subagent, weil Receiver-Worker-Code-Pattern jeweils unterschiedlich
    • general-purpose für Step 10 (Admin-UI)
    • general-purpose für Step 11 (CLI)
    • general-purpose für Step 12 (Doku — sehr ausführlich)
  • Tests: Separater general-purpose für E2E-Integration-Test (Step 13) — kompliziertes testcontainers-Setup, eigener Kontext
  • Verification: gsd-verifier für Acceptance-Criteria; manuelles 24h-Staging-Monitoring
  • Code-Review: gsd-code-reviewer für jeden Migration-Step separat (Stripe vs WorkflowEngine vs Mail haben unterschiedliche Risiken)

Historical Staging-Deployment-Steps

This section is retained as the original implementation sketch. The current production topology uses separate persistent worker applications (general, mail, import, webhook, plugin) and worker-general owns the outbox dispatcher; do not provision the old worker-dispatcher/WORKER_ROLE layout.

  1. Branch agent/03-outbox-pattern pushen, schrittweise Commits 1-13
  2. Coolify-Setup-Prerequisites: PgBouncer (#40), R/W-Split (#41), Shared-Store (#37/#38) müssen schon laufen
  3. Migration:
    • Pre-Deploy-Check: pnpm prisma migrate status auf Staging
    • Deploy mit prisma migrate deploy (Coolify-Pre-Start-Hook)
    • OutboxEvent-Tabelle erstellt, leer
  4. Worker-Replicas:
    • Historical only: Dispatcher-Worker als separate Replica vom General-Worker (skaliert unabhängig)
    • Historical only: Coolify-Service worker-dispatcher mit process.env.WORKER_ROLE=dispatcher
    • Historical only: Mindestens 1 Replica (Dispatcher ist idempotent via SKIP LOCKED, 2 Replicas sind safe für HA)
    • Historical receiver sketch: domain receivers such as mail, webhook, notification, audit-log, and workflow should run on the current worker groups unless a future dedicated receiver group is added. The current Coolify topology is worker-general, worker-mail, worker-import, worker-webhook, and worker-plugin.
  5. Env-Vars:
    OUTBOX_DISPATCHER_ENABLED=true
    OUTBOX_DISPATCH_INTERVAL_MS=1000
    OUTBOX_DISPATCH_BATCH_SIZE=100
    OUTBOX_MAX_RETRIES=6
    OUTBOX_DONE_RETENTION_HOURS=720
    
  6. Smoke-Tests:
    • Stripe-Webhook-E2E:
      1. Stripe-CLI: stripe trigger customer.subscription.updated
      2. Innerhalb 2s: SELECT * FROM "OutboxEvent" WHERE "eventType" = 'tenant.subscription.updated' ORDER BY "createdAt" DESC LIMIT 1 → erwarte 1 Row, status PENDING oder DONE
      3. Nach 5s: Row sollte DONE sein
      4. Mail-Worker-Log zeigt: Mail an Tenant-Admin versendet
    • Crash-Recovery-Test:
      1. Stripe-CLI: trigger Event
      2. Sofort docker kill worker-mail (während Event noch PENDING)
      3. OutboxEvent bleibt PENDING (oder geht auf FAILED nach Retries — je nach Timing)
      4. docker compose up worker-mail neu starten
      5. Innerhalb 30s: Event wird verarbeitet, Mail gesendet
    • Concurrency-Test:
      1. 1000 Events via Script publishen
      2. Historical only: docker scale worker-dispatcher=3
      3. Verifiziere: jedes Event genau 1x in Receiver-Logs (über Event-ID grep)
    • Dead-Letter-Test:
      1. Mail-Mock so konfigurieren dass es immer wirft
      2. Event triggern, beobachten dass es nach 8 Retries auf FAILED geht
      3. Admin-UI /admin/outbox zeigt Event in Failed-Liste
      4. CLI: toolctrl-cli outbox requeue --event-id <id> → Event geht zurück auf PENDING
  7. Monitoring-Setup:
    • Grafana-Panels:
      • „Outbox Pending Count" (Alert: > 1000 für > 5min)
      • „Outbox Failed Count Daily" (Alert: > 50/day)
      • „Dispatcher Throughput" (events/sec)
      • „Receiver Latency P95" pro Queue
  8. 24h-Beobachtungs-Phase:
    • Logs auf Sentry-Errors prüfen
    • SELECT status, COUNT(*) FROM "OutboxEvent" GROUP BY status regelmäßig
    • Acceptance: 0 unverarbeitete Events > 60s alt im 24h-Fenster

Deviations from Plan

  • Step 5 (Stripe webhook) — DONE in 2026-05-25 session. Wrapped subscription update + outbox publishes in a single prisma.$transaction; legacy createLegacyEvent direct call moved into the new apps/worker/src/jobs/legacy-event-receiver.ts consumer on outbox.notification, guarded by withOutboxIdempotency.
  • Step 11 (CLI re-queue) — DONE in 2026-05-25 session. toolctrl-cli outbox requeue --event-id <id> and outbox requeue --type <eventType> --since <iso> [--apply] ship under packages/cli/src/commands/outbox/. Two modes (single + bulk), dry-run by default for bulk, refuses to re-queue events already PENDING/PROCESSING, JSON output for scripting.
  • Step 13 (end-to-end integration test) — DONE in 2026-05-25 session. packages/db/src/outbox-e2e.test.ts exercises producer → claim → receiver → marker-row insert against real Postgres, with idempotency re-delivery assertion. Opt-in via OUTBOX_E2E_DATABASE_URL (or any DATABASE_URL that is not the vitest stub).
  • Step 6 (WorkflowEngine triggers) — ROUTE ROW PRODUCERS LANDED in 2026-05-26 session. apps/worker/src/jobs/workflow-trigger.ts now parses workflow.* outbox jobs, isolates redeliveries with withOutboxIdempotency, fans out to live event-trigger workflows, applies preflight, and starts executions through an injected executor port. apps/worker/src/jobs/workflow-trigger-store.ts adds the worker-native Prisma lookup/preflight store without importing web server modules, and bootstrapWorkflowTrigger is registered from apps/worker/src/index.ts for the general worker. Route-backed row.created, row.updated, and row.deleted producers publish workflow.trigger.event inside the same DB transaction as the row write and application event persistence when a request context is passed. Internal requestless mutations remain eventless by default to avoid worker feedback loops, while the direct internal trigger bridge remains the only allowed type: "trigger" execution call-site.
  • Steps 7, 8, 9 (generic legacy mail/webhook/notification/audit-log from web actions) — DEFERRED. Form-submission delivery is already outbox-backed: the general worker routes form.submission.* work and worker-mail bridges webhook.form-submission.deliver / mail.transactional.form-submission through the current outbox.webhook and outbox.mail subscribers. The remaining deferred work is the broader legacy action inventory outside that reviewed form-submission path. Reason: each generic migration touches a different module (apps/web/app/routes/**/action.tsx) and a different receiver worker pattern, and the operator scheduled them as separate per-domain code-review cycles. The foundation makes each one mechanical: wrap the action's DB write in prisma.$transaction, replace the direct side-effect call with publishOutboxEvent(tx, { type: '<domain>.<entity>.<verb>', payload }), and add a queue subscriber that mirrors legacy-event-receiver.ts (already merged this session). Acceptance criterion per migration: at least one Playwright assertion that crashing the worker between DB commit and queue-pop still delivers the side effect on restart.
  • Step 10 (admin UI under /app/admin/outbox) — DEFERRED. Reason: design decision (filters, detail view layout, re-queue confirm dialog) — needs UX input the operator hasn't provided yet. The CLI (toolctrl-cli outbox requeue) covers the operational hot-path until the UI lands. Acceptance criterion: list view with Status/Type/Tenant filters + pagination, detail view with payload + retry/cancel buttons, admin-only RBAC.
  • The dispatcher uses an in-process setTimeout-based polling loop rather than a BullMQ-repeatable cron job. This avoids needing a BullMQ Queue just to schedule the dispatcher itself, keeps the dispatcher independently testable with vi.useFakeTimers(), and matches the way other internal pollers (metrics-scheduler) are wired.
  • Default routing maps billing.* and tenant.subscription.* to outbox.notification. The Stripe-webhook migration uses billing.subscription.ended + billing.subscription.cancelled which land on outbox.notification via the existing billing. prefix rule — no router changes were needed.

Progress Log

  • 2026-05-17 08:25 — 6ac38a11 — Add OutboxEvent prisma model + migration + regenerated client
  • 2026-05-17 08:30 — 62fa3436 — Add publishOutboxEvent publisher helper with validation + dispatchKey idempotency
  • 2026-05-17 08:36 — 1427a4ef — Add atomic claim-batch query (FOR UPDATE SKIP LOCKED) with integration tests
  • 2026-05-17 08:41 — 386dc9fc — Add dispatcher worker + event-router with exponential-backoff retry and dead-letter
  • 2026-05-17 08:49 — 5a19fdf3 — Register dispatcher in worker bootstrap (gated by OUTBOX_DISPATCHER_ENABLED)
  • 2026-05-17 08:52 — 9a79b212 — Add withOutboxIdempotency receiver helper for at-least-once safety
  • 2026-05-17 08:55 — 41c277ab — Document outbox pattern in docs/architecture/outbox-pattern.md + update progress logs
  • 2026-05-17 08:57 — 5a57e18a — Catalog OUTBOX_DISPATCHER_ENABLED / BATCH_SIZE / INTERVAL_MS / MAX_RETRIES and add to .env.example
  • 2026-05-25 — 0bebef90 — feat(outbox): migrate stripe webhook to outbox pattern (DB update + billing.subscription.{ended,cancelled} publish in single prisma.$transaction, new legacy-event-receiver consumer on outbox.notification guarded by withOutboxIdempotency, bootstrap wired into worker general-group with shutdown integration, stripe-webhook + receiver unit tests cover the new transactional + idempotent paths)
  • 2026-05-25 — 9e40294f — feat(cli): add outbox requeue command for failed and bulk replay (--event-id single mode, --type --since bulk mode with --apply confirm flag, refuses to re-queue PENDING/PROCESSING, JSON output, 15 unit tests)
  • 2026-05-25 — a5ac964b — test(outbox): end-to-end integration via real postgres opt-in (publisher → claim → custom routing → receiver → marker-insert → idempotent re-delivery; opt-in via OUTBOX_E2E_DATABASE_URL; suite passes live against local postgres and skips when no DB env is set)

Lessons Learned

(Optional, am Ende des Tasks für nachfolgende Tasks)