Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,445 @@
|
||||
# Outbox-Pattern + Dispatcher
|
||||
|
||||
> **Current status source:** this detail plan is historical implementation
|
||||
> context with reconciled notes. Use `docs/architecture/outbox-pattern.md`
|
||||
> and `docs/ops/worker-queues.md` for current workflow/outbox runtime truth.
|
||||
|
||||
**Status:** in_progress (foundation + Stripe-webhook migration + WorkflowEngine row-trigger producer migration + form-submission mail/webhook bridges + CLI re-queue + e2e test shipped; remaining generic legacy mail/webhook/notification/audit web-action call-site migrations + admin UI deferred)
|
||||
**Task-ID:** #42
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Aktuell sind viele Side-Effects nicht transaktional mit DB-Writes gekoppelt:
|
||||
|
||||
- **Stripe-Webhook (`apps/api/src/billing/stripe-webhook-route.ts:60-138`):** `updateTenantSubscriptionFromStripe` updated DB → ruft danach `createLegacyEvent` → direkt callLegacyEvent (Mail, Webhook, Audit-Log). Wenn der Prozess zwischen Commit und `callLegacyEvent` crasht, ist DB konsistent aber Mail wurde nie versendet
|
||||
- **WorkflowEngine (`apps/web/app/modules/workflowEngine/*`):** This is now
|
||||
migrated for the row event producer path. `workflow.*` events route to
|
||||
`outbox.workflow`, the worker-general bridge starts matching workflows
|
||||
through the signed internal web API, and `row.created`, `row.updated`, and
|
||||
`row.deleted` are published in the same transaction as their row changes.
|
||||
Remaining WorkflowEngine gaps are semantic/runtime follow-ups: native
|
||||
branch/switch execution, delay/schedule blocks, and broader non-row producer
|
||||
coverage.
|
||||
- **Notification/Mail/Webhook-Outbound** aus Web-Actions: direkt aus dem Request-Handler triggered, bei Worker-Crash verloren
|
||||
|
||||
Das Outbox-Pattern löst das durch:
|
||||
1. Side-Effects werden **innerhalb der gleichen Transaction** wie der DB-Write als „Event"-Row in eine `OutboxEvent`-Tabelle geschrieben
|
||||
2. Ein separater **Dispatcher-Worker** liest pending Events, dispatched zu der jeweiligen Queue (mail, webhook, notification, audit-log), und markiert sie als processed
|
||||
3. Failures → Retries mit Exponential-Backoff, max-Retries, dann dead-letter
|
||||
4. At-least-once-Delivery garantiert; Receiver brauchen Idempotenz (Idempotency-Store aus #38)
|
||||
|
||||
Vorteile:
|
||||
- Transaction-Konsistenz: Event existiert ↔ DB-Change committed (atomic)
|
||||
- Crash-Safety: bei Crash zwischen Commit und Side-Effect wird Event beim Recovery dispatched
|
||||
- Audit-Trail: jede ausgelöste Aktion ist in DB sichtbar
|
||||
- Decoupling: Event-Produzent muss Receiver nicht kennen
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/read-write-split.md` — `publishOutboxEvent` läuft auf `prisma.write` (Outbox-Writes sind kritische Source-of-Truth, niemals Replica)
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Dispatcher nutzt BullMQ (gleiche Redis-Connection) für Queue-Dispatch
|
||||
- `02-observability/sentry-wiring-api-worker.md` — Dispatcher-Failures müssen captured werden
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/job-priority.md` — Dispatcher nutzt BullMQ-Priority für Event-Types (z.B. high für payment-success-mail)
|
||||
- `04-search-rag/embedding-pipeline.md` — Search-Indexer hängt an Outbox-Events (entity-created, entity-updated → re-index)
|
||||
- `05-business-features/marketing-funnel-phase-1.md` — Email-Sequence-Triggers via Outbox
|
||||
- alle nachfolgenden Features mit asynchronen Side-Effects
|
||||
|
||||
## Approach
|
||||
|
||||
**Neue Prisma-Tabelle:**
|
||||
|
||||
```prisma
|
||||
model OutboxEvent {
|
||||
id String @id @default(cuid())
|
||||
tenantId String? // null = system-event
|
||||
eventType String // z.B. "tenant.subscription.updated"
|
||||
payload Json
|
||||
status OutboxStatus @default(PENDING)
|
||||
createdAt DateTime @default(now())
|
||||
processedAt DateTime?
|
||||
retries Int @default(0)
|
||||
lastError String?
|
||||
scheduledAt DateTime @default(now()) // delay-fähig
|
||||
dispatchKey String? // für Idempotency / Dedup
|
||||
priority Int @default(0) // für #47
|
||||
|
||||
@@index([status, scheduledAt])
|
||||
@@index([tenantId, eventType])
|
||||
@@index([dispatchKey])
|
||||
}
|
||||
|
||||
enum OutboxStatus {
|
||||
PENDING
|
||||
PROCESSING
|
||||
DONE
|
||||
FAILED // alle retries verbraucht, dead-letter
|
||||
CANCELLED
|
||||
}
|
||||
```
|
||||
|
||||
**Publisher-Helper:**
|
||||
|
||||
```ts
|
||||
// packages/db/src/outbox.ts
|
||||
export async function publishOutboxEvent(
|
||||
tx: Prisma.TransactionClient,
|
||||
{ tenantId, type, payload, dispatchKey, scheduledAt, priority }: PublishArgs
|
||||
): Promise<OutboxEvent>
|
||||
```
|
||||
|
||||
Wird **nur innerhalb einer Transaction** aufgerufen — Typ-Constraint verhindert direkte Nutzung gegen `prisma`-Singleton.
|
||||
|
||||
**Dispatcher:**
|
||||
|
||||
Implementierungs-Variante A (gewählt): **BullMQ-Cron-Polling**, alle 1s pollt der Dispatcher-Worker pending Events (`WHERE status = 'PENDING' AND scheduledAt <= now()` LIMIT 100, ORDER BY priority DESC, createdAt ASC), markiert sie atomic als `PROCESSING` (via `UPDATE ... WHERE status = 'PENDING' RETURNING *` für Race-Condition-Safety), dispatched zu BullMQ-Queue (Queue-Name = Event-Type-Prefix, z.B. `mail`, `webhook`, `notification`, `audit-log`).
|
||||
|
||||
Variante B (verworfen für jetzt, aber dokumentiert): **LISTEN/NOTIFY-basiert.** Postgres-Trigger published auf `outbox_event_created`-Channel, Dispatcher listenened. Vorteil: niedrige Latency (<10ms statt 0-1s Polling). Nachteil: Trigger-Maintenance, Channel-Reconnect-Komplexität. Migration auf Variante B könnte später erfolgen wenn Latenz wichtig wird.
|
||||
|
||||
**Receiver-Idempotenz:**
|
||||
|
||||
Jeder Receiver (Mail-Worker, Webhook-Worker, etc.) prüft via `IdempotencyStore` (aus #38) ob `OutboxEvent.id` bereits verarbeitet wurde. Bei Retry: skip. Garantiert exactly-once-Effect bei at-least-once-Delivery.
|
||||
|
||||
**Retry-Strategie:**
|
||||
- Initial-Try: sofort
|
||||
- Retry-1: nach 1 min (exponential 2^0)
|
||||
- Retry-2: nach 2 min
|
||||
- Retry-3: nach 4 min
|
||||
- ... bis max-Retries (Default 8, also bis ~4h)
|
||||
- Danach: `status = FAILED`, manuelle Inspection oder Bulk-Re-Queue via CLI
|
||||
|
||||
**Dead-Letter-Handling:**
|
||||
- Admin-UI-Stub in `apps/web/app/routes/admin/outbox.tsx` zeigt failed Events
|
||||
- CLI-Tool `toolctrl-cli outbox requeue --event-id <id>` setzt ein einzelnes
|
||||
Event wieder auf PENDING. Bulk-Requeues laufen über
|
||||
`toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply]`.
|
||||
|
||||
**Migration:**
|
||||
|
||||
Status reconciliation (2026-05-27): Stripe webhook, row-event
|
||||
WorkflowEngine producers, CLI re-queue, and the opt-in outbox integration test
|
||||
have shipped. Notification/Mail/Webhook-Outbound migrations from remaining web
|
||||
actions and the Admin UI remain deferred.
|
||||
|
||||
1. **Stripe-Webhook (`apps/api/src/billing/stripe-webhook-route.ts:60-138`):**
|
||||
- Transaction wrappen: `prisma.write.$transaction(async (tx) => { await updateTenantSubscriptionFromStripe(tx, ...); await publishOutboxEvent(tx, { tenantId, type: 'tenant.subscription.updated', payload: {...} }); })`
|
||||
- `callLegacyEvent` direkt-Aufruf entfernen
|
||||
- Dispatcher routet Event zu `notification`-Queue (alter callLegacyEvent-Code wird Receiver)
|
||||
|
||||
2. **WorkflowEngine (`apps/web/app/modules/workflowEngine/*`):**
|
||||
- Trigger werden zu Outbox-Events (`type: 'workflow.trigger.<eventName>'`)
|
||||
- WorkflowEngine-Worker subscribed auf entsprechende Queue, executiert Workflow
|
||||
|
||||
3. **Notification / Mail / Webhook-Outbound aus Web-Actions:**
|
||||
- Web-Actions ersetzen direkte `sendEmail(...)`-Calls durch `await publishOutboxEvent(tx, { type: 'mail.send', payload: { to, template, vars }})`
|
||||
- Dispatcher routet zu `mail`-Queue, Mail-Worker (kommt aus #45) verarbeitet
|
||||
|
||||
**Warum nicht Alternativen:**
|
||||
- ❌ Saga-Pattern: Overkill für unsere Use-Cases, zu viel Orchestration
|
||||
- ❌ Event-Sourcing Full: massive Architektur-Umstellung, nicht in Phase 03 leistbar
|
||||
- ❌ CDC (Change Data Capture) via Debezium/wal2json: Infra-Overhead, kompliziertes Setup
|
||||
- ❌ Direkter Queue-Push aus Request-Handler (kein Outbox): bricht Transaction-Konsistenz, genau das Problem das wir lösen
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Prisma-Migration: OutboxEvent-Tabelle**
|
||||
- Files: `packages/db/prisma/schema.prisma`, neue Migration `prisma/migrations/<timestamp>_add_outbox_event/migration.sql`
|
||||
- Model + Enum wie oben
|
||||
- 3 Indizes für die Dispatcher-Queries
|
||||
- `pnpm prisma migrate dev --name add_outbox_event`
|
||||
- Commit: `03-scaling/outbox: add OutboxEvent model and migration`
|
||||
|
||||
2. **publishOutboxEvent-Helper**
|
||||
- Files: `packages/db/src/outbox.ts`, `packages/db/src/outbox.test.ts`
|
||||
- Signatur akzeptiert `Prisma.TransactionClient` (typed) — Aufrufer muss innerhalb $transaction sein
|
||||
- Validierung: eventType muss `<domain>.<noun>.<verb>`-Pattern matchen (z.B. `tenant.subscription.updated`)
|
||||
- DispatchKey-Validation (wenn gesetzt: max 256 Zeichen, alphanumeric)
|
||||
- Tests: erfolgreicher Insert, Race-Condition mit dispatchKey (unique-Konflikt sauber gehandled)
|
||||
- Commit: `03-scaling/outbox: add publishOutboxEvent helper with transaction enforcement`
|
||||
|
||||
3. **Dispatcher-Worker-Implementation**
|
||||
- Files: `apps/worker/src/outbox/dispatcher.ts`, `apps/worker/src/outbox/dispatcher.test.ts`, `apps/worker/src/outbox/event-router.ts`
|
||||
- Cron-Job (BullMQ-Repeatable) alle 1s
|
||||
- Claim-Query (atomic):
|
||||
```sql
|
||||
UPDATE "OutboxEvent"
|
||||
SET status = 'PROCESSING'
|
||||
WHERE id IN (
|
||||
SELECT id FROM "OutboxEvent"
|
||||
WHERE status = 'PENDING' AND "scheduledAt" <= NOW()
|
||||
ORDER BY priority DESC, "createdAt" ASC
|
||||
LIMIT 100
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *
|
||||
```
|
||||
- Pro claimed Event: dispatch via `event-router.ts` zu der richtigen BullMQ-Queue
|
||||
- On dispatch-success: mark `DONE`, set `processedAt`
|
||||
- On dispatch-error: increment retries, set `lastError`, `scheduledAt = now + backoff(retries)`. Wenn `retries >= max`: status `FAILED`
|
||||
- Tests: claim-race-condition (zwei Dispatcher-Instances, ein Event), retry-backoff-Berechnung, dead-letter-Übergang
|
||||
- Commit: `03-scaling/outbox: add dispatcher worker with skip-locked claim`
|
||||
|
||||
4. **Event-Router: Type → Queue-Mapping**
|
||||
- Files: `apps/worker/src/outbox/event-router.ts`
|
||||
- Config-basiertes Mapping:
|
||||
```ts
|
||||
const routing = {
|
||||
'mail.': 'mail',
|
||||
'webhook.': 'webhook',
|
||||
'notification.': 'notification',
|
||||
'audit.': 'audit-log',
|
||||
'workflow.': 'workflow',
|
||||
'tenant.subscription.': 'notification', // Composite-Routing
|
||||
'search.index.': 'search-index', // für Phase 04
|
||||
};
|
||||
```
|
||||
- Unknown-Type → DLQ + Sentry-Alert
|
||||
- Tests: alle Mappings, Fallback-Verhalten
|
||||
- Commit: `03-scaling/outbox: add event router with type to queue mapping`
|
||||
|
||||
5. **Migration: Stripe-Webhook**
|
||||
- Files: `apps/api/src/billing/stripe-webhook-route.ts`, neue Tests
|
||||
- Wrappen in `prisma.write.$transaction`
|
||||
- `callLegacyEvent`-Call durch `publishOutboxEvent` ersetzen
|
||||
- Receiver-Worker (`apps/worker/src/notification/legacy-event.worker.ts`) baut neuen Worker der den alten `callLegacyEvent`-Code als BullMQ-Handler ausführt
|
||||
- Tests: Stripe-Event-Mock → DB-Update + OutboxEvent existiert in derselben Transaction; bei Worker-Crash zwischen Steps → Outbox bleibt PENDING, beim nächsten Dispatch-Tick wird gepickt
|
||||
- Commit: `03-scaling/outbox: migrate stripe webhook to outbox pattern`
|
||||
|
||||
6. **Migration: WorkflowEngine-Triggers**
|
||||
- Files: `apps/web/app/modules/workflowEngine/routes/workflow-engine.api.server.tsx`, weitere Trigger-Stellen, neuer Worker `apps/worker/src/workflow/trigger.worker.ts`
|
||||
- Workflow-Trigger publishen Outbox-Event mit eventType `workflow.trigger.<eventName>`, payload enthält trigger-context
|
||||
- Worker konsumiert, lädt Workflow-Definition, executiert via existing Engine-Code
|
||||
- Tests: Trigger-Publish + Worker-Pickup; failed-Workflow markiert Event als FAILED nach max-retries
|
||||
- Commit: `03-scaling/outbox: migrate workflow engine triggers to outbox`
|
||||
|
||||
7. **Migration: Mail-Sending aus Web-Actions**
|
||||
- Files: alle `apps/web/app/routes/**/action.tsx`-Stellen die `sendEmail` direkt aufrufen, neuer Worker `apps/worker/src/mail/mail.worker.ts`
|
||||
- Inventarisierung via Subagent (Explore-Pass): liste alle `sendEmail`-Calls in Web
|
||||
- Pro Stelle: in Transaction wrappen + `publishOutboxEvent({ type: 'mail.send', payload: { to, template, vars }})` statt direkter Call
|
||||
- Mail-Worker: konsumiert `mail`-Queue, ruft Mail-Service (aus `packages/mail`)
|
||||
- Tests: Web-Action publisht Event, Worker verarbeitet, Mail wird gesendet (Mock-Mail-Provider)
|
||||
- Commit: `03-scaling/outbox: migrate web action mail sends to outbox`
|
||||
|
||||
8. **Migration: Webhook-Outbound aus Web-Actions**
|
||||
- Files: analog zu #7, aber für `dispatchWebhook`-Calls
|
||||
- Commit: `03-scaling/outbox: migrate web action webhook dispatches to outbox`
|
||||
|
||||
9. **Migration: Notification + Audit-Log aus Web-Actions**
|
||||
- Files: analog für `createNotification` und `auditLog`-Stellen
|
||||
- Commit: `03-scaling/outbox: migrate notification and audit log writes to outbox`
|
||||
|
||||
10. **Admin-UI für Outbox-Inspection**
|
||||
- Files: `apps/web/app/routes/admin/outbox.tsx`, `apps/web/app/routes/admin/outbox.$id.tsx`
|
||||
- Listing: Status-Filter, Type-Filter, Tenant-Filter, Pagination
|
||||
- Detail-View: Payload anzeigen, Retry-Manual-Button, Cancel-Button
|
||||
- Tests: Loader/Action, Berechtigungs-Check (admin-only)
|
||||
- Commit: `03-scaling/outbox: add admin ui for outbox event inspection`
|
||||
|
||||
11. **CLI-Tool für Re-Queue**
|
||||
- Files: `scripts/outbox/requeue.ts`, `package.json` script
|
||||
- Superseded by `toolctrl-cli outbox requeue --event-id <id>` for a single event
|
||||
- Superseded by `toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply]` for bulk
|
||||
- Tests: CLI-Smoke-Test
|
||||
- Commit: `03-scaling/outbox: add cli for requeueing failed events`
|
||||
|
||||
12. **Doku**
|
||||
- Files: `docs/architecture/outbox-pattern.md` (original plan named `docs/database/outbox-pattern.md`)
|
||||
- Sections:
|
||||
- **Konzept:** Warum Outbox? Transaction-Konsistenz, Crash-Safety, Decoupling
|
||||
- **Publisher-Usage:** Code-Beispiel für `publishOutboxEvent` in Transaction
|
||||
- **Event-Type-Konvention:** `<domain>.<noun>.<verb>`, Registry pflegen
|
||||
- **Dispatcher:** wie er pollt, claim-Strategy, Retry-Backoff, Dead-Letter
|
||||
- **Receiver-Idempotenz-Pflicht:** Code-Beispiel mit IdempotencyStore
|
||||
- **Operational:** Monitoring (Pending-Count-Alert > 1000), Admin-UI, CLI-Re-Queue
|
||||
- **Migration-Guide:** wie weitere Side-Effects migrieren
|
||||
- **Trade-offs:** Latenz (1s Polling-Tick), zukünftige LISTEN/NOTIFY-Migration
|
||||
- Commit: `03-scaling/outbox: add outbox pattern documentation`
|
||||
|
||||
13. **Integration-Test: End-to-End Outbox-Flow**
|
||||
- Files: `apps/worker/src/__integration__/outbox-e2e.test.ts`
|
||||
- testcontainers: Postgres + Redis + Mail-Mock-Service
|
||||
- Test-Cases:
|
||||
- Publisher → Dispatcher → Receiver → Effect (Mail-Mock empfängt)
|
||||
- Crash-Recovery: Receiver wirft Error → Event geht zurück auf PENDING mit retry+1
|
||||
- Dead-Letter: max-Retries → status FAILED, Admin-UI zeigt's
|
||||
- Race-Condition: 2 Dispatcher-Instances → kein Event wird doppelt dispatched
|
||||
- Idempotency: Receiver mit gleicher Event-ID 2x aufgerufen → Effekt nur 1x
|
||||
- Commit: `03-scaling/outbox: add end-to-end integration test suite`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** `publishOutboxEvent` (Transaction-Constraint, dispatchKey-Unique, Validation); Dispatcher (Claim-Logic, Retry-Backoff, Dead-Letter); Event-Router (alle Mappings)
|
||||
- **Integration-Tests:** End-to-End-Test (Step 13) — Publisher in API, Dispatcher im Worker, Receiver-Effekt verifiziert. testcontainers für Postgres + Redis
|
||||
- **Concurrency-Tests:** 2 Dispatcher-Instances parallel, 1000 Events — alle exakt 1x dispatched
|
||||
- **E2E-Tests (Playwright):** Admin-UI für Outbox-Inspection (Listing, Detail, Re-Queue-Button)
|
||||
- **Coverage-Target:** ≥ 90 % im Outbox-Code (Dispatcher, Helper, Router)
|
||||
- **Test-Files:**
|
||||
- `packages/db/src/outbox.test.ts`
|
||||
- `apps/worker/src/outbox/dispatcher.test.ts`
|
||||
- `apps/worker/src/outbox/event-router.test.ts`
|
||||
- `apps/worker/src/__integration__/outbox-e2e.test.ts`
|
||||
- `apps/web/app/routes/admin/outbox.test.tsx`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- Alle Tests grün, inkl. Concurrency + E2E
|
||||
- Staging-Verifikation: 24h-Lauf zeigt 0 unverarbeitete Events > 60s alt (Monitoring-Alert)
|
||||
- Stripe-Webhook-Test auf Staging: Crash zwischen DB-Commit und Mail-Send → nach Worker-Restart wird Mail gesendet
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `OutboxEvent`-Model in Prisma-Schema + Migration applied
|
||||
- [ ] `publishOutboxEvent`-Helper in `packages/db/src/outbox.ts`, akzeptiert nur TransactionClient
|
||||
- [ ] Dispatcher-Worker im `apps/worker` mit Cron-Polling (1s), atomic Claim via `FOR UPDATE SKIP LOCKED`
|
||||
- [ ] Event-Router mapped Event-Types zu BullMQ-Queues
|
||||
- [ ] Retry-Logic mit Exponential-Backoff (max 8 Retries), Dead-Letter bei Erschöpfung
|
||||
- [x] Stripe-Webhook migriert: DB-Update + Outbox-Event in einer Transaction; Worker-Crash zwischen Commit + Dispatch wird beim Recovery aufgeholt
|
||||
- [x] WorkflowEngine row-event triggers migriert; non-row producers remain follow-ups
|
||||
- [ ] Mail/Webhook/Notification/Audit-Log aus Web-Actions migriert
|
||||
- [ ] Admin-UI unter `/admin/outbox` listed pending/failed Events, erlaubt Re-Queue
|
||||
- [x] CLI `toolctrl-cli outbox requeue` funktioniert (single + bulk)
|
||||
- [x] `docs/architecture/outbox-pattern.md` enthält die current runtime/operator sections; the original `docs/database/outbox-pattern.md` target was superseded
|
||||
- [x] Opt-in integration test grün; Concurrency-Test (2 Dispatcher) bleibt offen
|
||||
- [ ] Staging-24h-Test: 0 unverarbeitete Events > 60s alt
|
||||
- [ ] Grafana-Panel: Outbox-Pending-Count, Dispatch-Rate, Retry-Rate, Failed-Count
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
**Per-Migration-Rollback:** Jede Migration (Steps 5-9) ist separat rollback-bar via `git revert`. Code-seitig fällt zurück auf direkte Side-Effect-Calls; OutboxEvent-Rows bleiben in DB (kein Daten-Verlust), werden ignoriert.
|
||||
|
||||
**Dispatcher-Disable:**
|
||||
- Env-Var `OUTBOX_DISPATCHER_ENABLED=false` im Worker → Cron stoppt, Events stauen sich
|
||||
- Useful für Maintenance oder Failure-Investigation
|
||||
- Pending-Count-Alert in Grafana fängt das ab
|
||||
|
||||
**Schema-Rollback:**
|
||||
- `prisma migrate resolve --rolled-back <migration>` + manuelle `DROP TABLE OutboxEvent`
|
||||
- Daten-Verlust für FAILED/PENDING Events — vor Rollback CSV-Export via `\COPY "OutboxEvent" TO ...`
|
||||
- Code muss vor Rollback zurück auf direct-Side-Effect-Calls (sonst breakt publishOutboxEvent)
|
||||
|
||||
**Receiver-Rollback einzeln:**
|
||||
- Wenn z.B. Mail-Worker buggy ist → BullMQ-Queue `mail` pausieren (`bull-board`-UI oder CLI), Events stauen sich harmlos, Fix deployen, Queue resumen
|
||||
|
||||
**DB-Migration:** non-destructive (nur CREATE TABLE + CREATE INDEX). Down-Migration: `DROP TABLE OutboxEvent CASCADE` (destructive — Daten-Verlust). Vor Down-Migration Backup ausführen.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Tenant-Isolation:** OutboxEvent hat `tenantId`. Receiver MUSS prüfen dass Payload-Operationen den richtigen Tenant treffen. Cross-Tenant-Spoofing-Risiko wenn Receiver Payload blind vertraut → Workflow-Worker validiert `payload.tenantId === event.tenantId`
|
||||
- **Payload-Validation:** Payloads sind `Json` — Receiver muss Zod-Schema-Validation pro Event-Type machen, sonst Crash bei Malformed
|
||||
- **Sensitive-Data im Payload:** keine Passwords, keine PII-Mehr-als-nötig. Email-Templates referenzieren statt embed
|
||||
- **Replay-Attacks:** dispatchKey + Idempotency-Store schützen. Receiver MUSS Idempotency-Check machen
|
||||
- **Dead-Letter-Access:** Admin-UI darf nur Admins lesen (RBAC-Check). Failed-Events können sensitive Daten enthalten (z.B. Stripe-Payload)
|
||||
- **Re-Queue-Berechtigung:** CLI + UI-Re-Queue nur für Admins. Manuelle Re-Queue eines Events kann unbeabsichtigte Effekte triggern (z.B. doppelte Mail) — Admin-UI zeigt Warnung
|
||||
- **DoS via Outbox-Spam:** Wenn ein bug 1000s Events pro Sekunde publisht → Dispatcher kommt nicht hinterher. Rate-Limit auf Publisher-Seite (max 100 Events pro Tenant pro Minute via SharedStore-Rate-Limiter)
|
||||
- **Backup-Implications:** OutboxEvent-Tabelle in Backup einbeziehen (sind kritische Source-of-Truth)
|
||||
- **OutboxEvent-Retention:** completed Events nach 30 Tagen löschen (Cron-Job), sonst Tabellen-Bloat. Failed-Events behalten (Compliance)
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit `apps/api/src/billing/stripe-webhook-route.ts:60-138` als Hauptbeispiel + Outbox-Pattern-Reference-Artikel
|
||||
- **Research:**
|
||||
- `Explore`-Subagent für „inventarisiere alle `sendEmail`, `dispatchWebhook`, `createNotification`, `auditLog`-Calls in `apps/web/`. Output: Migration-Backlog für Steps 7-9"
|
||||
- `Explore`-Subagent für „suche bestehende BullMQ-Queue-Setups in `apps/worker/` — was kann wiederverwendet werden?"
|
||||
- **Implementation:**
|
||||
- `general-purpose` für Steps 1-4 (Foundation: Schema + Helper + Dispatcher + Router)
|
||||
- Eigener `general-purpose` pro Migration-Step (5-9) — pro Bereich ein Subagent, weil Receiver-Worker-Code-Pattern jeweils unterschiedlich
|
||||
- `general-purpose` für Step 10 (Admin-UI)
|
||||
- `general-purpose` für Step 11 (CLI)
|
||||
- `general-purpose` für Step 12 (Doku — sehr ausführlich)
|
||||
- **Tests:** Separater `general-purpose` für E2E-Integration-Test (Step 13) — kompliziertes testcontainers-Setup, eigener Kontext
|
||||
- **Verification:** `gsd-verifier` für Acceptance-Criteria; manuelles 24h-Staging-Monitoring
|
||||
- **Code-Review:** `gsd-code-reviewer` für jeden Migration-Step separat (Stripe vs WorkflowEngine vs Mail haben unterschiedliche Risiken)
|
||||
|
||||
## Historical Staging-Deployment-Steps
|
||||
|
||||
This section is retained as the original implementation sketch. The current
|
||||
production topology uses separate persistent worker applications (`general`,
|
||||
`mail`, `import`, `webhook`, `plugin`) and `worker-general` owns the outbox
|
||||
dispatcher; do not provision the old `worker-dispatcher`/`WORKER_ROLE` layout.
|
||||
|
||||
1. Branch `agent/03-outbox-pattern` pushen, schrittweise Commits 1-13
|
||||
2. Coolify-Setup-Prerequisites: PgBouncer (#40), R/W-Split (#41), Shared-Store (#37/#38) müssen schon laufen
|
||||
3. Migration:
|
||||
- Pre-Deploy-Check: `pnpm prisma migrate status` auf Staging
|
||||
- Deploy mit `prisma migrate deploy` (Coolify-Pre-Start-Hook)
|
||||
- OutboxEvent-Tabelle erstellt, leer
|
||||
4. Worker-Replicas:
|
||||
- Historical only: Dispatcher-Worker als **separate Replica** vom General-Worker (skaliert unabhängig)
|
||||
- Historical only: Coolify-Service `worker-dispatcher` mit `process.env.WORKER_ROLE=dispatcher`
|
||||
- Historical only: Mindestens 1 Replica (Dispatcher ist idempotent via SKIP LOCKED, 2 Replicas sind safe für HA)
|
||||
- Historical receiver sketch: domain receivers such as mail, webhook, notification, audit-log, and workflow should run on the current worker groups unless a future dedicated receiver group is added. The current Coolify topology is `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin`.
|
||||
5. Env-Vars:
|
||||
```
|
||||
OUTBOX_DISPATCHER_ENABLED=true
|
||||
OUTBOX_DISPATCH_INTERVAL_MS=1000
|
||||
OUTBOX_DISPATCH_BATCH_SIZE=100
|
||||
OUTBOX_MAX_RETRIES=6
|
||||
OUTBOX_DONE_RETENTION_HOURS=720
|
||||
```
|
||||
6. Smoke-Tests:
|
||||
- **Stripe-Webhook-E2E:**
|
||||
1. Stripe-CLI: `stripe trigger customer.subscription.updated`
|
||||
2. Innerhalb 2s: `SELECT * FROM "OutboxEvent" WHERE "eventType" = 'tenant.subscription.updated' ORDER BY "createdAt" DESC LIMIT 1` → erwarte 1 Row, status PENDING oder DONE
|
||||
3. Nach 5s: Row sollte DONE sein
|
||||
4. Mail-Worker-Log zeigt: Mail an Tenant-Admin versendet
|
||||
- **Crash-Recovery-Test:**
|
||||
1. Stripe-CLI: trigger Event
|
||||
2. Sofort `docker kill worker-mail` (während Event noch PENDING)
|
||||
3. OutboxEvent bleibt PENDING (oder geht auf FAILED nach Retries — je nach Timing)
|
||||
4. `docker compose up worker-mail` neu starten
|
||||
5. Innerhalb 30s: Event wird verarbeitet, Mail gesendet
|
||||
- **Concurrency-Test:**
|
||||
1. 1000 Events via Script publishen
|
||||
2. Historical only: `docker scale worker-dispatcher=3`
|
||||
3. Verifiziere: jedes Event genau 1x in Receiver-Logs (über Event-ID grep)
|
||||
- **Dead-Letter-Test:**
|
||||
1. Mail-Mock so konfigurieren dass es immer wirft
|
||||
2. Event triggern, beobachten dass es nach 8 Retries auf FAILED geht
|
||||
3. Admin-UI `/admin/outbox` zeigt Event in Failed-Liste
|
||||
4. CLI: `toolctrl-cli outbox requeue --event-id <id>` → Event geht zurück auf PENDING
|
||||
7. Monitoring-Setup:
|
||||
- Grafana-Panels:
|
||||
- „Outbox Pending Count" (Alert: > 1000 für > 5min)
|
||||
- „Outbox Failed Count Daily" (Alert: > 50/day)
|
||||
- „Dispatcher Throughput" (events/sec)
|
||||
- „Receiver Latency P95" pro Queue
|
||||
8. 24h-Beobachtungs-Phase:
|
||||
- Logs auf Sentry-Errors prüfen
|
||||
- `SELECT status, COUNT(*) FROM "OutboxEvent" GROUP BY status` regelmäßig
|
||||
- Acceptance: 0 unverarbeitete Events > 60s alt im 24h-Fenster
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Step 5 (Stripe webhook) — DONE in 2026-05-25 session.** Wrapped subscription update + outbox publishes in a single `prisma.$transaction`; legacy `createLegacyEvent` direct call moved into the new `apps/worker/src/jobs/legacy-event-receiver.ts` consumer on `outbox.notification`, guarded by `withOutboxIdempotency`.
|
||||
- **Step 11 (CLI re-queue) — DONE in 2026-05-25 session.** `toolctrl-cli outbox requeue --event-id <id>` and `outbox requeue --type <eventType> --since <iso> [--apply]` ship under `packages/cli/src/commands/outbox/`. Two modes (single + bulk), dry-run by default for bulk, refuses to re-queue events already PENDING/PROCESSING, JSON output for scripting.
|
||||
- **Step 13 (end-to-end integration test) — DONE in 2026-05-25 session.** `packages/db/src/outbox-e2e.test.ts` exercises producer → claim → receiver → marker-row insert against real Postgres, with idempotency re-delivery assertion. Opt-in via `OUTBOX_E2E_DATABASE_URL` (or any `DATABASE_URL` that is not the vitest stub).
|
||||
- **Step 6 (WorkflowEngine triggers) — ROUTE ROW PRODUCERS LANDED in 2026-05-26 session.** `apps/worker/src/jobs/workflow-trigger.ts` now parses `workflow.*` outbox jobs, isolates redeliveries with `withOutboxIdempotency`, fans out to live event-trigger workflows, applies preflight, and starts executions through an injected executor port. `apps/worker/src/jobs/workflow-trigger-store.ts` adds the worker-native Prisma lookup/preflight store without importing web server modules, and `bootstrapWorkflowTrigger` is registered from `apps/worker/src/index.ts` for the `general` worker. Route-backed `row.created`, `row.updated`, and `row.deleted` producers publish `workflow.trigger.event` inside the same DB transaction as the row write and application event persistence when a request context is passed. Internal requestless mutations remain eventless by default to avoid worker feedback loops, while the direct internal trigger bridge remains the only allowed `type: "trigger"` execution call-site.
|
||||
- **Steps 7, 8, 9 (generic legacy mail/webhook/notification/audit-log from web actions) — DEFERRED.** Form-submission delivery is already outbox-backed: the general worker routes `form.submission.*` work and `worker-mail` bridges `webhook.form-submission.deliver` / `mail.transactional.form-submission` through the current `outbox.webhook` and `outbox.mail` subscribers. The remaining deferred work is the broader legacy action inventory outside that reviewed form-submission path. Reason: each generic migration touches a different module (`apps/web/app/routes/**/action.tsx`) and a different receiver worker pattern, and the operator scheduled them as separate per-domain code-review cycles. The foundation makes each one mechanical: wrap the action's DB write in `prisma.$transaction`, replace the direct side-effect call with `publishOutboxEvent(tx, { type: '<domain>.<entity>.<verb>', payload })`, and add a queue subscriber that mirrors `legacy-event-receiver.ts` (already merged this session). Acceptance criterion per migration: at least one Playwright assertion that crashing the worker between DB commit and queue-pop still delivers the side effect on restart.
|
||||
- **Step 10 (admin UI under `/app/admin/outbox`) — DEFERRED.** Reason: design decision (filters, detail view layout, re-queue confirm dialog) — needs UX input the operator hasn't provided yet. The CLI (`toolctrl-cli outbox requeue`) covers the operational hot-path until the UI lands. Acceptance criterion: list view with Status/Type/Tenant filters + pagination, detail view with payload + retry/cancel buttons, admin-only RBAC.
|
||||
- The dispatcher uses an in-process `setTimeout`-based polling loop rather than a BullMQ-repeatable cron job. This avoids needing a BullMQ Queue just to schedule the dispatcher itself, keeps the dispatcher independently testable with `vi.useFakeTimers()`, and matches the way other internal pollers (metrics-scheduler) are wired.
|
||||
- Default routing maps `billing.*` and `tenant.subscription.*` to `outbox.notification`. The Stripe-webhook migration uses `billing.subscription.ended` + `billing.subscription.cancelled` which land on `outbox.notification` via the existing `billing.` prefix rule — no router changes were needed.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 08:25 — 6ac38a11 — Add OutboxEvent prisma model + migration + regenerated client
|
||||
- 2026-05-17 08:30 — 62fa3436 — Add publishOutboxEvent publisher helper with validation + dispatchKey idempotency
|
||||
- 2026-05-17 08:36 — 1427a4ef — Add atomic claim-batch query (FOR UPDATE SKIP LOCKED) with integration tests
|
||||
- 2026-05-17 08:41 — 386dc9fc — Add dispatcher worker + event-router with exponential-backoff retry and dead-letter
|
||||
- 2026-05-17 08:49 — 5a19fdf3 — Register dispatcher in worker bootstrap (gated by OUTBOX_DISPATCHER_ENABLED)
|
||||
- 2026-05-17 08:52 — 9a79b212 — Add withOutboxIdempotency receiver helper for at-least-once safety
|
||||
- 2026-05-17 08:55 — 41c277ab — Document outbox pattern in docs/architecture/outbox-pattern.md + update progress logs
|
||||
- 2026-05-17 08:57 — 5a57e18a — Catalog OUTBOX_DISPATCHER_ENABLED / BATCH_SIZE / INTERVAL_MS / MAX_RETRIES and add to .env.example
|
||||
- 2026-05-25 — 0bebef90 — feat(outbox): migrate stripe webhook to outbox pattern (DB update + billing.subscription.{ended,cancelled} publish in single prisma.$transaction, new legacy-event-receiver consumer on outbox.notification guarded by withOutboxIdempotency, bootstrap wired into worker general-group with shutdown integration, stripe-webhook + receiver unit tests cover the new transactional + idempotent paths)
|
||||
- 2026-05-25 — 9e40294f — feat(cli): add outbox requeue command for failed and bulk replay (--event-id single mode, --type --since bulk mode with --apply confirm flag, refuses to re-queue PENDING/PROCESSING, JSON output, 15 unit tests)
|
||||
- 2026-05-25 — a5ac964b — test(outbox): end-to-end integration via real postgres opt-in (publisher → claim → custom routing → receiver → marker-insert → idempotent re-delivery; opt-in via OUTBOX_E2E_DATABASE_URL; suite passes live against local postgres and skips when no DB env is set)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
_(Optional, am Ende des Tasks für nachfolgende Tasks)_
|
||||
Reference in new issue
Block a user