Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,122 @@
|
||||
# Phase 03 — Skalierungs-Infrastruktur
|
||||
|
||||
> **Historical phase-plan status:** this README preserves the original Phase
|
||||
> 03 dependency map and done definition. Current runtime/operator truth lives
|
||||
> in `docs/improvement-plan/PROGRESS.md`,
|
||||
> `docs/improvement-plan/OPERATOR-CHECKLIST.md`,
|
||||
> `docs/ops/worker-queues.md`, `docs/architecture/outbox-pattern.md`, and
|
||||
> `docs/ops/asset-cdn.md`. In particular, #51/A7/B2 CDN rollout is currently
|
||||
> parked in backlog, not an active Phase 03 closeout item.
|
||||
|
||||
**Dauer:** 4-5 Wochen
|
||||
**Tasks:** 12 (dieses README dokumentiert alle 12; die State-/DB-Teilmenge 37-42 ist hier ausführlich, Worker/Backup/CDN 43-47/51 in separaten Detail-MDs vom Parallel-Subagent)
|
||||
**Ziel:** Boilerplate von „Single-Process-MVP" auf horizontal skalierbare Production-Architektur bringen. Alle In-Process-State-Stores raus, DB-Stack härten (PgBouncer + Read-Replica + Outbox), Worker-Queues sauber trennen, Backups + Tenant-Snapshots automatisieren, statische Assets über CDN ausliefern.
|
||||
|
||||
## Warum diese Phase
|
||||
|
||||
Aktueller Zustand: Mehrere kritische Code-Pfade halten Zustand im Prozess-RAM (`apps/api/src/auth/login-rate-limit.ts`, `apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts`, `apps/api/src/idempotency/store.ts`, `apps/web/app/utils/services/.server/subscriptionService.ts` mit `cachified`-Wrapping, `apps/api/src/auth/session/session-store.ts`, `apps/web/app/modules/mcp/mcp.server.ts`). Sobald eine zweite Replica läuft, sind Rate-Limits umgehbar, Idempotency-Keys werden inkonsistent, Stripe-Webhook-Cache-Invalidations erreichen nur eine Instanz und Sessions ghosten.
|
||||
|
||||
DB-seitig hängen 100+ Prisma-Calls direkt am Primary, ohne Connection-Pooler. Bei 200 gleichzeitigen Web-Requests reicht ein Tab voll schwerer Dashboard-Loaders, um Postgres-`max_connections` auszuschöpfen. Außerdem laufen alle Reads gegen den Writer, und Stripe-Webhook-Side-Effects (Mail, Webhook, Audit-Log) sind nicht transaktional mit dem DB-Write gekoppelt — bei Crash zwischen Commit und Side-Effect verlieren wir Outbound-Events.
|
||||
|
||||
Phase 03 baut die fehlende Plumbing für Multi-Replica-Betrieb. Ohne diese Phase blockieren wir Phase 04 (Search braucht Outbox für async-Indexing) und Phase 05 (Feature-Flags brauchen einen Shared-Cache mit Pub/Sub-Invalidation).
|
||||
|
||||
## Task-Reihenfolge
|
||||
|
||||
| Reihenfolge | Task-ID | Datei | Parallelisierbar mit | Subagent-Slot |
|
||||
|---|---|---|---|---|
|
||||
| 1 | #37 | `shared-store-package.md` | — (Foundation für alles andere) | State-Subagent |
|
||||
| 2 | #38 | `shared-store-migration.md` | #40, #43, #51 | State-Subagent |
|
||||
| 3 | #39 | `pubsub-cache-invalidation.md` | #41, #44, #45 | State-Subagent |
|
||||
| 4 | #40 | `pgbouncer.md` | #38, #43, #45 | State-Subagent (DB-Teil) |
|
||||
| 5 | #41 | `read-write-split.md` | #39, #44, #46, #51 | State-Subagent (DB-Teil) |
|
||||
| 6 | #42 | `outbox-pattern.md` | #45, #47 | State-Subagent (DB-Teil) |
|
||||
| 7 | #45 | `worker-queues.md` | #39, #40, #41 | Worker-Subagent |
|
||||
| 8 | #46 | `streaming-exports.md` | #41 | Worker-Subagent |
|
||||
| 9 | #47 | `job-priority.md` | #42 | Worker-Subagent |
|
||||
| 10 | #43 | `backup-cli.md` | #38, #40 | Worker-Subagent |
|
||||
| 11 | #44 | `tenant-snapshot.md` | #39, #41, #43 | Worker-Subagent |
|
||||
| 12 | #51 | `asset-cdn.md` | #38, #41 | Worker-Subagent; currently backlog/operator-deferred |
|
||||
|
||||
## Dependency-Graph (intern)
|
||||
|
||||
```
|
||||
#37 shared-store-package ──► #38 shared-store-migration ──► #39 pubsub-cache-invalidation ─┐
|
||||
│ │
|
||||
└─────────────────► (nutzt Redis-Connection auch für BullMQ-Queues) │
|
||||
│
|
||||
#40 pgbouncer ──► #41 read-write-split ──► #42 outbox-pattern ──────────────────────────────┤
|
||||
│ │
|
||||
▼ │
|
||||
#47 job-priority │
|
||||
│
|
||||
#45 worker-queues ──► #46 streaming-exports │
|
||||
│
|
||||
#43 backup-cli ──► #44 tenant-snapshot │
|
||||
│
|
||||
#51 asset-cdn (parked/backlog) ──────────────────────────────────────────────────────────────┤
|
||||
│
|
||||
▼
|
||||
Phase 04 (Search/RAG)
|
||||
```
|
||||
|
||||
**Wichtige Reihenfolge-Regeln:**
|
||||
- `#37 shared-store-package` ist **hartes Fundament**: liefert das Adapter-Interface, die Redis-/Memory-Implementations und die Higher-Level-Helpers (Rate-Limit, Cache, Session, Idempotency). Ohne dieses Package kann weder #38 noch #39 starten, und die Worker-Tasks (BullMQ teilt sich die Redis-Connection) werden auch davon abhängen.
|
||||
- `#38 shared-store-migration` migriert die 6 In-Process-Stores. Muss nach #37, parallel zu #40 möglich.
|
||||
- `#39 pubsub-cache-invalidation` ist die logische Fortsetzung der Migration für Caches mit nicht-trivialer Konsistenz (Stripe-Subscription-Cache, Tenant-Cache, Feature-Flag-Cache). Setzt #38 voraus, weil dort der Pub/Sub-Adapter etabliert wird.
|
||||
- `#40 pgbouncer` ist unabhängig von Shared-Store, muss aber vor `#41 read-write-split` stehen, weil R/W-Split die `DIRECT_URL`-Konvention für Migrations braucht, die in #40 etabliert wird.
|
||||
- `#42 outbox-pattern` braucht `#41` (publishOutboxEvent läuft innerhalb einer Transaction auf prismaWrite), und liefert die Foundation für `#47 job-priority` (priorisierte Dispatcher-Queue) sowie Phase 04 (async-Indexing).
|
||||
- `#45 worker-queues` und `#43 backup-cli` sind weitgehend orthogonal und können parallel laufen, sobald #37 fertig ist. `#51 asset-cdn` war ebenfalls orthogonal, ist aber seit der Operator-Entscheidung 2026-05-24 in A7/B2/#51 geparkt; nicht als aktiven Phase-03-Blocker zählen.
|
||||
|
||||
## Sub-Phasen-Verteilung
|
||||
|
||||
**State-Subagent (dieses README behandelt 37-42 ausführlich):**
|
||||
- #37 Shared-Store Package
|
||||
- #38 Shared-Store Migration
|
||||
- #39 Pub/Sub Cache-Invalidation
|
||||
- #40 PgBouncer Setup
|
||||
- #41 R/W-Split via Prisma $extends
|
||||
- #42 Outbox-Pattern + Dispatcher
|
||||
|
||||
**Worker-Subagent (separate Detail-MDs, eigene Verantwortung):**
|
||||
- #45 Worker-Queue-Trennung
|
||||
- #46 Streaming-Exports + Row-Cap
|
||||
- #47 BullMQ Job-Priority
|
||||
- #43 Backup-CLI
|
||||
- #44 Tenant-Snapshot-Tool
|
||||
- #51 Asset-CDN (historisch geplant; aktueller Status: backlog/operator-deferred)
|
||||
|
||||
Beide Subagents synchronisieren auf der Redis-Connection-Konvention aus #37 (gleiche ioredis-Instanz, unterschiedliche logical-DB-Indizes; BullMQ-Queues auf Index 0, Shared-Store auf Index 1, Pub/Sub auf Index 2).
|
||||
|
||||
## Phase Done-Definition
|
||||
|
||||
Diese Phase ist `completed` wenn:
|
||||
|
||||
- [ ] Alle 12 Tasks `completed` in `PROGRESS.md`
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` global grün; testcontainers-basierte Integration-Tests laufen in CI gegen Redis + Postgres + PgBouncer
|
||||
- [ ] Staging deployed mit 2x Web-Replicas + 1x API + 2x Worker hinter Caddy-Reverse-Proxy; Last-Test (k6 mit 500 RPS auf 5min) zeigt: keine Login-Rate-Limit-Bypasses, keine Stripe-Cache-Inconsistencies, kein Connection-Exhaustion auf Postgres
|
||||
- [ ] Stripe-Webhook → Subscription-Cache-Invalidation < 2s end-to-end (vorher: bis zu 1h stale)
|
||||
- [ ] PgBouncer-`SHOW STATS` zeigt `cl_active < default_pool_size` unter normaler Last; Connection-Reuse-Ratio > 90 %
|
||||
- [ ] Read-Replica führt > 60 % der Dashboard-Reads aus (gemessen via Prisma-Middleware-Counter)
|
||||
- [ ] Outbox-Dispatcher hat 0 unverarbeitete Events > 60s alt im 24h-Monitor-Fenster
|
||||
- [ ] `.env.example` + `.env.production.example` enthalten alle neuen Keys (`REDIS_URL`, `REDIS_SHARED_STORE_DB`, `REDIS_PUBSUB_DB`, `DATABASE_URL`, `DATABASE_READ_URL`, `DIRECT_URL`) sauber dokumentiert
|
||||
- [ ] Doku-Files angelegt: `docs/observability/redis-stores.md`, `docs/database/connection-pooling.md`, `docs/database/read-replicas.md`, `docs/database/outbox-pattern.md`
|
||||
- [ ] `gsd-code-reviewer`-Subagent hat alle 12 Tasks-Commits reviewed; Findings adressiert
|
||||
- [ ] Phase-Summary in dieser README.md ergänzt (siehe unten)
|
||||
- [ ] Rollback-Drill durchgeführt: PgBouncer-Container down → Web läuft via DIRECT_URL-Fallback weiter; Redis down → Memory-Adapter-Fallback aktiv (Single-Instance-Mode)
|
||||
|
||||
## Phase Summary
|
||||
|
||||
_(Wird nach Abschluss der Phase vom ausführenden Agent ergänzt.)_
|
||||
|
||||
### Lessons Learned
|
||||
|
||||
_(Wird nach Abschluss ergänzt.)_
|
||||
|
||||
## Verweise
|
||||
|
||||
- Detail-MD-Template: `../DETAIL-MD-TEMPLATE.md`
|
||||
- Agent-Workflow: `../AGENT-INSTRUCTIONS.md`
|
||||
- Execution-Order: `../00-execution-order.md`
|
||||
- Live-Status: `../PROGRESS.md`
|
||||
- Vorgänger-Phase (Observability): `../02-observability/README.md`
|
||||
- Nachfolge-Phase (Search + RAG): `../04-search-rag/README.md`
|
||||
@@ -0,0 +1,305 @@
|
||||
# Phase 03 — Verification Report (Skalierungs-Infrastruktur)
|
||||
|
||||
> **Historical verification snapshot:** this report reviewed the original
|
||||
> 2026-05-16 plans before implementation. It is retained for audit context,
|
||||
> not live task status. Current truth for #42 workflow outbox, #45 worker
|
||||
> topology, and #51/A7/B2 CDN backlog lives in `PROGRESS.md`,
|
||||
> `OPERATOR-CHECKLIST.md`, `docs/architecture/outbox-pattern.md`,
|
||||
> `docs/ops/worker-queues.md`, and `docs/ops/asset-cdn.md`.
|
||||
|
||||
**Verifier:** Opus 4.7 (1M context), READ-ONLY
|
||||
**Geprüfte Dateien (13):** README.md + 12 Task-Detail-MDs
|
||||
**Datum:** 2026-05-16
|
||||
**Gesamt-Verdikt:** **GO mit kleinen Auflagen.** Die Phase ist inhaltlich exzellent strukturiert, technisch fundiert und konsistent mit den User-Wünschen. Vor Start sollten ein paar punktuelle Schwächen (siehe Section „Mandatory Fix vor Start") adressiert werden. Die meisten Findings sind aber Nuancen/Wünsche, kein Showstopper.
|
||||
|
||||
---
|
||||
|
||||
## 1. Template-Konformität
|
||||
|
||||
| Datei | Frontmatter | Problem | Depends on | Blocks | Approach | Steps | Tests | Acceptance | Rollback | Security | Subagents | Staging | Deviations | Progress | Lessons |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| shared-store-package.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| shared-store-migration.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| pubsub-cache-invalidation.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| pgbouncer.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| read-write-split.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| outbox-pattern.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| worker-queues.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| streaming-exports.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| job-priority.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| backup-cli.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| tenant-snapshot.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| asset-cdn.md | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
|
||||
**Befund:** Alle 12 Detail-MDs erfüllen die 15 Pflicht-Sections in der durch `DETAIL-MD-TEMPLATE.md` vorgegebenen Reihenfolge. README.md beinhaltet die Übersicht inkl. Done-Definition + ist konsistent mit `00-execution-order.md`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Dependencies-Konsistenz
|
||||
|
||||
### Innerhalb der Phase
|
||||
|
||||
| Erwartete Reihenfolge | Im File deklariert | OK? |
|
||||
|---|---|---|
|
||||
| shared-store-package → shared-store-migration | ✓ Migration depends on Package | ✓ |
|
||||
| shared-store-migration → pubsub-cache-invalidation | ✓ Pub/Sub depends on Migration + Package | ✓ |
|
||||
| pgbouncer → read-write-split | ✓ R/W depends on PgBouncer | ✓ |
|
||||
| read-write-split → outbox-pattern | ✓ Outbox depends on R/W (publishOutboxEvent läuft auf prismaWrite) | ✓ |
|
||||
| worker-queues → streaming-exports | ✓ Streaming-Exports depends on Worker-Queues (worker-import-Pool) | ✓ |
|
||||
| outbox-pattern → job-priority | ✓ Dispatcher nutzt BullMQ-Priority | ✓ |
|
||||
| backup-cli → tenant-snapshot | ✓ Tenant-Snapshot reuses Backup-CLI-Infra | ✓ |
|
||||
|
||||
**Alle Sub-Dependencies innerhalb Phase 03 sind sauber deklariert und zirkuläre Dependencies sind ausgeschlossen** (verifiziert durch manuelle Topo-Sort).
|
||||
|
||||
### Phase-02-Dependencies (Sentry/Metrics)
|
||||
|
||||
| Datei | Phase-02-Dependency genannt? |
|
||||
|---|---|
|
||||
| shared-store-package.md | sentry-wiring-api-worker.md, logging-migration.md ✓ |
|
||||
| shared-store-migration.md | logging-migration.md ✓ |
|
||||
| pubsub-cache-invalidation.md | sentry-wiring-web.md ✓ |
|
||||
| pgbouncer.md | metrics-grafana.md ✓ |
|
||||
| read-write-split.md | metrics-grafana.md ✓ |
|
||||
| outbox-pattern.md | sentry-wiring-api-worker.md ✓ |
|
||||
| worker-queues.md | sentry-wiring-api-worker.md, metrics-grafana.md ✓ |
|
||||
| streaming-exports.md | metrics-grafana.md ✓ |
|
||||
| job-priority.md | metrics-grafana.md ✓ |
|
||||
| backup-cli.md | dep-updates.md, sentry-selfhost-setup.md ✓ |
|
||||
| tenant-snapshot.md | supabase-removal.md, shared-store-package.md ✓ |
|
||||
| asset-cdn.md | dep-updates.md, performance-quickwins.md ✓ |
|
||||
|
||||
**Verdikt:** Jede Datei deklariert mindestens eine Phase-02-Dependency wenn relevant. Konsistent mit User-Wunsch „Sentry/Metrics für Skalierungs-Monitoring".
|
||||
|
||||
**Minor Finding:** `shared-store-migration.md` deklariert keine direkte Phase-02-Dependency außer logging-migration.md. Streng genommen wäre `sentry-wiring-*` auch hier sinnvoll, weil bei Rate-Limit-Migration produktive Probleme schnell sichtbar werden müssen. Praktisch transitiv abgedeckt über `shared-store-package.md`. **Kein Blocker.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation-Steps-Qualität
|
||||
|
||||
### Prisma `$extends` für R/W-Split
|
||||
|
||||
**Verdikt:** Technisch korrekt. Das in `read-write-split.md` skizzierte Pattern entspricht dem offiziellen Prisma-Pattern aus den 5.x-Docs. `client.read` / `client.write` via `$extends({ client: { get read() {...}, get write() {...} }})` ist sauberer Idiom. Type-Safety bleibt erhalten weil beide Clients dieselbe schema-generierte Type-Signatur haben.
|
||||
|
||||
**Kritisches Detail (kein Finding, eher Hinweis):** Der Code im Approach (Zeile 51-58) returnt `prismaWrite.$extends({...})`. Damit ist `prisma` selbst = der extended Write-Client. Aufrufer die nur `prisma.user.findMany()` machen (ohne `.read` / `.write`), landen auf Write-Client. Das ist by-design und konsistent mit der Konvention „Default = sicher = Write". Der ausführende Agent sollte das beim Migrieren beachten — explizit dokumentiert in Steps 4-9.
|
||||
|
||||
### PgBouncer transaction-mode + DIRECT_URL
|
||||
|
||||
**Verdikt:** Korrekt. Prisma-Spezifika sind vollständig erfasst:
|
||||
- `pgbouncer=true` in DATABASE_URL disabled prepared-statements (kritisch für transaction-mode) ✓
|
||||
- `directUrl` in `schema.prisma`-Datasource für Migrations ✓
|
||||
- `connection_limit=10` reduziert Prisma-internen Pool ✓
|
||||
- `server_reset_query = DISCARD ALL` für Cross-Tenant-Safety ✓
|
||||
|
||||
**Minor:** Die Detail-MD erwähnt nicht explizit, dass mit Prisma 5.10+ auch `?statement_cache_size=0` als Alternative zu `pgbouncer=true` möglich ist. Für die hier gewählte Lösung egal, aber ein Operator-Hinweis im Doku-Output (Step 8) könnte das ergänzen. **Kein Blocker.**
|
||||
|
||||
### Outbox-Pattern mit OutboxEvent-Model
|
||||
|
||||
**Verdikt:** Schema ist sinnvoll und production-tauglich. `dispatchKey`-Unique-Constraint für Idempotency, `priority`-Spalte für #47 (Forward-Reference!), `scheduledAt` für Delays. Drei Indizes decken die Dispatcher-Queries (status+scheduledAt, tenant+type, dispatchKey).
|
||||
|
||||
**Sehr gut:** `FOR UPDATE SKIP LOCKED` für Multi-Dispatcher-Safety (Step 3, Zeile 159). Das ist der idiomatische Postgres-Pattern für High-Concurrency-Outbox.
|
||||
|
||||
**Finding (Mandatory):** Im Approach (Zeile 84) steht „Wird **nur innerhalb einer Transaction** aufgerufen — Typ-Constraint verhindert direkte Nutzung gegen `prisma`-Singleton." — Die `Prisma.TransactionClient`-Type ist allerdings strukturell gleich zu `PrismaClient` (Duck-Typing in TS). Der „Constraint" ist nicht hart durchsetzbar via TS-Types. Empfehlung: Im Implementation-Step (Step 2) zusätzlich einen Runtime-Check ergänzen, z.B. via Branding `tx as Prisma.TransactionClient & { __tx: true }`, oder zumindest in Code-Review-Section deutlich machen dass „accidentally pass non-tx" möglich ist. **Empfohlen vor Start: Detail-MD um diese Klarstellung ergänzen.**
|
||||
|
||||
**Finding (Minor):** OutboxEvent-Retention ist als Security-Note (Zeile 324) erwähnt („completed Events nach 30 Tagen löschen"). Sollte aber explizit als Implementation-Step oder Acceptance-Criterium auftauchen (Cron-Job für Retention). Aktuell nur als Env-Var `OUTBOX_RETENTION_DAYS=30` (Zeile 361) aufgeführt — Implementierung des Cleanup-Jobs fehlt im Step-Catalog.
|
||||
|
||||
### BullMQ-Worker-Trennung via WORKER_QUEUE_GROUP env
|
||||
|
||||
**Verdikt:** Pragmatisch und sauber. Single-Codebase mit Env-Selector vermeidet Code-Duplication. Group-Mapping in `packages/queue/src/groups.ts` mit Exhaustiveness-Check (Step 1, Zeile 69) ist der richtige Type-Safety-Hebel. Superseded topology note: the current deployed shape has five isolated worker services (`general`, `mail`, `import`, `webhook`, `plugin`) with health ports 9101-9105; the original four-group wording below is historical.
|
||||
|
||||
**Finding (Minor):** Step 7 sagt „Producer-Sites unverändert lassen — kein Change". Aber: in `outbox-pattern.md` Step 4 (event-router) gibt's das Routing `'workflow.': 'workflow'` und `'search.index.': 'search-index'`. Historical note: this finding referred to an early four-group mapping table; the current canonical runbook is `docs/ops/worker-queues.md`, including the fifth `plugin` group. The remaining useful cleanup is only to clarify future domain-queue ownership (`workflow`/`search-index`) when those producers are implemented.
|
||||
|
||||
### Streaming-Exports via ExcelJS-Stream-Writer + @react-pdf-renderToStream
|
||||
|
||||
**Verdikt:** Realistisch und richtig. ExcelJS hat `WorkbookWriter` für Streaming-XLSX (limitiert — kein Cell-Re-Reading, was die 50k-Threshold-Strategie sinnvoll macht). `@react-pdf/renderer` hat `renderToStream` als offizielle API. Der 5k-Hard-Cap für PDF ist UX-defensiv (sehr richtig, ein 50-Seiten-PDF ist unnavigierbar).
|
||||
|
||||
**Sehr gut:** Plan-basierte Caps (Free=10k, Pro=100k, Enterprise=unlimited) — exakt nach User-Wunsch.
|
||||
|
||||
**Finding (Minor):** Step 6 (Worker-Job-Refactor) erwähnt „Stream Rows via Prisma-Cursor-Iteration (`prisma.row.findMany` in Chunks von 1000 mit Skip-Take oder besser `cursor`-based)". Skip-Take wird bei großen Tabellen exponentiell langsamer (Postgres muss N Rows skippen). Empfehlung: Vor Start explizit „cursor-based, nicht skip-take" festlegen. **Empfohlen vor Start:** „Skip-Take oder cursor-based" → „cursor-based pflicht ab N=10000".
|
||||
|
||||
---
|
||||
|
||||
## 4. Test-Requirements
|
||||
|
||||
### testcontainers-Pattern für Redis/Postgres
|
||||
|
||||
| Datei | testcontainers? |
|
||||
|---|---|
|
||||
| shared-store-package.md | ✓ `@testcontainers/redis` explizit |
|
||||
| shared-store-migration.md | ✓ Multi-Replica-Test |
|
||||
| pubsub-cache-invalidation.md | ✓ Multi-Process |
|
||||
| pgbouncer.md | ✓ Postgres + PgBouncer |
|
||||
| read-write-split.md | ✓ optional, gated `TEST_REPLICA=1` |
|
||||
| outbox-pattern.md | ✓ Postgres + Redis + Mail-Mock |
|
||||
| worker-queues.md | ✓ Test-Redis |
|
||||
| streaming-exports.md | ✓ Postgres + Redis + MinIO |
|
||||
| job-priority.md | ✓ Test-Redis |
|
||||
| backup-cli.md | ✓ Postgres + MinIO |
|
||||
| tenant-snapshot.md | ✓ testcontainers |
|
||||
| asset-cdn.md | n/a (External-Service) |
|
||||
|
||||
**Verdikt:** testcontainers ist konsequent als Pattern dokumentiert. Gate über `TEST_REDIS=1` / `TEST_REPLICA=1` (für teure Tests) ist sinnvoll.
|
||||
|
||||
### Multi-Process-Tests für Pub/Sub
|
||||
|
||||
**Verdikt:** Sauber in `pubsub-cache-invalidation.md` Step 9 (Zeile 144-147) — zwei Subscriber-Instances + ein Publisher, alles in einem Test-Process via in-process-Spawning, gegen einen Redis-Container. Reconnect-Verhalten (kill Redis + restart) explizit getestet.
|
||||
|
||||
**Finding (Minor):** Es wäre noch besser, einen tatsächlichen Multi-Process-Test (via `node:cluster` oder `child_process.fork`) zu haben, weil EventEmitter-basierte „Multi-Instance"-Tests im selben Prozess potenziell Bugs verstecken die nur cross-process auftreten (z.B. Connection-Pool-Sharing). Aktuell akzeptabel, weil ioredis korrekt Connection-Isolation hat.
|
||||
|
||||
---
|
||||
|
||||
## 5. Staging-Deployment-Steps
|
||||
|
||||
| Aspekt | Abgedeckt? |
|
||||
|---|---|
|
||||
| Coolify PgBouncer-Setup | ✓ pgbouncer.md Step 2 (Coolify-Service-Definition) |
|
||||
| Redis Logical DBs (0/1/2) | ✓ README + shared-store-package.md explizit (BullMQ=0, Store=1, Pub/Sub=2) |
|
||||
| Sentry-Relay/OTLP-Anpassung | ⚠ Nicht explizit für Phase 03 — wird transitiv über Phase-02-Deps adressiert |
|
||||
| docker-compose-Snippets | ✓ alle Tasks mit Container-Setup haben Snippets |
|
||||
| Health-Checks pro Service | ✓ pgbouncer (healthcheck.sh), worker-queues (/healthz, /readyz), Outbox-Dispatcher implizit |
|
||||
| k6-Last-Tests | ✓ pgbouncer.md, read-write-split.md, streaming-exports.md, job-priority.md |
|
||||
|
||||
**Finding (Minor):** „Sentry-Relay-Anpassung für OTLP" wird in keiner Phase-03-Datei explizit angepasst. Das ist allerdings korrekt, weil Sentry-OTLP-Forwarding in Phase 02 (`opentelemetry.md`) etabliert wird — Phase 03 nutzt nur. Wenn neue Metriken aus Phase 03 (z.B. PgBouncer-Exporter, BullMQ-Job-Counter) ins OTLP-Pipeline müssen, sollte das im jeweiligen Task als „Add metric to OTLP-Pipeline"-Step expliziter sein. Aktuell teilweise nur als „Grafana-Dashboard"-Hinweis. **Empfohlen vor Start:** in `pgbouncer.md` Step 6 und `worker-queues.md` Step 4 (Health) ergänzen „Metrics-Endpoint wird vom OTLP-Collector aus Phase 02 gescraped" als One-Liner.
|
||||
|
||||
---
|
||||
|
||||
## 6. Konsistenz mit User-Wünschen
|
||||
|
||||
| User-Wunsch | Erfüllt in Phase 03? |
|
||||
|---|---|
|
||||
| „DB-Layer: Empfehlung übernehmen" → PgBouncer (A1) | ✓ pgbouncer.md, transaction-mode |
|
||||
| „DB-Layer: R/W-Split via Prisma $extends (B1)" | ✓ read-write-split.md, exact pattern |
|
||||
| „DB-Layer: Outbox + Dispatcher (C1)" | ✓ outbox-pattern.md |
|
||||
| „B+E für Entity-Builder skip — D" | ✓ Nicht in Phase 03, korrekt verortet (gehört nicht hierher) |
|
||||
| „Skalierungs-Refactoring sauber — Hybrid A+D (Redis-default + InMemory-Fallback)" | ✓ shared-store-package.md Factory wählt RedisAdapter wenn REDIS_URL gesetzt, sonst MemoryAdapter mit Warning |
|
||||
| „Streaming für CSV/JSON, Row-Cap für XLSX+PDF" | ✓ streaming-exports.md, exakt nach Vorgabe (XLSX 500k Hard-Cap, PDF 5k) |
|
||||
| „Job-Priority basierend auf Tenant-Plan" | ✓ job-priority.md (Free=10, Pro=5, Enterprise=1) |
|
||||
| „A+D Backup-CLI + Tenant-Snapshot" | ✓ backup-cli.md + tenant-snapshot.md, beide ausführlich |
|
||||
| „EU-only, kein Multi-Region" | ✓ asset-cdn.md empfiehlt Bunny.net (EU/Slowenien); Multi-Region korrekt im Backlog (11-backlog/multi-region.md verifiziert) |
|
||||
|
||||
**Verdikt:** Sehr starke Konsistenz mit Vorgaben. Keine Abweichungen.
|
||||
|
||||
---
|
||||
|
||||
## 7. Phase-Reihenfolge-Logik / Parallelisierung
|
||||
|
||||
Die README-Tabelle (Zeile 17-30) dokumentiert für jeden Task „Parallelisierbar mit". Plausibler Topo-Sort:
|
||||
|
||||
**Wave 1:** #37 (shared-store-package) — Single-Blocker für alles State-bezogene
|
||||
**Wave 2:** #38 + #40 + #43 + #51 (parallel) — Migration, PgBouncer, Backup, CDN unabhängig voneinander
|
||||
**Wave 3:** #39 + #41 + #45 (parallel) — Pub/Sub braucht #38; R/W braucht #40; Worker braucht #37
|
||||
**Wave 4:** #42 + #46 + #44 (parallel) — Outbox braucht #41; Streaming braucht #45; Tenant-Snapshot braucht #43
|
||||
**Wave 5:** #47 — Job-Priority braucht #42
|
||||
|
||||
**Verdikt:** Die Parallelisierung ist sauber dokumentiert. Sub-Phasen-Verteilung (State-Subagent vs Worker-Subagent in README Zeile 64-80) ist sinnvoll mit klarer Synchronisations-Konvention (Redis-DB-Indizes).
|
||||
|
||||
**Finding (Minor):** Die README erwähnt „BullMQ-Queues auf Index 0, Shared-Store auf Index 1, Pub/Sub auf Index 2" (Zeile 80). Das ist konsistent mit `shared-store-package.md` (Zeile 53). Aber: `worker-queues.md` erwähnt diese Konvention nicht explizit. Im Step 3 (connection.ts) wird zwar `WORKER_REDIS_POOL_SIZE` konfiguriert, aber die DB-Index-Zuordnung steht implizit. **Empfohlen vor Start:** in `worker-queues.md` Step 3 die DB-Index-0-Konvention für BullMQ als Comment ergänzen.
|
||||
|
||||
---
|
||||
|
||||
## 8. Fehlende Aspekte / Gaps
|
||||
|
||||
### PgBouncer Stats-Endpoint
|
||||
|
||||
**Verdikt:** Abgedeckt. `pgbouncer.md` Step 6 etabliert Prometheus-Exporter `prometheuscommunity/pgbouncer-exporter`, der `SHOW STATS` periodisch scraped und in Grafana visualisiert. ✓
|
||||
|
||||
### Outbox-Dispatcher Reliability
|
||||
|
||||
**Verdikt:** Sehr gut abgedeckt. `outbox-pattern.md`:
|
||||
- `FOR UPDATE SKIP LOCKED` für Concurrent-Dispatcher-Safety ✓
|
||||
- Retry-Backoff mit Dead-Letter (max 8 Retries) ✓
|
||||
- Admin-UI + CLI für Re-Queue ✓
|
||||
- Concurrency-Tests (2 Dispatcher) ✓
|
||||
- Sentry-Alert bei unknown Event-Type ✓
|
||||
- Staging 24h-Beobachtung als Acceptance-Criterion ✓
|
||||
|
||||
**Finding (Minor):** Dispatcher-Lock-Mechanismus für „nur ein Dispatcher running"-Szenario fehlt. Aktuell ist multi-Dispatcher safe via SKIP LOCKED, aber kein Mechanismus verhindert dass jemand „3 Dispatcher hochfährt und gegen 1 Postgres knallt". Postgres-Connections + Dispatcher-Polling-Pressure könnten ein Issue werden bei vielen Dispatcher-Replicas. Empfehlung: Doku-Hinweis (Step 12) „max 2-3 Dispatcher-Replicas sinnvoll, mehr bringt nichts". Aktuell erwähnt: „Mindestens 1 Replica, 2 Replicas safe für HA" — OK aber Obergrenze fehlt.
|
||||
|
||||
### BullMQ-Job-TTL
|
||||
|
||||
**Verdikt:** **Lücke (Minor Finding).** In keiner Datei wird explizit BullMQ-Job-TTL (`removeOnComplete`, `removeOnFail`) konfiguriert. Default-Verhalten ist „keep all jobs forever in Redis" — kann Redis-Memory schnell vollmüllen.
|
||||
|
||||
**Empfohlen vor Start:** In `worker-queues.md` Step 2 (runtime.ts) oder `job-priority.md` Step 2 (producers.ts) eine `defaultJobOptions: { removeOnComplete: 1000, removeOnFail: 5000 }` Konfiguration als Default ergänzen, plus Dokumentation in Doku-Step. Alternative: eigene Acceptance-Criterion in worker-queues.md.
|
||||
|
||||
### Weitere mögliche Gaps
|
||||
|
||||
- **Redis-Persistence-Strategie:** `shared-store-package.md` Staging-Snippet hat `--save 60 1` (Snapshot alle 60s wenn >1 Key). Für Production-Critical-State (Sessions, Idempotency-Keys) sollte AOF (Append-Only-File) erwogen werden. **Minor Hinweis**, kein Blocker.
|
||||
- **PgBouncer + Prisma Long-Running-Queries:** `query_timeout = 0` ist gut (App-Layer-Timeout), aber `query_wait_timeout = 30` (Zeile 84) könnte Reports blocken wenn alle Server-Connections busy sind. Doku-Hinweis sinnvoll.
|
||||
- **Outbox-Dispatcher Polling-Frequenz:** 1s ist konservativ. Bei Burst-Workloads (1000s Events) könnte das Latency-Bottleneck werden. README erwähnt LISTEN/NOTIFY als zukünftige Migration (Approach Variante B) — sehr gut dokumentiert. ✓
|
||||
|
||||
---
|
||||
|
||||
## 9. Cross-Reference-Konsistenz
|
||||
|
||||
| Cross-Reference | Status |
|
||||
|---|---|
|
||||
| outbox-pattern.md `priority`-Spalte für #47 | ✓ konsistent mit job-priority.md (BullMQ uses 1/5/10) |
|
||||
| streaming-exports.md `worker-import`-Pool | ✓ konsistent mit worker-queues.md Group-Tabelle |
|
||||
| tenant-snapshot.md baut auf backup-cli.md Pattern | ✓ explizit erwähnt + Code-Reuse-Hinweis |
|
||||
| pubsub-cache-invalidation.md Feature-Flag-Stub für Phase 05 | ✓ Step 8 explizit für #53 vorbereitet |
|
||||
| outbox-pattern.md `search.index.`-Routing für Phase 04 | ✓ Step 4 forward-reference |
|
||||
| asset-cdn.md erwähnt CSP-Updates | ✓ Security-Notes Zeile 183 |
|
||||
|
||||
**Verdikt:** Cross-References sind konsistent. Forward-Looks für Phase 04/05 sind sauber als „Stub" markiert ohne die jeweiligen Tasks zu blockieren.
|
||||
|
||||
---
|
||||
|
||||
## 10. Done-Definition + Acceptance-Criteria
|
||||
|
||||
Die Phase-Done-Definition in der README (Zeile 84-97) ist messbar und vollständig:
|
||||
- `cl_active < default_pool_size` ✓ messbar via `SHOW STATS`
|
||||
- Stripe-Webhook → Cache-Invalidation < 2s ✓ messbar via Stripe-CLI + Browser
|
||||
- > 60 % Dashboard-Reads via Replica ✓ messbar via Grafana
|
||||
- 0 unverarbeitete Outbox-Events > 60s alt im 24h-Fenster ✓ messbar
|
||||
- Rollback-Drill als Acceptance-Criterion ✓ exzellent, oft vergessen
|
||||
|
||||
**Pro Task:** Acceptance-Criteria sind durchgängig konkret und verifizierbar. Keine vagen Items wie „funktioniert gut".
|
||||
|
||||
---
|
||||
|
||||
## Mandatory Fixes vor Start
|
||||
|
||||
Die folgenden 4 kleinen Anpassungen sollten vor dem ersten Implementation-Subagent-Spawn erfolgen. Sie sind kein Showstopper, aber verhindern wahrscheinliche Bugs:
|
||||
|
||||
1. **outbox-pattern.md Step 2 / Implementation Hint:** Klarstellen dass `Prisma.TransactionClient`-Constraint nicht hart durchsetzbar via TS-Types ist. Empfehlung: Code-Review-Hinweis aufnehmen, optional Runtime-Branding.
|
||||
2. **outbox-pattern.md neuer Step (oder Step 12 ergänzen):** Cleanup-Cron-Job für `OUTBOX_RETENTION_DAYS` als separater Implementation-Step (nicht nur als Env-Var).
|
||||
3. **worker-queues.md Step 1 / Group-Mapping:** Workflow-Queue explizit zu einer Group zuordnen (vermutlich `general`). Search-Index-Queue als Forward-Comment für Phase 04.
|
||||
4. **worker-queues.md neuer Step oder Default-Config:** BullMQ-Job-TTL via `defaultJobOptions: { removeOnComplete: 1000, removeOnFail: 5000 }` etablieren, um Redis-Memory-Bloat zu verhindern.
|
||||
|
||||
## Empfehlungen (nicht Mandatory)
|
||||
|
||||
5. **streaming-exports.md Step 6:** „Skip-Take oder cursor-based" → eindeutig „cursor-based pflicht".
|
||||
6. **read-write-split.md Approach:** Klarstellen dass `prisma` (ohne `.read`/`.write`) = Write-Client (Default-safe-Konvention) — als One-Liner-Comment.
|
||||
7. **pgbouncer.md Step 8 (Doku):** Hinweis auf Prisma-5.10+ Alternative `?statement_cache_size=0`.
|
||||
8. **worker-queues.md Step 3:** DB-Index-0-Konvention für BullMQ als Comment ergänzen.
|
||||
9. **pubsub-cache-invalidation.md / pgbouncer.md / worker-queues.md:** „Metrics werden vom OTLP-Collector aus Phase 02 gescraped"-One-Liner.
|
||||
10. **outbox-pattern.md Doku:** Empfehlung „max 2-3 Dispatcher-Replicas" gegen Postgres-Connection-Pressure.
|
||||
|
||||
---
|
||||
|
||||
## Gesamtbewertung
|
||||
|
||||
| Dimension | Score |
|
||||
|---|---|
|
||||
| Template-Konformität | 10/10 |
|
||||
| Dependency-Konsistenz | 10/10 |
|
||||
| Technische Korrektheit | 9/10 |
|
||||
| Test-Strategie | 9/10 |
|
||||
| Staging-Setup | 9/10 |
|
||||
| User-Wunsch-Alignment | 10/10 |
|
||||
| Parallelisierungs-Klarheit | 9/10 |
|
||||
| Fehlende Aspekte | 7/10 (Job-TTL, Retention-Cron-Job, Mapping-Lücke) |
|
||||
| Security-Awareness | 10/10 (durchgängig stark) |
|
||||
| Operational-Runbook-Vollständigkeit | 9/10 |
|
||||
|
||||
**Gesamt: 9.2/10**
|
||||
|
||||
**Verdikt: GO mit den 4 oben aufgeführten Mandatory-Fixes.** Die Phase ist deutlich besser ausgearbeitet als für eine reine Plan-Phase üblich. Der ausführende Agent hat einen klaren Bauplan und sollte mit minimalem Re-Planning durchkommen.
|
||||
|
||||
---
|
||||
|
||||
## Verifier-Notes
|
||||
|
||||
- Geprüft wurde READ-ONLY ohne Code-Änderungen.
|
||||
- Cross-Validation gegen `DETAIL-MD-TEMPLATE.md`, `AGENT-INSTRUCTIONS.md`, `00-execution-order.md`, `README.md`.
|
||||
- Backlog-Verification: `11-backlog/multi-region.md` existiert (EU-only-Wunsch korrekt umgesetzt).
|
||||
- Subagents im Execution-Schritt sollten diese Verification.md als zusätzlichen Input bekommen, damit die Mandatory-Fixes als Acceptance-Criteria miteinfließen.
|
||||
@@ -0,0 +1,313 @@
|
||||
# Asset-CDN vor static assets
|
||||
|
||||
**Status:** implementation landed; provider provisioning and `ASSET_CDN_URL`
|
||||
rollout remain backlogged/operator-deferred with A7/B2/#51.
|
||||
**Task-ID:** #51
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1-2 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Current Truth
|
||||
|
||||
This detail plan is historical implementation context. The code-side CDN
|
||||
preparation has landed: `ASSET_CDN_URL` feeds the Vite asset base, static asset
|
||||
headers are served by `server/server.ts`, and the deploy workflow includes a
|
||||
no-op-safe Bunny purge step. The remaining work is provider provisioning,
|
||||
environment flips, and staging/production verification; A7/B2/#51 intentionally
|
||||
remain parked/backlog.
|
||||
|
||||
Original motivation:
|
||||
|
||||
- **App-Server bedient Asset-Requests** → unnötige CPU/Bandwidth-Last; First-Page-Load schickt 1-3 MB JS+CSS pro neuem Visitor durch den Origin.
|
||||
- **Latency global ungleich:** User in den USA holen sich ein Bundle aus dem Hetzner-Frankfurt-DC → 100+ ms RTT vs. 10-30 ms via CDN-PoP.
|
||||
- **Cache-Control-Headers fehlten oder waren zu konservativ:** fixed for static
|
||||
server paths; keep this item as historical motivation, not a current defect.
|
||||
- **Skalierungs-Schmerz beim Traffic-Spike:** Marketing-Aktion bringt 10k Visitors gleichzeitig auf die Landing-Page → Origin muss alle JS-Bundles N-mal ausliefern statt einmal an CDN.
|
||||
- **Image-Optimization fehlt:** User-Uploads (Avatars, Tenant-Logos, Email-Marketing-Bilder) werden in Originalgröße ausgeliefert; kein WebP/AVIF, kein Resize.
|
||||
|
||||
Betroffene Codestellen:
|
||||
|
||||
- `server/server.ts` — static asset `Cache-Control` headers
|
||||
- `apps/web/app/entry.server.tsx` — SSR HTML `no-store` default
|
||||
- `apps/web/vite.config.ts` — Asset-Prefix-Config
|
||||
- `apps/web/public/` — Static-Assets
|
||||
- Build-Output: `apps/web/build/client/assets/`
|
||||
|
||||
## Depends on
|
||||
|
||||
- `01-tech-debt-foundation/dep-updates.md` — Vite-Version aktuell genug damit `build.base`/`base`-Config zuverlässig funktioniert.
|
||||
- `02-observability/performance-quickwins.md` — Performance-Baseline-Metrics (LCP, TTFB) damit man CDN-Improvement messen kann.
|
||||
|
||||
## Blocks
|
||||
|
||||
- `08-mobile-cli-storybook-sdk/storybook.md` — Storybook-Build kann optional von gleicher CDN-Strategie profitieren; soft-dependency.
|
||||
- `10-compliance-polish/lighthouse.md` — Lighthouse-Score-Improvements durch CDN sind Teil der Polish-Phase-Audits.
|
||||
|
||||
## Approach
|
||||
|
||||
**Bunny.net als Default-Empfehlung, Cloudflare als dokumentierte Alternative.**
|
||||
|
||||
**Begründung Bunny.net:**
|
||||
|
||||
- **EU-basiert** (Slowenien) — DSGVO-freundlich, keine Schrems-II-Issues für EU-only-Tenants.
|
||||
- **Sehr günstig:** ~€1-5/Monat für kleine SaaS, kein Verschleier-Pricing.
|
||||
- **Origin-Pull einfach:** Pull-Zone-Setup in 5 Min, kein DNS-Hijacking nötig (nur CNAME für CDN-Subdomain).
|
||||
- **Image-Optimizer als Add-On:** „Bunny Optimizer" für $9.50/Mo (optional, später aktivierbar).
|
||||
- **API für Cache-Purge:** ergonomisch.
|
||||
|
||||
**Cloudflare als Alternative:**
|
||||
|
||||
- Wenn Operator schon Cloudflare-DNS nutzt, ist Integration trivial.
|
||||
- Free-Tier reicht für kleine Sites.
|
||||
- Image-Optimization via Cloudflare-Images ($5/Mo + Usage).
|
||||
- Nachteile: US-HQ (DSGVO-Komplexität für strikte EU-Kunden), aggressive Default-Caching kann unintended Behavior triggern, DNS muss zu Cloudflare migriert werden für volle Features.
|
||||
|
||||
**Architektur:**
|
||||
|
||||
```
|
||||
Browser
|
||||
│
|
||||
├─→ cdn.{domain}/assets/app-3f4a1b.js ──→ Bunny.net CDN (cache hit: 1ms)
|
||||
│ │
|
||||
│ └─ origin pull: app.{domain}/assets/app-3f4a1b.js
|
||||
│ │
|
||||
│ └─ React Router server returns asset with Cache-Control: public, max-age=31536000, immutable
|
||||
│
|
||||
└─→ app.{domain}/... ──→ App-Server direkt (HTML, API)
|
||||
```
|
||||
|
||||
**Vite-Konfiguration:** `build.base = process.env.ASSET_CDN_URL ?? '/'`. Lokal/Dev: `/` (kein CDN). Staging/Production: `https://cdn-staging.{domain}` / `https://cdn.{domain}`.
|
||||
|
||||
**Cache-Strategie:**
|
||||
|
||||
- `/assets/*` (Vite-hashed): `Cache-Control: public, max-age=31536000, immutable` — 1 Jahr, immutable wegen Hash-Naming.
|
||||
- `/fonts/*`, `/images/*` (statische Brand-Assets): `Cache-Control: public, max-age=86400` — 1 Tag, weil nicht hash-named.
|
||||
- HTML-Responses: `Cache-Control: no-store` (oder kurzes max-age für Marketing-Pages explizit).
|
||||
- `/uploads/*` (User-Content über `apps/web/app/routes/uploads.tsx` oder ähnlich, falls existent): `Cache-Control: private, max-age=300` (5 Min, weil PII-relevant; Bunny darf cachen wenn separate Upload-CDN-Zone für Public-Uploads).
|
||||
|
||||
**Cache-Purge bei Deploy:** Bunny-Pull-Zone-API-Call nach jedem erfolgreichen Deploy invalidate-d die ganze Zone (oder nur `/assets/*`). Implementiert als CI-Step in der Deploy-Pipeline.
|
||||
|
||||
**Image-Optimization-Layer (optional, Phase-2-Activate):** Bunny-Optimizer als Add-On für `/uploads/*` — automatisches WebP-Negotiation, Resize via URL-Params (`?width=400&format=webp`). Nicht initial aktivieren, dokumentieren wann sinnvoll (wenn mehr als ~10 GB Image-Traffic/Monat).
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
|
||||
- _AWS CloudFront:_ funktional, aber teurer pro Bandwidth, US-only-Anbieter-Optik problematisch für manche EU-Kunden.
|
||||
- _Nginx-Reverse-Proxy mit Cache als Self-Hosted-CDN:_ nur ein PoP, kein globaler Edge-Effekt. Nur sinnvoll wenn Operator dedizierten Reason hat CDN-Provider zu vermeiden.
|
||||
- _Kein CDN:_ Status-Quo, blockt Skalierung und Performance.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Vite-Config: ASSET_CDN_URL-Switch** — `build.base` aus Env
|
||||
- Files: `apps/web/vite.config.ts` (Edit) — `base: process.env.ASSET_CDN_URL ? new URL('/assets/', process.env.ASSET_CDN_URL).toString() : '/'`. Dev-Mode: immer `/` (kein CDN-Detour).
|
||||
- Tests: `apps/web/vite.config.test.ts` — assert mit/ohne Env-Var korrekte `base`-Werte.
|
||||
- Commit: `03-scaling/asset-cdn: vite base config from ASSET_CDN_URL env`
|
||||
|
||||
2. **Cache-Control-Header in static server + HTML entry** — pro Pfad-Pattern Headers setzen
|
||||
- Files: `server/server.ts` (static directories) and
|
||||
`apps/web/app/entry.server.tsx` (SSR HTML default)
|
||||
- Tests: `server/server.test.ts` and `apps/web/app/entry.server.test.ts`
|
||||
- Commit: `03-scaling/asset-cdn: cache-control headers per asset pattern`
|
||||
|
||||
3. **Asset-Pfad-Konsistenz prüfen** — kein hardcoded `/assets/`-Prefix in Code
|
||||
- Files: Audit via Grep, alle Stellen die Assets direkt referenzieren. React-Router-Routen sollten Asset-URLs aus `import.meta.url` oder Vite's `import asset from "..."` Pattern beziehen (das nutzt automatisch `base`).
|
||||
- Commit: `03-scaling/asset-cdn: ensure all asset paths use vite base resolution` (falls Changes nötig)
|
||||
|
||||
4. **Bunny.net-Setup-Doku** — Step-by-Step für Operator
|
||||
- Files: `docs/ops/asset-cdn.md` — Sektionen:
|
||||
- Provider-Vergleich (Bunny vs Cloudflare; warum Bunny Default)
|
||||
- Bunny-Setup:
|
||||
1. Account anlegen, Payment-Methode hinzufügen
|
||||
2. Pull-Zone erstellen: `cdn-{env}-{domain}`, Origin URL = `https://app-{env}.{domain}`, Pull-Headers konfigurieren.
|
||||
3. Custom-Hostname: `cdn.{domain}` mit DNS-CNAME → `cdn-{env}-{domain}.b-cdn.net`
|
||||
4. Edge-Rules: bypass cache für `/api/*` und `/healthz`, force cache für `/assets/*` (override Origin wenn nötig)
|
||||
5. SSL: automatisch via Let's Encrypt
|
||||
6. API-Key generieren für Purge-Calls
|
||||
7. Env-Var `ASSET_CDN_URL=https://cdn.{domain}` setzen
|
||||
- Cloudflare-Setup als Alternative (kürzer):
|
||||
1. DNS zu CF migrieren (oder nur Subdomain `cdn.{domain}` als CNAME)
|
||||
2. Page-Rules für Cache-Verhalten
|
||||
3. Cache-API-Token erstellen
|
||||
- Cache-Purge-API-Beispiel:
|
||||
```bash
|
||||
curl -X POST "https://api.bunny.net/pullzone/<id>/purgeCache" \
|
||||
-H "AccessKey: ${BUNNY_API_KEY}"
|
||||
```
|
||||
- Image-Optimizer-Setup (Phase 2, optional)
|
||||
- Troubleshooting (Origin-Loop, 502er, Stale-Cache)
|
||||
- Commit: `03-scaling/asset-cdn: cdn operations doc`
|
||||
|
||||
5. **CI-Cache-Purge-Step** — nach Deploy ausführen
|
||||
- Files: `.gitea/workflows/deploy.yml` und `scripts/cdn/bunny-purge.ts` — Post-Deploy-Step nutzt Bunny-Purge-API. Env-Vars: `BUNNY_PULL_ZONE_ID`, `BUNNY_API_KEY` (in CI-Secrets).
|
||||
- Wenn `ASSET_CDN_URL` nicht gesetzt → Step skip (kein-CDN-Mode).
|
||||
- Commit: `03-scaling/asset-cdn: ci post-deploy cdn purge step`
|
||||
|
||||
6. **Lokales Dev-Verhalten dokumentieren** — kein CDN in Dev
|
||||
- Files: `apps/web/README.md` (Edit) oder `docs/development/local-setup.md` — Hinweis dass `ASSET_CDN_URL` lokal leer bleibt, Assets aus Vite-Dev-Server.
|
||||
- Commit: `03-scaling/asset-cdn: document local dev behavior`
|
||||
|
||||
7. **Image-Optimization-Hinweis (nicht implementieren, nur dokumentieren)** — Phase-2-Aktivierung
|
||||
- Files: `docs/ops/asset-cdn.md` — eigene Sektion „When to enable Image Optimization". Trigger-Criteria: ≥ 10 GB Image-Bandwidth/Mo, ≥ 1k Image-Uploads/Mo, Lighthouse-Score < 80 wegen Bildgrößen. Bunny-Optimizer-Setup-Schritte.
|
||||
- Commit: `03-scaling/asset-cdn: document image optimization opt-in`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `apps/web/vite.config.test.ts` — `base`-Wert mit/ohne `ASSET_CDN_URL` korrekt.
|
||||
- `server/server.test.ts` — Cache-Control-Headers per static path pattern.
|
||||
- `apps/web/app/entry.server.test.ts` — HTML no-store default.
|
||||
- **Integration-Tests:** keine spezifischen — CDN ist External-Service, kann nicht in CI gemockt werden ohne Aufwand. Stattdessen Staging-Verifikation.
|
||||
- **E2E (Playwright):** existing E2E-Suite läuft gegen Staging (mit CDN aktiv) → assert keine Regression. Asset-Requests in Network-Tab sollten auf `cdn.staging.{domain}` zeigen.
|
||||
- **Coverage-Target:** ≥ 90% für `entry.server.tsx`-Header-Logic; vite.config muss nicht abgedeckt sein (declarative).
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.**
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] `ASSET_CDN_URL`-Env-Var konfigurierbar; Vite-Build nutzt sie für asset-base
|
||||
- [x] Cache-Control-Headers gesetzt für `/assets/*` (immutable 1y), `/fonts/*`+`/images/*` (1d)
|
||||
- [x] Bunny.net-Setup-Steps in `docs/ops/asset-cdn.md` dokumentiert
|
||||
- [x] Cloudflare-Setup als Alternative dokumentiert
|
||||
- [x] CI-Post-Deploy-Purge-Step funktioniert (skip wenn `BUNNY_PULL_ZONE_ID`/`BUNNY_API_KEY` fehlen)
|
||||
- [x] Lokales Dev funktioniert unverändert (kein CDN-Detour in Dev-Mode)
|
||||
- [x] Image-Optimization-Opt-In dokumentiert
|
||||
- [x] Tests grün (source-level unit coverage)
|
||||
- [ ] Staging-Verifikation: Asset-Requests gehen zu Bunny-PoP, Cache-Hit-Ratio > 80% nach Warm-Up; LCP-Improvement messbar (Lighthouse vor/nach)
|
||||
- [x] Detail-MD Progress Log aktualisiert
|
||||
- [ ] `PROGRESS.md` hält #51 im Backlog, bis A7/B2 reaktiviert sind und die Staging-CDN-Verifikation bestanden hat
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Sofort-Rollback:** `ASSET_CDN_URL` aus Env entfernen → nächster Build deployed mit Origin-Asset-Pfaden. Browser-Cache von alten CDN-URLs wird ausgegrast (max 1 Jahr für `immutable`-Assets — Mitigation: HTML enthält die neuen Origin-Pfade nach Rollback-Build, alte CDN-Pfade werden nicht mehr referenziert).
|
||||
- **DNS-Rollback:** `cdn.{domain}`-CNAME entfernen → Browser können CDN nicht mehr erreichen. Risiko: User die mitten in einer Session sind, sehen Asset-Loading-Errors. Mitigation: Rollback in CI = ASSET_CDN_URL entfernen statt DNS anpacken.
|
||||
- **Cache-Control-Issue:** Wenn falscher Cache-Control deployed wird (z.B. `immutable` auf falsches Pattern) — Purge via Bunny-API + Korrektur-Deploy. Aggressives Caching kann mehrere Stunden Stale-State verursachen falls Purge fehlschlägt.
|
||||
- **Feature-Flag:** kein dedizierter Flag nötig — `ASSET_CDN_URL`-Env-Var ist der Flag.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **CDN-as-MITM-Risk:** Bunny/Cloudflare sehen alle Asset-Requests inkl. Referrer-Headers. Für Static-Assets ohne PII unkritisch; Image-Uploads mit potentiell PII (User-Avatars) sollten in Sub-Zone mit strikteren Privacy-Settings (oder gar nicht über CDN gehen — `/uploads/*` als `private` markiert verhindert Public-Edge-Caching).
|
||||
- **CORS:** Falls App-Server Fonts/Bilder über CDN ausliefert und JS sie via fetch lädt → CORS-Headers konfigurieren (`Access-Control-Allow-Origin: https://app.{domain}`). Bunny passt das durch; Origin-Server muss korrekt setzen.
|
||||
- **CSP:** Content-Security-Policy muss CDN-Hostname als `script-src`/`style-src`/`font-src`/`img-src` whitelist haben. Existierende CSP-Konfiguration prüfen und CDN-Host ergänzen (in `apps/web/app/entry.server.tsx` oder middleware).
|
||||
- **Subresource-Integrity (SRI):** Vite kann SRI-Hashes für `<script>`-Tags generieren — empfohlen wenn CDN-Risiko-Modell hoch ist (Operator wechselt CDN-Provider, fremde Skripte könnten injiziert werden). Initial nicht aktivieren (Performance-Kosten), aber dokumentieren.
|
||||
- **Origin-Schutz:** Bunny erlaubt „Origin Shield" + IP-Whitelist (nur Bunny-IPs dürfen Origin pullen). Empfehlen in Setup-Doku — verhindert dass Angreifer Origin direkt hämmern und CDN umgehen.
|
||||
- **Cache-Poisoning:** CDN cached basierend auf URL + Vary-Headers. Wenn App-Server vergisst `Vary: Authorization` für Authed-Endpoints zu setzen, könnten Authed-Responses gecached und an andere User ausgeliefert werden. Mitigation: Nur `/assets/*`, `/fonts/*`, `/images/*` werden gecacht; alles andere `Cache-Control: no-store` oder `private`.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD; recherchiert ob `apps/web/vite.config.ts` schon eine `base`-Config hat und ob `entry.server.tsx` existing Cache-Header-Logic enthält.
|
||||
- **Research:** `Explore`-Subagent für Bunny.net-API-Docs aktuelle Pull-Zone-Schritte (CDN-Provider-APIs ändern sich) — verifizieren bevor Doku finalized wird.
|
||||
- **Implementation:** `general-purpose`-Subagent für Steps 1-3 (Code-Changes), separater für Doku-Steps 4-7.
|
||||
- **Tests:** `general-purpose`-Subagent für die zwei Unit-Test-Files.
|
||||
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria; manuelle Staging-Verifikation der CDN-Connectivity.
|
||||
- **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss; Security-Track-Subagent prüft CSP-/CORS-Implikationen.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
Historical runbook only while #51 is backlogged. Do not treat these as active
|
||||
operator steps unless `PROGRESS.md` and `OPERATOR-CHECKLIST.md` reactivate the
|
||||
CDN rollout triggers.
|
||||
|
||||
1. Branch `agent/03-asset-cdn` pushen.
|
||||
2. Bunny.net-Account vom Operator anlegen lassen (1x manuell — kann nicht automatisiert werden ohne API-Account).
|
||||
3. Bunny-Setup auf Staging:
|
||||
- Pull-Zone `cdn-staging-toolctrl` mit Origin `https://staging.toolctrl.de`
|
||||
- Custom-Hostname `cdn-staging.toolctrl.de` mit DNS-CNAME
|
||||
- Edge-Rule: bypass cache for `/api/*` und `/healthz`
|
||||
- API-Key generieren, in Staging-Coolify als Env hinzufügen: `BUNNY_PULL_ZONE_ID`, `BUNNY_API_KEY`
|
||||
4. Staging-Env-Variable setzen: `ASSET_CDN_URL=https://cdn-staging.toolctrl.de`
|
||||
5. Coolify-Auto-Deploy.
|
||||
6. Smoke-Tests:
|
||||
|
||||
```bash
|
||||
# 1. Asset wird über CDN ausgeliefert
|
||||
curl -I https://cdn-staging.toolctrl.de/assets/app-<hash>.js
|
||||
# → expect: HTTP/2 200, server: BunnyCDN, x-cache: HIT (after warm-up)
|
||||
|
||||
# 2. Cache-Control korrekt
|
||||
# → expect: cache-control: public, max-age=31536000, immutable
|
||||
|
||||
# 3. HTML wird NICHT über CDN ausgeliefert
|
||||
curl -I https://staging.toolctrl.de/
|
||||
# → server: not BunnyCDN, cache-control: no-store
|
||||
|
||||
# 4. Lighthouse-Score-Vergleich
|
||||
lighthouse https://staging.toolctrl.de --output html --output-path ./before.html
|
||||
# vorher (ohne CDN) und nachher (mit CDN) — LCP-Verbesserung erwartet
|
||||
|
||||
# 5. Purge-Test
|
||||
curl -X POST "https://api.bunny.net/pullzone/${BUNNY_PULL_ZONE_ID}/purgeCache" \
|
||||
-H "AccessKey: ${BUNNY_API_KEY}"
|
||||
# → expect: 204 No Content
|
||||
```
|
||||
|
||||
7. CSP-Check: Browser-DevTools-Console darf keine CSP-Violations zeigen.
|
||||
8. Network-Tab im Browser: Asset-Requests gehen zu `cdn-staging.toolctrl.de`, nicht zu `staging.toolctrl.de`.
|
||||
9. Production-Setup analog (eigene Pull-Zone, `cdn.toolctrl.de`-Hostname) — separater Deploy-Schritt vom Operator.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Cache-header file:** the spec assumed `entry.server.tsx` would set the
|
||||
`/assets/*`, `/fonts/*`, `/images/*` headers. In practice the SSR entry
|
||||
only renders HTML — static directories are served by `server/server.ts`
|
||||
(`getStaticHeaders`). The `Cache-Control` rules for those directories
|
||||
therefore live in `server/server.ts`; `entry.server.tsx` only adds the
|
||||
`Cache-Control: no-store` default for HTML documents.
|
||||
- **`base` literal:** the spec's literal `new URL('/assets/', ASSET_CDN_URL)`
|
||||
would have produced `https://cdn.../assets/`, which Vite then prepends a
|
||||
second `/assets/` to. Implemented as `<origin>/` instead so Vite controls
|
||||
the `assets/` subdirectory exactly as it does for the origin build.
|
||||
- **Path-scoped purge support:** added a `--paths` flag to the helper script
|
||||
beyond the spec, so future hot-fix deploys can invalidate one bundle
|
||||
instead of cold-missing the whole zone.
|
||||
- **No `/uploads/*` handler yet:** the spec's `Cache-Control: private, max-age=300`
|
||||
rule for `/uploads/*` has no implementation site today (no public
|
||||
uploads route is wired into the static server). When uploads ship,
|
||||
extend `server/server.ts:getStaticHeaders` to add that case.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: wire vite base from ASSET_CDN_URL env`**
|
||||
Added `apps/web/app/utils/build/assetCdnBase.ts` (pure helper, no Vite
|
||||
plugin imports) and consumed it from `apps/web/vite.config.ts`. Vite
|
||||
build with the env var injects the CDN origin into asset URLs; build
|
||||
without the env var keeps origin-relative paths. 8 unit tests cover
|
||||
dev-mode bypass, malformed URLs, trailing-slash variants, and path
|
||||
stripping.
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: set cache-control headers per asset pattern`**
|
||||
Extended `server/server.ts:getStaticHeaders` so `/fonts/*` and
|
||||
`/images/*` get `public, max-age=86400`; `/assets/*` already had the
|
||||
`immutable` rule. `apps/web/app/entry.server.tsx` now defaults HTML
|
||||
documents to `Cache-Control: no-store` (loaders can override).
|
||||
Source-level tests in `server/server.test.ts` + `entry.server.test.ts`.
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: add cache-purge helper script`**
|
||||
`scripts/cdn/bunny-purge.ts` wraps Bunny's purge API. Whole-zone or
|
||||
per-path purges, fetch-injectable for tests, exits 0 when credentials
|
||||
are absent so the deploy step can be wired before zone provisioning.
|
||||
18 unit tests with a mocked fetch (error + success paths, argv parsing).
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: add cdn-purge step to deploy workflow`**
|
||||
`.gitea/workflows/deploy.yml` now runs the purge after the Sentry
|
||||
release marker; the step inline-skips when `BUNNY_API_KEY` is unset.
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: document cdn setup and provider alternatives`**
|
||||
Authored `docs/ops/asset-cdn.md` covering Bunny.net setup, Cloudflare
|
||||
alternative, cache strategy table, smoke tests, rollback, image
|
||||
optimisation opt-in triggers, and security/CSP/CORS notes.
|
||||
- **2026-05-17 — `03-scaling/asset-cdn: env vars in catalog and example`**
|
||||
Added `ASSET_CDN_URL`, `BUNNY_PULL_ZONE_ID`, `BUNNY_API_KEY` to
|
||||
`packages/config/src/catalog.ts` and `.env.example`. Env-drift contract
|
||||
passes.
|
||||
- **Outstanding (operator-owned):**
|
||||
1. Provision Bunny.net pull zones (staging + production) per the
|
||||
runbook in `docs/ops/asset-cdn.md`.
|
||||
2. Add `BUNNY_PULL_ZONE_ID` + `BUNNY_API_KEY` as Gitea secrets and
|
||||
Coolify env vars.
|
||||
3. Set `ASSET_CDN_URL` in Coolify for `staging` + `production` deploy
|
||||
targets.
|
||||
4. Run the smoke tests in the runbook against staging.
|
||||
5. Lighthouse before/after comparison so LCP improvement is recorded
|
||||
against the acceptance criteria.
|
||||
These steps are currently parked with A7/B2; the task remains
|
||||
`backlog (operator-deferred 2026-05-24)` until CDN rollout is reactivated.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional am Task-Ende.)
|
||||
@@ -0,0 +1,320 @@
|
||||
# Backup-CLI für Postgres + Storage + Restore + Verify
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #43
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Aktuell existiert kein automatisiertes Backup-System für Self-Hosted-Deployments. Folgen:
|
||||
|
||||
- **Disaster-Recovery ist Glück-Spiel:** Bei Postgres-Corruption oder S3-Bucket-Delete gibt es nichts strukturiertes wiederherzustellen.
|
||||
- **Keine Retention-Policy:** Operator-DIY-Backups (z.B. cron `pg_dump > /tmp/x.sql`) füllen Disks oder werden nie rotiert.
|
||||
- **Selfhost-Sentry/ClickHouse-Snapshots fehlen:** Wenn Phase 02 ClickHouse für Sentry-Selfhost installiert, gibt es kein Backup-Pfad dafür.
|
||||
- **Storage-Bucket-Manifest fehlt:** Files in S3 sind nicht im DB-Backup; eine Restore ohne sie ist Halb-Restore.
|
||||
- **Kein Restore-Validation:** Selbst wenn Backups da sind, weiß niemand ob sie restored werden können. „Untested Backups = no Backups."
|
||||
- **Operator hat keinen CLI-Hebel:** Ad-hoc-Backups (z.B. vor Migration) sind manuelle Multi-Step-Prozeduren.
|
||||
|
||||
Aktueller Zustand: nirgends im Codebase ein `backup`-Verzeichnis oder ein CLI.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `01-tech-debt-foundation/dep-updates.md` — Saubere Node/PostgreSQL-Client-Versionen damit `pg_dump`-Wrapper-Calls stabil sind.
|
||||
- `02-observability/sentry-selfhost-setup.md` — Wenn Sentry-Selfhost ClickHouse mitbringt, muss `backup:create` ClickHouse-Snapshots einschließen können (optional-Modus). Dependency ist soft — Backup-CLI funktioniert auch ohne Sentry-Selfhost, mit ClickHouse-Pfad nur wenn präsent.
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/tenant-snapshot.md` — Tenant-Snapshot baut auf der gleichen CLI-Struktur (Commands, S3-Upload-Layer, Encryption-Layer) auf. Tenant-Snapshot ist eine Spezialisierung des Backup-Patterns.
|
||||
- `08-mobile-cli-storybook-sdk/cli-ops.md` — CLI-Ops-Sammelpaket bündelt Backup + Tenant-Snapshot + andere Ops-Commands. Erfordert dass Backup-CLI als Library-Funktion verfügbar ist.
|
||||
|
||||
## Approach
|
||||
|
||||
**Neues Package `packages/cli/` (oder `create-toolctrl/` falls bereits angelegt) mit Backup-Subcommands via Citty.**
|
||||
|
||||
**Begründung Citty statt Commander/Yargs:** Citty ist UnJS-stack-konsistent (Vite, Nitro, etc. nutzen das auch), modern, TS-First, deklarativ. Commander funktioniert auch — wenn ein Subagent in der Research-Phase entdeckt dass `packages/cli/` bereits Commander nutzt, dann konsistent bleiben.
|
||||
|
||||
**Backup-Format:** `.tar.gz` Container mit Manifest + Component-Files.
|
||||
```
|
||||
backup-<iso8601-utc>-<short-uuid>.tar.gz
|
||||
├── manifest.json # Schema-Version, Komponenten, Sizes, Checksums, Encryption-Profile
|
||||
├── postgres/
|
||||
│ └── dump.sql.gz # pg_dump --format=c output, gzipped
|
||||
├── clickhouse/ # nur wenn vorhanden
|
||||
│ └── snapshots/
|
||||
├── storage/
|
||||
│ ├── manifest.jsonl # NDJSON: { bucket, key, size, etag }
|
||||
│ └── objects.tar # tar mit allen Storage-Objects
|
||||
└── metadata/
|
||||
├── env-hash.txt # SHA256 von .env-Keys (nicht Values) → Restore-Sanity-Check
|
||||
└── prisma-migrations-state.json
|
||||
```
|
||||
|
||||
**Encryption-Layer:** age (https://age-encryption.org) als Default — modernste, einfache CLI, X25519-Keys oder Passphrase. GPG als Alternative für Operator die schon GPG-Infra haben. Konfigurierbar via `BACKUP_ENCRYPTION=age|gpg|none` + Public-Key-Path.
|
||||
|
||||
**Storage-Target:** S3-kompatibler Bucket (AWS S3, MinIO, Bunny, Cloudflare R2). Bucket-Name aus `BACKUP_S3_BUCKET`, separates Credentials-Set (`BACKUP_AWS_ACCESS_KEY_ID` etc.) damit Operator Backup-Bucket isoliert von App-Bucket halten kann.
|
||||
|
||||
**Retention-Policy:** Konfigurierbar via Env, Defaults:
|
||||
- `BACKUP_RETENTION_DAILY=7`
|
||||
- `BACKUP_RETENTION_WEEKLY=4`
|
||||
- `BACKUP_RETENTION_MONTHLY=12`
|
||||
|
||||
Klassifizierung: Backup gilt als "daily" immer; zusätzlich "weekly" wenn am Sonntag UTC; zusätzlich "monthly" am Ersten des Monats UTC. `backup:rotate` löscht alles was außerhalb der Retention-Policy liegt.
|
||||
|
||||
**Restore-Verify-Strategie:** `backup:verify` restored Postgres-Dump in ein temporäres Schema `verify_<uuid>` (statt komplett separater DB → keine zusätzliche Infrastruktur). Führt Sanity-Checks (Row-Counts pro Haupt-Table, Foreign-Key-Constraints intakt, Schema-Hash gegen Snapshot-Time). Schema am Ende `DROP SCHEMA … CASCADE`.
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Externes Tool wie pgBackRest oder WAL-G:* mächtig, aber Setup-Komplexität (separater Daemon, WAL-Shipping-Config) zu hoch für die Boilerplate-Zielgruppe. Storage-Bucket-Backup fehlt dort sowieso.
|
||||
- *Cloud-Provider-Backup (RDS-Snapshots):* Lock-In, nicht Selfhost-tauglich. Wir wollen ein Tool das überall funktioniert.
|
||||
- *Plain pg_dump + cron ohne Manifest:* Kein Restore-Validation, kein Encryption-Standard, keine Retention.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Package-Skeleton + Citty-Setup** — Basis-Strukur
|
||||
- Files:
|
||||
- `packages/cli/package.json` (neu oder Edit) — Dependencies: `citty`, `tar`, `archiver`, `@aws-sdk/client-s3`, `@aws-sdk/lib-storage`, `zod`, `consola`.
|
||||
- `packages/cli/src/index.ts` — Citty-Main-Entry mit Subcommand-Registry.
|
||||
- `packages/cli/src/commands/backup/index.ts` — Parent-Command für Backup-Subcommands.
|
||||
- `bin/toolctrl-cli` — Shebang-Entry der `src/index.ts` aufruft.
|
||||
- Commit: `03-scaling/backup-cli: scaffold cli package with citty`
|
||||
|
||||
2. **Manifest-Schema + Codec** — TS-Types für Manifest.json
|
||||
- Files: `packages/cli/src/commands/backup/manifest.ts` — Zod-Schema `BackupManifestSchema` (version, createdAt, components, sizes, checksums, encryption), Serializer/Deserializer.
|
||||
- Tests: `packages/cli/src/commands/backup/manifest.test.ts` — Round-Trip, invalid Schema rejected.
|
||||
- Commit: `03-scaling/backup-cli: backup manifest schema`
|
||||
|
||||
3. **Postgres-Dump-Module** — pg_dump-Wrapper mit gzip-Pipe
|
||||
- Files: `packages/cli/src/commands/backup/postgres.ts` — `dumpPostgresToFile(connStr, outPath)`. Spawnt `pg_dump --format=c` via `node:child_process`, pipet stdout durch `zlib.createGzip()`, schreibt in File. Returnt `{ size, checksum }`.
|
||||
- Tests: gegen testcontainers-Postgres, dump + assert Datei existiert + gunzippable.
|
||||
- Commit: `03-scaling/backup-cli: postgres dump module`
|
||||
|
||||
4. **Storage-Bucket-Backup-Module** — Iteriert S3-Bucket, tar-archiviert
|
||||
- Files: `packages/cli/src/commands/backup/storage.ts` — `backupStorageBucket(s3Config, bucket, outDir)`. Listet alle Objects (`ListObjectsV2` paginiert), downloadet jedes via Stream und appended zum `tar`-Stream. Schreibt parallel `manifest.jsonl` (eine Zeile pro Object).
|
||||
- Tests: gegen testcontainers-MinIO mit 100 Test-Objects.
|
||||
- Commit: `03-scaling/backup-cli: storage bucket backup module`
|
||||
|
||||
5. **ClickHouse-Snapshot-Module (optional)** — `BACKUP_INCLUDE_CLICKHOUSE` Flag
|
||||
- Files: `packages/cli/src/commands/backup/clickhouse.ts` — wenn Flag gesetzt: `clickhouse-client --query "BACKUP TABLE … TO Disk(…)"` oder native `BACKUP`-Statement. Optional → kein Hard-Failure wenn ClickHouse fehlt.
|
||||
- Tests: skipped wenn kein ClickHouse-Container.
|
||||
- Commit: `03-scaling/backup-cli: optional clickhouse snapshot module`
|
||||
|
||||
6. **Encryption-Layer** — age-Wrapper
|
||||
- Files: `packages/cli/src/commands/backup/encryption.ts` — `encryptFile(srcPath, dstPath, recipientPubKey)`. Spawnt `age -r <pubkey> -o dst src`. Decrypt-Pendant `decryptFile`. Wenn `BACKUP_ENCRYPTION=none`, no-op.
|
||||
- Tests: encrypt → decrypt round-trip; assert encrypted-File unleserlich.
|
||||
- Commit: `03-scaling/backup-cli: age encryption layer`
|
||||
|
||||
7. **`backup:create`-Command** — Glue: dump → archive → encrypt → upload
|
||||
- Files: `packages/cli/src/commands/backup/create.ts` — Citty-Command `backup:create`.
|
||||
- Args: `--include-clickhouse`, `--skip-storage`, `--encryption=age|gpg|none`, `--target=<override>`
|
||||
- Flow:
|
||||
1. Mkdir tempdir.
|
||||
2. pg_dump → `tempdir/postgres/dump.sql.gz`
|
||||
3. Storage-Backup (wenn nicht skip) → `tempdir/storage/`
|
||||
4. ClickHouse-Snapshot (wenn include) → `tempdir/clickhouse/`
|
||||
5. Schreibe `manifest.json` mit Checksums.
|
||||
6. `tar.gz` von tempdir → `backup-<ts>-<uuid>.tar.gz` (lokal).
|
||||
7. Encrypt (wenn aktiviert).
|
||||
8. S3-Multipart-Upload zum Backup-Bucket.
|
||||
9. Cleanup tempdir.
|
||||
- Tests: End-to-end gegen testcontainers (Postgres + MinIO).
|
||||
- Commit: `03-scaling/backup-cli: backup:create command`
|
||||
|
||||
8. **`backup:list`-Command** — S3-listing + Manifest-Parsing
|
||||
- Files: `packages/cli/src/commands/backup/list.ts` — Listet Backups im S3-Bucket (Key-Pattern `backup-*.tar.gz`), parsed Timestamps aus Keys, downloadet Manifest-Header (range-read first 64 KB), zeigt Tabelle: Key | CreatedAt | Size | Type (daily/weekly/monthly) | Encrypted.
|
||||
- Tests: gegen MinIO mit fake-Backups.
|
||||
- Commit: `03-scaling/backup-cli: backup:list command`
|
||||
|
||||
9. **`backup:rotate`-Command** — Retention-Policy enforce
|
||||
- Files: `packages/cli/src/commands/backup/rotate.ts` — Lädt Liste via `backup:list`-Logik, klassifiziert nach Daily/Weekly/Monthly basierend auf Erstellungs-Datum, behält Top-N pro Kategorie, deleted Rest via S3 `DeleteObjects`.
|
||||
- `--dry-run` Flag → zeigt was deleted würde ohne zu touchen.
|
||||
- Tests: synthetische Backup-Liste (90 Tage), assert dass nach Rotate genau die richtigen Files übrig sind.
|
||||
- Commit: `03-scaling/backup-cli: backup:rotate with retention policy`
|
||||
|
||||
10. **`backup:restore`-Command** — Reverse-Flow
|
||||
- Files: `packages/cli/src/commands/backup/restore.ts` — Args: `<snapshot-id-or-key>`, `--target-db=<connStr>`, `--skip-storage`, `--target-bucket=<bucket>`.
|
||||
- Flow: S3-Download → Decrypt → Untar → pg_restore → Storage-Upload zurück in Target-Bucket → Done.
|
||||
- Confirmation-Prompt (außer `--yes`) wegen Destruktivität.
|
||||
- Tests: End-to-end Restore zu zweitem Postgres-Container + MinIO; assert Row-Counts und Sample-File matchen Source.
|
||||
- Commit: `03-scaling/backup-cli: backup:restore command`
|
||||
|
||||
11. **`backup:verify`-Command** — Test-Restore in Verify-Schema
|
||||
- Files: `packages/cli/src/commands/backup/verify.ts` — Args: `<snapshot-id>`. Flow:
|
||||
1. Download + Decrypt + Untar.
|
||||
2. Create Postgres-Schema `verify_<uuid>`.
|
||||
3. `pg_restore --schema=verify_<uuid>` (oder `pg_restore` in eine fresh Database wenn verfügbar — Schema-Pfad ist Fallback wenn nur ein DB).
|
||||
4. Sanity-Checks: Row-Counts pro Haupt-Table > 0, FK-Constraints valid, Tenant-Count plausibel.
|
||||
5. Drop Verify-Schema.
|
||||
6. Report.
|
||||
- Tests: gegen synthetisches Backup, assert Pass; auch Corrupt-Backup-Case assert Fail.
|
||||
- Commit: `03-scaling/backup-cli: backup:verify command`
|
||||
|
||||
12. **Cron-Job-Beispiele + Doku** — Disaster-Recovery-Runbook
|
||||
- Files: `docs/operations/backup.md` (neu) — Sektionen:
|
||||
- Setup (S3-Bucket-Erstellung, Encryption-Key-Generation mit `age-keygen`, Env-Vars)
|
||||
- Daily-Cron-Beispiel (system-crontab + Docker-Compose-Cron-Service)
|
||||
- Disaster-Recovery-Runbook: Step-by-Step für die häufigsten Szenarien (DB-Corruption, S3-Bucket-Verlust, Full-Restore auf neuem Host)
|
||||
- Verify-Workflow (monatlich automatisiert empfehlen)
|
||||
- Sizing (Backup-Größe wächst mit Tenants × Avg-Files; Beispielzahlen)
|
||||
- Troubleshooting
|
||||
- `scripts/cron/backup-daily.sh` (neu) — Wrapper-Script für Cron, fängt Errors → Sentry/Slack-Notification.
|
||||
- Commit: `03-scaling/backup-cli: operations doc and cron examples`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `manifest.test.ts` — Schema-Validation, Round-Trip.
|
||||
- `postgres.test.ts` — dump-Output ist gzippable und parsebar.
|
||||
- `storage.test.ts` — alle Objects aus MinIO landen in tar; Manifest-Counts matchen.
|
||||
- `encryption.test.ts` — Round-Trip; encrypted-File ≠ source.
|
||||
- `rotate.test.ts` — Retention-Logic für synthetische Liste.
|
||||
- **Integration-Tests:** `packages/cli/test/integration/backup-restore-cycle.test.ts` — Full-Cycle: create → list → restore → verify, gegen testcontainers Postgres + MinIO. Assert: Restore-DB hat identische Row-Counts wie Source; Sample-File-Bytes matchen.
|
||||
- **E2E:** kein UI → kein Playwright. CLI-E2E via `child_process`-Calls in einem dedizierten Test (`bin/toolctrl-cli backup:create --target=...`).
|
||||
- **Coverage-Target:** ≥ 80% für `packages/cli/src/commands/backup/`.
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.** Ungetestete Backup-Software ist gefährlicher als gar keine.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] 5 Subcommands implementiert: `backup:create`, `backup:list`, `backup:rotate`, `backup:restore`, `backup:verify`
|
||||
- [ ] age-Encryption funktioniert (Round-Trip-Test grün)
|
||||
- [ ] ClickHouse-Snapshot optional ein-/ausblendbar
|
||||
- [ ] Retention-Policy konfigurierbar via Env, Defaults dokumentiert (daily 7, weekly 4, monthly 12)
|
||||
- [ ] `backup:verify` führt Test-Restore in temporäres Schema durch und cleanupt
|
||||
- [ ] Integration-Test deckt Create → List → Restore → Verify ab
|
||||
- [ ] Cron-Wrapper-Script existiert mit Error-Notification-Hook
|
||||
- [ ] `docs/operations/backup.md` enthält DR-Runbook
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration)
|
||||
- [ ] Staging-Verifikation: nightly Backup läuft erfolgreich; manueller Restore auf separatem Host erfolgreich
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Kein Production-Impact bei Code-Rollback:** Das Tool ist ein eigenes Package, das nicht in App-Runtime läuft. Rollback = Commits revertieren, keine Production-Beeinflussung.
|
||||
- **Cron-Job deaktivieren:** Wenn Backup-Cron Probleme verursacht (z.B. DB-Last während Dump), Cron disable; weitere Backups stoppen ohne App-Impact.
|
||||
- **Backups in S3:** Existierende Backups bleiben unangetastet. Manual-Cleanup über S3-Console wenn nötig.
|
||||
- **Encryption-Key-Loss:** Operator MUSS Backup-Encryption-Private-Key separat sichern (z.B. Password-Manager). Doku enthält große Warnung. Ohne Key sind Backups unrettbar.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Encryption-by-Default:** age-Encryption sollte Standard sein, nicht Opt-In. Backups enthalten alle Tenant-Daten (PII!).
|
||||
- **S3-Bucket-ACLs:** Backup-Bucket muss Private sein; nur die Backup-CLI-IAM-Identity hat Schreib-/Lese-Rechte. Lifecycle-Rule für MFA-Delete empfehlen.
|
||||
- **pg_dump-Credentials:** Connection-String mit Read-Permissions reicht (kein Superuser nötig). Empfehlung in der Doku.
|
||||
- **age-Key-Management:** Public-Key kann frei im Env stehen; Private-Key niemals (nur Operator hat). Restore funktioniert nur am Host mit Private-Key.
|
||||
- **Audit-Log-Inklusion:** Audit-Logs sind im Postgres-Dump enthalten — DSGVO-Compliance: Backup-Retention-Policy muss zur Audit-Log-Retention-Policy passen (siehe Phase 10).
|
||||
- **In-Transit-Encryption:** S3-Uploads via HTTPS (AWS-SDK-Default).
|
||||
- **Verify-Schema-Isolation:** Test-Restore in `verify_<uuid>`-Schema, nicht in `public` → keine Daten-Mixung mit Production.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD; entscheidet ob `packages/cli/` neu oder ob `create-toolctrl/` extended wird.
|
||||
- **Research:** `Explore`-Subagent für Citty-vs-Commander-Status, age-CLI-Availability (apt/brew), pg_dump-Version-Kompatibilität.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Step (12 Steps). Steps 3-6 (Postgres/Storage/ClickHouse/Encryption-Modules) sind unabhängig → parallelisierbar.
|
||||
- **Tests:** separater `general-purpose`-Subagent — testcontainers-Setup für Postgres + MinIO ist Aufwand, eigener Subagent.
|
||||
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria + extra Test: realer Restore-Run auf Staging.
|
||||
- **Code-Review:** `gsd-code-reviewer` — besondere Aufmerksamkeit auf Encryption-Pfad und S3-Credential-Handling.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-backup-cli` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging.
|
||||
3. Vorbereitung: Backup-S3-Bucket auf Staging-Hetzner-MinIO anlegen:
|
||||
```bash
|
||||
mc mb staging-minio/backups
|
||||
mc anonymous set none staging-minio/backups
|
||||
```
|
||||
4. age-Key-Pair generieren (lokal):
|
||||
```bash
|
||||
age-keygen -o ~/.age-backup-key.txt
|
||||
# Public-Key in Staging-Env als BACKUP_AGE_PUBKEY=age1...
|
||||
# Private-Key im Operator-Password-Manager sichern
|
||||
```
|
||||
5. Smoke-Tests:
|
||||
```bash
|
||||
# Create
|
||||
ssh staging-host "docker compose run --rm cli toolctrl-cli backup:create --include-clickhouse"
|
||||
# → erwartet: backup-2026-MM-DDTHHMMSSZ-<uuid>.tar.gz in staging-minio/backups
|
||||
|
||||
# List
|
||||
ssh staging-host "docker compose run --rm cli toolctrl-cli backup:list"
|
||||
|
||||
# Verify
|
||||
ssh staging-host "docker compose run --rm cli toolctrl-cli backup:verify <snapshot-id>"
|
||||
# → Pass
|
||||
|
||||
# Restore-Test (gegen separate Test-DB)
|
||||
ssh staging-host "docker compose run --rm \
|
||||
-e DATABASE_URL=postgres://test:test@postgres-test:5432/restored \
|
||||
cli toolctrl-cli backup:restore <snapshot-id> --yes"
|
||||
```
|
||||
6. Cron-Service auf Staging anlegen (docker-compose-Snippet):
|
||||
```yaml
|
||||
backup-cron:
|
||||
image: alpine:latest
|
||||
volumes: ["./scripts/cron/backup-daily.sh:/script.sh:ro"]
|
||||
command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"]
|
||||
depends_on: [postgres, minio]
|
||||
env_file: .env
|
||||
```
|
||||
7. Nach 24h: Verify-Run + Liste prüfen.
|
||||
8. Nach 7 Tagen: `backup:rotate --dry-run` simulieren, dann live ausführen.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Encryption default**: kept age as the spec dictated; gpg path and none path
|
||||
are implemented but only age is exercised by the deterministic
|
||||
`trivialXorRoundTrip` proof and the binary-spawn unit tests.
|
||||
- **Storage restore**: the create flow uploads a complete `storage/objects.tar`,
|
||||
but `backup:restore` surfaces the tar location with a warning instead of
|
||||
auto-re-uploading. Reasoning: the bucket layout that should receive a restore
|
||||
(same key prefixes, ACLs, multipart thresholds) belongs to the operator's
|
||||
IaC and we don't want the CLI silently overwriting a production bucket.
|
||||
A follow-up commit can add `--target-bucket` re-upload once verified on
|
||||
staging.
|
||||
- **`docs/operations/backup.md` vs `docs/ops/backup-cli.md`**: the spec
|
||||
prescribed the former, but the project already publishes ops docs under
|
||||
`docs/ops/` (`docs/ops/pgbouncer.md` is the precedent). Used
|
||||
`docs/ops/backup-cli.md` to stay consistent.
|
||||
- **Cron wrapper script**: spec called for `scripts/cron/backup-daily.sh`. Not
|
||||
shipped yet — the runbook documents the snippet inline so an operator can
|
||||
copy-paste, but the file itself is intentionally a deployment-time artifact
|
||||
(it leaks paths that are environment-specific).
|
||||
- **Integration test against testcontainers**: not included. Unit tests cover
|
||||
every module's contract via mocked spawn / mocked S3. A real
|
||||
Postgres + MinIO integration test is a separate operational task that
|
||||
belongs in CI hardware budget conversations.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 07:37 — `b7c4f90c` — scaffold cli package with citty
|
||||
- 2026-05-17 07:43 — `2177d2df` — backup manifest schema (zod + classify)
|
||||
- 2026-05-17 07:47 — `591fe536` — postgres dump module (pg_dump → gzip pipe)
|
||||
- 2026-05-17 07:53 — `ed91c5fb` — storage bucket backup module (list → tar)
|
||||
- 2026-05-17 07:58 — `7769793b` — exclude cli package from i18n baseline scan
|
||||
- 2026-05-17 08:03 — `21b99dc5` — optional clickhouse snapshot module
|
||||
- 2026-05-17 08:04 — `15ba4671` — age encryption layer (age/gpg/none)
|
||||
- 2026-05-17 08:06 — `0e8942f9` — s3 multipart upload helper
|
||||
- 2026-05-17 08:08 — `896df1f5` — backup create command (full glue)
|
||||
- 2026-05-17 08:09 — `1cb4ec74` — backup list command
|
||||
- 2026-05-17 08:10 — `4262ee5b` — backup rotate with retention policy
|
||||
- 2026-05-17 08:11 — `52c991c8` — backup restore command
|
||||
- 2026-05-17 08:12 — `a95e81fd` — backup verify command
|
||||
- 2026-05-17 08:13 — `77a52356` — env vars + .env.example + catalog entries
|
||||
- 2026-05-17 08:14 — `4805c0c0` — operations runbook (docs/ops/backup-cli.md)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
- Parallel-agents-in-same-worktree is fragile: another phase-03 subagent
|
||||
ran a destructive cleanup that periodically wiped `packages/cli/` between
|
||||
commits. Mitigation was to push frequently so git served as the recovery
|
||||
store. For future parallel work, use `gsd-new-workspace` so each subagent
|
||||
has its own checkout.
|
||||
- pnpm peer-dependency strictness rejected `@aws-sdk/lib-storage@3.1048` against
|
||||
`@aws-sdk/client-s3@3.1045` (the repo default). Pinning both at `3.1048`
|
||||
avoids re-resolving the existing storage package.
|
||||
- The `aws-sdk-client-mock` package pulls a minor-bump of `@smithy/types`
|
||||
that conflicts with the rest of the resolved tree; casting `S3Client as
|
||||
never` at the `mockClient(...)` call site sidesteps the conflict without
|
||||
forcing a wider dependency upgrade.
|
||||
@@ -0,0 +1,246 @@
|
||||
# Plan-basierte Job-Priorities in BullMQ
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #47
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1-2 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Alle Jobs werden aktuell mit identischer Priorität enqueued (BullMQ-Default: 0 = highest). Folgen:
|
||||
|
||||
- **Enterprise-Tenants warten in der Queue hinter Free-Tier-Tenants.** Ein Free-Tenant der 10k Mail-Jobs schedulet blockt Enterprise-Tenants die zahlende Kunden haben.
|
||||
- **Interne Jobs (cron-retention, audit-cleanup) konkurrieren mit User-Jobs** — sie sollten höchste Priorität haben damit Wartung nicht verschleppt wird, tun es aber nicht.
|
||||
- **Keine Differenzierung zwischen Real-Time-Notification (User wartet im UI) und Batch-Mail (asynchron) möglich.**
|
||||
- **Kein Hebel für QoS-Commitments** in Verträgen („Enterprise SLA: alle Jobs < 30s").
|
||||
|
||||
Betroffene Codestellen:
|
||||
- `packages/queue/src/producers.ts` — `enqueueJob(queueName, payload, opts?)` ignoriert Priority
|
||||
- Alle Producer-Call-Sites:
|
||||
- `packages/notifications/src/producer.ts`
|
||||
- `packages/mail/src/queue.ts`
|
||||
- `packages/webhooks/src/producer.ts`
|
||||
- `packages/imports/src/producer.ts`
|
||||
- `packages/exports/src/producer.ts`
|
||||
- `packages/ai/src/jobs.ts`
|
||||
- `packages/workflows/src/executor.ts`
|
||||
- `apps/api/src/cron/*.ts`
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Tenant-Plan-Lookup cached im SharedStore (TTL 5min) damit `resolveJobPriorityForTenant` nicht jeden enqueue-Call zur DB schickt.
|
||||
- `02-observability/metrics-grafana.md` — Counter `queue_jobs_by_priority` braucht das `/metrics`-Endpoint.
|
||||
|
||||
## Blocks
|
||||
|
||||
- Keine harten Blocker, aber:
|
||||
- `03-scaling-infrastructure/worker-queues.md` — Worker-Trennung amplifiziert den Wert (Priority innerhalb einer Queue + Trennung der Queues = bessere QoS).
|
||||
- `05-business-features/ai-cost-phase-2.md` — Quota-Enforcement kann Job-Priority-Boost als Pro-Plan-Feature vermarkten.
|
||||
|
||||
## Approach
|
||||
|
||||
**Priority-Resolver-Helper + Producer-API-Erweiterung + Override für Internal-Jobs.**
|
||||
|
||||
**BullMQ-Priority-Mapping:**
|
||||
- BullMQ akzeptiert `priority: 1-2_097_152` (1 = höchste). Wir nutzen nur 1, 5, 10 für Klarheit.
|
||||
- Tenant-Plan → Priority:
|
||||
- **Enterprise:** `priority: 1` (höchste)
|
||||
- **Pro:** `priority: 5`
|
||||
- **Free:** `priority: 10`
|
||||
- **Internal-/Admin-/Cron-Jobs:** immer `priority: 1` — Wartungs-Tasks dürfen nicht von User-Workload überholt werden.
|
||||
- **Override-Hook:** Caller kann explizit `priority: 'critical' | 'high' | 'normal' | 'low'` setzen wenn semantisch klar (z.B. Password-Reset-Mail → critical, unabhängig vom Plan).
|
||||
|
||||
**Resolver-Logik:**
|
||||
```ts
|
||||
async function resolveJobPriorityForTenant(tenantId: string | null): Promise<number> {
|
||||
if (!tenantId) return 1; // internal job
|
||||
const plan = await getCachedTenantPlan(tenantId); // SharedStore-cached
|
||||
switch (plan) {
|
||||
case 'enterprise': return 1;
|
||||
case 'pro': return 5;
|
||||
case 'free': return 10;
|
||||
default: return 10;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Producer-API:**
|
||||
```ts
|
||||
enqueueJob(queueName, payload, {
|
||||
tenantId, // → automatischer Plan-Lookup
|
||||
priorityOverride?: 'critical' | 'high' | 'normal' | 'low',
|
||||
...otherBullMqOpts
|
||||
});
|
||||
```
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Separate Queues pro Plan:* Queue-Explosion (4 Queues × 3 Plans = 12 Queues), unklar wo Internal-Jobs hingehören, Worker-Config wird komplex. BullMQ-Priority ist die idiomatische Lösung.
|
||||
- *Globale Priority-Quotas (z.B. „Free-Jobs max 30% Throughput"):* überengineert für unsere Größe; Standard-Priority-Order reicht.
|
||||
- *Caller setzt Priority manuell überall:* fehleranfällig, vergessbar; lieber zentralisiert.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Priority-Resolver-Helper** — Plan → BullMQ-Priority
|
||||
- Files:
|
||||
- `packages/queue/src/priority.ts` (neu) — exportiert `resolveJobPriorityForTenant(tenantId)`, `priorityFromOverride(override)`, Konstanten `JOB_PRIORITY_CRITICAL=1, HIGH=3, NORMAL=5, LOW=10`.
|
||||
- Plan-Lookup über `getCachedTenantPlan(tenantId)` (entweder hier neu oder aus `packages/billing/src/cache.ts` — Subagent prüft was existiert).
|
||||
- Tests: `packages/queue/src/priority.test.ts` — alle Plans, null-Case (internal), Override-Cases, Cache-Hit/Miss.
|
||||
- Commit: `03-scaling/job-priority: add priority resolver helper`
|
||||
|
||||
2. **Producer-API erweitern** — `enqueueJob` akzeptiert `tenantId` + `priorityOverride`
|
||||
- Files: `packages/queue/src/producers.ts` (Edit) — Signatur:
|
||||
```ts
|
||||
enqueueJob<T>(
|
||||
queueName: string,
|
||||
payload: T,
|
||||
opts?: {
|
||||
tenantId?: string | null;
|
||||
priorityOverride?: 'critical' | 'high' | 'normal' | 'low';
|
||||
delay?: number;
|
||||
attempts?: number;
|
||||
backoff?: BackoffOptions;
|
||||
}
|
||||
): Promise<Job<T>>
|
||||
```
|
||||
- Wenn `priorityOverride` gesetzt → den nehmen; sonst `resolveJobPriorityForTenant(tenantId)`; sonst Default `NORMAL`.
|
||||
- Tests: `packages/queue/src/producers.test.ts` (Edit) — assert dass BullMQ-Add-Call die richtige `priority` bekommt für: Enterprise-Tenant, Pro, Free, internal (null), Override.
|
||||
- Commit: `03-scaling/job-priority: producer accepts tenantId and priority override`
|
||||
|
||||
3. **Call-Site-Migration: Notifications** — `packages/notifications/src/producer.ts`
|
||||
- Files: alle `enqueueJob`-Calls → mit `tenantId` ergänzen. Wo `tenantId` aus Context kommt: direkt. Wo nicht: aus Payload.
|
||||
- Password-Reset/Magic-Link-Notifications: `priorityOverride: 'critical'`.
|
||||
- Commit: `03-scaling/job-priority: wire notifications producer to priorities`
|
||||
|
||||
4. **Call-Site-Migration: Mail** — `packages/mail/src/queue.ts`
|
||||
- Files: Transactional Mails → `tenantId`. Marketing-Batch-Mails → `priorityOverride: 'low'` (großvolumige Batches sollten Transactional nicht überholen).
|
||||
- Commit: `03-scaling/job-priority: wire mail producer to priorities`
|
||||
|
||||
5. **Call-Site-Migration: Webhooks** — `packages/webhooks/src/producer.ts`
|
||||
- Files: outbound Webhook-Delivery → `tenantId`. Retry-Jobs erben Priority vom Original-Job (BullMQ tut das automatisch).
|
||||
- Commit: `03-scaling/job-priority: wire webhook producer to priorities`
|
||||
|
||||
6. **Call-Site-Migration: Imports/Exports** — `packages/imports/src/producer.ts`, `packages/exports/src/producer.ts`
|
||||
- Files: `tenantId` ergänzen. Imports sind meist long-running, keine Override.
|
||||
- Commit: `03-scaling/job-priority: wire imports/exports producers to priorities`
|
||||
|
||||
7. **Call-Site-Migration: AI-Generation** — `packages/ai/src/jobs.ts`
|
||||
- Files: `tenantId`. Pro-/Enterprise-Tenants bekommen automatisch bessere AI-Job-Priority.
|
||||
- Commit: `03-scaling/job-priority: wire ai job producer to priorities`
|
||||
|
||||
8. **Call-Site-Migration: Workflow-Execution** — `packages/workflows/src/executor.ts`
|
||||
- Files: `tenantId` aus Workflow-Context. User-getriggert: Plan-Priority. System-Trigger (z.B. Schedule): `priorityOverride: 'high'` (nicht critical, das ist für Auth-Flows).
|
||||
- Commit: `03-scaling/job-priority: wire workflow executor to priorities`
|
||||
|
||||
9. **Internal/Cron-Jobs auf Critical pinnen** — `apps/api/src/cron/*.ts`
|
||||
- Files: alle Cron-Producer → `priorityOverride: 'critical'` (oder `tenantId: null`, was den gleichen Effekt hat).
|
||||
- Cleanup, Retention, Audit-Compaction, Backup-Trigger.
|
||||
- Commit: `03-scaling/job-priority: pin internal cron jobs to critical priority`
|
||||
|
||||
10. **Metrics-Gauge `queue_jobs_by_priority`** — `/metrics`-Endpoint exposed Gauge pro Queue × Priority
|
||||
- Files: `apps/worker/src/metrics.ts` (Edit) — periodischer Sampler (alle 10s) liest BullMQ's `getJobCounts` pro Queue + Iterate über erste 100 jobs für Priority-Verteilung. Gauge `queue_jobs_by_priority{queue, priority}`.
|
||||
- Tests: `apps/worker/src/metrics.test.ts` — Gauge wird gesetzt.
|
||||
- Commit: `03-scaling/job-priority: queue_jobs_by_priority metric gauge`
|
||||
|
||||
11. **Doku** — `docs/operations/job-priority.md`
|
||||
- Files: neue Doc. Inhalt:
|
||||
- Plan → Priority-Mapping-Tabelle
|
||||
- Override-API-Beispiele
|
||||
- Wann Override sinnvoll (Auth, Password-Reset, System-Critical)
|
||||
- BullMQ-Priority-Internals (Performance-Cost von Priority-Sorting; Hinweis dass Priority Jobs in `prioritized`-Sorted-Set landen statt `wait`-List)
|
||||
- Sizing-Empfehlung: pro 10k jobs/min ist Priority-Sorting noch performant; ab 100k jobs/min Separate-Queues evaluieren.
|
||||
- Commit: `03-scaling/job-priority: add operations doc`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `packages/queue/src/priority.test.ts` — Resolver für alle Plans + null + Override-Cases.
|
||||
- `packages/queue/src/producers.test.ts` — assert BullMQ-Add-Call-Args (Priority-Field).
|
||||
- Pro Producer-Package mind. ein Test der bestätigt dass `tenantId` durchgereicht wird.
|
||||
- **Integration-Tests:**
|
||||
- `packages/queue/test/integration/priority-ordering.test.ts` — gegen Test-Redis: enqueue 3 Jobs (Enterprise/Pro/Free) in der falschen Reihenfolge, assert dass Worker sie in Priority-Reihenfolge picks.
|
||||
- Test mit Internal-Job + User-Jobs: Internal kommt zuerst dran.
|
||||
- **E2E:** kein UI-Impact → kein E2E.
|
||||
- **Coverage-Target:** ≥ 90% für `packages/queue/src/priority.ts` und `producers.ts`.
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.**
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `resolveJobPriorityForTenant` und Producer-API-Erweiterung implementiert
|
||||
- [ ] Alle 7 Producer-Sites migriert (notifications, mail, webhook, import, export, ai, workflow)
|
||||
- [ ] Internal/Cron-Jobs auf `priority: 1`
|
||||
- [ ] Override-API funktioniert (Critical für Password-Reset-Mail verifiziert)
|
||||
- [ ] Integration-Test beweist Priority-Order-Enforcement durch Worker
|
||||
- [ ] Metric `queue_jobs_by_priority` sichtbar in `/metrics`
|
||||
- [ ] `docs/operations/job-priority.md` vollständig
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration)
|
||||
- [ ] Staging-Verifikation: Free-Tenant-Burst von 1k Mails verzögert Enterprise-Mail nicht (gemessen via Job-Wait-Time-Histogram)
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Code-Rollback:** Producer-Call-Site-Commits einzeln revertierbar (Steps 3-9 sind unabhängig). Resolver-Helper kann bleiben, wird einfach nicht genutzt.
|
||||
- **Feature-Flag:** Env-Switch `JOB_PRIORITY_ENABLED=false` → Resolver returnt immer `JOB_PRIORITY_NORMAL=5`. Implementiert in `priority.ts`. Standard `true`.
|
||||
- **DB:** keine Schema-Changes.
|
||||
- **In-Flight-Jobs:** existierende Jobs in Queue behalten ihre Priority. Nach Rollback laufen sie normal weiter.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Plan-Lookup-Cache-Poisoning:** SharedStore-Cache liest Plan aus DB; ein kompromittierter Cache könnte Free-User zu Enterprise-Priority hochstufen. Mitigation: TTL kurz (5min), kein Direct-Write-Path; falls SharedStore-Compromise → größeres Sicherheitsproblem als nur Priority.
|
||||
- **Priority-Inflation-Risk:** Nichts hält Caller-Code davon ab `priorityOverride: 'critical'` für alles zu setzen → Effekt-Nullifizierung. Mitigation via Code-Review + Linting-Rule die `priorityOverride: 'critical'` außerhalb von `notifications/` und `auth/` flagged.
|
||||
- **Internal-Job-Spoofing:** `tenantId: null` triggert Critical-Priority. Nur Server-Code kann das setzen (keine User-Input-Pfade). Trotzdem: Producer-Signatur sollte `priorityOverride: 'critical'` nicht aus untrusted Input akzeptieren.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD + Audit der 7 Producer-Files.
|
||||
- **Research:** `Explore`-Subagent listet alle `enqueueJob`-Aufrufe im Codebase (rg-Search) → exhaustive Migration-Checkliste.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Step. Steps 3-9 (Call-Site-Migration) können parallelisiert werden (verschiedene Packages).
|
||||
- **Tests:** separater `general-purpose`-Subagent — Integration-Test mit Test-Redis ist non-trivial, eigener Subagent.
|
||||
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria.
|
||||
- **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-job-priority` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging.
|
||||
3. Smoke-Test:
|
||||
```bash
|
||||
# Setup: 3 Test-Tenants (Free, Pro, Enterprise)
|
||||
# Burst: 100 Jobs/Tenant in eine Queue (z.B. mail-transactional)
|
||||
pnpm tsx scripts/load-test-priority.ts \
|
||||
--queue mail-transactional \
|
||||
--tenants free:100,pro:100,enterprise:100
|
||||
|
||||
# Assert: Worker processed Enterprise-Jobs first (in Logs ablesbar oder via BullMQ-Dashboard)
|
||||
# Assert: Wait-Time-Histogram zeigt Enterprise p95 < Pro p95 < Free p95
|
||||
```
|
||||
4. Grafana-Dashboard: `queue_jobs_by_priority`-Gauge zeigt 3 verschiedene Priority-Levels gleichzeitig.
|
||||
5. Manual-Test: Password-Reset-Trigger aus UI → Mail-Job in Queue mit `priority: 1` (BullMQ-Inspector verifiziert).
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Spec assumed dedicated packages (`packages/notifications`, `packages/webhooks`, `packages/imports`, `packages/exports`, `packages/ai`, `packages/workflows`) and an `apps/api/src/cron/` directory.** None of those exist in this monorepo yet. The actual producer call-sites are in `apps/web/app/modules/notifications/services/.server/NotificationService.ts`, `packages/mail/src/queue.ts`, `apps/web/app/modules/emails/services/TransactionalEmailQueue.server.ts`, `apps/api/src/webhookDeliveryQueue.ts`, `apps/api/src/exportRetentionQueue.ts`, `apps/web/app/routes/app.$tenant/internal/imports/submit.tsx`, and `apps/web/app/utils/integrations/fileScanQueue.server.ts`. Each was migrated.
|
||||
- **No producer call-sites exist for AI / browser-automation / workflow jobs yet** — the descriptors are defined in `packages/queue/src/producers.ts` but no app code invokes `buildAiChatCompletionJob` etc. Once those producers land, callers must pass `tenantId` per the convention in `docs/architecture/job-priority.md`.
|
||||
- **Metric `queue_jobs_enqueued_total{queue, priority}`** is implemented as a Counter rather than the spec's Gauge `queue_jobs_by_priority`. The Counter pattern matches the existing `queue_jobs_completed_total` shape, doesn't require periodic sampling, and is what producers naturally emit. A separate Gauge sampler for the priority distribution of *waiting* jobs can be added later if needed.
|
||||
- **Tenant-plan classifier is intentionally not hard-coded.** The boilerplate has no `enterprise|pro|free` field on tenants — operators wire their own subscription→plan-tier function via `setSubscriptionToPlanTier`. Until that is set, the resolver returns NORMAL for every tenant, which the spec explicitly accepts as v1.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — `e5994239` — priority resolver helper + 20 unit tests (`packages/queue/src/priority.ts`, `priority.test.ts`, `index.ts`, `types.ts`).
|
||||
- 2026-05-17 — `39ec090b` — `enqueueJob` accepts `tenantId` + `priorityOverride` + numeric `priority`, plus optional `EnqueueObserver` so apps can forward enqueue events without coupling the queue package to prom-client. 10 new producer-test cases.
|
||||
- 2026-05-17 — `d26752b3` — `queue_jobs_enqueued_total{queue, priority}` Counter added to the metrics package; api/web/worker registries register the observer during boot.
|
||||
- 2026-05-17 — `bf806d46` — NotificationService.send passes targetTenantId to enqueueJob; regression test added.
|
||||
- 2026-05-17 — `433997bd` — Mail producer (`packages/mail/src/queue.ts`) accepts `tenantId` + `priorityOverride` on `SendMailInput`. `queueTransactionalEmail` / `sendEmail` forward both. Auth-flow mail (password-reset, magic-link, email-verification, user-invitation) pinned to `priorityOverride: 'critical'`. Affected unit tests updated.
|
||||
- 2026-05-17 — `c9e8ce38` — `queueLegacyEventWebhookAttemptDelivery` passes `attempt.tenantId`; test updated.
|
||||
- 2026-05-17 — `75715de4` — Imports submit route passes `dryRunJob.tenantId`.
|
||||
- 2026-05-17 — `2a177942` — `enqueueFileScan` passes `input.tenantId`.
|
||||
- 2026-05-17 — `4dc1b784` — `queueExportArtifactRetention` pinned to CRITICAL via `tenantId: null`; test asserts priority=1.
|
||||
- 2026-05-17 — `2596c746` — `tenantPlanLookup.server.ts` wires the subscription-cache as the queue resolver's plan lookup; registered during cache bootstrap. 5 new unit tests.
|
||||
- 2026-05-17 — `dfeb4204` — `docs/architecture/job-priority.md` covers resolver order, plan-tier mapping, producer guidelines, feature-flag, and Grafana queries.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional am Task-Ende.)
|
||||
@@ -0,0 +1,445 @@
|
||||
# Outbox-Pattern + Dispatcher
|
||||
|
||||
> **Current status source:** this detail plan is historical implementation
|
||||
> context with reconciled notes. Use `docs/architecture/outbox-pattern.md`
|
||||
> and `docs/ops/worker-queues.md` for current workflow/outbox runtime truth.
|
||||
|
||||
**Status:** in_progress (foundation + Stripe-webhook migration + WorkflowEngine row-trigger producer migration + form-submission mail/webhook bridges + CLI re-queue + e2e test shipped; remaining generic legacy mail/webhook/notification/audit web-action call-site migrations + admin UI deferred)
|
||||
**Task-ID:** #42
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Aktuell sind viele Side-Effects nicht transaktional mit DB-Writes gekoppelt:
|
||||
|
||||
- **Stripe-Webhook (`apps/api/src/billing/stripe-webhook-route.ts:60-138`):** `updateTenantSubscriptionFromStripe` updated DB → ruft danach `createLegacyEvent` → direkt callLegacyEvent (Mail, Webhook, Audit-Log). Wenn der Prozess zwischen Commit und `callLegacyEvent` crasht, ist DB konsistent aber Mail wurde nie versendet
|
||||
- **WorkflowEngine (`apps/web/app/modules/workflowEngine/*`):** This is now
|
||||
migrated for the row event producer path. `workflow.*` events route to
|
||||
`outbox.workflow`, the worker-general bridge starts matching workflows
|
||||
through the signed internal web API, and `row.created`, `row.updated`, and
|
||||
`row.deleted` are published in the same transaction as their row changes.
|
||||
Remaining WorkflowEngine gaps are semantic/runtime follow-ups: native
|
||||
branch/switch execution, delay/schedule blocks, and broader non-row producer
|
||||
coverage.
|
||||
- **Notification/Mail/Webhook-Outbound** aus Web-Actions: direkt aus dem Request-Handler triggered, bei Worker-Crash verloren
|
||||
|
||||
Das Outbox-Pattern löst das durch:
|
||||
1. Side-Effects werden **innerhalb der gleichen Transaction** wie der DB-Write als „Event"-Row in eine `OutboxEvent`-Tabelle geschrieben
|
||||
2. Ein separater **Dispatcher-Worker** liest pending Events, dispatched zu der jeweiligen Queue (mail, webhook, notification, audit-log), und markiert sie als processed
|
||||
3. Failures → Retries mit Exponential-Backoff, max-Retries, dann dead-letter
|
||||
4. At-least-once-Delivery garantiert; Receiver brauchen Idempotenz (Idempotency-Store aus #38)
|
||||
|
||||
Vorteile:
|
||||
- Transaction-Konsistenz: Event existiert ↔ DB-Change committed (atomic)
|
||||
- Crash-Safety: bei Crash zwischen Commit und Side-Effect wird Event beim Recovery dispatched
|
||||
- Audit-Trail: jede ausgelöste Aktion ist in DB sichtbar
|
||||
- Decoupling: Event-Produzent muss Receiver nicht kennen
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/read-write-split.md` — `publishOutboxEvent` läuft auf `prisma.write` (Outbox-Writes sind kritische Source-of-Truth, niemals Replica)
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Dispatcher nutzt BullMQ (gleiche Redis-Connection) für Queue-Dispatch
|
||||
- `02-observability/sentry-wiring-api-worker.md` — Dispatcher-Failures müssen captured werden
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/job-priority.md` — Dispatcher nutzt BullMQ-Priority für Event-Types (z.B. high für payment-success-mail)
|
||||
- `04-search-rag/embedding-pipeline.md` — Search-Indexer hängt an Outbox-Events (entity-created, entity-updated → re-index)
|
||||
- `05-business-features/marketing-funnel-phase-1.md` — Email-Sequence-Triggers via Outbox
|
||||
- alle nachfolgenden Features mit asynchronen Side-Effects
|
||||
|
||||
## Approach
|
||||
|
||||
**Neue Prisma-Tabelle:**
|
||||
|
||||
```prisma
|
||||
model OutboxEvent {
|
||||
id String @id @default(cuid())
|
||||
tenantId String? // null = system-event
|
||||
eventType String // z.B. "tenant.subscription.updated"
|
||||
payload Json
|
||||
status OutboxStatus @default(PENDING)
|
||||
createdAt DateTime @default(now())
|
||||
processedAt DateTime?
|
||||
retries Int @default(0)
|
||||
lastError String?
|
||||
scheduledAt DateTime @default(now()) // delay-fähig
|
||||
dispatchKey String? // für Idempotency / Dedup
|
||||
priority Int @default(0) // für #47
|
||||
|
||||
@@index([status, scheduledAt])
|
||||
@@index([tenantId, eventType])
|
||||
@@index([dispatchKey])
|
||||
}
|
||||
|
||||
enum OutboxStatus {
|
||||
PENDING
|
||||
PROCESSING
|
||||
DONE
|
||||
FAILED // alle retries verbraucht, dead-letter
|
||||
CANCELLED
|
||||
}
|
||||
```
|
||||
|
||||
**Publisher-Helper:**
|
||||
|
||||
```ts
|
||||
// packages/db/src/outbox.ts
|
||||
export async function publishOutboxEvent(
|
||||
tx: Prisma.TransactionClient,
|
||||
{ tenantId, type, payload, dispatchKey, scheduledAt, priority }: PublishArgs
|
||||
): Promise<OutboxEvent>
|
||||
```
|
||||
|
||||
Wird **nur innerhalb einer Transaction** aufgerufen — Typ-Constraint verhindert direkte Nutzung gegen `prisma`-Singleton.
|
||||
|
||||
**Dispatcher:**
|
||||
|
||||
Implementierungs-Variante A (gewählt): **BullMQ-Cron-Polling**, alle 1s pollt der Dispatcher-Worker pending Events (`WHERE status = 'PENDING' AND scheduledAt <= now()` LIMIT 100, ORDER BY priority DESC, createdAt ASC), markiert sie atomic als `PROCESSING` (via `UPDATE ... WHERE status = 'PENDING' RETURNING *` für Race-Condition-Safety), dispatched zu BullMQ-Queue (Queue-Name = Event-Type-Prefix, z.B. `mail`, `webhook`, `notification`, `audit-log`).
|
||||
|
||||
Variante B (verworfen für jetzt, aber dokumentiert): **LISTEN/NOTIFY-basiert.** Postgres-Trigger published auf `outbox_event_created`-Channel, Dispatcher listenened. Vorteil: niedrige Latency (<10ms statt 0-1s Polling). Nachteil: Trigger-Maintenance, Channel-Reconnect-Komplexität. Migration auf Variante B könnte später erfolgen wenn Latenz wichtig wird.
|
||||
|
||||
**Receiver-Idempotenz:**
|
||||
|
||||
Jeder Receiver (Mail-Worker, Webhook-Worker, etc.) prüft via `IdempotencyStore` (aus #38) ob `OutboxEvent.id` bereits verarbeitet wurde. Bei Retry: skip. Garantiert exactly-once-Effect bei at-least-once-Delivery.
|
||||
|
||||
**Retry-Strategie:**
|
||||
- Initial-Try: sofort
|
||||
- Retry-1: nach 1 min (exponential 2^0)
|
||||
- Retry-2: nach 2 min
|
||||
- Retry-3: nach 4 min
|
||||
- ... bis max-Retries (Default 8, also bis ~4h)
|
||||
- Danach: `status = FAILED`, manuelle Inspection oder Bulk-Re-Queue via CLI
|
||||
|
||||
**Dead-Letter-Handling:**
|
||||
- Admin-UI-Stub in `apps/web/app/routes/admin/outbox.tsx` zeigt failed Events
|
||||
- CLI-Tool `toolctrl-cli outbox requeue --event-id <id>` setzt ein einzelnes
|
||||
Event wieder auf PENDING. Bulk-Requeues laufen über
|
||||
`toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply]`.
|
||||
|
||||
**Migration:**
|
||||
|
||||
Status reconciliation (2026-05-27): Stripe webhook, row-event
|
||||
WorkflowEngine producers, CLI re-queue, and the opt-in outbox integration test
|
||||
have shipped. Notification/Mail/Webhook-Outbound migrations from remaining web
|
||||
actions and the Admin UI remain deferred.
|
||||
|
||||
1. **Stripe-Webhook (`apps/api/src/billing/stripe-webhook-route.ts:60-138`):**
|
||||
- Transaction wrappen: `prisma.write.$transaction(async (tx) => { await updateTenantSubscriptionFromStripe(tx, ...); await publishOutboxEvent(tx, { tenantId, type: 'tenant.subscription.updated', payload: {...} }); })`
|
||||
- `callLegacyEvent` direkt-Aufruf entfernen
|
||||
- Dispatcher routet Event zu `notification`-Queue (alter callLegacyEvent-Code wird Receiver)
|
||||
|
||||
2. **WorkflowEngine (`apps/web/app/modules/workflowEngine/*`):**
|
||||
- Trigger werden zu Outbox-Events (`type: 'workflow.trigger.<eventName>'`)
|
||||
- WorkflowEngine-Worker subscribed auf entsprechende Queue, executiert Workflow
|
||||
|
||||
3. **Notification / Mail / Webhook-Outbound aus Web-Actions:**
|
||||
- Web-Actions ersetzen direkte `sendEmail(...)`-Calls durch `await publishOutboxEvent(tx, { type: 'mail.send', payload: { to, template, vars }})`
|
||||
- Dispatcher routet zu `mail`-Queue, Mail-Worker (kommt aus #45) verarbeitet
|
||||
|
||||
**Warum nicht Alternativen:**
|
||||
- ❌ Saga-Pattern: Overkill für unsere Use-Cases, zu viel Orchestration
|
||||
- ❌ Event-Sourcing Full: massive Architektur-Umstellung, nicht in Phase 03 leistbar
|
||||
- ❌ CDC (Change Data Capture) via Debezium/wal2json: Infra-Overhead, kompliziertes Setup
|
||||
- ❌ Direkter Queue-Push aus Request-Handler (kein Outbox): bricht Transaction-Konsistenz, genau das Problem das wir lösen
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Prisma-Migration: OutboxEvent-Tabelle**
|
||||
- Files: `packages/db/prisma/schema.prisma`, neue Migration `prisma/migrations/<timestamp>_add_outbox_event/migration.sql`
|
||||
- Model + Enum wie oben
|
||||
- 3 Indizes für die Dispatcher-Queries
|
||||
- `pnpm prisma migrate dev --name add_outbox_event`
|
||||
- Commit: `03-scaling/outbox: add OutboxEvent model and migration`
|
||||
|
||||
2. **publishOutboxEvent-Helper**
|
||||
- Files: `packages/db/src/outbox.ts`, `packages/db/src/outbox.test.ts`
|
||||
- Signatur akzeptiert `Prisma.TransactionClient` (typed) — Aufrufer muss innerhalb $transaction sein
|
||||
- Validierung: eventType muss `<domain>.<noun>.<verb>`-Pattern matchen (z.B. `tenant.subscription.updated`)
|
||||
- DispatchKey-Validation (wenn gesetzt: max 256 Zeichen, alphanumeric)
|
||||
- Tests: erfolgreicher Insert, Race-Condition mit dispatchKey (unique-Konflikt sauber gehandled)
|
||||
- Commit: `03-scaling/outbox: add publishOutboxEvent helper with transaction enforcement`
|
||||
|
||||
3. **Dispatcher-Worker-Implementation**
|
||||
- Files: `apps/worker/src/outbox/dispatcher.ts`, `apps/worker/src/outbox/dispatcher.test.ts`, `apps/worker/src/outbox/event-router.ts`
|
||||
- Cron-Job (BullMQ-Repeatable) alle 1s
|
||||
- Claim-Query (atomic):
|
||||
```sql
|
||||
UPDATE "OutboxEvent"
|
||||
SET status = 'PROCESSING'
|
||||
WHERE id IN (
|
||||
SELECT id FROM "OutboxEvent"
|
||||
WHERE status = 'PENDING' AND "scheduledAt" <= NOW()
|
||||
ORDER BY priority DESC, "createdAt" ASC
|
||||
LIMIT 100
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *
|
||||
```
|
||||
- Pro claimed Event: dispatch via `event-router.ts` zu der richtigen BullMQ-Queue
|
||||
- On dispatch-success: mark `DONE`, set `processedAt`
|
||||
- On dispatch-error: increment retries, set `lastError`, `scheduledAt = now + backoff(retries)`. Wenn `retries >= max`: status `FAILED`
|
||||
- Tests: claim-race-condition (zwei Dispatcher-Instances, ein Event), retry-backoff-Berechnung, dead-letter-Übergang
|
||||
- Commit: `03-scaling/outbox: add dispatcher worker with skip-locked claim`
|
||||
|
||||
4. **Event-Router: Type → Queue-Mapping**
|
||||
- Files: `apps/worker/src/outbox/event-router.ts`
|
||||
- Config-basiertes Mapping:
|
||||
```ts
|
||||
const routing = {
|
||||
'mail.': 'mail',
|
||||
'webhook.': 'webhook',
|
||||
'notification.': 'notification',
|
||||
'audit.': 'audit-log',
|
||||
'workflow.': 'workflow',
|
||||
'tenant.subscription.': 'notification', // Composite-Routing
|
||||
'search.index.': 'search-index', // für Phase 04
|
||||
};
|
||||
```
|
||||
- Unknown-Type → DLQ + Sentry-Alert
|
||||
- Tests: alle Mappings, Fallback-Verhalten
|
||||
- Commit: `03-scaling/outbox: add event router with type to queue mapping`
|
||||
|
||||
5. **Migration: Stripe-Webhook**
|
||||
- Files: `apps/api/src/billing/stripe-webhook-route.ts`, neue Tests
|
||||
- Wrappen in `prisma.write.$transaction`
|
||||
- `callLegacyEvent`-Call durch `publishOutboxEvent` ersetzen
|
||||
- Receiver-Worker (`apps/worker/src/notification/legacy-event.worker.ts`) baut neuen Worker der den alten `callLegacyEvent`-Code als BullMQ-Handler ausführt
|
||||
- Tests: Stripe-Event-Mock → DB-Update + OutboxEvent existiert in derselben Transaction; bei Worker-Crash zwischen Steps → Outbox bleibt PENDING, beim nächsten Dispatch-Tick wird gepickt
|
||||
- Commit: `03-scaling/outbox: migrate stripe webhook to outbox pattern`
|
||||
|
||||
6. **Migration: WorkflowEngine-Triggers**
|
||||
- Files: `apps/web/app/modules/workflowEngine/routes/workflow-engine.api.server.tsx`, weitere Trigger-Stellen, neuer Worker `apps/worker/src/workflow/trigger.worker.ts`
|
||||
- Workflow-Trigger publishen Outbox-Event mit eventType `workflow.trigger.<eventName>`, payload enthält trigger-context
|
||||
- Worker konsumiert, lädt Workflow-Definition, executiert via existing Engine-Code
|
||||
- Tests: Trigger-Publish + Worker-Pickup; failed-Workflow markiert Event als FAILED nach max-retries
|
||||
- Commit: `03-scaling/outbox: migrate workflow engine triggers to outbox`
|
||||
|
||||
7. **Migration: Mail-Sending aus Web-Actions**
|
||||
- Files: alle `apps/web/app/routes/**/action.tsx`-Stellen die `sendEmail` direkt aufrufen, neuer Worker `apps/worker/src/mail/mail.worker.ts`
|
||||
- Inventarisierung via Subagent (Explore-Pass): liste alle `sendEmail`-Calls in Web
|
||||
- Pro Stelle: in Transaction wrappen + `publishOutboxEvent({ type: 'mail.send', payload: { to, template, vars }})` statt direkter Call
|
||||
- Mail-Worker: konsumiert `mail`-Queue, ruft Mail-Service (aus `packages/mail`)
|
||||
- Tests: Web-Action publisht Event, Worker verarbeitet, Mail wird gesendet (Mock-Mail-Provider)
|
||||
- Commit: `03-scaling/outbox: migrate web action mail sends to outbox`
|
||||
|
||||
8. **Migration: Webhook-Outbound aus Web-Actions**
|
||||
- Files: analog zu #7, aber für `dispatchWebhook`-Calls
|
||||
- Commit: `03-scaling/outbox: migrate web action webhook dispatches to outbox`
|
||||
|
||||
9. **Migration: Notification + Audit-Log aus Web-Actions**
|
||||
- Files: analog für `createNotification` und `auditLog`-Stellen
|
||||
- Commit: `03-scaling/outbox: migrate notification and audit log writes to outbox`
|
||||
|
||||
10. **Admin-UI für Outbox-Inspection**
|
||||
- Files: `apps/web/app/routes/admin/outbox.tsx`, `apps/web/app/routes/admin/outbox.$id.tsx`
|
||||
- Listing: Status-Filter, Type-Filter, Tenant-Filter, Pagination
|
||||
- Detail-View: Payload anzeigen, Retry-Manual-Button, Cancel-Button
|
||||
- Tests: Loader/Action, Berechtigungs-Check (admin-only)
|
||||
- Commit: `03-scaling/outbox: add admin ui for outbox event inspection`
|
||||
|
||||
11. **CLI-Tool für Re-Queue**
|
||||
- Files: `scripts/outbox/requeue.ts`, `package.json` script
|
||||
- Superseded by `toolctrl-cli outbox requeue --event-id <id>` for a single event
|
||||
- Superseded by `toolctrl-cli outbox requeue --type <eventType> --since <iso> [--apply]` for bulk
|
||||
- Tests: CLI-Smoke-Test
|
||||
- Commit: `03-scaling/outbox: add cli for requeueing failed events`
|
||||
|
||||
12. **Doku**
|
||||
- Files: `docs/architecture/outbox-pattern.md` (original plan named `docs/database/outbox-pattern.md`)
|
||||
- Sections:
|
||||
- **Konzept:** Warum Outbox? Transaction-Konsistenz, Crash-Safety, Decoupling
|
||||
- **Publisher-Usage:** Code-Beispiel für `publishOutboxEvent` in Transaction
|
||||
- **Event-Type-Konvention:** `<domain>.<noun>.<verb>`, Registry pflegen
|
||||
- **Dispatcher:** wie er pollt, claim-Strategy, Retry-Backoff, Dead-Letter
|
||||
- **Receiver-Idempotenz-Pflicht:** Code-Beispiel mit IdempotencyStore
|
||||
- **Operational:** Monitoring (Pending-Count-Alert > 1000), Admin-UI, CLI-Re-Queue
|
||||
- **Migration-Guide:** wie weitere Side-Effects migrieren
|
||||
- **Trade-offs:** Latenz (1s Polling-Tick), zukünftige LISTEN/NOTIFY-Migration
|
||||
- Commit: `03-scaling/outbox: add outbox pattern documentation`
|
||||
|
||||
13. **Integration-Test: End-to-End Outbox-Flow**
|
||||
- Files: `apps/worker/src/__integration__/outbox-e2e.test.ts`
|
||||
- testcontainers: Postgres + Redis + Mail-Mock-Service
|
||||
- Test-Cases:
|
||||
- Publisher → Dispatcher → Receiver → Effect (Mail-Mock empfängt)
|
||||
- Crash-Recovery: Receiver wirft Error → Event geht zurück auf PENDING mit retry+1
|
||||
- Dead-Letter: max-Retries → status FAILED, Admin-UI zeigt's
|
||||
- Race-Condition: 2 Dispatcher-Instances → kein Event wird doppelt dispatched
|
||||
- Idempotency: Receiver mit gleicher Event-ID 2x aufgerufen → Effekt nur 1x
|
||||
- Commit: `03-scaling/outbox: add end-to-end integration test suite`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** `publishOutboxEvent` (Transaction-Constraint, dispatchKey-Unique, Validation); Dispatcher (Claim-Logic, Retry-Backoff, Dead-Letter); Event-Router (alle Mappings)
|
||||
- **Integration-Tests:** End-to-End-Test (Step 13) — Publisher in API, Dispatcher im Worker, Receiver-Effekt verifiziert. testcontainers für Postgres + Redis
|
||||
- **Concurrency-Tests:** 2 Dispatcher-Instances parallel, 1000 Events — alle exakt 1x dispatched
|
||||
- **E2E-Tests (Playwright):** Admin-UI für Outbox-Inspection (Listing, Detail, Re-Queue-Button)
|
||||
- **Coverage-Target:** ≥ 90 % im Outbox-Code (Dispatcher, Helper, Router)
|
||||
- **Test-Files:**
|
||||
- `packages/db/src/outbox.test.ts`
|
||||
- `apps/worker/src/outbox/dispatcher.test.ts`
|
||||
- `apps/worker/src/outbox/event-router.test.ts`
|
||||
- `apps/worker/src/__integration__/outbox-e2e.test.ts`
|
||||
- `apps/web/app/routes/admin/outbox.test.tsx`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- Alle Tests grün, inkl. Concurrency + E2E
|
||||
- Staging-Verifikation: 24h-Lauf zeigt 0 unverarbeitete Events > 60s alt (Monitoring-Alert)
|
||||
- Stripe-Webhook-Test auf Staging: Crash zwischen DB-Commit und Mail-Send → nach Worker-Restart wird Mail gesendet
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `OutboxEvent`-Model in Prisma-Schema + Migration applied
|
||||
- [ ] `publishOutboxEvent`-Helper in `packages/db/src/outbox.ts`, akzeptiert nur TransactionClient
|
||||
- [ ] Dispatcher-Worker im `apps/worker` mit Cron-Polling (1s), atomic Claim via `FOR UPDATE SKIP LOCKED`
|
||||
- [ ] Event-Router mapped Event-Types zu BullMQ-Queues
|
||||
- [ ] Retry-Logic mit Exponential-Backoff (max 8 Retries), Dead-Letter bei Erschöpfung
|
||||
- [x] Stripe-Webhook migriert: DB-Update + Outbox-Event in einer Transaction; Worker-Crash zwischen Commit + Dispatch wird beim Recovery aufgeholt
|
||||
- [x] WorkflowEngine row-event triggers migriert; non-row producers remain follow-ups
|
||||
- [ ] Mail/Webhook/Notification/Audit-Log aus Web-Actions migriert
|
||||
- [ ] Admin-UI unter `/admin/outbox` listed pending/failed Events, erlaubt Re-Queue
|
||||
- [x] CLI `toolctrl-cli outbox requeue` funktioniert (single + bulk)
|
||||
- [x] `docs/architecture/outbox-pattern.md` enthält die current runtime/operator sections; the original `docs/database/outbox-pattern.md` target was superseded
|
||||
- [x] Opt-in integration test grün; Concurrency-Test (2 Dispatcher) bleibt offen
|
||||
- [ ] Staging-24h-Test: 0 unverarbeitete Events > 60s alt
|
||||
- [ ] Grafana-Panel: Outbox-Pending-Count, Dispatch-Rate, Retry-Rate, Failed-Count
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
**Per-Migration-Rollback:** Jede Migration (Steps 5-9) ist separat rollback-bar via `git revert`. Code-seitig fällt zurück auf direkte Side-Effect-Calls; OutboxEvent-Rows bleiben in DB (kein Daten-Verlust), werden ignoriert.
|
||||
|
||||
**Dispatcher-Disable:**
|
||||
- Env-Var `OUTBOX_DISPATCHER_ENABLED=false` im Worker → Cron stoppt, Events stauen sich
|
||||
- Useful für Maintenance oder Failure-Investigation
|
||||
- Pending-Count-Alert in Grafana fängt das ab
|
||||
|
||||
**Schema-Rollback:**
|
||||
- `prisma migrate resolve --rolled-back <migration>` + manuelle `DROP TABLE OutboxEvent`
|
||||
- Daten-Verlust für FAILED/PENDING Events — vor Rollback CSV-Export via `\COPY "OutboxEvent" TO ...`
|
||||
- Code muss vor Rollback zurück auf direct-Side-Effect-Calls (sonst breakt publishOutboxEvent)
|
||||
|
||||
**Receiver-Rollback einzeln:**
|
||||
- Wenn z.B. Mail-Worker buggy ist → BullMQ-Queue `mail` pausieren (`bull-board`-UI oder CLI), Events stauen sich harmlos, Fix deployen, Queue resumen
|
||||
|
||||
**DB-Migration:** non-destructive (nur CREATE TABLE + CREATE INDEX). Down-Migration: `DROP TABLE OutboxEvent CASCADE` (destructive — Daten-Verlust). Vor Down-Migration Backup ausführen.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Tenant-Isolation:** OutboxEvent hat `tenantId`. Receiver MUSS prüfen dass Payload-Operationen den richtigen Tenant treffen. Cross-Tenant-Spoofing-Risiko wenn Receiver Payload blind vertraut → Workflow-Worker validiert `payload.tenantId === event.tenantId`
|
||||
- **Payload-Validation:** Payloads sind `Json` — Receiver muss Zod-Schema-Validation pro Event-Type machen, sonst Crash bei Malformed
|
||||
- **Sensitive-Data im Payload:** keine Passwords, keine PII-Mehr-als-nötig. Email-Templates referenzieren statt embed
|
||||
- **Replay-Attacks:** dispatchKey + Idempotency-Store schützen. Receiver MUSS Idempotency-Check machen
|
||||
- **Dead-Letter-Access:** Admin-UI darf nur Admins lesen (RBAC-Check). Failed-Events können sensitive Daten enthalten (z.B. Stripe-Payload)
|
||||
- **Re-Queue-Berechtigung:** CLI + UI-Re-Queue nur für Admins. Manuelle Re-Queue eines Events kann unbeabsichtigte Effekte triggern (z.B. doppelte Mail) — Admin-UI zeigt Warnung
|
||||
- **DoS via Outbox-Spam:** Wenn ein bug 1000s Events pro Sekunde publisht → Dispatcher kommt nicht hinterher. Rate-Limit auf Publisher-Seite (max 100 Events pro Tenant pro Minute via SharedStore-Rate-Limiter)
|
||||
- **Backup-Implications:** OutboxEvent-Tabelle in Backup einbeziehen (sind kritische Source-of-Truth)
|
||||
- **OutboxEvent-Retention:** completed Events nach 30 Tagen löschen (Cron-Job), sonst Tabellen-Bloat. Failed-Events behalten (Compliance)
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit `apps/api/src/billing/stripe-webhook-route.ts:60-138` als Hauptbeispiel + Outbox-Pattern-Reference-Artikel
|
||||
- **Research:**
|
||||
- `Explore`-Subagent für „inventarisiere alle `sendEmail`, `dispatchWebhook`, `createNotification`, `auditLog`-Calls in `apps/web/`. Output: Migration-Backlog für Steps 7-9"
|
||||
- `Explore`-Subagent für „suche bestehende BullMQ-Queue-Setups in `apps/worker/` — was kann wiederverwendet werden?"
|
||||
- **Implementation:**
|
||||
- `general-purpose` für Steps 1-4 (Foundation: Schema + Helper + Dispatcher + Router)
|
||||
- Eigener `general-purpose` pro Migration-Step (5-9) — pro Bereich ein Subagent, weil Receiver-Worker-Code-Pattern jeweils unterschiedlich
|
||||
- `general-purpose` für Step 10 (Admin-UI)
|
||||
- `general-purpose` für Step 11 (CLI)
|
||||
- `general-purpose` für Step 12 (Doku — sehr ausführlich)
|
||||
- **Tests:** Separater `general-purpose` für E2E-Integration-Test (Step 13) — kompliziertes testcontainers-Setup, eigener Kontext
|
||||
- **Verification:** `gsd-verifier` für Acceptance-Criteria; manuelles 24h-Staging-Monitoring
|
||||
- **Code-Review:** `gsd-code-reviewer` für jeden Migration-Step separat (Stripe vs WorkflowEngine vs Mail haben unterschiedliche Risiken)
|
||||
|
||||
## Historical Staging-Deployment-Steps
|
||||
|
||||
This section is retained as the original implementation sketch. The current
|
||||
production topology uses separate persistent worker applications (`general`,
|
||||
`mail`, `import`, `webhook`, `plugin`) and `worker-general` owns the outbox
|
||||
dispatcher; do not provision the old `worker-dispatcher`/`WORKER_ROLE` layout.
|
||||
|
||||
1. Branch `agent/03-outbox-pattern` pushen, schrittweise Commits 1-13
|
||||
2. Coolify-Setup-Prerequisites: PgBouncer (#40), R/W-Split (#41), Shared-Store (#37/#38) müssen schon laufen
|
||||
3. Migration:
|
||||
- Pre-Deploy-Check: `pnpm prisma migrate status` auf Staging
|
||||
- Deploy mit `prisma migrate deploy` (Coolify-Pre-Start-Hook)
|
||||
- OutboxEvent-Tabelle erstellt, leer
|
||||
4. Worker-Replicas:
|
||||
- Historical only: Dispatcher-Worker als **separate Replica** vom General-Worker (skaliert unabhängig)
|
||||
- Historical only: Coolify-Service `worker-dispatcher` mit `process.env.WORKER_ROLE=dispatcher`
|
||||
- Historical only: Mindestens 1 Replica (Dispatcher ist idempotent via SKIP LOCKED, 2 Replicas sind safe für HA)
|
||||
- Historical receiver sketch: domain receivers such as mail, webhook, notification, audit-log, and workflow should run on the current worker groups unless a future dedicated receiver group is added. The current Coolify topology is `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin`.
|
||||
5. Env-Vars:
|
||||
```
|
||||
OUTBOX_DISPATCHER_ENABLED=true
|
||||
OUTBOX_DISPATCH_INTERVAL_MS=1000
|
||||
OUTBOX_DISPATCH_BATCH_SIZE=100
|
||||
OUTBOX_MAX_RETRIES=6
|
||||
OUTBOX_DONE_RETENTION_HOURS=720
|
||||
```
|
||||
6. Smoke-Tests:
|
||||
- **Stripe-Webhook-E2E:**
|
||||
1. Stripe-CLI: `stripe trigger customer.subscription.updated`
|
||||
2. Innerhalb 2s: `SELECT * FROM "OutboxEvent" WHERE "eventType" = 'tenant.subscription.updated' ORDER BY "createdAt" DESC LIMIT 1` → erwarte 1 Row, status PENDING oder DONE
|
||||
3. Nach 5s: Row sollte DONE sein
|
||||
4. Mail-Worker-Log zeigt: Mail an Tenant-Admin versendet
|
||||
- **Crash-Recovery-Test:**
|
||||
1. Stripe-CLI: trigger Event
|
||||
2. Sofort `docker kill worker-mail` (während Event noch PENDING)
|
||||
3. OutboxEvent bleibt PENDING (oder geht auf FAILED nach Retries — je nach Timing)
|
||||
4. `docker compose up worker-mail` neu starten
|
||||
5. Innerhalb 30s: Event wird verarbeitet, Mail gesendet
|
||||
- **Concurrency-Test:**
|
||||
1. 1000 Events via Script publishen
|
||||
2. Historical only: `docker scale worker-dispatcher=3`
|
||||
3. Verifiziere: jedes Event genau 1x in Receiver-Logs (über Event-ID grep)
|
||||
- **Dead-Letter-Test:**
|
||||
1. Mail-Mock so konfigurieren dass es immer wirft
|
||||
2. Event triggern, beobachten dass es nach 8 Retries auf FAILED geht
|
||||
3. Admin-UI `/admin/outbox` zeigt Event in Failed-Liste
|
||||
4. CLI: `toolctrl-cli outbox requeue --event-id <id>` → Event geht zurück auf PENDING
|
||||
7. Monitoring-Setup:
|
||||
- Grafana-Panels:
|
||||
- „Outbox Pending Count" (Alert: > 1000 für > 5min)
|
||||
- „Outbox Failed Count Daily" (Alert: > 50/day)
|
||||
- „Dispatcher Throughput" (events/sec)
|
||||
- „Receiver Latency P95" pro Queue
|
||||
8. 24h-Beobachtungs-Phase:
|
||||
- Logs auf Sentry-Errors prüfen
|
||||
- `SELECT status, COUNT(*) FROM "OutboxEvent" GROUP BY status` regelmäßig
|
||||
- Acceptance: 0 unverarbeitete Events > 60s alt im 24h-Fenster
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Step 5 (Stripe webhook) — DONE in 2026-05-25 session.** Wrapped subscription update + outbox publishes in a single `prisma.$transaction`; legacy `createLegacyEvent` direct call moved into the new `apps/worker/src/jobs/legacy-event-receiver.ts` consumer on `outbox.notification`, guarded by `withOutboxIdempotency`.
|
||||
- **Step 11 (CLI re-queue) — DONE in 2026-05-25 session.** `toolctrl-cli outbox requeue --event-id <id>` and `outbox requeue --type <eventType> --since <iso> [--apply]` ship under `packages/cli/src/commands/outbox/`. Two modes (single + bulk), dry-run by default for bulk, refuses to re-queue events already PENDING/PROCESSING, JSON output for scripting.
|
||||
- **Step 13 (end-to-end integration test) — DONE in 2026-05-25 session.** `packages/db/src/outbox-e2e.test.ts` exercises producer → claim → receiver → marker-row insert against real Postgres, with idempotency re-delivery assertion. Opt-in via `OUTBOX_E2E_DATABASE_URL` (or any `DATABASE_URL` that is not the vitest stub).
|
||||
- **Step 6 (WorkflowEngine triggers) — ROUTE ROW PRODUCERS LANDED in 2026-05-26 session.** `apps/worker/src/jobs/workflow-trigger.ts` now parses `workflow.*` outbox jobs, isolates redeliveries with `withOutboxIdempotency`, fans out to live event-trigger workflows, applies preflight, and starts executions through an injected executor port. `apps/worker/src/jobs/workflow-trigger-store.ts` adds the worker-native Prisma lookup/preflight store without importing web server modules, and `bootstrapWorkflowTrigger` is registered from `apps/worker/src/index.ts` for the `general` worker. Route-backed `row.created`, `row.updated`, and `row.deleted` producers publish `workflow.trigger.event` inside the same DB transaction as the row write and application event persistence when a request context is passed. Internal requestless mutations remain eventless by default to avoid worker feedback loops, while the direct internal trigger bridge remains the only allowed `type: "trigger"` execution call-site.
|
||||
- **Steps 7, 8, 9 (generic legacy mail/webhook/notification/audit-log from web actions) — DEFERRED.** Form-submission delivery is already outbox-backed: the general worker routes `form.submission.*` work and `worker-mail` bridges `webhook.form-submission.deliver` / `mail.transactional.form-submission` through the current `outbox.webhook` and `outbox.mail` subscribers. The remaining deferred work is the broader legacy action inventory outside that reviewed form-submission path. Reason: each generic migration touches a different module (`apps/web/app/routes/**/action.tsx`) and a different receiver worker pattern, and the operator scheduled them as separate per-domain code-review cycles. The foundation makes each one mechanical: wrap the action's DB write in `prisma.$transaction`, replace the direct side-effect call with `publishOutboxEvent(tx, { type: '<domain>.<entity>.<verb>', payload })`, and add a queue subscriber that mirrors `legacy-event-receiver.ts` (already merged this session). Acceptance criterion per migration: at least one Playwright assertion that crashing the worker between DB commit and queue-pop still delivers the side effect on restart.
|
||||
- **Step 10 (admin UI under `/app/admin/outbox`) — DEFERRED.** Reason: design decision (filters, detail view layout, re-queue confirm dialog) — needs UX input the operator hasn't provided yet. The CLI (`toolctrl-cli outbox requeue`) covers the operational hot-path until the UI lands. Acceptance criterion: list view with Status/Type/Tenant filters + pagination, detail view with payload + retry/cancel buttons, admin-only RBAC.
|
||||
- The dispatcher uses an in-process `setTimeout`-based polling loop rather than a BullMQ-repeatable cron job. This avoids needing a BullMQ Queue just to schedule the dispatcher itself, keeps the dispatcher independently testable with `vi.useFakeTimers()`, and matches the way other internal pollers (metrics-scheduler) are wired.
|
||||
- Default routing maps `billing.*` and `tenant.subscription.*` to `outbox.notification`. The Stripe-webhook migration uses `billing.subscription.ended` + `billing.subscription.cancelled` which land on `outbox.notification` via the existing `billing.` prefix rule — no router changes were needed.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 08:25 — 6ac38a11 — Add OutboxEvent prisma model + migration + regenerated client
|
||||
- 2026-05-17 08:30 — 62fa3436 — Add publishOutboxEvent publisher helper with validation + dispatchKey idempotency
|
||||
- 2026-05-17 08:36 — 1427a4ef — Add atomic claim-batch query (FOR UPDATE SKIP LOCKED) with integration tests
|
||||
- 2026-05-17 08:41 — 386dc9fc — Add dispatcher worker + event-router with exponential-backoff retry and dead-letter
|
||||
- 2026-05-17 08:49 — 5a19fdf3 — Register dispatcher in worker bootstrap (gated by OUTBOX_DISPATCHER_ENABLED)
|
||||
- 2026-05-17 08:52 — 9a79b212 — Add withOutboxIdempotency receiver helper for at-least-once safety
|
||||
- 2026-05-17 08:55 — 41c277ab — Document outbox pattern in docs/architecture/outbox-pattern.md + update progress logs
|
||||
- 2026-05-17 08:57 — 5a57e18a — Catalog OUTBOX_DISPATCHER_ENABLED / BATCH_SIZE / INTERVAL_MS / MAX_RETRIES and add to .env.example
|
||||
- 2026-05-25 — 0bebef90 — feat(outbox): migrate stripe webhook to outbox pattern (DB update + billing.subscription.{ended,cancelled} publish in single prisma.$transaction, new legacy-event-receiver consumer on outbox.notification guarded by withOutboxIdempotency, bootstrap wired into worker general-group with shutdown integration, stripe-webhook + receiver unit tests cover the new transactional + idempotent paths)
|
||||
- 2026-05-25 — 9e40294f — feat(cli): add outbox requeue command for failed and bulk replay (--event-id single mode, --type --since bulk mode with --apply confirm flag, refuses to re-queue PENDING/PROCESSING, JSON output, 15 unit tests)
|
||||
- 2026-05-25 — a5ac964b — test(outbox): end-to-end integration via real postgres opt-in (publisher → claim → custom routing → receiver → marker-insert → idempotent re-delivery; opt-in via OUTBOX_E2E_DATABASE_URL; suite passes live against local postgres and skips when no DB env is set)
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
_(Optional, am Ende des Tasks für nachfolgende Tasks)_
|
||||
@@ -0,0 +1,303 @@
|
||||
# PgBouncer Setup
|
||||
|
||||
**Status:** completed (PgBouncer deployed; web/api/worker `DATABASE_URL` via PgBouncer and `DIRECT_URL` direct to Postgres per operator checklist A4)
|
||||
**Task-ID:** #40
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 2 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Vor #40 hingen alle Prisma-Clients (apps/web, apps/api, apps/worker) direkt am Postgres-Primary. Pro Prozess öffnet Prisma standardmäßig bis zu **`num_physical_cpus + 1` Connections**, auf einem 4-Core-Container also 5 — pro Replica. Bei 2 Web + 1 API + 2 Worker = 25 Connections im Idle-Fall. Unter Last (lange Loader, Streaming-Queries) explodiert das schnell, weil Prisma pro Query eine eigene Connection holt bevor sie returned wird.
|
||||
|
||||
Postgres-Default `max_connections=100` ist bei ≥ 4 Replicas mit etwas Last (Dashboard-Loader, Bulk-Import) bereits ausgeschöpft. Symptome: `FATAL: sorry, too many clients already` im API-Log, sporadische 500er beim Login während Cron-Jobs laufen.
|
||||
|
||||
Lösung: **PgBouncer** als Connection-Pooler davor. Jede App-Replica hält ihre Prisma-Pool gegen PgBouncer (statt direkt Postgres), PgBouncer multiplexed das auf eine viel kleinere Anzahl Server-Connections.
|
||||
|
||||
Prisma-Specials:
|
||||
- Prisma nutzt **Prepared Statements** intensiv. Im `transaction`-Pool-Mode (was wir für Web brauchen) hält PgBouncer Server-Connections nur während einer Transaction, danach landet die nächste Query potenziell auf einer anderen Server-Connection → Prepared-Statements gehen kaputt
|
||||
- Prisma-Migrations müssen direkt gegen Postgres laufen (nicht über PgBouncer), weil sie DDL + Prepared-Statements für DDL nutzen
|
||||
- Lösung: `DATABASE_URL` zeigt auf PgBouncer (für Queries), `DIRECT_URL` zeigt direkt auf Postgres (für Migrations). Prisma `schema.prisma` deklariert beide.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `02-observability/metrics-grafana.md` — PgBouncer-Metrics werden in Grafana visualisiert; ohne Metrics-Infrastruktur können wir nicht beobachten, ob Pool tatsächlich hilft
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/read-write-split.md` — Read-Replica-Setup braucht DIRECT_URL-Konvention die hier etabliert wird
|
||||
- `03-scaling-infrastructure/backup-cli.md` — `pg_dump` muss DIRECT_URL nutzen (PgBouncer kann das nicht)
|
||||
- alle nachfolgenden Tasks die Prisma-Schema-Migrations laufen lassen
|
||||
|
||||
## Approach
|
||||
|
||||
**PgBouncer als Container vor Postgres**, mit `transaction`-Pool-Mode als Default. Prisma-Connection-Strings strikt aufgeteilt: `DATABASE_URL` (über PgBouncer, Port 6432) für Application-Queries, `DIRECT_URL` (direkt Postgres, Port 5432) für `prisma migrate` und alles was Prepared-Statement-Persistence braucht.
|
||||
|
||||
**Pool-Mode-Wahl:**
|
||||
- `session`: Connection bleibt für gesamte Client-Session reserviert — zu wenig Multiplexing, fast wie ohne Pooler. **Verworfen.**
|
||||
- `transaction`: Connection wird pro Transaction reserviert — Standard für Web-Workloads, beste Skalierung. **Gewählt.**
|
||||
- `statement`: Connection wird pro Statement reserviert — bricht alles was nicht-trivial ist (kein Multi-Statement-Tx, keine Prepared-Statements). **Verworfen.**
|
||||
|
||||
**Sizing-Default für Hetzner-Staging (4 vCPU Postgres):**
|
||||
- `default_pool_size = 25` (Server-Connections pro DB-User × DB-Name)
|
||||
- `max_client_conn = 1000` (Wie viele Frontend-Connections PgBouncer akzeptiert)
|
||||
- `reserve_pool_size = 5` (Burst-Buffer wenn default_pool gefüllt)
|
||||
- `pool_mode = transaction`
|
||||
|
||||
**Health-Check:** PgBouncer hat eingebaute virtuelle DB `pgbouncer` mit Commands `SHOW STATS`, `SHOW POOLS`, `SHOW LISTS`. Coolify-Health-Check pingt `psql -h pgbouncer -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS"`.
|
||||
|
||||
**Monitoring:** Prometheus-Exporter (`prometheus-pgbouncer-exporter`) sammelt Pool-Stats, in Grafana visualisiert.
|
||||
|
||||
**Deploy-Varianten:** Snippets für Coolify (Hetzner), Fly.io (managed Postgres + sidecar PgBouncer), Docker-Compose lokal.
|
||||
|
||||
**Warum nicht Alternativen:**
|
||||
- ❌ Prisma-Accelerate: Cloud-only, Vendor-Lockin, latency-overhead (HTTP statt TCP)
|
||||
- ❌ Prisma-Data-Proxy: deprecated zugunsten Accelerate
|
||||
- ❌ Application-Layer-Pool (z.B. generic-pool-Wrapper): zu invasiv, Prisma nutzt eh eigenen Pool, doppelte Buchhaltung
|
||||
- ❌ pgcat (Rust-PgBouncer-Replacement): jünger, weniger Battle-tested. Für später als Tuning-Option
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **PgBouncer-Container in docker-compose**
|
||||
- Files: `docker-compose.yml`, `docker-compose.staging.yml`, `infra/pgbouncer/pgbouncer.ini`, `infra/pgbouncer/userlist.txt.template`
|
||||
- Container `edoburu/pgbouncer:latest` (gut gepflegt, schlank)
|
||||
- Volume-Mount für config-files
|
||||
- Environment-Vars für DB_USER/DB_PASSWORD aus secrets
|
||||
- Network: gleicher Docker-Network wie Postgres + Apps
|
||||
- Expose nur intern (Port 6432 nur im Docker-Network, kein Public-Listen)
|
||||
- Commit: `03-scaling/pgbouncer: add pgbouncer container with transaction pool mode`
|
||||
|
||||
2. **pgbouncer.ini mit Tuning-Defaults**
|
||||
- Files: `infra/pgbouncer/pgbouncer.ini`
|
||||
- Komplette Konfiguration mit Comments:
|
||||
```ini
|
||||
[databases]
|
||||
toolctrl = host=postgres port=5432 dbname=toolctrl
|
||||
|
||||
[pgbouncer]
|
||||
listen_addr = 0.0.0.0
|
||||
listen_port = 6432
|
||||
pool_mode = transaction
|
||||
default_pool_size = 25
|
||||
max_client_conn = 1000
|
||||
reserve_pool_size = 5
|
||||
reserve_pool_timeout = 3
|
||||
server_idle_timeout = 600
|
||||
server_lifetime = 3600
|
||||
query_timeout = 0
|
||||
query_wait_timeout = 30
|
||||
server_reset_query = DISCARD ALL
|
||||
server_check_query = SELECT 1
|
||||
server_check_delay = 30
|
||||
auth_type = scram-sha-256
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
admin_users = pgb_admin
|
||||
stats_users = pgb_stats
|
||||
ignore_startup_parameters = extra_float_digits,search_path
|
||||
```
|
||||
- Commit: `03-scaling/pgbouncer: configure pgbouncer.ini with documented defaults`
|
||||
|
||||
3. **userlist.txt-Generation**
|
||||
- Files: `scripts/generate-pgbouncer-userlist.sh`, `infra/pgbouncer/userlist.txt.template`
|
||||
- Script generiert userlist.txt aus Env-Vars (`PG_USER`, `PG_PASSWORD`, `PGB_ADMIN_PASSWORD`, `PGB_STATS_PASSWORD`) im SCRAM-SHA-256-Format
|
||||
- Pre-Start-Hook im Container ruft Script auf
|
||||
- Tests: Bash-Test dass Script korrekt SCRAM-Strings generiert
|
||||
- Commit: `03-scaling/pgbouncer: add scram-sha-256 userlist generation script`
|
||||
|
||||
4. **Prisma-Migrate URL-Routing** — Current shipped implementation routes migrations through `prisma.config.ts`, not `schema.prisma directUrl`; Prisma 7 rejects `url`/`directUrl` in schema. Runtime PrismaClient reads `DATABASE_URL`, while migrate/db push/db pull/db seed use `DIRECT_URL` with fallback to `DATABASE_URL`.
|
||||
- Files: `prisma.config.ts`, `scripts/db/check-direct-url.sh`
|
||||
- Commit: `03-scaling/pgbouncer: route Prisma migrate through DIRECT_URL`
|
||||
|
||||
5. **Env-Var-Konvention dokumentieren + Beispiele aktualisieren**
|
||||
- Files: `.env.example`, `.env.production.example`, `packages/config/src/env-schema.ts` (oder analog)
|
||||
- Env-Vars:
|
||||
```
|
||||
# Application queries (via PgBouncer)
|
||||
DATABASE_URL=postgresql://app_user:pass@pgbouncer:6432/toolctrl?pgbouncer=true&connection_limit=10
|
||||
|
||||
# Migrations + introspection (direct to Postgres)
|
||||
DIRECT_URL=postgresql://app_user:pass@postgres:5432/toolctrl?connection_limit=5
|
||||
```
|
||||
- `pgbouncer=true`-Flag in URL tells Prisma to disable prepared-statements (kritisch für transaction-mode!)
|
||||
- `connection_limit=10` reduziert Prisma-internen Pool (PgBouncer pooled schon)
|
||||
- Zod-Schema-Validierung: beide URLs required in Production, DIRECT_URL optional in Dev (fallback DATABASE_URL)
|
||||
- Commit: `03-scaling/pgbouncer: document database and direct urls in env catalogue`
|
||||
|
||||
6. **Health-Check + Monitoring**
|
||||
- Files: `docker-compose.yml` (healthcheck-Section), `infra/pgbouncer/healthcheck.sh`, `infra/monitoring/prometheus-pgbouncer-exporter.yml`
|
||||
- Healthcheck-Script: `psql -h localhost -p 6432 -U pgb_stats pgbouncer -c "SHOW STATS" > /dev/null`
|
||||
- Prometheus-Exporter-Container `prometheuscommunity/pgbouncer-exporter`
|
||||
- Grafana-Dashboard-JSON in `infra/monitoring/grafana-dashboards/pgbouncer.json` (Pool-Usage, Wait-Time, Server-Connection-Count)
|
||||
- Commit: `03-scaling/pgbouncer: add healthcheck and prometheus exporter`
|
||||
|
||||
7. **Application-Update: Prisma-Clients verifizieren**
|
||||
- Files: `packages/db/src/client.ts`, alle App-Boot-Files
|
||||
- Sicherstellen dass `DATABASE_URL` aus Env genutzt wird (sollte schon sein); kein Hardcoded-Direct-URL irgendwo
|
||||
- Boot-Time-Validation: bei fehlendem `DIRECT_URL` in Production → Hard-Fail mit klarer Message
|
||||
- Tests: Boot-Tests mit beiden URLs gesetzt; Missing-DIRECT_URL-Test
|
||||
- Commit: `03-scaling/pgbouncer: validate database urls at boot time`
|
||||
|
||||
8. **Deploy-Variants dokumentieren**
|
||||
- Files: `docs/database/connection-pooling.md`
|
||||
- Sections:
|
||||
- **Coolify (Hetzner):** docker-compose-Snippet, Coolify-Service-Definition mit Health-Check
|
||||
- **Fly.io:** PgBouncer als Sidecar im selben `fly.toml`, oder managed Postgres mit eingebautem Pooler
|
||||
- **Bare Docker-Compose (Self-Host):** Standard-Snippet
|
||||
- **Tuning-Guidelines:** wie `default_pool_size` berechnen (Faustregel: `max_connections / N_app_instances`), wann auf `statement_mode` wechseln (read-only-replicas können das)
|
||||
- **Pool-Mode-Vergleich:** Session vs Transaction vs Statement (warum transaction Default ist)
|
||||
- **Monitoring:** `SHOW STATS`, `SHOW POOLS`, was alarmiert (cl_waiting > 0 ist Engpass)
|
||||
- Commit: `03-scaling/pgbouncer: add connection pooling deployment and tuning docs`
|
||||
|
||||
9. **Integration-Test gegen Container-Stack**
|
||||
- Files: `packages/db/src/__integration__/pgbouncer.test.ts`
|
||||
- testcontainers: Postgres + PgBouncer beide hochgezogen
|
||||
- Tests: Prisma-Client connected via PgBouncer kann Query laufen lassen; Prisma-Migrate gegen DIRECT_URL funktioniert; Prepared-Statement-Tests (mit `pgbouncer=true` muss `EXECUTE` fallback auf `EXECUTE_NOW` greifen)
|
||||
- Concurrency-Test: 50 parallele Queries, alle erfolgreich
|
||||
- Commit: `03-scaling/pgbouncer: add testcontainers integration suite`
|
||||
|
||||
10. **Staging-Deployment + Smoke-Tests**
|
||||
- siehe Staging-Deployment-Steps unten
|
||||
- Commit: `03-scaling/pgbouncer: enable pgbouncer in staging coolify deployment`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** Env-Schema-Validation (DATABASE_URL + DIRECT_URL Pflicht in Production); Boot-Time-Check
|
||||
- **Integration-Tests:** testcontainers-basiert mit Postgres + PgBouncer. Test-Suite muss:
|
||||
- Prisma-Client via PgBouncer kann normale Queries laufen
|
||||
- `prisma migrate dev` gegen DIRECT_URL erfolgreich
|
||||
- Prepared-Statements werden nicht persistent gehalten (verify durch zwei aufeinanderfolgende Queries die auf verschiedenen Server-Connections landen können)
|
||||
- Concurrency: 50 parallele Prisma-Calls funktionieren
|
||||
- **E2E-Tests:** keine neuen — bestehende Web-Tests müssen mit PgBouncer in Test-Compose grün bleiben
|
||||
- **Coverage-Target:** N/A (Infrastructure-Setup, Code-Änderungen minimal)
|
||||
- **Test-Files:** `packages/db/src/__integration__/pgbouncer.test.ts`; Smoke-Test-Script unter `scripts/smoke-test-pgbouncer.sh`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- testcontainers-Integration grün
|
||||
- Staging-Last-Test (k6 mit 100 RPS auf `/api/me`, 5 min) zeigt: PgBouncer `cl_active < 25`, `cl_waiting = 0`, keine `too many clients`-Errors
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] PgBouncer-Container in `docker-compose.yml` + `docker-compose.staging.yml` definiert
|
||||
- [ ] `infra/pgbouncer/pgbouncer.ini` mit dokumentierten Tuning-Defaults
|
||||
- [ ] `userlist.txt` wird zur Boot-Zeit aus Env-Vars generiert (SCRAM-SHA-256)
|
||||
- [ ] `packages/db/prisma/schema.prisma` deklariert `directUrl`
|
||||
- [ ] `.env.example` + `.env.production.example` enthalten beide URLs mit Comments
|
||||
- [ ] Boot-Time-Validation in allen Apps: fehlende `DIRECT_URL` in Production → Hard-Fail
|
||||
- [ ] Health-Check-Script funktioniert, in Coolify als Liveness/Readiness konfiguriert
|
||||
- [ ] Prometheus-Exporter sammelt PgBouncer-Stats; Grafana-Dashboard zeigt Pool-Usage
|
||||
- [ ] Integration-Test (testcontainers) grün
|
||||
- [ ] Staging-Last-Test: `SHOW STATS` zeigt `cl_active < default_pool_size`, `cl_waiting = 0` unter normaler Last
|
||||
- [ ] `docs/database/connection-pooling.md` enthält alle 4 Sections (Coolify/Fly/Docker/Tuning)
|
||||
- [ ] `pnpm prisma migrate dev` funktioniert lokal (gegen DIRECT_URL)
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
**PgBouncer ausschalten ohne Code-Änderungen:**
|
||||
- Env-Var ändern: `DATABASE_URL=postgresql://app_user:pass@postgres:5432/toolctrl` (Port 5432 statt 6432, kein `pgbouncer=true`)
|
||||
- App-Restart → bypasst PgBouncer komplett
|
||||
- DIRECT_URL bleibt unverändert
|
||||
- Connection-Limit (`connection_limit=10`) hochsetzen falls nötig (jetzt direkt gegen Postgres)
|
||||
|
||||
**Container entfernen:**
|
||||
- `docker compose down pgbouncer` → Container weg
|
||||
- Nur möglich nachdem DATABASE_URL umgestellt
|
||||
|
||||
**Tuning-Rollback bei Pool-Engpass:**
|
||||
- `default_pool_size` rauf, `pool_mode` auf `session` (wenn Prepared-Statement-Probleme), `reserve_pool_size` rauf
|
||||
- Bei `transaction`-Mode-Crash mit fremden Postgres-Features: temporär `session`-Mode + Migration zurück planen
|
||||
|
||||
**DB-Migrations:** keine in diesem Task (nur schema.prisma `directUrl` directive — non-destructive)
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **PgBouncer-Public-Listen verboten:** Container exposed Port 6432 nur im Docker-Network, kein `ports:` mapping nach außen. Anders als der Postgres-Container, der für Migrations vom Host erreichbar sein muss (DIRECT_URL aus dev-Host)
|
||||
- **SCRAM-SHA-256:** kein `md5` oder `plain` auth (md5 ist deprecated und unsicher)
|
||||
- **Admin-User getrennt:** `pgb_admin` und `pgb_stats` separat von App-User; Stats-User darf nur READ auf virtual `pgbouncer`-DB
|
||||
- **`server_reset_query = DISCARD ALL`:** kritisch — räumt Session-State (temp tables, cursors, prepared statements) zwischen Tx auf, verhindert Cross-Tenant-State-Leak im transaction-Mode
|
||||
- **`ignore_startup_parameters`:** wir whitelist nur `extra_float_digits` und `search_path` — andere unbekannte Parameter werden rejected (gegen Injection-Angriffe via Connection-String)
|
||||
- **`query_timeout = 0`** (kein Timeout in PgBouncer) — Timeouts kommen aus Application-Layer (Prisma). Setting auf > 0 würde Long-Running-Queries für Reports killen
|
||||
- **Userlist-Datei:** chmod 600, owned by pgbouncer-user. Generation-Script darf Passwort nicht in History oder Logs leaken (mit `set +x` rahmen)
|
||||
- **Audit:** PgBouncer-Logs auf `failed login`, `closed: socket error` monitoren — Anzeichen für Brute-Force oder Misconfig
|
||||
- **Postgres `max_connections`:** auf min. `default_pool_size × N_databases × 1.5` setzen (für Admin-Zugriff + Headroom)
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Prisma-Docs-Ausschnitt zu „PgBouncer + connection_limit + pgbouncer=true" + bestehender `docker-compose.yml` als Input
|
||||
- **Research:** `Explore`-Subagent für „suche nach `DATABASE_URL`-Referenzen in allen Apps, dokumentiere die Liste der Stellen die nicht über packages/db/src/client.ts gehen" — wir wollen wissen ob es Direct-Connect-Stellen gibt die wir übersehen
|
||||
- **Implementation:**
|
||||
- `general-purpose` für Steps 1-3 (Container + Config + userlist)
|
||||
- `general-purpose` für Steps 4-5 (Prisma-Schema + Env)
|
||||
- `general-purpose` für Steps 6-7 (Health + App-Validation)
|
||||
- `general-purpose` für Step 8 (Doku, weil umfangreich)
|
||||
- **Tests:** Separater `general-purpose` für Integration-Test (Step 9) — testcontainers-Setup ist nicht trivial
|
||||
- **Verification:** `gsd-verifier` + manueller k6-Last-Test auf Staging
|
||||
- **Code-Review:** `gsd-code-reviewer` mit Fokus auf Env-Var-Konvention und Security-Notes-Items
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-pgbouncer` pushen
|
||||
2. Coolify-Setup vorab:
|
||||
- Postgres-Container hat schon `max_connections=200` (defensive für Pool-Setup)
|
||||
- Coolify-Service `pgbouncer` neu anlegen mit Image `edoburu/pgbouncer:latest`, Network = toolctrl-net
|
||||
- Coolify-Env-Vars (für pgbouncer-Service): `PG_HOST=postgres`, `PG_PORT=5432`, `PG_DATABASE=toolctrl`, `PG_USER=...`, `PG_PASSWORD=...`, `PGB_ADMIN_PASSWORD=...`, `PGB_STATS_PASSWORD=...`
|
||||
- Coolify-Volume: `infra/pgbouncer/` als `/etc/pgbouncer/` mounted
|
||||
- Healthcheck-Command: `/healthcheck.sh` (aus Step 6)
|
||||
3. App-Env-Vars umstellen (Coolify):
|
||||
- `DATABASE_URL=postgresql://app_user:pass@pgbouncer:6432/toolctrl?pgbouncer=true&connection_limit=10`
|
||||
- `DIRECT_URL=postgresql://app_user:pass@postgres:5432/toolctrl?connection_limit=5` (neu)
|
||||
4. Coolify-Auto-Deploy triggert Rebuild aller Apps mit neuen Env-Vars
|
||||
5. Smoke-Tests nach Deploy:
|
||||
- **Connection-Test:** `docker exec apps-api node -e "require('@toolctrl/db').prisma.\$queryRaw\`SELECT 1\`.then(console.log)"` → expected `[ { '?column?': 1 } ]`
|
||||
- **PgBouncer-Stats:** `docker exec pgbouncer psql -h localhost -p 6432 -U pgb_stats pgbouncer -c "SHOW STATS"` → erwarte `total_xact_count > 0`
|
||||
- **Pool-Usage:** `SHOW POOLS` → erwarte `cl_active > 0`, `sv_active < default_pool_size`
|
||||
- **Migration-Test:** `pnpm prisma migrate status` (vom Coolify-Build-Container, sollte DIRECT_URL nutzen) → erwarte „Database schema is up to date"
|
||||
6. Last-Test (k6):
|
||||
```sh
|
||||
k6 run --vus 50 --duration 5m scripts/k6/api-me-load.js
|
||||
```
|
||||
- Während Test parallel `SHOW STATS` alle 30s
|
||||
- Acceptance: `cl_waiting = 0`, keine `too many connections`-Fehler in API-Logs
|
||||
7. Grafana-Dashboard live ansehen: Pool-Usage-Graph zeigt Sägezahn (typisches Pool-Pattern), `Server Connection Count` stabil unter `default_pool_size`
|
||||
8. Bei Issues: `docker logs pgbouncer | grep -E '(WARNING|ERROR|FATAL)'` für Misconfig-Hinweise
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Prisma 7 dropped `url` and `directUrl` from `schema.prisma`.** The
|
||||
`validate` command on Prisma 7.8 rejects both properties (P1012). The
|
||||
plan's step 4 prescribed adding `url = env("DATABASE_URL")` and
|
||||
`directUrl = env("DIRECT_URL")` to the datasource block. We instead
|
||||
route `DIRECT_URL` through `prisma.config.ts`
|
||||
(`datasource.url = env("DIRECT_URL" || "DATABASE_URL")`). Migrate now
|
||||
uses the unpooled URL while the PrismaClient at runtime continues to
|
||||
read `DATABASE_URL` from `process.env`. Documented in
|
||||
`docs/ops/pgbouncer.md`.
|
||||
- **Step 3 (SCRAM-SHA-256 userlist generation script)** is not shipped
|
||||
as `scripts/generate-pgbouncer-userlist.sh`. The ops runbook documents
|
||||
pulling the already-SCRAM-hashed entry straight out of
|
||||
`pg_authid.rolpassword`, since Postgres 16 stores the hash in the
|
||||
correct format when `password_encryption=scram-sha-256`. Avoids
|
||||
duplicating the SCRAM derivation logic in shell.
|
||||
- **2026-05-27 status reconciliation:** `docs/improvement-plan/OPERATOR-CHECKLIST.md`
|
||||
A4 verifies web/api/worker now use the PgBouncer `DATABASE_URL` and
|
||||
keep `DIRECT_URL` pointed at raw Postgres. This supersedes the stale
|
||||
app-DATABASE_URL-flip-pending caveat and matches #40 `completed` in
|
||||
`PROGRESS.md`.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 07:39 — `3d1e32d6` — add pgbouncer container to docker-compose (transaction pool mode, no host port)
|
||||
- 2026-05-17 07:43 — `b7c4f90c` — configure infra/pgbouncer/pgbouncer.ini with transaction-mode defaults + userlist.txt.template
|
||||
- 2026-05-17 07:45 — `fe8809b6` — route Prisma Migrate through DIRECT_URL via prisma.config.ts (schema.prisma `directUrl` not supported on Prisma 7 — see Deviations)
|
||||
- 2026-05-17 07:48 — `2bee026c` — update .env.example, .env.production.example, packages/config/src/catalog.ts to document DATABASE_URL (pgbouncer:6432) + DIRECT_URL (postgres:5432)
|
||||
- 2026-05-17 07:50 — `4147d4ef` — add docs/ops/pgbouncer.md runbook (Coolify deploy, tuning, troubleshooting, rollback)
|
||||
- 2026-05-17 07:52 — `5e26da4e` — add docker-compose.staging.yml overlay (Postgres+pgbouncer+app depends_on wiring)
|
||||
- 2026-05-17 07:55 — `8eec215b` — add scripts/db/check-direct-url.sh + wire into prisma:migrate:deploy
|
||||
- 2026-05-18 00:48 UTC — staging-deploy — `edoburu/pgbouncer:latest` container `boilerplate-pgbouncer` brought up on `/opt/pgbouncer/` via `docker compose up -d`, attached to the `coolify` network so apps can resolve it as `boilerplate-pgbouncer:6432`. `pgbouncer.ini` points `boilerplate_validation` + `boilerplate` databases at `rs3pdxczfhmxuqyx3k0uy2co:5432`. `userlist.txt` populated from `pg_authid.rolpassword` (SCRAM-SHA-256). Smoke green: `psql -h boilerplate-pgbouncer -p 6432 -U boilerplate -d boilerplate_validation -c "SELECT 1"` returned `1` row from inside the `coolify` network. The old `nc`-based healthcheck note is superseded by the current `pg_isready` healthcheck in `docker-compose.yml` and `docs/ops/pgbouncer.md`.
|
||||
- 2026-05-27 — docs/status reconciliation — A4 verifies the app env flip is live: web/api/worker `DATABASE_URL` goes through PgBouncer and `DIRECT_URL` goes direct to Postgres. The previous pending-operator note is superseded.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
_(Optional, am Ende des Tasks für nachfolgende Tasks)_
|
||||
+261
@@ -0,0 +1,261 @@
|
||||
# Pub/Sub Cache-Invalidation
|
||||
|
||||
**Status:** in_progress (subscription + tenant + feature-flag two-tier caches shipped; CLI cross-process publisher closed 2026-05-20; reconnect-flush + testcontainers integration closed 2026-05-25 — remaining work is the cross-app namespace fan-out documented in Deviations)
|
||||
**Task-ID:** #39
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 3-4 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Selbst nach #38 (Shared-Store-Migration) verbleibt ein Konsistenz-Problem: einige hot caches sollen weiterhin **lokal in-process** (cachified-LRU) gehalten werden, weil Redis-Roundtrips pro Request zu teuer sind. Beispiele:
|
||||
|
||||
- **Subscription-Cache (Web):** Pro Request mehrfach abgefragt (Entitlement-Checks, Limit-Counter, UI-Visibility). Redis-Roundtrip 1-3ms × 20 Calls = 20-60ms Overhead
|
||||
- **Tenant-Cache:** Pro Request fix
|
||||
- **Entitlements-Cache:** Pro Request fix
|
||||
- **Feature-Flag-Cache (kommt in Phase 05):** Pro UI-Render fix
|
||||
|
||||
Lösung: **Zwei-Stufiger Cache.** L1 = in-process cachified (schnell, nicht-shared), L2 = SharedStore (langsam, shared). Invalidations werden über Pub/Sub propagiert: wenn ein Event den Cache invalidieren soll, published derjenige, der die Quelle ändert (z.B. Stripe-Webhook in API), und ALLE Web-Replicas hören mit und purgen ihren L1.
|
||||
|
||||
Aktueller Zustand: Stripe-Webhook in `apps/api/src/billing/stripe-webhook-route.ts:60-138` updated DB + ruft `clearSubscriptionsCache(tenantId)` lokal in der API. Aber: der Cache liegt nicht in der API, sondern in **jeder Web-Replica**. Heute: Web-Cache wird erst nach Ablauf von 1h-TTL frisch. Resultat: User kann z.B. nach Plan-Upgrade noch bis zu 1h alte Limits sehen.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — liefert `createCacheStore` mit Pub/Sub-Invalidation
|
||||
- `03-scaling-infrastructure/shared-store-migration.md` — Subscription-Cache muss schon über SharedStore laufen, sonst kein Channel zum Subscriben
|
||||
- `02-observability/sentry-wiring-web.md` — Subscriber-Errors müssen captured werden, sonst tappt der Operator im Dunkeln wenn Subscriber stillschweigend stirbt
|
||||
|
||||
## Blocks
|
||||
|
||||
- `05-business-features/feature-flags.md` — Feature-Flag-Cache nutzt identisches Pattern
|
||||
- `03-scaling-infrastructure/tenant-snapshot.md` — Tenant-Reset published Invalidation für Tenant-Cache
|
||||
- alle nachfolgenden Caches mit Cross-Replica-Konsistenz-Anforderung
|
||||
|
||||
## Approach
|
||||
|
||||
**Pattern: Source-of-Truth-Owner published, Cache-Holder subscribed.**
|
||||
|
||||
1. **Publisher-Seite:** Wer immer die SoT ändert (API für Subscription via Stripe-Webhook, Worker für Tenant-Snapshot, etc.) ruft nach Commit `cache.invalidate(key)` auf dem SharedStore-Cache. Das published auf Channel `<namespace>:invalidate` mit Payload `{ key }`.
|
||||
|
||||
2. **Subscriber-Seite:** Jede Web-Replica (und API, wo sie auch L1 hält) startet beim Boot einen Subscriber pro relevantem Cache-Channel. Der Subscriber-Callback purged den lokalen L1 (cachified.delete) und optional auch den L2 (für Cleanup).
|
||||
|
||||
3. **Auto-Reconnect:** ioredis hat eingebauten Reconnect. Wir wrappen das mit einem `onReconnect`-Handler, der nach erfolgreichem Reconnect einmal alle L1-Caches komplett flushed (`cache.clearAll()`), weil wir nicht wissen welche Invalidations während Disconnect verloren gingen.
|
||||
|
||||
4. **Replay-fähig:** Pub/Sub hat keine Persistence — bei Disconnect verloren. Mitigation: L1-TTL bleibt kurz (5-15min), so dass im Worst-Case-Disconnect-Szenario nach kurzer Zeit auto-refresh greift. Für strikte Konsistenz: Outbox-Pattern (#42) für kritische Events, dann Worker konsumiert Outbox und published Invalidation. Doppelte Sicherheit.
|
||||
|
||||
5. **Multi-Process-Setup:**
|
||||
- **API (1 Replica):** Publisher für Subscription, Tenant, Entitlements
|
||||
- **Web (2+ Replicas):** Subscriber für alle obigen
|
||||
- **Worker (2+ Replicas):** Publisher für lange laufende Updates (Bulk-Imports, Tenant-Snapshot-Restore)
|
||||
|
||||
## Vier konkrete Caches migrieren
|
||||
|
||||
### A. Subscription-Cache (Stripe)
|
||||
- **Source:** `apps/api/src/billing/stripe-webhook-route.ts:60-138`
|
||||
- **Trigger:** Stripe-Event (`customer.subscription.updated`, `invoice.payment_succeeded`, etc.)
|
||||
- **Channel:** `cache:subscription:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** API in Webhook-Handler nach `updateTenantSubscriptionFromStripe`
|
||||
- **Subscriber:** Web (jede Replica) ruft `clearSubscriptionsCache(tenantId)`
|
||||
|
||||
### B. Entitlements-Cache
|
||||
- **Source:** `apps/web/app/utils/entitlements/.server/productAccess.server.ts`
|
||||
- **Trigger:** Subscription-Change (impliziert Entitlement-Change), Manual-Admin-Edit
|
||||
- **Channel:** `cache:entitlements:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** Identisch zur Subscription-Trigger-Stelle (gleiche Webhook-Handler) + Admin-Edit-Route
|
||||
- **Subscriber:** Web purged in-process Entitlements-Map
|
||||
|
||||
### C. Tenant-Cache
|
||||
- **Source:** `apps/web/app/utils/data/.server/appData.ts`
|
||||
- **Trigger:** Tenant-Settings-Update, Tenant-Logo-Upload, Member-Add/Remove
|
||||
- **Channel:** `cache:tenant:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** Web (eigene Action) UND API (bei API-getriebener Mutation)
|
||||
- **Subscriber:** Alle Web-Replicas + API (wo Tenant gelesen wird)
|
||||
|
||||
### D. Feature-Flag-Cache (Vorbereitung für #53)
|
||||
- **Source:** kommt in Phase 05
|
||||
- **Channel:** `cache:feature-flag:invalidate`
|
||||
- **Payload:** `{ flagKey?: string; tenantId?: string; all?: boolean }`
|
||||
- **Publisher:** Admin-Edit in Web
|
||||
- **Subscriber:** Web + API + Worker
|
||||
- In diesem Task nur das **Channel-Naming + Subscriber-Stub** vorbereiten, damit #53 nur noch den Publisher hinzufügt
|
||||
|
||||
**Warum nicht andere Pattern:**
|
||||
- ❌ Polling: schwergewichtig, schlechte Reaktionszeit
|
||||
- ❌ Cache-TTL niedrig halten (5s): Hammers gegen Redis und DB
|
||||
- ❌ WebSocket-basierte Push-Invalidation: Overkill, wir haben Redis schon
|
||||
- ❌ Kafka/NATS für ein einziges Use-Case: zu viel Infra
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Subscriber-Lifecycle-Framework**
|
||||
- Files: `packages/shared-store/src/helpers/subscriber-lifecycle.ts`, `subscriber-lifecycle.test.ts`
|
||||
- `createSubscriberRegistry({ store })` mit `register(channel, handler)` + `start()` + `stop()`
|
||||
- Auto-Reconnect-Handler: bei Redis-Reconnect alle registered Handler einmalig mit `{ type: 'flush' }` aufrufen
|
||||
- Sentry-Integration: jeder Subscriber-Error wird captured + re-thrown an unhandled (Subscribers dürfen nicht stillschweigend sterben)
|
||||
- Tests: register/start/stop, Reconnect-Flush, Error-Capture
|
||||
- Commit: `03-scaling/pubsub: add subscriber lifecycle framework with reconnect handling`
|
||||
|
||||
2. **App-Boot-Integration Web**
|
||||
- Files: `apps/web/app/entry.server.tsx` (oder analoger Boot-Hook), `apps/web/app/utils/subscribers/index.server.ts`
|
||||
- SubscriberRegistry beim Boot starten, registriert alle Web-Subscriber (Subscription, Entitlements, Tenant, Feature-Flag-Stub)
|
||||
- Graceful-Shutdown-Handler: `process.on('SIGTERM', () => registry.stop())`
|
||||
- Tests: Boot-Integration-Test (Mock-Redis, check dass alle Channels subscribed sind)
|
||||
- Commit: `03-scaling/pubsub: wire subscriber registry into web boot`
|
||||
|
||||
3. **App-Boot-Integration API**
|
||||
- Files: `apps/api/src/server.ts`, `apps/api/src/subscribers/index.ts`
|
||||
- Analog Web, API hat eigene Subscriber-Liste (Tenant + Feature-Flag)
|
||||
- Commit: `03-scaling/pubsub: wire subscriber registry into api boot`
|
||||
|
||||
4. **Subscription-Cache: Publisher in Stripe-Webhook**
|
||||
- Files: `apps/api/src/billing/stripe-webhook-route.ts`, Test
|
||||
- Nach `updateTenantSubscriptionFromStripe` → `subscriptionCache.invalidate(tenantId)` (cache aus #38)
|
||||
- Tests: Mock Redis, verify Pub/Sub-Message wurde published
|
||||
- Commit: `03-scaling/pubsub: publish subscription invalidation from stripe webhook`
|
||||
|
||||
5. **Subscription-Cache: Subscriber in Web**
|
||||
- Files: `apps/web/app/utils/subscribers/subscription-cache.subscriber.ts`, Test
|
||||
- Handler ruft `clearSubscriptionsCache(tenantId)` + zusätzlich `subscriptionCache.invalidate(tenantId, { localOnly: true })` für L1
|
||||
- Integration-Test: 2 Web-Instances, Webhook auf API → beide Web-Instances sehen neuen Wert < 2s
|
||||
- Commit: `03-scaling/pubsub: subscribe subscription cache invalidation in web`
|
||||
|
||||
6. **Entitlements-Cache: Publisher + Subscriber**
|
||||
- Files: `apps/web/app/utils/entitlements/.server/productAccess.server.ts`, `apps/web/app/utils/subscribers/entitlements-cache.subscriber.ts`, Tests
|
||||
- Publisher in API (gleiche Webhook-Stelle wie Subscription, weil korreliert)
|
||||
- Subscriber in Web purged in-process Entitlements-Map
|
||||
- Commit: `03-scaling/pubsub: add entitlements cache pub/sub channel`
|
||||
|
||||
7. **Tenant-Cache: Publisher + Subscriber**
|
||||
- Files: `apps/web/app/utils/data/.server/appData.ts`, `apps/web/app/utils/subscribers/tenant-cache.subscriber.ts`, Tests
|
||||
- Publisher in Web-Action für Tenant-Settings-Update; API-Side wo Tenant-Mutation passiert
|
||||
- Subscriber sowohl Web als auch API
|
||||
- Commit: `03-scaling/pubsub: add tenant cache pub/sub channel`
|
||||
|
||||
8. **Feature-Flag-Cache-Stub (für #53)**
|
||||
- Files: `apps/web/app/utils/subscribers/feature-flag-cache.subscriber.ts` (Stub), `apps/api/src/subscribers/feature-flag-cache.subscriber.ts` (Stub)
|
||||
- Subscriber registered Handler, der derzeit nur loggt (`"feature-flag cache invalidation received, publisher not yet implemented"`)
|
||||
- Ermöglicht #53, nur Publisher-Seite zu ergänzen
|
||||
- Commit: `03-scaling/pubsub: add feature-flag cache subscriber stubs`
|
||||
|
||||
9. **Multi-Process-Integration-Test**
|
||||
- Files: `apps/web/__integration__/cross-replica-invalidation.test.ts`
|
||||
- testcontainers spinnt 1 Redis-Container + 2 in-process Web-Subscriber-Instances + 1 in-process Publisher
|
||||
- Test-Cases: invalidate triggert beide Subscriber; Reconnect-Flush nach Disconnect-Simulation; Concurrent-Invalidation
|
||||
- Commit: `03-scaling/pubsub: add multi-process invalidation integration tests`
|
||||
|
||||
10. **Doku**
|
||||
- Files: `docs/observability/redis-stores.md` (Erweiterung): Sektion „Pub/Sub-Channels" mit allen 4 Channels, Owner-Tabelle (wer published, wer subscribed)
|
||||
- Operational-Runbook: „Cache-Invalidation funktioniert nicht — Debug-Steps" (Redis-MONITOR, Subscriber-Health-Check)
|
||||
- Commit: `03-scaling/pubsub: document pub/sub channels and runbook`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** SubscriberRegistry (register, start, stop, reconnect-flush, error-capture); jeder einzelne Subscriber-Handler (purged korrekt)
|
||||
- **Integration-Tests:** Cross-Replica-Invalidation pro Channel (Multi-Process-Setup via testcontainers); Reconnect-Verhalten (kill Redis, restart, verify Flush)
|
||||
- **E2E-Tests (Playwright):** Stripe-Webhook-Simulation → check ob Web-UI direkt neuen Subscription-Status zeigt (Replication-Test)
|
||||
- **Coverage-Target:** ≥ 90 % im Subscriber-Code, ≥ 85 % an Publisher-Stellen
|
||||
- **Test-Files:** Subscriber-Files co-located; Integration in `apps/web/__integration__/` und `apps/api/src/__integration__/`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- Multi-Process-Integration-Test grün
|
||||
- Staging-Test: Stripe-Webhook auf Replica A → Web-Replica B + C sehen neuen Wert < 2s
|
||||
- Reconnect-Test auf Staging: `docker restart redis` → nach 30s sind alle Subscriber wieder online
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] SubscriberRegistry-Framework in `packages/shared-store/src/helpers/subscriber-lifecycle.ts` implementiert
|
||||
- [ ] Web + API booten SubscriberRegistry, registrieren alle relevanten Channels
|
||||
- [ ] Vier Channels live: `cache:subscription:invalidate`, `cache:entitlements:invalidate`, `cache:tenant:invalidate`, `cache:feature-flag:invalidate`
|
||||
- [ ] Stripe-Webhook published Subscription + Entitlements-Invalidation nach DB-Update
|
||||
- [ ] Tenant-Mutation-Stellen published Tenant-Invalidation
|
||||
- [ ] Feature-Flag-Subscriber-Stubs vorhanden (für #53)
|
||||
- [ ] Reconnect-Flush funktioniert: nach Redis-Restart sind alle L1-Caches geleert
|
||||
- [ ] Multi-Process-Integration-Test grün (2 Subscriber-Instances, ein Publisher, alle Channels)
|
||||
- [ ] Staging-Verifikation: End-to-End-Latenz Stripe-Event → Web-Cache-Invalidation < 2s
|
||||
- [ ] `docs/observability/redis-stores.md` enthält Pub/Sub-Channel-Tabelle + Operational-Runbook
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
Pro Channel separat rollback-bar:
|
||||
|
||||
- **Subscription-Channel:** Revert Publisher- + Subscriber-Commits → zurück zu 1h-Stale-Verhalten, funktional
|
||||
- **Entitlements/Tenant:** Analog, zurück zu vorherigem Pattern (z.B. Loader-Refetch oder Page-Refresh)
|
||||
- **SubscriberRegistry-Framework:** Wenn das ganze Framework Probleme macht → Boot-Integration revert, Subscribers schweigen, Caches werden nicht invalidiert. Mitigation: L1-TTL kurz (5-15min) als Safety-Net
|
||||
|
||||
**Global-Kill-Switch:** Env-Var `DISABLE_CACHE_PUBSUB=true` → SubscriberRegistry started nicht, Publisher no-op. Erlaubt schnelles Disable ohne Redeploy.
|
||||
|
||||
**DB-Migrations:** keine in diesem Task.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Pub/Sub-Message-Validation:** Subscriber-Handler validieren Payload strict (Zod-Schema pro Channel), reject Malformed mit Sentry-Capture. Verhindert Crash bei Garbage-Messages
|
||||
- **Channel-Spoofing-Risk:** Wer Redis-Zugriff hat, kann beliebig invalidieren. Mitigation: Redis-AUTH zwingend in Production, kein Public-Listen, Network-Isolation
|
||||
- **Cache-Stampede bei Mass-Invalidation:** Wenn Angreifer `cache:tenant:invalidate` für alle Tenants published → alle Replicas refetchen aus DB. Mitigation: Subscriber-Handler implementiert lokales Debounce (max 1 invalidation pro key pro 500ms)
|
||||
- **Sensitive-Data in Invalidation-Payload:** Payload enthält nur tenantId (UUID), keine PII. Klare Regel: Pub/Sub-Channels sind nicht zum Datentransport, nur Invalidation
|
||||
- **Disconnect-During-Critical-Operation:** Wenn Redis während Stripe-Webhook down ist, schlägt Publisher fehl. Mitigation: Outbox-Pattern (#42) puffert Events, Dispatcher retried Publish bis Redis wieder up
|
||||
- **Sentry-Breadcrumbs:** SubscriberRegistry meldet jede Disconnect/Reconnect-Event als Breadcrumb, damit Stale-Cache-Bugs in Produktion debugbar sind
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Channel-Tabelle (A-D) + bestehender `subscriptionService.ts`-Implementation als Input
|
||||
- **Research:** `Explore`-Subagent für „aktuelle cachified-Nutzung in `apps/web/app/utils/services/.server/` und `apps/web/app/utils/data/.server/` — welche Caches haben non-trivial Konsistenz-Anforderungen?"
|
||||
- **Implementation:** Ein `general-purpose`-Subagent pro Channel (A-D), max 1 Channel pro Subagent — Publisher + Subscriber + Test gehören zusammen
|
||||
- **Framework-Subagent:** Separater `general-purpose`-Subagent für Schritte 1-3 (SubscriberRegistry-Framework + Boot-Integration), weil Framework-Bug alle Channels bricht
|
||||
- **Tests:** `general-purpose`-Subagent für Multi-Process-Integration-Test mit testcontainers (Step 9)
|
||||
- **Verification:** `gsd-verifier` für Acceptance-Criteria + Staging-Latenz-Messung
|
||||
- **Code-Review:** `gsd-code-reviewer` nach allen Channels — Fokus auf Reconnect-Pfade und Error-Handling im Subscriber
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-pubsub-cache-invalidation` pushen
|
||||
2. Voraussetzung: #38 muss auf Staging erfolgreich getestet sein (Subscription läuft schon über SharedStore)
|
||||
3. Coolify-Setup:
|
||||
- **2x Web-Replicas** (Cross-Replica-Test braucht ≥ 2)
|
||||
- **1x API-Replica** (Publisher-Seite)
|
||||
- Redis-Container läuft schon aus #37/#38
|
||||
4. Env-Var setzen: `REDIS_PUBSUB_DB=2` (eigener logischer Index)
|
||||
5. Smoke-Tests:
|
||||
- **End-to-End-Stripe-Test:**
|
||||
1. Stripe-CLI: `stripe trigger customer.subscription.updated --customer cus_xxx`
|
||||
2. Innerhalb 2s: GET `/api/tenants/<id>/subscription` auf Web-Replica A → erwarte neuen Status
|
||||
3. Gleiche GET auf Web-Replica B → erwarte ebenfalls neuen Status
|
||||
- **Reconnect-Test:**
|
||||
1. `docker exec redis redis-cli SHUTDOWN NOSAVE`
|
||||
2. Warte 5s, Web-Replicas loggen Disconnect
|
||||
3. `docker compose up redis` neu starten
|
||||
4. Innerhalb 30s sehen Logs `"subscriber reconnected, flushing local caches"`
|
||||
5. Stripe-Test wiederholen, muss wieder funktionieren
|
||||
- **Pub/Sub-Monitor:** Auf Staging `docker exec redis redis-cli -n 2 PSUBSCRIBE '*'` laufen lassen, parallel Webhook triggern → verifiziere dass Message published wird
|
||||
6. Sentry-Check: Keine Subscriber-Errors in Issues-Liste
|
||||
7. Coolify-Logs prüfen: Beim Boot jeder Web-Replica steht `"subscriber registry started: 4 channels"`
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Entitlements cache** is NOT a separate two-tier cache. The repo has no in-process entitlements cache today — `resolveTenantProductAccess` (`apps/web/app/utils/entitlements/.server/productAccess.server.ts`) recomputes from the subscription on every call. Entitlement freshness piggy-backs on subscription invalidation: once `cache:subscription:invalidate` clears the subscription L1, the next entitlement check pulls a fresh subscription and derives a fresh decision. Channel inventory in `docs/architecture/two-tier-cache.md` documents the chain.
|
||||
- ~~**Feature-Flag-Cache subscriber stub** was NOT shipped.~~ — superseded: Phase 05 #53 has since shipped the full feature-flag two-tier cache (`apps/web/app/utils/services/.server/featureFlagCache.server.ts` + `subscribeFeatureFlagCache()` in `cache-bootstrap.server.ts`) and the web-route mutators all call `invalidateFlagCache(name)` after the DB write. **2026-05-20 follow-up `9edc1357`** closed the last cross-process gap by giving the operator CLI a one-shot publisher (`publishFeatureFlagInvalidationOnce` in `@toolctrl/feature-flags`, wired into `toolctrl-cli feature-flags {enable,disable,set-rules}` via `broadcastFlagInvalidation`); without it the CLI-driven mutation left long-lived web replicas serving stale evaluations from L1 until the 60s TTL expired.
|
||||
- **Subscriber-lifecycle framework** was NOT extracted as a separate `createSubscriberRegistry`. Each two-tier cache exposes its own idempotent `subscribe()` method (singleton-guarded). `apps/web/app/cache-bootstrap.server.ts` starts the two subscribers in parallel and logs errors via `webLogger`. A registry can be extracted later if we exceed ~4 channels.
|
||||
- ~~**Reconnect-flush** is provided as `flushLocal()` on the two-tier cache but not yet wired to a Redis `'ready'` event handler — the L1 TTL safety-net (default 300s via `CACHE_L1_TTL_SECONDS`) covers reconnect-loss scenarios.~~ — **resolved 2026-05-25 (`47c37b88`)**. `SharedStoreAdapter.onReconnect(handler)` now exposes ioredis `'ready'`-after-disconnect events (noop on `MemoryAdapter`); `createTwoTierCache(...).subscribe()` automatically registers a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also removes the reconnect handler.
|
||||
- **Cross-app fan-out (API publisher → Web subscriber)** uses per-app SharedStore namespaces (`api`, `web`). For broadcasts to reach across processes both apps must share the same SharedStore namespace; this is documented as a follow-up in `docs/architecture/two-tier-cache.md`. The local-only mutation paths (web-side `clearTenantSubscriptionCache`) still publish on the web channel so in-app fan-out across web replicas works today. **2026-05-20** — CLI feature-flag mutators sidestep the per-app namespace mismatch by explicitly publishing on the `web` namespace (configurable via `FEATURE_FLAGS_PUBLISH_NAMESPACE`), proving the pattern that the broader API↔Web platform-namespace refactor can adopt.
|
||||
- ~~**Multi-Process testcontainers integration test** (Step 9) is deferred. In-process tests already prove cross-replica fan-out by routing two `SharedStore` facades through one `MemoryAdapter`, which is the same fan-out path Redis exercises.~~ — **resolved 2026-05-25 (`f4b290bb`)**. New `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` brings up a `redis:7-alpine` container via `testcontainers` (or honours `PUBSUB_TEST_REDIS_URL`) and exercises cross-instance pub/sub fan-out, reconnect-flush, and peer-L2-mutation-after-reconnect. Skipped by default (no env var + no Docker) — opt-in mirrors the `packages/db/src/outbox-claim.test.ts` pattern. Verified live against the local Docker daemon (3/3 green).
|
||||
|
||||
## Progress Log
|
||||
|
||||
- **2026-05-17** — Commit `db117362` — `03-scaling/pubsub-cache: add two-tier cache helper combining L1 cachified + L2 shared-store`. New `createTwoTierCache<T>` in `packages/shared-store/src/helpers/two-tier-cache.ts` pairs an in-process L1 (Map default, swappable for LRUCache) with a SharedStore L2 + pub/sub channel. Public surface: `get/set/invalidate/subscribe/deleteLocal/flushLocal` + `l1`/`l2`/`namespace` accessors. `pubsubEnabled: false` flips `invalidate` to local-only + makes `subscribe()` a no-op. 9 unit tests cover round-trip, L2 fallback, cross-replica fan-out, localOnly, wildcard flush, kill-switch, custom subscriber-extra-handler. Exported from `@toolctrl/shared-store`.
|
||||
- **2026-05-17** — Commit `b258ba0c` — `03-scaling/pubsub-cache: migrate subscription + tenant caches to two-tier with pub/sub`. New `apps/web/app/utils/services/.server/subscriptionCache.server.ts` (subscription namespace) and `apps/web/app/utils/db/tenant-cache.server.ts` (tenant namespace). Existing cachified LRU keeps the hot read-path; the new SharedStore L2 + pub/sub broadcasts cross-replica invalidations. `clearTenantSubscriptionCache(tenantId)` now also calls `getSubscriptionCache().invalidate(tenantId)`; subscribers bridge into `clearCacheKey('tenantSubscription:<id>')` / `clearCacheKey('tenant:<id>'/'tenantSimple:<id>'/'tenantIdOrSlug:<id>')`. New `apps/web/app/cache-bootstrap.server.ts` is wired into `entry.server.tsx` and is idempotent + env-gated. 12 new unit tests (5 subscriptionCache + 3 tenant-cache + 4 subscriptionService updates).
|
||||
- **2026-05-17** — Commit `15bf34d5` — `03-scaling/pubsub-cache: publish subscription invalidation from api stripe webhook`. New `apps/api/src/cache/subscription-cache-publisher.ts` exposes `publishSubscriptionInvalidation(tenantId)`. Stripe webhook handler calls it fire-and-forget after `updateTenantSubscriptionProductFromStripe`. Publish errors are logged via `apiLogger.warn` so a Redis outage can never 5xx the webhook (Stripe would retry). New test asserts publisher fires with the correct tenantId.
|
||||
- **2026-05-17** — `03-scaling/pubsub-cache: env vars, docs, progress log` (this commit). Adds `CACHE_INVALIDATION_ENABLED` + `CACHE_L1_TTL_SECONDS` to `packages/config/src/catalog.ts` and `.env.example`. Adds `docs/architecture/two-tier-cache.md` covering pattern decision rubric, channel naming, reconnect semantics, channel inventory, implementation recipe, feature-flag pre-written recipe for #53, kill-switch operational guidance.
|
||||
- **2026-05-20** — Commit `9edc1357` — `feat(feature-flags+cli): cross-process invalidation publisher (#39)`. New `publishFeatureFlagInvalidationOnce` + `publishFeatureFlagInvalidationViaStore` exports in `@toolctrl/feature-flags` provide a one-shot SharedStore-open → publish → close lifecycle so short-lived processes (CLI, scripts) can broadcast `cache:feature-flags:invalidate` without keeping a long-lived Redis client. `toolctrl-cli feature-flags {enable,disable,set-rules}` now call `broadcastFlagInvalidation(name)` after the DB mutation; channel namespace defaults to `web` and honours `FEATURE_FLAGS_PUBLISH_NAMESPACE` env override. 9 new unit tests in `packages/feature-flags/src/publisher.test.ts` + `packages/cli/src/commands/feature-flags/invalidate.test.ts` cover the wire format, env-namespace resolution, kill-switch path, and error-swallowing contract. Web + feature-flags + CLI typecheck green.
|
||||
- **2026-05-25** — Commits `47c37b88` + `f4b290bb` — closes the two remaining Deviations. `47c37b88` (`feat(shared-store): flush L1 caches on redis reconnect`) adds `SharedStoreAdapter.onReconnect(handler)` to the adapter surface (ioredis `'ready'`-after-first-connect on `RedisAdapter`, noop on `MemoryAdapter`), re-exposes it on the `SharedStore` facade, and wires `createTwoTierCache().subscribe()` to register a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also unregisters the reconnect handler. New unit test in `two-tier-cache.test.ts` exercises the wiring through a proxy adapter (fire-then-flush, debounce-collapses-bursts, disposer-removes-handler). `f4b290bb` (`test(shared-store): real-redis testcontainers integration for two-tier cache`) adds `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` with three integration scenarios (cross-instance pub/sub fan-out, reconnect-flush, peer-L2-mutation-visible-after-reconnect). Opt-in via `PUBSUB_TEST_REDIS_URL` or a reachable Docker daemon; module-top-level boot so `describe.skipIf` sees a concrete decision pre-registration. Verified live against `redis:7-alpine` via testcontainers — 3/3 green. `testcontainers ^10.28.0` added as devDependency of `@toolctrl/shared-store`. Workspace typecheck + full test suite green (only pre-existing `uiInventory` failure remains).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
- The two-tier helper's `invalidate({ localOnly: true })` mode is the bridge that lets the subscriber callback purge local L1 without re-publishing — important to avoid invalidation loops.
|
||||
- Keeping the legacy `cachified` LRU as the read path (rather than ripping it out) made the migration zero-risk: the new two-tier cache is purely additive (it owns the pub/sub channel) and existing route handlers see no behaviour change.
|
||||
- vitest 0.29 lacks `vi.hoisted()`; for ESM mock factories that need a shared mock fn the workaround is to attach the fn to `globalThis` and reference it via getter inside the factory.
|
||||
- Concurrent multi-agent setups need a serializer for shared files (config catalog, .env.example, PROGRESS.md). Stashes + autostash-pulls race, and another agent's `git stash`+`git reset` can wipe unstaged changes. Mitigation: `git add` immediately after any edit to a shared file so the change lives in the index rather than the working tree.
|
||||
@@ -0,0 +1,306 @@
|
||||
# Read-Write-Split via Prisma $extends
|
||||
|
||||
**Status:** in_progress (code-complete; read-replica deferred by operator; single-DB fallback live)
|
||||
**Task-ID:** #41
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Sämtliche Prisma-Queries (auch reine Read-Loaders für Dashboards, KB-Public-Seiten, Analytics, Listings) gehen heute gegen den Primary-Postgres. Das Primary ist gleichzeitig Write-Target — bei einem Bulk-Write (Import von 10k CRM-Records) verlangsamen sich parallele Reads spürbar. Mit dem PgBouncer-Setup aus #40 ist das Connection-Limit zwar entschärft, aber Disk-I/O und CPU-Last bleiben auf einer Maschine.
|
||||
|
||||
Postgres unterstützt Streaming-Replication out of the box. Eine Read-Replica entlastet den Primary für alle Read-Only-Workloads. Voraussetzung: die Application muss bewusst entscheiden, ob eine Query an Replica (eventually consistent, Lag bis ~1s) oder Primary (strikt aktuell) geht.
|
||||
|
||||
Aktuelle Hot-Read-Pfade (Stichproben):
|
||||
- `apps/web/app/modules/crm/db/*.db.server.ts` — List-Queries (Contacts, Companies, Deals)
|
||||
- `apps/web/app/modules/knowledgeBase/db/*.db.server.ts` — Public KB-Routes (hohe Lese-Last, kein Konsistenz-Bedarf)
|
||||
- `apps/web/app/modules/analytics/*` — Dashboard-Loaders (aggregierte Reports, oft minutenlang gecached)
|
||||
- `apps/web/app/routes/app.$tenant/dashboard.tsx` — Loader pro Page-Load
|
||||
- `apps/web/app/utils/data/.server/appData.ts` — Read-Side (Tenant-Lookups, Member-Listings)
|
||||
- `apps/web/app/modules/sites/*` und `apps/web/app/modules/portals/*` — Public-Facing Loaders (Marketing-Sites, Customer-Portals)
|
||||
|
||||
Klassische Write-Pfade müssen explizit Primary nutzen: Mutations, Session-Writes, Stripe-Webhook-Updates, Outbox-Writes (siehe #42).
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/pgbouncer.md` — `DIRECT_URL`-Konvention und Pool-Setup ist Voraussetzung; Read-Replica läuft optional auch über eigenen PgBouncer
|
||||
- `02-observability/metrics-grafana.md` — Read/Write-Verteilung wird über Prisma-Middleware-Counter erfasst und in Grafana visualisiert
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/outbox-pattern.md` — Outbox-Writes müssen explizit Primary nutzen, Konvention wird hier etabliert
|
||||
- `03-scaling-infrastructure/tenant-snapshot.md` — Snapshot-Reads können von Replica kommen, Restore-Writes Primary
|
||||
- `04-search-rag/embedding-pipeline.md` — Indexer liest große Mengen — gut für Replica
|
||||
- alle nachfolgenden Read-Heavy-Features
|
||||
|
||||
## Approach
|
||||
|
||||
**Zwei Prisma-Clients hinter sauberem Interface.**
|
||||
|
||||
1. **`packages/db/src/client.ts` zentral umbauen:**
|
||||
```ts
|
||||
const writeUrl = env.DATABASE_URL;
|
||||
const readUrl = env.DATABASE_READ_URL ?? writeUrl; // Fallback
|
||||
|
||||
const prismaWrite = new PrismaClient({ datasources: { db: { url: writeUrl } } });
|
||||
const prismaRead = readUrl === writeUrl
|
||||
? prismaWrite // Same instance, no double connection
|
||||
: new PrismaClient({ datasources: { db: { url: readUrl } } });
|
||||
|
||||
export const prisma = prismaWrite.$extends({
|
||||
name: 'rw-split',
|
||||
client: {
|
||||
get write() { return prismaWrite; },
|
||||
get read() { return prismaRead; },
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
2. **Konvention:** `prisma.write.user.findUnique(...)` für strikte Konsistenz, `prisma.read.user.findUnique(...)` für eventually-consistent.
|
||||
|
||||
3. **Migration der Hot-Read-Pfade** (siehe Liste oben) explizit pro Funktion auf `prisma.read` umstellen. **Kein Default-Routing** — wir wollen bewusste Entscheidung pro Call.
|
||||
|
||||
4. **Telemetry:** Prisma-Middleware zählt pro Client (write vs read), exposed via OpenTelemetry-Counter `prisma.queries.total{client="read|write"}`. Grafana-Dashboard zeigt Verteilung.
|
||||
|
||||
5. **Fallback-Logik:** Wenn `DATABASE_READ_URL` nicht gesetzt ist → Read-Client = Write-Client, keine Replication-Nutzung. So bleibt Code-Konvention auch in Dev ohne Replica funktional.
|
||||
|
||||
6. **Test-Setup:**
|
||||
- In CI gegen Single-DB (read = write Fallback) → kein Replica-Mock nötig, alle Tests grün bleiben
|
||||
- Optional Integration-Test gegen Replica via testcontainers (Postgres-Primary + Postgres-Standby mit Streaming-Replication)
|
||||
|
||||
**Warum dieses Pattern:**
|
||||
- Prisma `$extends` ist offizieller Mechanismus, type-safe, keine Magic
|
||||
- Explizite Auswahl pro Call statt Auto-Routing (z.B. nach Operation-Type) — Auto-Routing macht Konsistenz-Bugs schwer auffindbar
|
||||
- Fallback-Konstruktion: Code bleibt funktional auch ohne Replica (Dev/Self-Hosted-Setups ohne Replica-Setup)
|
||||
|
||||
**Warum nicht Alternativen:**
|
||||
- ❌ Prisma-Middleware mit Auto-Routing nach Operation-Type: Bricht „read-after-write" in einer Request-Lifecycle (z.B. nach `create` direkt `findUnique` auf Replica → Race)
|
||||
- ❌ Separate Module pro DB-Access (read-models package): zu viel Doppelung, schwer zu maintainen
|
||||
- ❌ Externe Library wie prisma-replicas: Maintenance unsicher, wir haben nur einen Use-Case
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Two-Client-Setup in packages/db**
|
||||
- Files: `packages/db/src/client.ts`, `packages/db/src/client.test.ts`
|
||||
- Implementierung wie oben skizziert, mit Fallback wenn READ_URL nicht gesetzt
|
||||
- Type-Test: `prisma.read.user.findFirst()` und `prisma.write.user.findFirst()` beide typed
|
||||
- Tests: Fallback-Verhalten, Singleton-Pattern, Beide-Clients-Different bei gesetzter READ_URL
|
||||
- Commit: `03-scaling/rw-split: add prisma write/read clients with fallback`
|
||||
|
||||
2. **Env-Var-Schema erweitern**
|
||||
- Files: `packages/config/src/env-schema.ts` (oder analog), `.env.example`, `.env.production.example`
|
||||
- `DATABASE_READ_URL` als optional, Zod-Schema
|
||||
- Comment: „Optional Postgres Read-Replica URL. Wenn leer, fallen alle Reads auf Primary zurück (DATABASE_URL)."
|
||||
- Validation: wenn gesetzt, muss valides postgres:// URL sein
|
||||
- Commit: `03-scaling/rw-split: add DATABASE_READ_URL to env catalogue`
|
||||
|
||||
3. **Telemetry-Middleware**
|
||||
- Files: `packages/db/src/telemetry.ts`, `telemetry.test.ts`
|
||||
- Prisma-Middleware (`$use`) registriert pro Client-Instance, zählt Queries (counter), Duration (histogram), labels mit `model`, `operation`, `client`
|
||||
- Integration mit OpenTelemetry-Metrics aus Phase 02
|
||||
- Tests: Counter incrementiert korrekt, Labels gesetzt
|
||||
- Commit: `03-scaling/rw-split: add prisma telemetry middleware for read/write split observability`
|
||||
|
||||
4. **Migration: apps/web/app/modules/crm/db/*.db.server.ts**
|
||||
- Files: alle List-Query-Funktionen in `apps/web/app/modules/crm/db/`
|
||||
- Pro Funktion: wenn Read-Only (z.B. `getContacts`, `getCompanies`, `getDealsForBoard`) → `prisma.read`. Mutations bleiben `prisma.write` (oder default-`prisma` weil das = `prisma.write`)
|
||||
- Pro Funktion bewusster Choice — Subagent dokumentiert Entscheidung pro Funktion in Commit-Message
|
||||
- Tests: bestehende Tests laufen weiter; Single-DB-Fallback macht keinen Unterschied
|
||||
- Commit: `03-scaling/rw-split: route crm list queries to read replica`
|
||||
|
||||
5. **Migration: apps/web/app/modules/knowledgeBase/db/*.db.server.ts**
|
||||
- Files: KB-DB-Files
|
||||
- Public-KB-Routes haben höchste Read-Last, beste Migration-Candidate
|
||||
- Mutations (KB-Editor) bleiben Write
|
||||
- Commit: `03-scaling/rw-split: route knowledge base public reads to replica`
|
||||
|
||||
6. **Migration: apps/web/app/modules/analytics/***
|
||||
- Files: alle Analytics-Loader
|
||||
- Aggregations + Reports sind Read-Only, oft langsame Queries → ideal für Replica
|
||||
- Commit: `03-scaling/rw-split: route analytics dashboard queries to read replica`
|
||||
|
||||
7. **Migration: apps/web/app/routes/app.$tenant/dashboard.tsx**
|
||||
- Files: Dashboard-Route + verlinkter Loader-Code
|
||||
- Dashboard-Loader pure Read → `prisma.read`
|
||||
- Commit: `03-scaling/rw-split: route tenant dashboard loader to read replica`
|
||||
|
||||
8. **Migration: apps/web/app/utils/data/.server/appData.ts**
|
||||
- Files: `appData.ts` (Read-Side)
|
||||
- Tenant/Member-Lookups sind Read-Only — Subscription-Cache (#38) cached oben drauf
|
||||
- **Vorsicht:** wenn nach Login direkt `getUserAndTenants` aufgerufen wird, hat Replica evtl. den frisch erstellten User noch nicht (Replication-Lag). Für Auth-Pfade weiterhin `prisma.write` nutzen, dokumentieren
|
||||
- Commit: `03-scaling/rw-split: route appdata read side to replica with auth path exception`
|
||||
|
||||
9. **Migration: apps/web/app/modules/sites/* + portals/***
|
||||
- Files: alle Site/Portal-Loaders
|
||||
- Public-Facing, hohe Last, Read-Only → `prisma.read`
|
||||
- Commit: `03-scaling/rw-split: route public sites and portals to read replica`
|
||||
|
||||
10. **Telemetry-Verifikation + Grafana-Dashboard**
|
||||
- Files: `infra/monitoring/grafana-dashboards/prisma-rw-split.json`
|
||||
- Panels: Total Queries pro Client, Latency-P95 pro Client, Queries pro Model+Operation
|
||||
- Doku-Update: `docs/observability/redis-stores.md` ergänzen um „wir beobachten read/write-split via Grafana"
|
||||
- Commit: `03-scaling/rw-split: add grafana dashboard for read/write split observability`
|
||||
|
||||
11. **Doku**
|
||||
- Files: `docs/database/read-replicas.md`
|
||||
- Sections:
|
||||
- **Setup:** wie Postgres-Streaming-Replication aufgesetzt wird (`primary_conninfo`, `hot_standby = on`)
|
||||
- **Coolify/Hetzner-Setup:** zwei Postgres-Container, eines als Standby, `pg_basebackup`-Init
|
||||
- **Fly.io:** managed Postgres-Replicas
|
||||
- **Konvention:** „Use read for X, write for Y" — Tabelle:
|
||||
- **Use `prisma.read` für:** Listings, Dashboards, Public-Pages, Reports, Search-Result-Display
|
||||
- **Use `prisma.write` für:** Mutations, Auth-Pfade (Login/Session), Stripe-Webhook-Reads-die-Writes-vorangehen, Outbox-Reads im Dispatcher
|
||||
- **Replication-Lag-Considerations:** typisch < 1s, aber Burst-Writes können Lag auf 5-10s pushen. Read-after-write nicht garantiert auf Replica
|
||||
- **Monitoring:** `pg_stat_replication` auf Primary; `pg_last_xact_replay_timestamp()` auf Replica
|
||||
- **Failover:** Manuell (Coolify-Restart von Standby zu Primary); automatisch via Patroni/repmgr (out of scope für jetzt)
|
||||
- Commit: `03-scaling/rw-split: add read replicas documentation with usage guidelines`
|
||||
|
||||
12. **Integration-Test (optional, gated)**
|
||||
- Files: `packages/db/src/__integration__/rw-split.test.ts`
|
||||
- testcontainers mit Postgres-Primary + Postgres-Standby (Streaming-Replication)
|
||||
- Tests: Write auf Primary → Read auf Replica nach 100ms (mit kleinem Sleep, lebensnah)
|
||||
- Gated über `TEST_REPLICA=1`-Env-Var (nicht in default-CI weil Replica-Setup teuer)
|
||||
- Commit: `03-scaling/rw-split: add optional integration test with replicated postgres`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** `client.ts` Fallback-Logik (READ_URL gesetzt vs nicht); Telemetry-Middleware Counter
|
||||
- **Integration-Tests:** Bestehende CI-Suite läuft weiter (Single-DB-Fallback macht keinen Unterschied); optionaler Replica-Test gated über Env-Var
|
||||
- **E2E-Tests:** keine neuen, bestehende E2E müssen grün bleiben
|
||||
- **Manual-Test:** auf Staging mit Replica: read-Query auf Replica, write-Query auf Primary, verify via `pg_stat_activity`
|
||||
- **Coverage-Target:** keine Regression; `client.ts` + `telemetry.ts` ≥ 90 %
|
||||
- **Test-Files:** `packages/db/src/client.test.ts`, `packages/db/src/telemetry.test.ts`, `packages/db/src/__integration__/rw-split.test.ts`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- Bestehende Tests alle grün
|
||||
- Migration-Subagent hat pro Funktion in Commit-Message dokumentiert WARUM read oder write gewählt wurde
|
||||
- Staging-Test (mit Replica) zeigt: > 60 % der Dashboard-Reads gehen an Replica (Grafana-Panel)
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `packages/db/src/client.ts` exportiert `prisma` mit `.write` und `.read` Accessors via `$extends`
|
||||
- [ ] Fallback funktioniert: ohne `DATABASE_READ_URL` ist `prisma.read === prisma.write`
|
||||
- [ ] Env-Schema kennt `DATABASE_READ_URL` als optional, dokumentiert
|
||||
- [ ] Telemetry-Middleware aktiv auf beiden Clients, OpenTelemetry-Metrics live
|
||||
- [ ] Alle 6 genannten Hot-Path-Module nutzen `prisma.read` für reine Read-Queries
|
||||
- [ ] Auth-Pfade in `appData.ts` bleiben bewusst auf `prisma.write` (read-after-write Garantie)
|
||||
- [ ] Grafana-Dashboard `prisma-rw-split.json` exists und visualisiert die Verteilung
|
||||
- [ ] Staging-Verifikation: Read-Replica-Container hochgezogen, > 60 % Dashboard-Reads landen auf Replica
|
||||
- [ ] `docs/database/read-replicas.md` enthält Setup, Konvention, Tabelle „Use read for X, write for Y", Lag-Considerations
|
||||
- [ ] Bestehende Test-Suite grün (Single-DB-Fallback in CI)
|
||||
- [ ] Optional: Integration-Test mit replizierten Postgres-Containern (gated, läuft in nightly-Job)
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
**Pro Modul-Migration separat rollback-bar:** `git revert <migration-commit>` → Modul fällt zurück auf `prisma.write` (= Primary). Kein Datenverlust, kein DB-Schema-Change.
|
||||
|
||||
**Global-Disable:** `DATABASE_READ_URL` unsetzen + App-Restart → Read-Client = Write-Client. Replica wird ignoriert, alles geht an Primary. Keine Code-Änderung nötig.
|
||||
|
||||
**Replica-down-Szenario:** Wenn Replica ausfällt:
|
||||
- Prisma-Read-Client wirft Connection-Error
|
||||
- Sentry-Alert
|
||||
- Manuell `DATABASE_READ_URL` auf Primary umstellen + Restart → läuft weiter (degraded Performance)
|
||||
- Future: Auto-Fallback in `client.ts` (mit Circuit-Breaker) — out of scope für jetzt
|
||||
|
||||
**DB-Schema:** keine Migrations in diesem Task.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Read-Replica-Credentials:** separater User mit READ-ONLY-Rechten auf Replica (`GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_user`). Selbst wenn `prisma.read` versehentlich für Write genutzt wird → Postgres rejected
|
||||
- **Connection-String-Handling:** `DATABASE_READ_URL` darf nie geloggt werden (Passwort)
|
||||
- **Replication-Slot-Security:** Streaming-Replication-User braucht REPLICATION-Recht, das ist sensibel. Separater `replicator`-User, nur von Replica-IP erlaubt
|
||||
- **Cross-Tenant-Data-Leak via Replica-Lag:** wenn Tenant-Switch direkt nach Write erfolgt und Loader auf Replica liest, könnte alte Tenant-ID-Daten gelesen werden. Mitigation: Auth-Pfade strikt `prisma.write`. Code-Review check
|
||||
- **PII-Considerations:** Replica hat full data copy. Backup-/Snapshot-Policy für Replica gleich strikt wie Primary
|
||||
- **DSGVO:** wenn EU-Customer-Data: Replica im selben Land hosten. Hetzner-Falkenstein für Primary, Hetzner-Helsinki für Replica wäre OK (beide EU)
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit `packages/db/src/client.ts` als Input + Prisma-`$extends`-Docs — soll Type-Safety prüfen
|
||||
- **Research:** `Explore`-Subagent für „Liste alle Prisma-Aufrufe in `apps/web/app/modules/`, kategorisiere als Read-Only vs Write" — Output ist Migration-Backlog für Steps 4-9
|
||||
- **Implementation:**
|
||||
- `general-purpose` für Step 1-3 (Foundation: Client + Env + Telemetry)
|
||||
- Ein eigener `general-purpose` pro Migration-Step (4-9), weil pro Modul Tradeoffs überlegt werden müssen — der Subagent muss pro Funktion Read/Write begründen
|
||||
- `general-purpose` für Step 11 (Doku, umfangreich)
|
||||
- **Tests:** Separater `general-purpose` für Integration-Test mit testcontainers + Streaming-Replication (Step 12)
|
||||
- **Verification:** `gsd-verifier` für Acceptance-Criteria; Grafana-Panel-Check als manueller Final-Verify
|
||||
- **Code-Review:** `gsd-code-reviewer` für alle Migration-Commits — Read/Write-Choices reviewen, Auth-Pfad-Ausnahmen prüfen
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-rw-split` pushen, schrittweise Commits 1-12
|
||||
2. Coolify-Setup für Postgres-Replica:
|
||||
- Neuen Coolify-Service `postgres-replica` mit Image `postgres:16-alpine`
|
||||
- Custom-Init-Script in `infra/postgres/init-replica.sh`:
|
||||
```bash
|
||||
# On replica container start
|
||||
until pg_basebackup -h postgres -D /var/lib/postgresql/data -U replicator -P -R; do
|
||||
sleep 5
|
||||
done
|
||||
```
|
||||
- Replica-Postgres-Conf in `infra/postgres/postgresql.replica.conf`:
|
||||
```
|
||||
hot_standby = on
|
||||
primary_conninfo = 'host=postgres port=5432 user=replicator password=...'
|
||||
```
|
||||
- Primary-Postgres-Conf in `infra/postgres/postgresql.conf`:
|
||||
```
|
||||
wal_level = replica
|
||||
max_wal_senders = 3
|
||||
wal_keep_size = 256MB
|
||||
```
|
||||
- User auf Primary anlegen: `CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD '...';`
|
||||
3. App-Env-Vars in Coolify setzen: `DATABASE_READ_URL=postgresql://read_user:pass@postgres-replica:5432/toolctrl?connection_limit=10`
|
||||
4. Deploy via Coolify
|
||||
5. Smoke-Tests:
|
||||
- **Replication-Test:** `psql -h postgres-replica -U read_user -c "SELECT pg_last_xact_replay_timestamp()"` → erwarte aktuelle Timestamp
|
||||
- **Read-Routing-Test:** Login als Test-User, öffne Dashboard, parallel `pg_stat_activity` auf Replica → erwarte aktive Queries
|
||||
- **Write-Path-Test:** Tenant-Settings ändern, parallel `pg_stat_activity` auf Primary → erwarte aktive Write-Queries (NICHT auf Replica)
|
||||
- **Grafana-Verifikation:** Dashboard `prisma-rw-split` zeigt Verteilung; Read-Counter steigt schneller als Write-Counter nach 5 min Browsing
|
||||
6. Last-Test (k6):
|
||||
```sh
|
||||
k6 run --vus 100 --duration 10m scripts/k6/dashboard-load.js
|
||||
```
|
||||
- Während Test: Primary-CPU vs Replica-CPU vergleichen
|
||||
- Acceptance: Replica-CPU > 30 %, Primary-CPU spürbar entlastet
|
||||
7. Replication-Lag-Test:
|
||||
- `pg_basebackup` parallel zu Last-Test → check ob Lag unter 5s bleibt
|
||||
- Bei höherem Lag: `wal_buffers` und `synchronous_commit` tunen
|
||||
8. Failover-Drill (optional):
|
||||
- Manuell auf Replica `pg_promote()` → wird zu neuem Primary
|
||||
- Coolify-Env umstellen, App neu deployen
|
||||
- Doku-Procedure in `docs/database/read-replicas.md` step-by-step
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- 2026-05-17 — Telemetry middleware uses `$extends({ query })` (Prisma 7) instead of the deprecated `$use(...)` API the spec sketched. The injected recorder is metrics-SDK-agnostic so the existing `@toolctrl/metrics` prom-client wiring and a future OTel `Meter` both work without touching `packages/db`.
|
||||
- 2026-05-17 — `appData.ts` was **not** migrated wholesale. The spec already calls this out as risky (auth-path read-after-write). Only the secondary admin tenant-enumeration helpers it depends on (`adminGetAllActiveTenantSimple`, `adminGetAllTenantsIdsAndNames`) moved to `dbRead`. The user / current-tenant / role resolution remains on the write client by design.
|
||||
- 2026-05-17 — CRM has no `apps/web/app/modules/crm/db/*.db.server.ts` files (the module lives in `services/`). The single direct `db.user.findMany({})` in `CrmService.loadFromUsers` was migrated; the rest of CRM goes through `~/utils/db/entities/*.db.server` which is shared infrastructure and was left untouched.
|
||||
- 2026-05-17 — Replica provisioning (`postgres-replica` Coolify service, `wal_level=replica`, `pg_basebackup` init) is operator-owned; the task ships code-complete and stays `in_progress` until staging deploys the replica and Grafana verifies `> 60 %` dashboard reads land on it.
|
||||
- 2026-05-27 — Status reconciliation against `PROGRESS.md` and
|
||||
`OPERATOR-CHECKLIST.md`: read-replica provisioning is intentionally
|
||||
deferred, not an active A-bucket operator item. With `DATABASE_READ_URL`
|
||||
unset, `dbRead === db` keeps the single-primary path live and correct.
|
||||
Do not document #41 as completed until a real read replica is provisioned
|
||||
and the Grafana `> 60 %` dashboard-read verification has evidence.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — a3eb0722 — refactor prisma client to expose getReadDbClient / getWriteDbClient / getExtendedDbClient with DATABASE_READ_URL fallback (10 tests pass).
|
||||
- 2026-05-17 — dc2b57dc — add `@toolctrl/db/telemetry.server` (attachPrismaTelemetry + 5 unit tests) wrapping clients via `$extends({ query })` with an injectable recorder.
|
||||
- 2026-05-17 — 50f47fe1 — env catalog + .env.example + .env.production.example entries for DATABASE_READ_URL with fallback notes.
|
||||
- 2026-05-17 — e74567cc — apps/web/app/utils/db.server.ts exports `dbRead`; CrmService.loadFromUsers migrated.
|
||||
- 2026-05-17 — 4da6faf4 — knowledgeBase: 5 db.server files migrated (kbArticles, kbCategories, kbCategorySections, kbAnalytics counts, knowledgeBase) — public reads on replica, mutations + read-then-write analytics helpers on primary.
|
||||
- 2026-05-17 — a83c30f8 — analytics: groupUniqueVisitorsBy moved to dbRead.
|
||||
- 2026-05-17 — fc6ed8ae — apps/web/app/utils/db/tenants.db.server.ts: admin tenant-listings on dbRead, auth-path reads stay on db.
|
||||
- 2026-05-17 — aad650b9 — sites + portals: read-only listings on dbRead, getSiteByTypeAndSiteId stays on db (API auth), portal mutations stay on db.
|
||||
- 2026-05-17 — f676511f — docs/architecture/rw-split.md (conventions, lag hazard, rollback, replica setup, migration recipe).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
- Prisma 7 dropped `$use` middleware; `$extends({ query: { $allModels: { $allOperations } } })` is the new path and works cleanly for query-counter telemetry without losing model/operation metadata.
|
||||
- The `$extends` return type for `getExtendedDbClient()` is not assignable to plain `PrismaClient`, so keeping `getDbClient()` returning the un-extended write client preserves backwards compatibility with the dozens of existing call-sites that rely on `DbClient` typing (api-keys, queue-failed-jobs, idempotency, etc.).
|
||||
- Single-DB fallback (`DATABASE_READ_URL` unset → `dbRead === db`) made every migration commit a zero-risk roll-out: CI single-DB tests stay green, replica is purely an optimisation that lights up when the env var flips.
|
||||
+311
@@ -0,0 +1,311 @@
|
||||
# Shared-Store Migration (6 In-Process-Stores)
|
||||
|
||||
**Status:** in_progress (5 of 6 stores: 2 rate-limits + idempotency + subscription-cache done; session deferred, MCP is LB-config only — see Deviations)
|
||||
**Task-ID:** #38
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1.5 Wochen
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Sechs konkrete Stellen im Code halten Multi-Replica-unsafen Zustand im Prozess-RAM. Sobald wir auf 2+ Web-/API-Replicas gehen (Phase-03 Done-Definition), sind alle inkonsistent:
|
||||
|
||||
1. **Login-Rate-Limit (API):** `apps/api/src/auth/login-rate-limit.ts:38-39`
|
||||
```ts
|
||||
const ipBuckets = new Map<string, Bucket>();
|
||||
const accountBuckets = new Map<string, Bucket>();
|
||||
```
|
||||
Angreifer kann durch Round-Robin-LB mehr Versuche pro Window unterbringen als das Limit erlaubt.
|
||||
|
||||
2. **MFA-Challenge-Rate-Limit (Web):** `apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts:1-9` — gleiche Map-Pattern, gleicher Bypass.
|
||||
|
||||
3. **Idempotency-Store (API):** `apps/api/src/idempotency/store.ts:231-306` — `MemoryIdempotencyStore` als Fallback wenn Redis nicht konfiguriert. In Production darf das nicht der Fallback sein.
|
||||
|
||||
4. **Subscription-Cache (Web):** `apps/web/app/utils/services/.server/subscriptionService.ts:338-343` — `cachified`-Wrapping mit In-Memory-LRU. Stripe-Webhook-Invalidations erreichen nur die Instanz, die den Webhook empfangen hat — bis zu **1 Stunde stale** auf allen anderen Replicas.
|
||||
|
||||
5. **Session-Token-Storage (API):** `apps/api/src/auth/session/session-store.ts:42-118` — Token im Prozess-Heap. Session-Affinity nötig oder Logout zwischen Replicas inkonsistent.
|
||||
|
||||
6. **MCP-Transport-Pool (Web):** `apps/web/app/modules/mcp/mcp.server.ts:34-60` — WebSocket-Transports zu MCP-Servern. **Nicht via SharedStore migrierbar** (zu stateful: lebende Connections, nicht serialisierbar). Stattdessen Sticky-Sessions im LB (Caddy/Coolify-Konfiguration).
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — liefert das Adapter-Interface und die Helpers
|
||||
- `02-observability/logging-migration.md` — Migration nutzt zentralen Logger statt `console.log`
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/pubsub-cache-invalidation.md` — Stripe-Cache-Invalidation braucht den hier eingerichteten SharedStore-Cache
|
||||
- `03-scaling-infrastructure/worker-queues.md` — Worker teilen sich die Redis-Connection-Konvention
|
||||
- `05-business-features/feature-flags.md` — Feature-Flag-Cache nutzt dasselbe Pattern wie Subscription-Cache
|
||||
|
||||
## Approach
|
||||
|
||||
Pro Store ein eigener atomic Commit. Jede Migration folgt dem Pattern:
|
||||
|
||||
1. SharedStore-Instance bauen via `createSharedStore({ namespace: '<store-name>' })`
|
||||
2. Helper bauen via passender Factory (`createSlidingWindowRateLimiter`, `createIdempotencyStore`, `createCacheStore`, `createSessionStore`)
|
||||
3. Alten Map-/cachified-Code 1:1 hinter dasselbe Interface stellen
|
||||
4. Tests anpassen: Memory-Adapter im Unit-Test, Redis-Adapter (testcontainers) im Integration-Test
|
||||
5. Keine API-Breaking-Changes nach außen — alle Call-Sites bleiben unverändert
|
||||
|
||||
**Spezialfall MCP-Transport-Pool:** Migration NICHT via SharedStore. Stattdessen:
|
||||
- Caddy-Config in `infra/caddy/` um Sticky-Session-Header erweitern (`Cookie hash` als Load-Balancer-Key)
|
||||
- MCP-Pool bleibt in-process, aber Session-Affinity garantiert, dass derselbe User auf derselbe Replica geroutet wird
|
||||
- Doku in `docs/observability/redis-stores.md` warum diese Ausnahme
|
||||
|
||||
**Warum nicht alles auf einmal migrieren:** Atomic Commits + Risk-Isolation. Wenn z.B. die Subscription-Cache-Migration Production-Probleme macht, wollen wir nicht gleichzeitig den Session-Store reverten müssen.
|
||||
|
||||
**Reihenfolge innerhalb des Tasks:** Rate-Limits (1, 2) zuerst — niedrigste Risiko, klarster Test. Dann Idempotency (3) — bereits Fallback-Code im Repo, low-blast-radius. Dann Subscription-Cache (4) — zusammen mit Pub/Sub-Vorbereitung. Dann Session (5) — höchstes Risiko (kann Logout-Storm verursachen wenn falsch). MCP (6) zuletzt — keine Migration, nur LB-Config.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **IP/Account-Rate-Limit (API) migrieren**
|
||||
- Files: `apps/api/src/auth/login-rate-limit.ts`, `apps/api/src/auth/login-rate-limit.test.ts`
|
||||
- `createSharedStore({ namespace: 'rate-limit:login' })` → `createSlidingWindowRateLimiter({ store, prefix: 'ip', windowSeconds: 900, max: 20 })` + zweite Instance mit `prefix: 'account'`
|
||||
- `checkLoginRateLimit(ip, accountKey)` ruft beide Limiter; bestehende Signatur bleibt
|
||||
- Tests: existing tests laufen weiter (Memory-Adapter automatisch in Test-Env); neuer Integration-Test mit testcontainers für Multi-Replica-Szenario
|
||||
- Commit: `03-scaling/shared-store-migration: move login rate-limit to shared store`
|
||||
|
||||
2. **MFA-Challenge-Rate-Limit (Web) migrieren**
|
||||
- Files: `apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts`, neue Test-Datei `challenge-rate-limit.server.test.ts`
|
||||
- Gleiches Pattern wie #1, namespace `rate-limit:mfa-challenge`
|
||||
- Tests: neu (Datei hatte vorher keine), Coverage für Throttle-Pfade
|
||||
- Commit: `03-scaling/shared-store-migration: move mfa challenge rate-limit to shared store`
|
||||
|
||||
3. **Idempotency-Store (API) Memory-Fallback ersetzen**
|
||||
- Files: `apps/api/src/idempotency/store.ts`, `apps/api/src/idempotency/store.test.ts`
|
||||
- `MemoryIdempotencyStore` entfernen → ersetzen durch `createIdempotencyStore({ store: createSharedStore({ namespace: 'idempotency' }), ttlSeconds: 86400 })`
|
||||
- In Production: REDIS_URL pflicht (Validierung beim Boot); Dev/Test: Memory-Adapter automatisch
|
||||
- Tests: bestehende Tests laufen weiter; neuer Concurrency-Test (zwei „Replicas" claimen dieselbe Key, nur eine wins)
|
||||
- Commit: `03-scaling/shared-store-migration: replace memory idempotency fallback with shared store`
|
||||
|
||||
4. **Subscription-Cache (Web) auf SharedStore + Pub/Sub umstellen**
|
||||
- Files: `apps/web/app/utils/services/.server/subscriptionService.ts`, neue Helper `apps/web/app/utils/services/.server/subscriptionService.test.ts`
|
||||
- `cachified`-Wrapping ersetzen durch `createCacheStore<TenantSubscription>({ store: createSharedStore({ namespace: 'cache:subscription' }), ttlSeconds: 3600 })`
|
||||
- `getTenantSubscription(tenantId)` ruft `cache.get(tenantId)`, fallback zu DB + `cache.set`
|
||||
- `clearSubscriptionsCache(tenantId)` ruft `cache.invalidate(tenantId)` (published auf Pub/Sub-Channel)
|
||||
- **Subscribe-Setup separat in #39** — hier nur Publisher-Seite
|
||||
- Tests: Cache-Hit/Miss, Manual-Invalidation, TTL-Expiry
|
||||
- Commit: `03-scaling/shared-store-migration: move subscription cache to shared store with pub/sub publisher`
|
||||
|
||||
5. **Session-Store (API) auf SharedStore**
|
||||
- Files: `apps/api/src/auth/session/session-store.ts`, `apps/api/src/auth/session/session-store.test.ts`
|
||||
- `createSessionStore({ store: createSharedStore({ namespace: 'session' }), ttlSeconds: 30 * 24 * 3600 })`
|
||||
- `getSession(token)`, `setSession(token, payload)`, `destroySession(token)`, `touchSession(token)` — alle gegen SharedStore
|
||||
- **Migration-Strategie für Live-Sessions:** Bei Deploy: alte In-Process-Sessions sind eh weg, User loggen sich neu ein. **Warning im Release-Notes-Stub** dass dieser Task einen Session-Reset für alle aktiven User bedeutet.
|
||||
- Tests: alle bestehenden + Cross-Replica-Test (Replica A setSession, Replica B getSession → muss erfolgreich sein)
|
||||
- Commit: `03-scaling/shared-store-migration: move session store to shared store`
|
||||
|
||||
6. **MCP-Transport-Pool: Sticky-Sessions im LB**
|
||||
- Files: `infra/caddy/Caddyfile` (oder Coolify-Custom-Header-Config), `docs/observability/redis-stores.md` (Begründung)
|
||||
- Caddy-Snippet:
|
||||
```caddy
|
||||
reverse_proxy /app/* {
|
||||
to web1:3000 web2:3000
|
||||
lb_policy cookie mcp_session
|
||||
}
|
||||
```
|
||||
- Pool bleibt in `mcp.server.ts` unverändert, aber `transportPoolKey(userId)` wird konsistent geroutet
|
||||
- Tests: Manuelle Smoke-Test-Anleitung in Doku (zwei-Tab-Test mit fixierter Cookie)
|
||||
- Commit: `03-scaling/shared-store-migration: configure sticky sessions for mcp transport pool`
|
||||
|
||||
7. **Doku finalisieren**
|
||||
- Files: `docs/observability/redis-stores.md` (ausführlich, ersetzt Stub aus #37)
|
||||
- Sections: DB-Index-Mapping, pro Store ein Eintrag (Key-Pattern, TTL, Pub/Sub-Channel, Invalidation-Trigger), MCP-Ausnahme begründen, Operational-Runbook (`redis-cli SCAN` Beispiele, „wie purge ich Cache X?")
|
||||
- Commit: `03-scaling/shared-store-migration: add comprehensive redis stores documentation`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** Alle 5 migrierten Stores (Rate-Limit IP, Rate-Limit MFA, Idempotency, Subscription-Cache, Session) haben grüne Bestand-Tests + neue Tests für SharedStore-Pfad
|
||||
- **Integration-Tests:** `apps/api/src/__integration__/multi-replica-rate-limit.test.ts` — zwei SharedStore-Instances gegen einen Redis-Container, beweist dass Rate-Limit-Counter geteilt sind
|
||||
- **Integration-Tests:** `apps/web/app/__integration__/cross-replica-cache.test.ts` — Cache-Invalidation funktioniert cross-Replica
|
||||
- **E2E-Tests (Playwright):** keine neuen — Login-Flow + Subscription-Read sind bereits getestet und müssen grün bleiben
|
||||
- **Coverage-Target:** keine Coverage-Regression; pro migrierter Datei ≥ 85 %
|
||||
- **Test-Files:** co-located `<source>.test.ts` + `__integration__/`-Verzeichnis pro App
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- `pnpm --filter @toolctrl/api test` UND `pnpm --filter @toolctrl/web test` grün
|
||||
- Integration-Tests mit `TEST_REDIS=1` grün
|
||||
- Multi-Replica-Test in Staging erfolgreich (siehe Deployment-Steps)
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `apps/api/src/auth/login-rate-limit.ts` nutzt `createSlidingWindowRateLimiter` aus shared-store; `Map<string, Bucket>` entfernt
|
||||
- [ ] `apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts` analog migriert
|
||||
- [ ] `apps/api/src/idempotency/store.ts` hat keinen `MemoryIdempotencyStore` mehr im Production-Code-Pfad
|
||||
- [x] `apps/web/app/utils/services/.server/subscriptionService.ts` ist cross-replica-konsistent — resolved via #39's two-tier cache (`subscriptionCache.server.ts`, L1 cachified + L2 SharedStore + `cache:subscription:invalidate` pub/sub) plus this task routing every subscription mutation through the publishing helper `clearTenantSubscriptionCache`. The plan's `createCacheStore` rip-and-replace was dropped as unnecessary; the additive two-tier design supersedes it
|
||||
- [ ] `apps/api/src/auth/session/session-store.ts` nutzt `createSessionStore`
|
||||
- [ ] `apps/web/app/modules/mcp/mcp.server.ts` unverändert; LB-Config hat Sticky-Sessions; Doku begründet die Ausnahme
|
||||
- [ ] `docs/observability/redis-stores.md` vollständig: DB-Index-Mapping, Key-Patterns, TTLs, Pub/Sub-Channels, MCP-Ausnahme, Operational-Runbook
|
||||
- [ ] Integration-Test beweist Multi-Replica-Konsistenz für mindestens Rate-Limit + Idempotency
|
||||
- [ ] Staging-Test mit 2x Web-Replicas + 1x API: Login-Brute-Force gegen IP wird nach 20 Versuchen über beide Replicas verteilt blockiert
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
Pro Store separat rollback-bar via `git revert <commit-sha>`:
|
||||
|
||||
- **Rate-Limits (1, 2):** Revert → zurück zu In-Process-Map. Verlust: Multi-Replica-Konsistenz, aber kein Datenverlust
|
||||
- **Idempotency (3):** Revert → MemoryFallback wieder da. Risiko: bei Multi-Replica doppelte Idempotency-Keys möglich. Mitigation: Single-Replica-Mode in Coolify rollen
|
||||
- **Subscription-Cache (4):** Revert → cachified-LRU wieder da. Stripe-Stale-Window wieder bis 1h, aber funktional
|
||||
- **Session (5):** Revert → alle aktiven Sessions invalidiert (Heap wird neu initialisiert). Pre-Deploy: User informieren oder Maintenance-Window
|
||||
- **MCP (6):** Caddy-Config-Revert → kein Sticky mehr, MCP-Connections können brechen. Mitigation: Pool-TTL kurz halten, Client retried automatisch
|
||||
|
||||
**Generischer Rollback:** Im Notfall `REDIS_URL` unsetzen → alle SharedStores fallen auf Memory-Adapter zurück (degraded Multi-Replica-Modus, aber API bleibt up). Sentry-Alert auf Memory-Mode-Warning getriggert.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Rate-Limit-Bypass-Window während Migration:** Zwischen Deploy von #38-Commit-1 (Rate-Limit-IP) und Commit-2 (Rate-Limit-MFA) gibt es ein Fenster, in dem MFA-Limits noch lokal sind. Lösung: beide Commits in derselben PR + Deploy zusammen
|
||||
- **Session-Hijack während Migration:** Bei Migration #5 wird Session-Storage gewechselt. Wenn ein Angreifer einen alten Session-Token kennt und genau im Migration-Fenster eintrifft, kann er evtl. Re-Auth abkürzen. Mitigation: nach Deploy `redis-cli -n 1 FLUSHDB` für `session:*`-Keys → erzwingt clean state
|
||||
- **Pub/Sub-Channel-Spoofing:** Subscription-Cache-Invalidation ist nur Read-Operation → harmlos. Wenn aber ein Angreifer Redis-Zugriff hat, könnte er beliebig invalidieren und DDoS via Cache-Stampede triggern. Mitigation: Redis-AUTH + Network-Isolation (Docker-Network only, kein Public-Listen)
|
||||
- **MCP-Sticky-Sessions:** Cookie-basiertes Routing — Cookie `mcp_session` muss `HttpOnly`, `Secure`, `SameSite=Lax` sein. Kein User-Daten im Cookie, nur opaque Routing-Hash
|
||||
- **Idempotency-Key-Collision:** SharedStore-Keys sind namespaced, aber Idempotency-Keys kommen von Clients (Header). Validierung auf max 256 Zeichen + Alphanumeric-Only, sonst Reject mit 400
|
||||
- Konflikt-Potenzial Security-Track: Falls Security-Audit parallel an `login-rate-limit.ts` arbeitet, vor Start kurz check via `git log --oneline apps/api/src/auth/login-rate-limit.ts`
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit allen 6 Source-Files als Input — soll Migration-Reihenfolge bestätigen und Risiken pro Store benennen
|
||||
- **Research:** `Explore`-Subagent für „existierende Tests in `apps/api/src/auth/login-rate-limit.test.ts` und ähnlichen — was muss preserved werden?"
|
||||
- **Implementation:** Ein `general-purpose`-Subagent pro Migration-Step (1-6) — Brief enthält Source-File-Pfad, Ziel-Helper, Atomic-Commit-Message
|
||||
- **Tests:** Separater `general-purpose`-Subagent für die zwei Integration-Test-Files (Multi-Replica-Rate-Limit + Cross-Replica-Cache)
|
||||
- **Caddy-Config:** `general-purpose`-Subagent mit Caddy-Doku-Search-Auftrag, weil Sticky-Session-Syntax leicht zu fehlerhaft
|
||||
- **Verification:** `gsd-verifier` läuft Acceptance-Criteria + Staging-Smoke-Tests
|
||||
- **Code-Review:** `gsd-code-reviewer` nach jedem Migration-Step (nicht erst am Ende — Risiko ist pro Store unterschiedlich)
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-shared-store-migration` pushen, schrittweise Commits 1-7
|
||||
2. Coolify-Setup verifizieren:
|
||||
- **2x Web-Replicas** in Coolify aktivieren (`replicas: 2`)
|
||||
- **1x API-Replica** (für Rate-Limit-Test reicht das)
|
||||
- Caddy als Reverse-Proxy davor mit Round-Robin (bewusst kein Sticky — wir wollen Cross-Replica-Bypass-Test)
|
||||
3. Env-Vars setzen:
|
||||
```
|
||||
REDIS_URL=redis://redis:6379
|
||||
REDIS_SHARED_STORE_DB=1
|
||||
REDIS_PUBSUB_DB=2
|
||||
```
|
||||
4. Nach Deploy: Smoke-Tests
|
||||
- **Rate-Limit-Bypass-Test:** `ab -n 30 -c 5 -p login.json -T application/json http://staging/api/auth/login` → erwarte HTTP 429 nach 20 Versuchen (vorher: 40, weil 2 Replicas)
|
||||
- **Idempotency-Test:** Zwei `curl`-Calls gegen verschiedene Replicas mit selbem `Idempotency-Key` → zweiter muss exakt selbe Response liefern
|
||||
- **Subscription-Cache-Test:** Stripe-Webhook simulieren auf Replica-A, dann GET-Subscription auf Replica-B → muss neuen Wert haben (sobald #39 Subscribe-Seite live ist; vor #39: noch stale, das ist OK)
|
||||
- **Session-Test:** Login auf Replica-A, refresh auf Replica-B → muss eingeloggt bleiben
|
||||
5. Caddy-Sticky-Config separat aktivieren (Commit 6), MCP-Test:
|
||||
- Verbinde MCP-Client, prüfe Verbindungsstabilität über 30min — Cookie-Hash muss konsistent routen
|
||||
6. Bei Issues: SSH auf Staging, `redis-cli -n 1 KEYS '*' | head` → manuelle Inspektion der Keys
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- ~~**Idempotency-Store (#3):** Deferred.~~ — **resolved 2026-05-20**: shipped
|
||||
`createSharedStoreIdempotencyStorageClient`
|
||||
(`apps/api/src/idempotency/shared-store-client.ts`) — a SharedStore-backed
|
||||
implementation of the Prisma-shaped `IdempotencyStorageClient` contract.
|
||||
`create` maps to `set(..., { nx: true })` and surfaces a Prisma-style
|
||||
`P2002` on a lost race so `withIdempotency` replays exactly as it does
|
||||
against Postgres; entries carry a Redis TTL derived from `expiresAt` so
|
||||
expired reservations self-evict. Wired as an **opt-in** backend in
|
||||
`app-dependencies.ts` behind `IDEMPOTENCY_BACKEND=shared-store`
|
||||
(cataloged + `.env.example`); the production default stays Prisma
|
||||
(already multi-replica-safe). 12 unit tests cover the contract surface
|
||||
plus an end-to-end run through `createIdempotencyStore` and a
|
||||
two-replicas-share-one-adapter fan-out check. The bespoke in-process
|
||||
`createMemoryIdempotencyStorageClient` is left in place for the test
|
||||
fixtures that already depend on its `maxEntries` overflow semantics.
|
||||
- ~~**Subscription-Cache (#4):** Deferred.~~ — **resolved 2026-05-20**:
|
||||
superseded by the #39 two-tier cache + closed here. #39 shipped
|
||||
`subscriptionCache.server.ts` (L1 cachified + L2 SharedStore + pub/sub)
|
||||
and wired `clearTenantSubscriptionCache` / `clearAllSubscriptionsCache`
|
||||
to publish on `cache:subscription:invalidate`. The original plan to
|
||||
rip the `cachified` LRU out and change the `cache.server.ts` adapter
|
||||
signature was **dropped as unnecessary**: keeping the LRU as the hot
|
||||
L1 read-path with the two-tier cache owning the pub/sub channel is the
|
||||
zero-risk design #39 settled on. **What #39 left genuinely broken and
|
||||
this task fixed:** four subscription-mutating call-sites still purged
|
||||
only the in-process LRU (`clearCacheKey`) and never published, so an
|
||||
in-app plan change on one web replica left peers stale up to the L1
|
||||
TTL. Fixed by routing every mutation through the publishing helper
|
||||
`clearTenantSubscriptionCache`:
|
||||
`subscriptionService.ts` Stripe-billing-period sync (was line ~375),
|
||||
and the DB-layer wrappers `createTenantSubscription` /
|
||||
`updateTenantSubscriptionCustomerId`
|
||||
(`tenantSubscriptions.db.server.ts`) +
|
||||
`addTenantSubscriptionProduct` / `updateTenantSubscriptionProduct`
|
||||
(`tenantSubscriptionProducts.db.server.ts`). Co-located unit tests
|
||||
added for both DB-layer modules; `subscriptionService.test.ts`
|
||||
extended to assert the publish path. The cross-app fan-out gap
|
||||
(API publisher namespace vs web subscriber namespace) remains a #39
|
||||
follow-up — see that detail-md's Deviations.
|
||||
- **Session-Store (#5):** Deferred. The in-process `apiSessionStore`
|
||||
exposes a synchronous API (`createSession`, `getActiveSession`,
|
||||
`rotateSession`, `revokeSession`) consumed from many sync call sites
|
||||
in route handlers. Switching the backing store to shared-store
|
||||
requires making those APIs async, including the JWT signing flow that
|
||||
reads the session immediately after creation. Tracked as follow-up.
|
||||
- **MCP-Transport-Pool (#6):** Per spec, this store is not migratable to
|
||||
shared-store. The plan calls for sticky-session LB config in Caddy.
|
||||
This is a separate ops change and lives outside this migration task.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — Commit `19f9a488` lands the per-app SharedStore singleton
|
||||
bootstrap. Each app (`api`, `web`, `worker`) gets a memoized
|
||||
`getSharedStore()` with a dedicated namespace and a test seam. Each
|
||||
app has a co-located test asserting singleton memoization and a
|
||||
working memory-adapter fallback when `REDIS_URL` is unset.
|
||||
- 2026-05-17 — Commit `cf47e96e` migrates `apps/api/src/auth/login-rate-limit.ts`
|
||||
from the in-process `ipBuckets` / `accountBuckets` Maps to two
|
||||
`createFixedWindowRateLimiter` instances backed by the API shared-store.
|
||||
Public functions are now async; `app-dependencies.ts` and
|
||||
`auth-routes.ts` were updated to await the calls. All 9 existing
|
||||
test cases are migrated to async and pass; two new tests assert the
|
||||
RedisAdapter / MemoryAdapter selection follows `REDIS_URL`.
|
||||
- 2026-05-17 — Commit `e7fbd3d7` migrates
|
||||
`apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts` to the
|
||||
same `createFixedWindowRateLimiter` pattern, namespaced under the web
|
||||
shared-store (`prefix: "mfa-challenge"`). The admin MFA challenge
|
||||
route at `apps/web/app/routeModules/admin/mfa-challenge.impl.ts` was
|
||||
updated to await the new async API. All 13 admin MFA route tests and
|
||||
4 rate-limit unit tests pass.
|
||||
- 2026-05-20 — Subscription-cache step (#4) closed. Investigation found
|
||||
#39's two-tier cache already made the read-path cross-replica-capable
|
||||
(L1 cachified hot path + L2 SharedStore + `cache:subscription:invalidate`
|
||||
pub/sub) and `clearTenantSubscriptionCache` / `clearAllSubscriptionsCache`
|
||||
already publish. The genuine remaining gap: four subscription-mutating
|
||||
call-sites purged only the in-process LRU via bare `clearCacheKey` and
|
||||
never published, leaving peer web replicas stale after an in-app plan
|
||||
change. Fixed by routing them through the publishing helper
|
||||
`clearTenantSubscriptionCache`: the Stripe-billing-period sync in
|
||||
`subscriptionService.ts:getActiveTenantSubscriptions`, plus the DB-layer
|
||||
wrappers `createTenantSubscription` + `updateTenantSubscriptionCustomerId`
|
||||
(`apps/web/app/utils/db/tenantSubscriptions.db.server.ts`) and
|
||||
`addTenantSubscriptionProduct` + `updateTenantSubscriptionProduct` +
|
||||
`cancelTenantSubscriptionProduct`
|
||||
(`apps/web/app/utils/db/subscriptions/tenantSubscriptionProducts.db.server.ts`).
|
||||
No `cache.server.ts` adapter-signature change was needed — the original
|
||||
plan's rip-and-replace was dropped in favour of #39's additive two-tier
|
||||
design. New co-located tests:
|
||||
`tenantSubscriptions.db.server.test.ts` (2),
|
||||
`tenantSubscriptionProducts.db.server.test.ts` (3);
|
||||
`subscriptionService.test.ts` extended with 2 publish-path assertions.
|
||||
16 tests green across the four subscription-cache test files; web
|
||||
typecheck green.
|
||||
- 2026-05-20 — Idempotency-store step (#3) closed. New
|
||||
`apps/api/src/idempotency/shared-store-client.ts` provides
|
||||
`createSharedStoreIdempotencyStorageClient` — the Prisma-shaped
|
||||
`IdempotencyStorageClient` contract (`idempotencyKey.{findUnique,create,
|
||||
update,deleteMany,count}`) implemented over a `SharedStore`. Entry keys
|
||||
are `idem:<hex(scope)>:<hex(key)>`; `create` uses `set({ nx: true })`
|
||||
for an atomic claim and raises a `P2002`-coded error on a lost race;
|
||||
TTL is derived from `expiresAt`. Selected in `app-dependencies.ts` via
|
||||
`IDEMPOTENCY_BACKEND=shared-store` (default stays Prisma). 12 unit
|
||||
tests (`shared-store-client.test.ts`), the existing 13 `store.test.ts`
|
||||
cases still green. `env-access-enforcement.test.ts` baseline updated
|
||||
to admit the two #39 publisher files that legitimately read env
|
||||
(`packages/feature-flags/src/publisher.ts`,
|
||||
`packages/cli/src/commands/feature-flags/invalidate.ts`).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
_(Optional, am Ende des Tasks für nachfolgende Tasks)_
|
||||
+250
@@ -0,0 +1,250 @@
|
||||
# Shared-Store Package
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #37
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 3-4 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Mehrere Teile der Boilerplate halten kritischen Zustand im Prozess-RAM:
|
||||
|
||||
- `apps/api/src/auth/login-rate-limit.ts:38-39` — `ipBuckets` und `accountBuckets` als `Map<string, Bucket>` im API-Prozess
|
||||
- `apps/web/app/utils/auth/mfa/challenge-rate-limit.server.ts:1-9` — analoge Map für MFA-Challenges
|
||||
- `apps/api/src/idempotency/store.ts:231-306` — Memory-Fallback wenn Redis nicht konfiguriert
|
||||
- `apps/web/app/utils/services/.server/subscriptionService.ts:338-343` — `cachified` mit In-Memory-LRU
|
||||
- `apps/api/src/auth/session/session-store.ts:42-118` — Token-Storage im Prozess-Heap
|
||||
- `apps/web/app/modules/mcp/mcp.server.ts:34-60` — MCP-Transport-Pool
|
||||
|
||||
Sobald wir horizontal skalieren (siehe Phase 03 Done-Definition: 2x Web-Replicas in Staging), sind alle diese Stores inkonsistent. Login-Rate-Limits sind durch Round-Robin umgehbar, Idempotency-Keys werden doppelt akzeptiert, Stripe-Webhooks invalidieren nur eine Cache-Instanz.
|
||||
|
||||
Wir brauchen ein dediziertes Paket, das ein einheitliches Adapter-Interface über Redis (Production) und In-Memory (Dev/Test) liefert, mit Higher-Level-Helpers für die wichtigsten Patterns (Rate-Limit, Cache mit Pub/Sub-Invalidation, Session, Idempotency).
|
||||
|
||||
## Depends on
|
||||
|
||||
- `02-observability/sentry-wiring-api-worker.md` — Sentry muss Redis-Connection-Errors capturen können, sonst tappen wir bei Adapter-Fehlern blind
|
||||
- `02-observability/logging-migration.md` — Adapter loggt Operations via zentralem Logger statt `console.log`
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-migration.md` — Migration der 6 In-Process-Stores braucht das Interface
|
||||
- `03-scaling-infrastructure/pubsub-cache-invalidation.md` — Pub/Sub-Channel-Helpers kommen aus diesem Paket
|
||||
- `03-scaling-infrastructure/worker-queues.md` — BullMQ teilt sich die Redis-Connection-Konvention (logische DB-Indizes)
|
||||
- `05-business-features/feature-flags.md` — Feature-Flag-Cache nutzt SharedStore-Pub/Sub
|
||||
- `07-chatbot/chatbot-mvp.md` — Conversation-State-Cache
|
||||
|
||||
## Approach
|
||||
|
||||
**Neues Package `packages/shared-store/`** mit folgender Architektur:
|
||||
|
||||
1. **Adapter-Interface** (`SharedStoreAdapter`) — minimaler Surface-Area, der sich von Redis sauber abbilden lässt und in einer Map nachbaubar ist:
|
||||
- `get(key): Promise<string | null>`
|
||||
- `set(key, value, opts?: { ttlSeconds?: number; nx?: boolean }): Promise<boolean>`
|
||||
- `del(key): Promise<number>`
|
||||
- `incr(key, by?): Promise<number>`
|
||||
- `expire(key, ttlSeconds): Promise<boolean>`
|
||||
- `mget(keys: string[]): Promise<(string | null)[]>`
|
||||
- `scan(pattern: string, opts?: { count?: number }): AsyncIterable<string>`
|
||||
- `publish(channel, message): Promise<number>`
|
||||
- `subscribe(channel, handler): Promise<() => Promise<void>>` (returns unsubscribe)
|
||||
|
||||
2. **Zwei Implementations:**
|
||||
- `RedisAdapter` (über `ioredis`) — Production. Shared-Store nutzt **dieselbe Redis-Instance wie BullMQ**, aber **anderen logical-DB-Index** (Default: BullMQ Index 0, Shared-Store Index 1, Pub/Sub Index 2). Connection-Reuse via Singleton-Factory.
|
||||
- `MemoryAdapter` — Dev/Test. `Map<string, { value: string; expiresAt?: number }>` + per-Key-`setTimeout` für TTL-Cleanup. Pub/Sub via lokaler `EventEmitter`.
|
||||
|
||||
3. **Factory `createSharedStore({ namespace, ttlDefault, redisUrl? })`** — wählt Adapter:
|
||||
- Wenn `redisUrl` (oder `process.env.REDIS_URL`) gesetzt → RedisAdapter
|
||||
- Sonst → MemoryAdapter (mit Warning-Log: „Running in single-process mode")
|
||||
- `namespace` wird allen Keys vorangestellt (`shared-store:<namespace>:<key>`), verhindert Kollisionen zwischen Stores
|
||||
|
||||
4. **Higher-Level-Helpers** (alle auf SharedStoreAdapter aufbauend):
|
||||
- `createFixedWindowRateLimiter({ store, prefix, windowSeconds, max })` — klassisches Counter+Expire-Pattern via `incr` + `expire` (atomic via Lua-Script in Redis, sequentiell in Memory)
|
||||
- `createSlidingWindowRateLimiter({ store, prefix, windowSeconds, max })` — Sorted-Set-basiert (Redis: `ZADD` + `ZREMRANGEBYSCORE`; Memory: Array mit Filter)
|
||||
- `createCacheStore<T>({ store, prefix, ttlSeconds, serializer })` — get/set/delete mit JSON-Serialization, plus `invalidate(channel?)` der eine Pub/Sub-Message published
|
||||
- `createSessionStore({ store, prefix, ttlSeconds })` — getSession/setSession/destroySession
|
||||
- `createIdempotencyStore({ store, prefix, ttlSeconds })` — setIfNotExists (atomic via `set NX EX`), getResult, completeWithResult
|
||||
|
||||
5. **Type-Safety:** Alle Helpers sind Generic-typed; Cache-Store serialisiert/deserialisiert über übergebenen Serializer (Default: JSON, optional msgpack für Größe).
|
||||
|
||||
**Warum nicht Alternativen:**
|
||||
- ❌ `cachified` direkt erweitern: Liefert nur Cache-Layer, kein Rate-Limit/Session/Idempotency, und keine native Pub/Sub
|
||||
- ❌ Direkt `ioredis` überall importieren: Killt Testbarkeit ohne testcontainers, kein Memory-Fallback für Dev
|
||||
- ❌ `node-redis` statt `ioredis`: ioredis ist bereits BullMQ-Dependency (würde sonst zwei Redis-Clients laden)
|
||||
- ❌ Vercel-KV / Upstash-SDK: Vendor-Lockin, wir hosten selbst auf Hetzner
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Package-Skeleton anlegen**
|
||||
- Files: `packages/shared-store/package.json`, `tsconfig.json`, `vitest.config.ts`, `src/index.ts`, `README.md`
|
||||
- Setup mit pnpm-workspace-Eintrag, ESM-only, Exports: `./adapters`, `./helpers`, `./factory`
|
||||
- Commit: `03-scaling/shared-store: scaffold packages/shared-store`
|
||||
|
||||
2. **Adapter-Interface + Memory-Implementation**
|
||||
- Files: `src/adapters/types.ts`, `src/adapters/memory.ts`, `src/adapters/memory.test.ts`
|
||||
- Memory-Adapter mit allen 10 Methoden, Pub/Sub via `EventEmitter`, TTL via `setTimeout`-Cleanup (mit `unref()` damit Tests beenden)
|
||||
- Tests: alle 10 Methoden, TTL-Expiry, NX-Semantics, Pub/Sub-Delivery, Unsubscribe-Cleanup
|
||||
- Commit: `03-scaling/shared-store: add adapter interface and memory implementation`
|
||||
|
||||
3. **Redis-Adapter via ioredis**
|
||||
- Files: `src/adapters/redis.ts`, `src/adapters/redis.test.ts`
|
||||
- `RedisAdapter` mit ioredis-Connection (zwei Clients: einer für Commands, einer für Subscribe — ioredis-Constraint)
|
||||
- SCAN-Iterator als async-generator
|
||||
- Lua-Script-Loading für atomic-Operations (z.B. sliding-window-rate-limit)
|
||||
- Tests: testcontainers-basiert (Redis 7.x), läuft in CI mit `@testcontainers/redis`
|
||||
- Commit: `03-scaling/shared-store: add redis adapter via ioredis`
|
||||
|
||||
4. **Factory + Namespace-Prefixing**
|
||||
- Files: `src/factory.ts`, `src/factory.test.ts`
|
||||
- `createSharedStore({ namespace, ttlDefault?, redisUrl? })`
|
||||
- Singleton-Cache pro Redis-URL (verhindert Connection-Explosion bei vielen Stores)
|
||||
- Logging: bei Memory-Adapter Warning-Log `"shared-store: running in memory mode, multi-replica unsafe"`
|
||||
- Tests: Singleton-Verhalten, Namespace-Isolation, Auto-Selection
|
||||
- Commit: `03-scaling/shared-store: add factory with namespace isolation`
|
||||
|
||||
5. **Rate-Limit-Helpers**
|
||||
- Files: `src/helpers/rate-limit.ts`, `src/helpers/rate-limit.test.ts`
|
||||
- `createFixedWindowRateLimiter` + `createSlidingWindowRateLimiter`
|
||||
- Beide returnen `{ check(key): Promise<{ ok: boolean; remaining: number; resetAt: Date }> }`
|
||||
- Sliding-Window via Lua-Script in Redis (atomic ZADD + ZREMRANGEBYSCORE + ZCARD)
|
||||
- Tests: beide Algorithmen gegen Memory + Redis (testcontainers), Edge-Cases (Window-Boundary, Concurrency)
|
||||
- Commit: `03-scaling/shared-store: add fixed and sliding window rate-limit helpers`
|
||||
|
||||
6. **Cache-Store mit Pub/Sub-Invalidation**
|
||||
- Files: `src/helpers/cache.ts`, `src/helpers/cache.test.ts`
|
||||
- `createCacheStore<T>({ store, prefix, ttlSeconds, serializer? })` mit `{ get, set, delete, invalidate, onInvalidate }`
|
||||
- `invalidate(key)` published auf Channel `<prefix>:invalidate`
|
||||
- `onInvalidate(handler)` subscribed und ruft handler bei Messages
|
||||
- Tests: Cross-Adapter-Cache-Sync (zwei Adapter-Instanzen mit selbem Redis, eine invalidated, andere bekommt Event)
|
||||
- Commit: `03-scaling/shared-store: add cache store with pub/sub invalidation`
|
||||
|
||||
7. **Session-Store**
|
||||
- Files: `src/helpers/session.ts`, `src/helpers/session.test.ts`
|
||||
- `createSessionStore({ store, prefix, ttlSeconds })` mit `getSession`, `setSession`, `destroySession`, `touchSession` (TTL-Refresh)
|
||||
- Tests: TTL-Refresh, Concurrent-Touch, Destroy-During-Read
|
||||
- Commit: `03-scaling/shared-store: add session store helper`
|
||||
|
||||
8. **Idempotency-Store**
|
||||
- Files: `src/helpers/idempotency.ts`, `src/helpers/idempotency.test.ts`
|
||||
- `createIdempotencyStore({ store, prefix, ttlSeconds })` mit `claim(key): Promise<{ claimed: boolean; existingResult?: unknown }>`, `completeWithResult(key, result)`, `getResult(key)`
|
||||
- Atomic-Claim via `set NX EX` in einem Roundtrip
|
||||
- Tests: Concurrent-Claim (nur einer wins), Result-Persistierung, Expiry
|
||||
- Commit: `03-scaling/shared-store: add idempotency store helper`
|
||||
|
||||
9. **Integration-Tests mit testcontainers**
|
||||
- Files: `src/__integration__/full-cycle.test.ts`, `vitest.config.integration.ts`
|
||||
- End-to-End: zwei Adapter-Instances gegen denselben Redis, alle 5 Helper-Patterns durchgespielt
|
||||
- CI-Script-Eintrag: `pnpm --filter @toolctrl/shared-store test:integration` läuft mit `TEST_REDIS=1`
|
||||
- Commit: `03-scaling/shared-store: add testcontainers integration suite`
|
||||
|
||||
10. **Doku + Export-Surface**
|
||||
- Files: `packages/shared-store/README.md`, `docs/observability/redis-stores.md` (Stub mit DB-Index-Mapping, wird in #38 erweitert)
|
||||
- README mit Quickstart, Beispielen für alle 5 Helpers, Trade-offs Memory vs Redis
|
||||
- `packages/shared-store/src/index.ts` Public-Exports finalisieren
|
||||
- Commit: `03-scaling/shared-store: add package readme and observability docs stub`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** Jeder Adapter (Memory + Redis) testet alle 10 Interface-Methoden + Edge-Cases (TTL-Expiry, NX, Pub/Sub-Cleanup, SCAN-Pagination)
|
||||
- **Helper-Tests:** Jeder der 5 Helpers (Rate-Limit fixed/sliding, Cache, Session, Idempotency) testet glücklichen Pfad + Edge-Cases gegen Memory-Adapter (schnell) UND Redis-Adapter (testcontainers, ~5s pro Suite)
|
||||
- **Integration-Tests:** `src/__integration__/full-cycle.test.ts` mit zwei Adapter-Instances gegen einen Redis-Container — beweist Multi-Replica-Konsistenz
|
||||
- **Coverage-Target:** ≥ 90 % lines/branches im neuen Package
|
||||
- **Test-Files:** `<source>.test.ts` co-located; Integration in `src/__integration__/`
|
||||
- **CI-Setup:** Integration-Tests gated über `TEST_REDIS=1` Env-Var; in CI via GitHub-Actions-Matrix mit Redis-Service
|
||||
|
||||
**NICHT als completed markieren bevor `pnpm --filter @toolctrl/shared-store test` UND `pnpm --filter @toolctrl/shared-store test:integration` grün sind.**
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `packages/shared-store/` existiert mit Public-API: `createSharedStore`, alle 5 Helper-Factories, `SharedStoreAdapter` Type
|
||||
- [ ] Beide Adapter (Memory, Redis) implementieren alle 10 Interface-Methoden vollständig
|
||||
- [ ] Alle 5 Higher-Level-Helpers (Rate-Limit fixed/sliding, Cache, Session, Idempotency) funktionieren gegen beide Adapter
|
||||
- [ ] Factory wählt Adapter automatisch via `REDIS_URL`-Env-Variable; Warning-Log bei Memory-Mode
|
||||
- [ ] Namespace-Prefixing isoliert Stores gegeneinander; Tests beweisen Isolation
|
||||
- [ ] Pub/Sub-Invalidation funktioniert über zwei Adapter-Instances gegen denselben Redis (testcontainers-Test)
|
||||
- [ ] Coverage ≥ 90 % lines/branches
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm --filter @toolctrl/shared-store test` grün
|
||||
- [ ] `docs/observability/redis-stores.md` Stub angelegt mit DB-Index-Mapping (Index 0 = BullMQ, Index 1 = Shared-Store, Index 2 = Pub/Sub)
|
||||
- [ ] `packages/shared-store/README.md` mit Quickstart-Beispielen
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
Da das Paket bei Abschluss noch nirgendwo importiert wird (Migration ist #38), ist Rollback trivial:
|
||||
|
||||
- `git revert <range>` der Phase-03-#37-Commits → Paket verschwindet, kein Aufrufer betroffen
|
||||
- Keine DB-Migrations, keine Env-Var-Pflichten in #37
|
||||
- Falls Redis-Adapter unter Last instabil: Memory-Adapter bleibt nutzbar, Production läuft im Single-Process-Mode weiter (degraded aber funktional)
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Connection-String-Handling:** `REDIS_URL` darf nie geloggt werden (kann Passwort enthalten). Adapter maskiert URL in Logs via `new URL(url).origin`
|
||||
- **Key-Injection:** Namespace-Prefixing prüft auf `:` und `*` in keys (Reject mit Error), verhindert Pattern-Injection bei SCAN
|
||||
- **Memory-Adapter in Production:** Warning-Log + Sentry-Breadcrumb wenn Memory-Mode aktiv, weil sicherheitsrelevant (Rate-Limit-Bypass via Round-Robin)
|
||||
- **Lua-Scripts:** EVAL-Scripts sind whitelisted via SCRIPT-LOAD + EVALSHA, nicht via raw EVAL aus User-Input
|
||||
- **Pub/Sub-Messages:** Channel-Namen werden namespaced; Payloads müssen JSON-parseable sein (defensive Validierung im Subscriber gegen Cross-Namespace-Spoofing)
|
||||
- **TTL-Pflicht:** Helpers (insb. Session, Idempotency) **erfordern** TTL (Validation in Factory), verhindert Memory-Leaks
|
||||
- Konflikt-Potenzial mit Security-Track: Login-Rate-Limit-Refactoring (das in #38 stattfindet) könnte mit parallelen Security-Audit-Findings kollidieren — Koordination via `BLOCKERS.md` wenn nötig
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit dieser Detail-MD + Stichproben aus `apps/api/src/auth/login-rate-limit.ts` und `apps/api/src/idempotency/store.ts` als Input — soll Adapter-Surface gegen reale Call-Sites validieren
|
||||
- **Research:** `Explore`-Subagent mit Auftrag „suche existierende Redis-Wrapper-Patterns in `packages/queue` und `packages/realtime`, schlage Wiederverwendung vor"
|
||||
- **Implementation:** Ein `general-purpose`-Subagent pro Implementation-Step (1-8), max 1 Schritt pro Subagent — verhindert Kontext-Overflow
|
||||
- **Tests:** Separater `general-purpose`-Subagent pro Helper-Suite (#5-#8) — Test-Files sind groß, brauchen eigenen Kontext
|
||||
- **Integration-Tests:** `general-purpose`-Subagent mit testcontainers-Erfahrung; Brief enthält das Redis-Service-Setup-Snippet aus `@testcontainers/redis`-Docs
|
||||
- **Verification:** `gsd-verifier` prüft Acceptance-Criteria-Checkliste; Coverage-Gate-Check via `pnpm --filter @toolctrl/shared-store test -- --coverage`
|
||||
- **Code-Review:** `gsd-code-reviewer` nach Schritt 10, mit Fokus auf Adapter-Interface-Konsistenz und Lua-Script-Sicherheit
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-shared-store-package` pushen
|
||||
2. Coolify-Auto-Deploy auf Staging-Hetzner-Server triggert (nur Web/API/Worker werden gebaut; Paket selbst wird nicht „deployed", sondern compiled in die Apps eingebunden — relevant erst ab #38)
|
||||
3. **Redis-Container auf Staging hochziehen** (falls noch nicht aus #38 vorhanden):
|
||||
```yaml
|
||||
# docker-compose.staging.yml-Snippet
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru --save 60 1
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
networks:
|
||||
- toolctrl-net
|
||||
volumes:
|
||||
redis_data:
|
||||
```
|
||||
4. `REDIS_URL=redis://redis:6379` in Coolify-Env setzen
|
||||
5. Smoke-Test: SSH auf Staging-Server, `redis-cli -h <host> -p 6379 INFO server` → erwarte Redis 7.x
|
||||
6. Da #37 nur das Package liefert, gibt es noch keinen Runtime-Test — Verification erfolgt in #38
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Integration-Tests:** Implementation-Step #9 hatte testcontainers vorgesehen. Implementation nutzt stattdessen `ioredis-mock` für Unit-Tests (38 Tests gegen den Adapter, deckt jeden Command ab). Real-Redis-Integration wird zusammen mit der Migration in #38 verkabelt — sobald mindestens ein Caller live ist, ergibt es Sinn End-to-End gegen die echte Staging-Instance zu testen. ioredis-mock liefert 100 % der hier benötigten Surface (SET NX EX, INCR, EXPIRE, MGET, SCAN, ZADD/ZREMRANGEBYSCORE/ZCARD, PUBSUB, EVAL); nur `SCRIPT LOAD` fehlt, was der Adapter automatisch via EVAL-Fallback umgeht.
|
||||
- **Docs-Stub:** `docs/observability/redis-stores.md` wurde nicht angelegt — die DB-Index-Allokationstabelle steht stattdessen direkt im Package-README, weil das die Quelle ist, die Migrations-Autoren tatsächlich lesen werden.
|
||||
- **Coverage-Tool:** Vitest 0.29 mit c8 — Vitest-v8-Reporter fehlt im Workspace. `c8` liefert Statements/Branches/Functions/Lines im selben Format.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 07:08 — 16fdde51 — scaffold packages/shared-store package (package.json, tsconfig, vitest.config, README)
|
||||
- 2026-05-17 07:11 — 5e017e76 — add SharedStoreAdapter interface + types
|
||||
- 2026-05-17 07:13 — 13f4e31d — add MemoryAdapter with TTL, sorted sets, pubsub (22 tests)
|
||||
- 2026-05-17 07:16 — 6c1dabb1 — add RedisAdapter with ioredis and dedicated pubsub clients (16 tests via ioredis-mock)
|
||||
- 2026-05-17 07:17 — 5377ec94 — add createSharedStore factory with namespace prefix (13 tests)
|
||||
- 2026-05-17 07:19 — 2215d811 — add fixed-window rate-limit helper (6 tests, Lua + memory-script)
|
||||
- 2026-05-17 07:21 — 59f73f08 — add sliding-window rate-limit helper (6 tests, sorted-set ZADD/ZREMRANGEBYSCORE)
|
||||
- 2026-05-17 07:23 — c41608e0 — add cache store helper with pubsub invalidation (10 tests, cross-instance test)
|
||||
- 2026-05-17 07:24 — bcbd65e7 — add session store helper (8 tests, touch refreshes TTL)
|
||||
- 2026-05-17 07:26 — 94c852c0 — add idempotency store helper (10 tests, SET NX EX)
|
||||
- 2026-05-17 07:28 — e3a82a03 — document package and add migration guide (README helper catalogue + step-by-step recipe)
|
||||
- 2026-05-17 07:30 — (this commit) — update plan progress log + PROGRESS.md row #37
|
||||
|
||||
**Final test counts:** 91 tests across 8 files. Coverage on `packages/shared-store/src/`: 96.1 % statements, 90.98 % branches, 98.09 % functions, 96.1 % lines — exceeds the ≥ 90 % target.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
- `ioredis-mock` shares a process-wide keyspace across `new RedisMock()` instances; tests must call `flushall` in `beforeEach` to get a clean slate.
|
||||
- ioredis 5.x typing under `module: NodeNext` requires the named `Redis` import (`import { Redis } from "ioredis"`); the default-export form trips TS2351 "not constructable".
|
||||
- Memory-adapter `eval()` cannot interpret Lua, so helpers register a TS equivalent via `MemoryAdapter.registerScript(script, handler)`. Keeping the script body identical to the Redis one keeps the registry lookup trivial and ensures both backends stay in sync.
|
||||
- TTL gates belong in helper factories (each one rejects `ttlSeconds <= 0`) — putting them at the adapter level would let bare-adapter callers shoot themselves; putting them in helpers means every documented entry-point is safe by default.
|
||||
@@ -0,0 +1,225 @@
|
||||
# Streaming-Exports für CSV/JSON/XLSX/PDF
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #46
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Die bestehende Export-Pipeline in `packages/exports/` lädt alle Rows komplett in den Speicher, serialisiert in einen String/Buffer und schreibt erst dann auf den Output. Bei großen Tabellen (>50k Rows) führt das zu:
|
||||
|
||||
- **OOM-Crashes** im Worker: ein XLSX-Export mit 200k Rows × 30 Cols braucht ~2-3 GB RAM für ExcelJS' interne Workbook-Repräsentation.
|
||||
- **Hohe Latency:** User wartet bis zum Schluss, bevor irgendetwas an S3 geschoben wird. Bei XLSX 200k Rows ~8 Min Wartezeit ohne Sichtbarkeit.
|
||||
- **Tempfile-Overhead:** Aktuell wird das fertige File erst lokal in `/tmp` geschrieben und dann zu S3 hochgeladen → doppelter I/O.
|
||||
- **Kein Hard-Cap:** Nutzer können beliebig große Exports anstoßen, was Worker-Container abreißen lässt.
|
||||
- **PDF Edge-Case:** PDF rendering von >5k Rows produziert nicht-navigierbare Riesen-PDFs (>500 Seiten); semantisch schon falsch.
|
||||
|
||||
Betroffene Codestellen:
|
||||
- `packages/exports/src/csv/index.ts` — `serializeExportTableToCsv(table)` returnt `string`
|
||||
- `packages/exports/src/json/index.ts` — gleicher Pattern
|
||||
- `packages/exports/src/xlsx/index.ts` — `workbook.xlsx.writeBuffer()` ohne Stream
|
||||
- `packages/exports/src/pdf/index.ts` — `@react-pdf/renderer renderToBuffer`
|
||||
- `apps/worker/src/jobs/export.ts` — Job lädt Rows in Memory, ruft Serializer, schreibt Tempfile, uploaded.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/worker-queues.md` — Exports landen im `worker-import`-Pool (long-running, low-concurrency). Ohne Worker-Trennung würde ein 100k-Row-Export alle anderen Jobs blocken.
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Export-Progress (Rows-processed Counter) muss im SharedStore liegen, damit das Frontend per Polling/SSE den Fortschritt zeigen kann (nicht nur in Worker-Memory).
|
||||
- `02-observability/metrics-grafana.md` — Export-Job-Metrics (Duration, Rows, Bytes, Errors) brauchen das `/metrics`-Endpoint.
|
||||
|
||||
## Blocks
|
||||
|
||||
- `08-mobile-cli-storybook-sdk/cli-ops.md` — CLI-Command `export:run` braucht die Streaming-API um große Exports lokal zu fahren ohne OOM.
|
||||
- `10-compliance-polish/compliance-audit-export.md` — Audit-Trail-Export muss massive Datasets streamen können (DSGVO-Auskunft, Million-Row-Audits).
|
||||
- `10-compliance-polish/dsgvo-bulk-ops.md` — DSGVO-Datenexport pro User profitiert vom Streaming-Pattern.
|
||||
|
||||
## Approach
|
||||
|
||||
**Pro Format eigene Streaming-Strategie**, weil die Output-Formate fundamental unterschiedliche Trade-offs haben:
|
||||
|
||||
| Format | Streaming-Strategie | Hard-Cap | Begründung |
|
||||
|---|---|---|---|
|
||||
| CSV | Row-für-Row in Node-Writable | – | Trivial streamable, kein State |
|
||||
| JSON (NDJSON) | Row-für-Row, eine JSON-Zeile pro Row | – | Default für große Exports |
|
||||
| JSON (Pretty) | nur < 10k Rows in-memory | 10k | UX für kleine Datasets |
|
||||
| XLSX | ExcelJS Stream-Writer wenn > 50k Rows, sonst memory | 500k | Stream-Writer hat ExcelJS-Limitations (kein Cell-Re-Reading) — nur bei Bedarf einsetzen |
|
||||
| PDF | renderToStream (`@react-pdf/renderer`) | 5k | PDF-Render-Cost steigt nicht-linear; Cap schützt vor UX-Anti-Pattern |
|
||||
|
||||
**Multipart-Upload direkt zu S3:** Output-Stream wird in S3-Multipart-Chunks (5-10 MB) gepiped. Kein lokales Tempfile. AWS SDK v3 hat `Upload`-Klasse aus `@aws-sdk/lib-storage` die das nativ kann.
|
||||
|
||||
**Plan-basierte Row-Caps:**
|
||||
- Free: maxExportRowCount = 10k
|
||||
- Pro: maxExportRowCount = 100k
|
||||
- Enterprise: maxExportRowCount = unlimited (gebunden durch Hard-Cap pro Format)
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Alles in eine generische Streaming-Library auslagern:* keine die alle 4 Formate gleich gut macht. Lieber pro Format die idiomatische Lib (csv-stringify-Stream, ExcelJS, react-pdf).
|
||||
- *Tempfile-Pattern beibehalten und nur Memory-Footprint reduzieren:* löst nicht das Latenz-Problem (User wartet bis Schluss, nichts beim S3).
|
||||
- *Erst alles im Worker, dann separater Streaming-Layer:* Doppelarbeit. Lieber direkt streamen.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **CSV-Streaming-API** — Async-Iterator-basierter Writer
|
||||
- Files: `packages/exports/src/csv/index.ts` (Edit) — neue Funktion `serializeExportTableToCsvStream(writeStream, table, options)`. Nutzt `csv-stringify` im Stream-Mode (`stringify.stringifier({ header: true })`), pipe in `writeStream`. Iteriert über `table.rows` (async iterable). Bestehende `serializeExportTableToCsv(table)` bleibt für Backward-Compat, ruft intern den Streamer mit `PassThrough` und Collect-Buffer.
|
||||
- Tests: `packages/exports/src/csv/stream.test.ts` — assert dass Memory-Footprint bei 1M generierten Rows < 100 MB bleibt (mit `process.memoryUsage`-Snapshots); assert NDJSON-Output korrekt (Header + Rows).
|
||||
- Commit: `03-scaling/streaming-exports: csv stream-writer with async iterator`
|
||||
|
||||
2. **JSON-Streaming-API (NDJSON + Pretty)** — Default NDJSON, Pretty nur für kleine Sets
|
||||
- Files: `packages/exports/src/json/index.ts` (Edit) — `serializeExportTableToJsonStream(writeStream, table, options)`. `options.mode = 'ndjson' | 'pretty'`. Bei `ndjson`: pro Row `JSON.stringify(row) + '\n'`. Bei `pretty` und `rowCount < 10000`: collect in Array, `JSON.stringify(arr, null, 2)`. Bei `pretty` und ≥ 10k: Error `EXPORT_PRETTY_JSON_TOO_LARGE` mit Hint auf NDJSON.
|
||||
- Tests: `packages/exports/src/json/stream.test.ts` — beide Modes, Error-Case.
|
||||
- Commit: `03-scaling/streaming-exports: json/ndjson stream-writer`
|
||||
|
||||
3. **XLSX Streaming-Writer mit Threshold** — ExcelJS-Stream wenn > 50k
|
||||
- Files: `packages/exports/src/xlsx/index.ts` (Edit) — `serializeExportTableToXlsxStream(writeStream, table, options)`. Estimate-RowCount aus `table.estimatedRowCount` (vom Caller geliefert, kein vorab-Count). Wenn `> 50000`: `new ExcelJS.stream.xlsx.WorkbookWriter({ stream: writeStream })`, Worksheet committed Row-by-Row mit `worksheet.addRow(row).commit()`. Wenn `≤ 50000`: bestehender In-Memory-Pfad (kein Regression-Risk für kleine Exports).
|
||||
- Hard-Cap 500k: prüfen am Anfang, wenn `> 500000` → `EXPORT_ROW_LIMIT_EXCEEDED`.
|
||||
- Tests: `packages/exports/src/xlsx/stream.test.ts` — Both-Paths-Tests, Hard-Cap-Test, Stream-Mode-Memory-Check (< 200 MB für 200k Rows × 20 Cols).
|
||||
- Commit: `03-scaling/streaming-exports: xlsx stream-writer with 50k threshold`
|
||||
|
||||
4. **PDF Streaming via renderToStream** — `@react-pdf/renderer`
|
||||
- Files: `packages/exports/src/pdf/index.ts` (Edit) — `serializeExportTableToPdfStream(writeStream, table, options)`. Hard-Cap 5000 — wenn `rowCount > 5000` → `EXPORT_PDF_ROW_LIMIT_EXCEEDED` mit Hint auf CSV/XLSX. Nutzt `renderToStream(<TableDocument table={table} />)` → Stream `pipe(writeStream)`.
|
||||
- Tests: `packages/exports/src/pdf/stream.test.ts` — Render-Success für 1k Rows, Error bei 5001.
|
||||
- Commit: `03-scaling/streaming-exports: pdf renderToStream with 5k cap`
|
||||
|
||||
5. **S3 Multipart-Upload-Sink** — Output-Stream landet direkt im S3-Bucket
|
||||
- Files: `packages/exports/src/sinks/s3-multipart.ts` (neu) — Factory `createS3MultipartUploadSink({ bucket, key, contentType })` returnt `{ writeStream, completionPromise }`. Nutzt `@aws-sdk/lib-storage`'s `Upload`-Klasse mit `partSize: 8 * 1024 * 1024` (8 MB) und `queueSize: 4`.
|
||||
- Tests: `packages/exports/src/sinks/s3-multipart.test.ts` — gegen `aws-sdk-client-mock` oder lokales MinIO im testcontainer.
|
||||
- Commit: `03-scaling/streaming-exports: s3 multipart upload sink`
|
||||
|
||||
6. **Worker-Export-Job refactoren** — Glue alles zusammen, kein Tempfile mehr
|
||||
- Files: `apps/worker/src/jobs/export.ts` (Edit) — Job-Handler:
|
||||
1. Resolve Tenant-Plan → maxExportRowCount.
|
||||
2. Stream Rows via Prisma-Cursor-Iteration (`prisma.row.findMany` in Chunks von 1000 mit Skip-Take oder besser `cursor`-based).
|
||||
3. Pick Format-Streamer.
|
||||
4. Create S3-Multipart-Sink.
|
||||
5. Pipe Streamer → Sink, await completionPromise.
|
||||
6. Update Job-Record mit S3-Key, RowCount, Size, Duration.
|
||||
7. Notify User (email/notification) mit Download-Link.
|
||||
- Progress-Tracking: alle 5k Rows `sharedStore.set(\`export:progress:\${jobId}\`, { processedRows, totalEstimate })` mit TTL 1h.
|
||||
- Tests: `apps/worker/src/jobs/export.test.ts` — Mock Prisma + S3-Sink, End-to-End-Job mit 100k Rows.
|
||||
- Commit: `03-scaling/streaming-exports: worker job pipes directly to s3 multipart`
|
||||
|
||||
7. **Tenant-Plan-Row-Caps** — Per-Plan-Limits resolver
|
||||
- Files: `packages/billing/src/limits.ts` (Edit oder neu wenn nicht da) — `getMaxExportRowCount(tenantId)` returns 10k/100k/Infinity je Plan. Cached via SharedStore (TTL 5min) damit nicht jeder Export-Job DB-roundtrips macht.
|
||||
- `apps/worker/src/jobs/export.ts` (Edit) — Limit-Check vor Job-Start; bei Überschreitung Job mit Error `EXPORT_ROW_LIMIT_PLAN` fail.
|
||||
- UI-Hinweis: existierendes Export-Modal zeigt das Limit (Story für Frontend-Task, hier nur Backend).
|
||||
- Tests: `packages/billing/src/limits.test.ts` — alle 3 Plans, Cache-Hit/Miss.
|
||||
- Commit: `03-scaling/streaming-exports: plan-based row caps for exports`
|
||||
|
||||
8. **Doku: Format-Capabilities-Matrix** — User-/Operator-Doc
|
||||
- Files: `docs/features/exports.md` (neu) — Matrix mit Format × Plan × MaxRows × Streaming-Strategie; Beispiel-Calls; S3-Bucket-Setup-Hinweis; Troubleshooting (Job hängt, Datei korrupt, etc.).
|
||||
- Commit: `03-scaling/streaming-exports: add exports feature doc`
|
||||
|
||||
9. **Metrics-Hook** — Export-Counters in /metrics
|
||||
- Files: `apps/worker/src/metrics.ts` (Edit) — Counter `export_jobs_total{format,status}`, Histogram `export_job_duration_seconds{format}`, Gauge `export_job_rows{format}`.
|
||||
- Commit: `03-scaling/streaming-exports: export job metrics`
|
||||
|
||||
10. **Deprecation der nicht-Streaming-APIs** — Markieren, nicht entfernen (Backward-Compat)
|
||||
- Files: `packages/exports/src/csv/index.ts`, `json/index.ts`, `xlsx/index.ts`, `pdf/index.ts` — JSDoc `@deprecated Use serializeExportTableTo*Stream`. Bleiben funktional bis nächste Major.
|
||||
- Commit: `03-scaling/streaming-exports: deprecate non-streaming serializer apis`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** je Format ein Stream-Test mit Memory-Footprint-Assertion (`process.memoryUsage().heapUsed`-Snapshots vor/nach). 1M-Row-Stress-Test für CSV/NDJSON.
|
||||
- **Integration-Tests:**
|
||||
- `apps/worker/src/jobs/export.integration.test.ts` — Full-Cycle gegen testcontainers (Postgres + Redis + MinIO). Enqueue Export-Job mit 200k synthetischen Rows, assert: S3-Object existiert, Größe plausibel, Job-Progress wird im SharedStore aktualisiert, finaler Job-Status = success.
|
||||
- Plan-Cap-Test: Free-Tenant mit 20k-Row-Export → Job fails mit klarem Error.
|
||||
- **E2E (Playwright):** Frontend-Export-Trigger → poll Job-Status bis success → click Download-Link → assert File-Download (file-size check). Test mit ~5k-Row-Dataset (schnell genug).
|
||||
- **Coverage-Target:** ≥ 85% für `packages/exports/src/*/stream.ts` und `apps/worker/src/jobs/export.ts`.
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.**
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] 4 Stream-Writer-APIs (CSV, NDJSON/Pretty-JSON, XLSX, PDF) implementiert
|
||||
- [ ] S3-Multipart-Sink funktioniert gegen Staging-MinIO + Production-S3
|
||||
- [ ] Worker-Export-Job nutzt Streaming-Pfad, kein Tempfile mehr
|
||||
- [ ] Per-Plan Row-Caps (10k/100k/unlimited) enforced
|
||||
- [ ] Hard-Caps per Format (XLSX 500k, PDF 5k) enforced
|
||||
- [ ] 200k-Row-CSV-Export läuft mit < 200 MB Worker-Memory
|
||||
- [ ] Progress-Tracking via SharedStore funktioniert (5k-Row-Granularität)
|
||||
- [ ] `docs/features/exports.md` enthält Capabilities-Matrix
|
||||
- [ ] Metrics in `/metrics` sichtbar
|
||||
- [ ] Alte Non-Stream-APIs deprecated (JSDoc), nicht entfernt
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration, e2e)
|
||||
- [ ] Staging-Smoke-Test: 100k-Row-Export endet erfolgreich in < 3 Min
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Code-Rollback:** Schrittweise Commits revertieren. Da alte Non-Stream-APIs bewusst deprecated aber funktional bleiben, kann der Worker-Job-Refactor (Step 6) einzeln zurückgerollt werden ohne die neue Streamer-Infrastruktur zu verlieren.
|
||||
- **Feature-Flag:** `EXPORT_STREAMING_ENABLED=false` im Env als Env-Switch → Worker-Job fällt auf alten Non-Stream-Pfad zurück. Im Code als if-branch in `apps/worker/src/jobs/export.ts`. Empfohlen für die ersten Wochen Production.
|
||||
- **S3-Multipart-Abort:** Falls Multipart-Upload mittendrin failt, AWS-SDK ruft auto `AbortMultipartUpload` — kein orphaned Multipart-Garbage.
|
||||
- **DB:** keine Schema-Changes.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Plan-Cap-Bypass-Prevention:** Limit-Check passiert im Worker (nicht nur Frontend) → kann nicht via DevTools umgangen werden.
|
||||
- **S3-Object-ACLs:** Export-Files in S3 sind Private (no public-read). Download-Link ist Signed-URL mit TTL 1h.
|
||||
- **Sensitive-Data in Export:** Tenant-Isolation muss im Prisma-Query stehen (WHERE tenantId = …). Add-Audit-Log-Entry pro Export für Compliance.
|
||||
- **DoS-Risiko:** Auch mit Streaming könnte ein Enterprise-Tenant 10M-Row-Exports anstoßen und Workers belegen. Mitigation via Job-Priority (siehe `job-priority.md`) und Hard-Caps pro Format.
|
||||
- **Multipart-Upload-Credentials:** S3-Creds aus Env, gleich wie für andere Storage-Ops. Keine Sonderbehandlung nötig.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD + Audit der bestehenden `packages/exports/`-Struktur als Input.
|
||||
- **Research:** `Explore`-Subagent um aktuelle Format-Implementierungen zu mappen (CSV-/JSON-/XLSX-/PDF-Module bereits da?) — verhindert Wheel-Reinvention.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Format-Step (4 Steps), getrennt vom Worker-Glue-Step. S3-Sink eigener Subagent.
|
||||
- **Tests:** separater `general-purpose`-Subagent — wichtig: Stress-Tests (Memory-Asserts) brauchen Sorgfalt damit sie nicht flaky werden.
|
||||
- **Verification:** `gsd-verifier`-Subagent gegen Acceptance-Criteria.
|
||||
- **Code-Review:** `gsd-code-reviewer` nach Phasen-Abschluss.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-streaming-exports` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging. Sicherstellen dass `worker-import`-Service vorhanden ist (`worker-queues.md` Task muss durch sein).
|
||||
3. MinIO auf Staging hochziehen (falls noch nicht):
|
||||
```bash
|
||||
docker run -d --name minio \
|
||||
-p 9000:9000 -p 9001:9001 \
|
||||
-e MINIO_ROOT_USER=admin \
|
||||
-e MINIO_ROOT_PASSWORD=admin12345 \
|
||||
minio/minio server /data --console-address ":9001"
|
||||
```
|
||||
4. Smoke-Tests:
|
||||
```bash
|
||||
# 1. Small export (CSV, 1k rows)
|
||||
curl -X POST https://staging.../api/exports -d '{ "format": "csv", "table": "users" }' -H "Authorization: Bearer ..."
|
||||
# poll job-status until success, download file, check row-count
|
||||
|
||||
# 2. Large stream export (CSV, 200k rows, Enterprise plan)
|
||||
# → assert duration < 60s, file size > 0, memory peak < 200 MB im Worker (Grafana)
|
||||
|
||||
# 3. Plan-cap test (Free plan, 20k requested)
|
||||
# → assert job fails with EXPORT_ROW_LIMIT_PLAN
|
||||
```
|
||||
5. Grafana-Dashboard prüfen: `export_jobs_total{status="success"}` zählt korrekt; `export_job_duration_seconds_bucket` zeigt Verteilung.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Plan-cap helper lives in `@toolctrl/exports/limits` instead of a separate `packages/billing`.** There is no `@toolctrl/billing` package today; placing the resolver alongside the streamers keeps the cap definitions and the per-format hard caps colocated, which is where the consumer (worker job + UI surface) already lives.
|
||||
- **No metrics counter / no JSDoc deprecation pass on the non-streaming APIs.** The detail plan called for `export_jobs_total` Prometheus counters and a wide JSDoc `@deprecated` sweep. These two follow-ups are intentionally deferred: (a) metrics wiring requires a worker bootstrap touch that overlaps with #45/#39, (b) the deprecation pass is a one-line annotation we can ship in a small follow-up commit without blocking the streaming pipeline.
|
||||
- **Worker job is a factory (`createExportJobRunner`) rather than a default-exported handler.** Keeps the unit tests deterministic without spinning up a real S3 / Prisma — the production wiring (resolveTenantPlan, rowSource, buildArtifactKey) is the operator's responsibility and lives outside the streaming pipeline.
|
||||
- **AWS SDK versions pinned to 3.1045.0** (exact, not caret) on both `apps/worker` and `packages/exports` to avoid the `@smithy/types` v4.14.1↔v4.14.2 incompatibility that surfaces when `@toolctrl/storage` (`^3.958.0`) resolves to 1045 but the unconstrained caret in newer packages resolves to 1048.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — 10 atomic commits land the streaming export pipeline:
|
||||
- `2a061a9d` CSV stream-writer with async-iterable row source + memory smoke test (100k rows < 100 MB).
|
||||
- `a28fbb6d` JSON/NDJSON stream-writer with 10k pretty-mode cap and `ExportPrettyJsonTooLargeError`.
|
||||
- `35712412` XLSX stream-writer choosing ExcelJS WorkbookWriter above 50k rows; 500k hard cap + dedicated error type.
|
||||
- `9b5057e0` PDF `renderToStream` with 5k row hard cap (`ExportPdfRowLimitExceededError`).
|
||||
- `f3e66865` S3 multipart upload sink (`@aws-sdk/lib-storage`, 8 MB parts, queueSize 4) returning `{ writeStream, completionPromise }`.
|
||||
- `309e7177` Plan-based row cap helper (`resolveExportRowCap` / `assertExportRowCap`) covering free / pro / enterprise.
|
||||
- `881219bb` Worker export-job factory `createExportJobRunner` wiring resolveTenantPlan → cap → rowSource → format streamer → S3 sink → shared-store progress + S3 SDK pin.
|
||||
- `56a732b0` `GET /v1/exports/:exportJobId/progress` endpoint backed by the shared-store snapshot.
|
||||
- `1ecbb9ad` `docs/ops/streaming-exports.md` runbook.
|
||||
- (this commit) detail-md progress log + PROGRESS.md row update.
|
||||
- 67 unit tests cover all four streamers, the multipart sink, the plan-cap matrix (3 plans × 4 formats), the worker job (4 scenarios incl. progress-flush) and the API progress reader. `pnpm --filter @toolctrl/exports test` + `apps/worker` + `apps/api` exports/progress test all green.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional am Task-Ende.)
|
||||
@@ -0,0 +1,311 @@
|
||||
# Tenant-Snapshot mit PII-Anonymisierung + Restore
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #44
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 1 Woche
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Dev/QA/Sales-Teams brauchen regelmäßig „Production-ähnliche" Test-Datasets, aber:
|
||||
|
||||
- **Direkter Prod-DB-Dump in Dev = DSGVO-Verstoß** (PII, Stripe-IDs, Emails, IPs).
|
||||
- **Manuelle Anonymisierung ist fehleranfällig** (vergessene Felder, inkonsistente Replacement-Patterns, kein Tracking welche Snapshots welches Anonymisierungs-Profile haben).
|
||||
- **Bug-Reproduction von einem spezifischen Tenant-State ist heute unmöglich** — Operator müsste komplette DB ziehen, was wegen DSGVO blockiert.
|
||||
- **Sales-Demos brauchen realistische Tenants** mit Workflows, Subscriptions, Audit-Trail — Synthetic-Data hat nie die Komplexität.
|
||||
- **QA-Test-Datasets sind hand-kuratierte Fakten-Sammlungen** → veralten, decken Edge-Cases nicht ab.
|
||||
|
||||
Use-Cases die wir freischalten:
|
||||
1. **Bug-Repro:** „Customer X meldet Bug in Workflow Y" → Snapshot ziehen, in Dev-Env restore, debuggen.
|
||||
2. **QA-Datasets:** monatlich pro Plan-Tier (Free/Pro/Enterprise) ein Snapshot eines repräsentativen Tenants → CI-Test-Fixture.
|
||||
3. **Sales-Demos:** Snapshot eines „idealen" Production-Tenants → in Demo-Env restore → realistisches Show-and-Tell.
|
||||
4. **Tenant-Migration:** Tenant zu separater Instance umziehen (z.B. Enterprise-Tenant auf dediziertem Server).
|
||||
|
||||
Betroffene Codestellen:
|
||||
- Prisma-Schema: alle Models mit `tenantId`-Spalte (User, Row, Entity, Workflow, Subscription, AuditLog, etc.) — Subagent enumeriert alle in Research-Phase.
|
||||
- Storage-Layout: `tenants/<tenantId>/...` Convention.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/backup-cli.md` — Tenant-Snapshot baut auf der gleichen CLI-Infrastruktur (Citty-Commands, Encryption-Layer, S3-Sink, Manifest-Schema-Pattern). Reuse statt Parallelentwicklung.
|
||||
- `01-tech-debt-foundation/supabase-removal.md` — Saubere einheitliche Storage-Abstraktion damit Tenant-Files konsistent iterierbar sind.
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Tenant-Plan-Info-Lookup im Anonymisierungs-Profile (verschiedene Anonymisierungs-Strenge je Plan möglich, z.B. Enterprise-Tenants wollen vielleicht „weniger" anonymisiert für Sales-Repros).
|
||||
|
||||
## Blocks
|
||||
|
||||
- `08-mobile-cli-storybook-sdk/cli-ops.md` — CLI-Ops-Package bündelt Tenant-Snapshot als weiteres Sub-Tool.
|
||||
- `10-compliance-polish/dsgvo-bulk-ops.md` — DSGVO-Datenexport pro User profitiert vom gleichen Selector-Pattern (tenantId + userId-Filter).
|
||||
|
||||
## Approach
|
||||
|
||||
**CLI-Command `tenant:snapshot` und `tenant:restore` im selben `packages/cli/`-Package wie Backup, separater Command-Tree.**
|
||||
|
||||
**Anonymisierungs-Profile (deklarativ):**
|
||||
- `profile=full-anonymized` (Default für Snapshots) — alle PII ersetzt, deterministisch (gleicher Input → gleicher Output für Foreign-Key-Konsistenz).
|
||||
- `profile=minimal-anonymized` — nur Felder die DSGVO-strikt sind (Email, Phone, Address, IP, Stripe-IDs); FirstName/LastName bleiben.
|
||||
- `profile=none` — KEINE Anonymisierung. Nur durch Operator mit Super-Admin-Recht erlaubt, hard-gated, separater Audit-Eintrag.
|
||||
|
||||
**Anonymisierungs-Mapping (`profile=full-anonymized`):**
|
||||
|
||||
| Feld-Typ | Mapping |
|
||||
|---|---|
|
||||
| Email | `user{seq}@anonymized.test` — deterministisch via SHA256(originalEmail)[:8] |
|
||||
| FirstName | `faker.person.firstName(seed=hashedId)` |
|
||||
| LastName | `faker.person.lastName(seed=hashedId)` |
|
||||
| Phone | `null` |
|
||||
| Address (Street/City/Country) | `faker.location.*(seed=hashedId)` |
|
||||
| ZipCode | erste 2 Ziffern behalten + Rest random (Postleitzahl-Bereich) |
|
||||
| IPAddress | `127.0.0.{seq % 255}` |
|
||||
| Stripe-CustomerID | `cus_anon{seq}` |
|
||||
| Stripe-SubscriptionID | `sub_anon{seq}` |
|
||||
| Stripe-PaymentMethodID | `pm_anon{seq}` |
|
||||
| API-Keys / Secrets (in Settings-JSON) | `[REDACTED]` |
|
||||
| Webhook-URLs (custom) | `https://anonymized.test/webhook/{seq}` |
|
||||
| User-Avatar-URLs | `https://anonymized.test/avatar/{seq}.png` (Storage-Objects werden trotzdem mitexportiert aber Public-Hosts darin rewritten) |
|
||||
|
||||
**Konsistenz-Hash:** Pro Snapshot wird ein Salt generiert, alle Anonymisierungen nutzen `hash(salt + originalValue)` als Seed. → Innerhalb eines Snapshots ist Anonymisierung deterministisch (User Alice referenziert in Workflow X = User Alice referenziert in Workflow Y), aber zwischen Snapshots verschieden (kein Cross-Snapshot-Reverse-Engineering).
|
||||
|
||||
**Snapshot-Format:** Gleiche Struktur wie Backup, aber tenant-scoped.
|
||||
```
|
||||
toolctrl-snapshot-<tenant-slug>-<iso8601>-<short-uuid>.tar.gz
|
||||
├── manifest.json # Tenant-ID, Source-Slug, Schema-Version, Anonymization-Profile, Anonymization-Salt-Hash, ExportDate, CreatedBy
|
||||
├── prisma/
|
||||
│ └── data.jsonl # NDJSON: { model: 'User', data: {...} } — pro Row eine Zeile
|
||||
├── storage/
|
||||
│ ├── manifest.jsonl # { originalKey, newKey, size, etag }
|
||||
│ └── objects.tar
|
||||
└── audit/
|
||||
└── snapshot-event.json # Audit-Trail-Entry für diesen Snapshot (im Source-System)
|
||||
```
|
||||
|
||||
**Restore-Logik:**
|
||||
- `tenant:restore <snapshot> --target-tenant-slug=<new-slug>` legt neuen Tenant in Target-DB an (oder failt wenn slug existiert ohne `--force`).
|
||||
- Rewrited alle FK-Beziehungen (Snapshot-Tenant-ID → Neu-Tenant-ID), generiert neue IDs für alle Rows (UUID-Remapping-Table im Memory).
|
||||
- Storage-Objects landen unter `tenants/<new-tenant-id>/...` in der lokal-konfigurierten Storage.
|
||||
- Sanity-Check am Ende: alle FKs valid, keine dangling references.
|
||||
|
||||
**Permission-Model:**
|
||||
- Source-System (Snapshot-Export): **nur Super-Admin** (Plattform-Owner). Tenant-Admin darf NICHT seinen eigenen Tenant snapshotten — könnte Daten anderer User des Tenants exfiltrieren in unverschlüsselter Form. Mitigation: User-DSGVO-Export ist separater Pfad (Task `10-compliance/dsgvo-bulk-ops.md`).
|
||||
- Target-System (Restore): jeder Operator des Ziel-Systems (Dev-Box etc.). Kein hardes Gate, weil Snapshot-File ist die Berechtigung.
|
||||
- Audit-Log-Eintrag pro Snapshot im Source-System (Actor, TenantID, Profile, Timestamp).
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Anonymisierung in einem separaten Post-Processing-Step (Snapshot raw, dann anonymize):* doppelter Speicher, Risiko dass raw Snapshot „vergessen" wird. Lieber inline.
|
||||
- *DB-Trigger-basierte Anonymisierung in einer separaten Anonymized-Replica:* zu viel Infra-Aufwand für Toolctrl-Boilerplate-Größe.
|
||||
- *Library wie pg_anonymize:* nicht TS-nativ, kein Storage-Backup, kein Foreign-Key-Awareness wie wir es brauchen.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Tenant-Model-Enumeration** — alle Tenant-scoped Models entdecken
|
||||
- Files: `packages/cli/src/commands/tenant/models.ts` (neu) — exportiert `TENANT_SCOPED_MODELS: Array<{ model: string; tenantField: string }>`. Enthält explizite Liste (User, Row, Entity, Workflow, Subscription, AuditLog, ApiKey, EmailMarketingCampaign, etc.). Subagent recherchiert Prisma-Schema und liefert vollständige Liste.
|
||||
- Tests: `models.test.ts` — assert dass jede Prisma-Model-Klasse mit `tenantId`-Spalte in der Liste ist (introspection via `Prisma._dmmf`).
|
||||
- Commit: `03-scaling/tenant-snapshot: enumerate tenant-scoped models`
|
||||
|
||||
2. **Anonymizer-Engine** — Profile-basierte Field-Transformations
|
||||
- Files:
|
||||
- `packages/cli/src/commands/tenant/anonymizer.ts` — Klasse `Anonymizer({ profile, salt })`. Methode `transform(model, row)` returnt anonymisiertes Row-Object. Lookup-Table pro Model: welche Felder welche Transform-Funktion (Email-Anonymizer, Faker-Person-Name, IP-Mask, etc.).
|
||||
- `packages/cli/src/commands/tenant/profiles.ts` — die 3 Profile als Konfiguration.
|
||||
- Faker-Seed-Mechanismus: `faker.seed(parseInt(hash(salt + rowId), 16) % MAX_INT)`.
|
||||
- Tests: `anonymizer.test.ts` — pro Profile pro Feld-Typ ein Test; Determinismus-Test (gleicher salt + input → gleicher output).
|
||||
- Commit: `03-scaling/tenant-snapshot: anonymizer engine with 3 profiles`
|
||||
|
||||
3. **Data-Extractor** — Prisma-Query pro Model filtered by tenantId
|
||||
- Files: `packages/cli/src/commands/tenant/extractor.ts` — `extractTenantData(prisma, tenantId, anonymizer)` async iterator yieldet `{ model, data }` Objects. Nutzt Cursor-basiertes Paging (1000er-Chunks) damit large Tenants nicht den Speicher sprengen. Foreign-Key-Order-aware: User vor Row, Row vor RowSetting, etc. (Topological-Sort der Models).
|
||||
- Tests: `extractor.test.ts` — Test-Tenant in testcontainers-Postgres, assert dass alle Models extrahiert werden, anonymisiert, in richtiger FK-Order.
|
||||
- Commit: `03-scaling/tenant-snapshot: cursor-based data extractor`
|
||||
|
||||
4. **Storage-Extractor** — Tenant-Files aus S3 holen, ggf. anonymized rewrites
|
||||
- Files: `packages/cli/src/commands/tenant/storage-extractor.ts` — listet alle Objects mit Prefix `tenants/<tenantId>/`, download streamt in tar, schreibt `manifest.jsonl` mit `{ originalKey, newKey, size, etag }`. `newKey` ist `tenants/SNAPSHOT_TENANT_ID/...` mit Platzhalter, der beim Restore ersetzt wird.
|
||||
- Tests: `storage-extractor.test.ts` — gegen MinIO mit 50 Test-Files unter `tenants/abc/...`, assert alle drin.
|
||||
- Commit: `03-scaling/tenant-snapshot: storage extractor for tenant files`
|
||||
|
||||
5. **`tenant:snapshot`-Command** — Glue
|
||||
- Files: `packages/cli/src/commands/tenant/snapshot.ts` — Citty-Command.
|
||||
- Args: `<tenantId>`, `--target=<output-file>` (default `./toolctrl-snapshot-<tenant-slug>-<ts>.tar.gz`), `--profile=full-anonymized|minimal-anonymized|none`, `--include-storage` (default true), `--encrypt=age|none`
|
||||
- Flow:
|
||||
1. Permission-Check: SUPER_ADMIN_API_KEY env-set, prompt for confirmation if `profile=none`.
|
||||
2. Generate Salt.
|
||||
3. Iterate `extractTenantData` → schreibe `prisma/data.jsonl` (im tempdir).
|
||||
4. Storage-Extract → `storage/`.
|
||||
5. Manifest erzeugen (inkl. Salt-Hash, nicht Salt selbst).
|
||||
6. tar.gz.
|
||||
7. Encrypt (optional).
|
||||
8. AuditLog-Entry im Source-System (separater Prisma-Write).
|
||||
- Tests: `snapshot.integration.test.ts` — gegen testcontainers, assert Output-Datei existiert + parsbar + Anonymisierung wirksam (Sample-Check: kein Email-Pattern `*@*.com` außer `*@anonymized.test`).
|
||||
- Commit: `03-scaling/tenant-snapshot: tenant:snapshot command`
|
||||
|
||||
6. **Restorer + FK-Remapper** — Snapshot in Target-DB einspielen
|
||||
- Files: `packages/cli/src/commands/tenant/restorer.ts` — `restoreTenantSnapshot(prisma, snapshotPath, { targetSlug })`. Flow:
|
||||
1. Untar.
|
||||
2. Neuer Tenant-Row erstellen (slug aus `--target-tenant-slug`).
|
||||
3. ID-Mapping-Table: `Map<oldId, newId>` pro Model (alle Primary-Keys neu generieren).
|
||||
4. Iterate NDJSON, FK-Felder via Mapping ersetzen, Insert.
|
||||
5. Storage-Objects: Tar entpacken, jeden Key `tenants/SNAPSHOT_TENANT_ID/...` → `tenants/<new-tenant-id>/...` umbenennen, upload.
|
||||
6. Sanity-Check: Foreign-Key-Integrity (`SELECT COUNT(*) FROM Row WHERE tenantId NOT IN (SELECT id FROM Tenant)` muss 0 sein, etc. pro Model).
|
||||
- Tests: `restorer.test.ts` — Full-Cycle: snapshot eines Tenants → restore in zweiter DB → assert Row-Counts matchen + Sample-File downloadbar.
|
||||
- Commit: `03-scaling/tenant-snapshot: restorer with fk remapping`
|
||||
|
||||
7. **`tenant:restore`-Command** — CLI-Wrapper für Restorer
|
||||
- Files: `packages/cli/src/commands/tenant/restore.ts` — Citty-Command. Args: `<snapshot-file>`, `--target-tenant-slug=<slug>`, `--target-db=<connStr>` (default aus Env), `--force` (existing slug erlaubt).
|
||||
- Confirmation-Prompt (außer `--yes`).
|
||||
- Commit: `03-scaling/tenant-snapshot: tenant:restore command`
|
||||
|
||||
8. **Audit-Log-Hook** — Im Source-System loggen
|
||||
- Files: `packages/cli/src/commands/tenant/audit.ts` — `logSnapshotEvent(prisma, { tenantId, actorUserId, profile, snapshotKey })` schreibt AuditLog-Entry mit Event `TENANT_SNAPSHOT_CREATED`. Permission-Check: nur Super-Admin-API-Key darf Snapshot triggern.
|
||||
- Tests: `audit.test.ts` — assert Entry geschrieben.
|
||||
- Commit: `03-scaling/tenant-snapshot: audit log entry for snapshot ops`
|
||||
|
||||
9. **Doku** — `docs/operations/tenant-snapshot.md`
|
||||
- Files: neu. Inhalt:
|
||||
- Use-Cases (Bug-Repro, QA, Sales-Demos, Migration)
|
||||
- Anonymisierungs-Profile + Mapping-Tabelle
|
||||
- Permission-Model (Source vs Target)
|
||||
- CLI-Beispiele für alle 3 Use-Cases
|
||||
- Restore-in-Dev-Workflow (mit lokal Setup)
|
||||
- Sicherheitswarnung für `profile=none`
|
||||
- Troubleshooting (FK-Errors beim Restore, Storage-Bucket-Mismatch)
|
||||
- Commit: `03-scaling/tenant-snapshot: operations doc`
|
||||
|
||||
10. **Lint-Rule für PII-Fields** — Optional aber empfohlen
|
||||
- Files: `packages/cli/src/commands/tenant/lint.ts` (oder als Test) — ESLint-Custom-Rule oder Vitest-Test der das Prisma-Schema scant und sicherstellt, dass alle Felder mit `@map("email")`, `@map("phone")`, etc. im Anonymizer-Mapping abgedeckt sind. Verhindert vergessene PII-Felder bei Schema-Erweiterungen.
|
||||
- Tests: `lint.test.ts` — synthetisches Schema mit „vergessenem" PII-Feld → Lint failed.
|
||||
- Commit: `03-scaling/tenant-snapshot: schema lint for pii field coverage`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `anonymizer.test.ts` — pro Profile pro Feld-Typ; Determinismus-Test.
|
||||
- `models.test.ts` — vollständige Tenant-Model-Coverage gegen Prisma-DMMF.
|
||||
- `extractor.test.ts` — Cursor-Paging, FK-Order.
|
||||
- `restorer.test.ts` — FK-Remapping, Sanity-Checks.
|
||||
- `audit.test.ts` — Permission-Gate funktioniert.
|
||||
- **Integration-Tests:**
|
||||
- `packages/cli/test/integration/tenant-snapshot-cycle.test.ts` — End-to-End: 2-Tenant-Test-DB → snapshot Tenant A → restore in zweite DB als Slug `tenant-a-clone` → assert:
|
||||
- alle Tenant-A-Rows da (Counts identisch)
|
||||
- alle Anonymisierungen angewendet (Sample-Emails matchen `*@anonymized.test`)
|
||||
- alle FKs valid
|
||||
- Storage-Objects unter neuem Tenant-Prefix
|
||||
- Audit-Eintrag im Source-System
|
||||
- **E2E:** kein UI → kein Playwright.
|
||||
- **Coverage-Target:** ≥ 85% für `packages/cli/src/commands/tenant/`.
|
||||
- **Test-Files:** wie aufgelistet.
|
||||
|
||||
**NICHT als completed markieren bevor alle Tests grün sind.** Daten-Anonymisierungs-Bugs sind DSGVO-Risiken.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `tenant:snapshot` und `tenant:restore` CLI-Commands implementiert
|
||||
- [ ] 3 Anonymisierungs-Profile (full/minimal/none) wählbar
|
||||
- [ ] Anonymisierung deterministisch (Salt-basiert) → FK-Konsistenz innerhalb Snapshot
|
||||
- [ ] Snapshot enthält Manifest mit Anonymisierungs-Profile-Hinweis
|
||||
- [ ] Storage-Objects mitexportiert und beim Restore unter neuem Tenant-Prefix landen
|
||||
- [ ] FK-Remapping beim Restore funktioniert (Sanity-Check grün)
|
||||
- [ ] Permission: Snapshot nur mit Super-Admin-API-Key
|
||||
- [ ] Audit-Log-Entry pro Snapshot im Source-System
|
||||
- [ ] PII-Coverage-Lint sichert alle bekannten PII-Felder
|
||||
- [ ] `docs/operations/tenant-snapshot.md` enthält alle 4 Use-Cases
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration)
|
||||
- [ ] Staging-Verifikation: snapshot real-Tenant → restore in Dev-Box → manueller Smoke-Test in Dev-UI
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Code-Rollback:** Eigenes CLI-Package, kein Production-Impact. Commits revertieren.
|
||||
- **Existierende Snapshots:** bleiben gültig (versioniert via Manifest-Schema-Version). Restore-Pfad muss alte Schema-Versionen weiterhin lesen können → Migration-Logic in `restorer.ts` falls Schema sich ändert.
|
||||
- **Audit-Log-Entries:** bleiben bestehen (read-only by design).
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **PII-Vollständigkeit:** Höchstes Risiko. Wenn ein PII-Feld in der Anonymizer-Map vergessen wird, leakt es im Snapshot. Mitigation:
|
||||
- PII-Coverage-Lint (Step 10).
|
||||
- Code-Review mit Security-Track-Subagent.
|
||||
- Stichproben-Test: Snapshot eines Test-Tenants generieren, `grep -E "[\\w.]+@[\\w.]+\\.[a-z]+" snapshot.tar.gz` (nach Decompression) sollte nur `@anonymized.test`-Matches finden.
|
||||
- **`profile=none` Missbrauch:** Hard-gated via Confirmation-Prompt + Audit-Log + Super-Admin-Permission. Doku enthält explizite Warnung dass das DSGVO-Verletzung sein kann wenn unverschlüsselt geteilt.
|
||||
- **Salt-Speicherung:** Nur Salt-Hash im Manifest (für Reproduzierbarkeits-Check), nicht Salt selbst. Salt wird verworfen nach Snapshot-Erstellung — keine Reverse-Engineerable.
|
||||
- **Snapshot-File-Distribution:** Snapshots sind selbst sensitiv (auch anonymisiert: Geschäftsdaten, Workflows). Encryption-by-Default (age) empfohlen. Doku weist darauf hin.
|
||||
- **Restore-in-Production-Schutz:** `tenant:restore` checkt `process.env.NODE_ENV !== 'production'` ODER explizites `--allow-production-restore`-Flag (gefährliche Operation). Standard: Production-Restore failed mit klarer Error-Message.
|
||||
- **Stripe-IDs:** Müssen ersetzt werden, sonst könnte ein Dev-Tenant-Restore versehentlich gegen die echte Stripe-API operieren. Im Mapping abgedeckt.
|
||||
- **Webhook-URLs:** Wenn ein Tenant einen Custom-Webhook bei Drittsystem registriert hat (`https://customer-x.com/webhook`), und der Restore-Tenant feuert das ab → Drittsystem bekommt Test-Daten. Mapping ersetzt durch `anonymized.test`-URL → kein Outgoing.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Detail-MD; recherchiert ob `packages/cli/` schon existiert (aus `backup-cli.md`-Task) für Code-Reuse.
|
||||
- **Research:** `Explore`-Subagent enumeriert vollständig alle Tenant-scoped Models aus Prisma-Schema und alle PII-Felder. Output → `models.ts` und `anonymizer.ts` Profile-Definition.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Step. Step 2 (Anonymizer) und Step 6 (Restorer) sind die kritischsten — eigene Subagents mit besonderer Sorgfalt.
|
||||
- **Tests:** separater `general-purpose`-Subagent, Schwerpunkt auf Determinismus-Tests und FK-Sanity-Checks.
|
||||
- **Verification:** `gsd-verifier`-Subagent + zusätzlich `gsd-security-reviewer` (falls existent) auf den Anonymizer für PII-Coverage.
|
||||
- **Code-Review:** `gsd-code-reviewer` mit Fokus auf Daten-Leak-Pfade.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-tenant-snapshot` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging.
|
||||
3. Setup: Staging-Tenant mit realistischem Test-Dataset (mind. 100 Users, 500 Rows, 10 Workflows, 50 Storage-Files).
|
||||
4. Smoke-Tests:
|
||||
```bash
|
||||
# Snapshot
|
||||
ssh staging-host "docker compose run --rm cli toolctrl-cli tenant:snapshot \
|
||||
<tenant-id> \
|
||||
--target=/snapshots/tenant-acme-test.tar.gz \
|
||||
--profile=full-anonymized \
|
||||
--include-storage \
|
||||
--encrypt=age"
|
||||
|
||||
# Download lokal
|
||||
scp staging-host:/snapshots/tenant-acme-test.tar.gz ~/
|
||||
|
||||
# PII-Stichprobe
|
||||
age -d -i ~/.age-key.txt ~/tenant-acme-test.tar.gz | tar -xzO prisma/data.jsonl | grep -E "@" | head
|
||||
# → erwartet: nur @anonymized.test, keine echten Emails
|
||||
|
||||
# Restore in lokale Dev-DB
|
||||
pnpm tsx packages/cli/bin/toolctrl-cli tenant:restore \
|
||||
~/tenant-acme-test.tar.gz \
|
||||
--target-tenant-slug=acme-restored \
|
||||
--target-db=postgres://dev:dev@localhost:5432/devdb
|
||||
```
|
||||
5. Manueller Smoke-Test in lokaler Dev-UI: Login als restorierte User, Workflows ansehen, Subscription-State plausibel.
|
||||
6. Permission-Test: ohne `SUPER_ADMIN_API_KEY` → Command failed mit klarer Message.
|
||||
7. Audit-Eintrag in Source-DB prüfen.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **No external faker dep.** Spec mentioned `@faker-js/faker`. Implementation uses a small curated word-pool seeded by `sha256(salt + value)` to keep the CLI free of an additional 500 KB dependency and to make every anonymizer output 100% deterministic. The mapping in the runbook lists every field-type and its strategy.
|
||||
- **Audit log is file/stdout, not a Prisma write.** The CLI deliberately does not depend on `@prisma/client` (mirroring the backup-cli approach). The audit module emits a structured JSON line that operators redirect into their audit pipeline. Future work can replace this with a DB write when the CLI adopts a runtime Prisma adapter.
|
||||
- **Prisma is dependency-injected.** Snapshot and restore expose a `PrismaRunner` / `PrismaRestoreRunner` interface; production consumers wire up the real client in `bin/toolctrl-cli.mjs`. This avoids a hard Prisma dep in the CLI package and makes unit tests trivial.
|
||||
- **FK remapping is `tenantId`-only.** None of the tenant-scoped models reference other tenant-scoped rows by id in a way that would survive a snapshot — everything roots at `tenantId`. Row primary keys are kept as the source emitted them.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- **2026-05-25 (Task G — globaler-Row-Extractor, schließt B7-Design-Limit):**
|
||||
- Commit 1 (`e9fe18fc`) — `models.ts::GLOBAL_REFERENCED_MODELS` + `PLATFORM_CATALOG_GLOBALS` enumerieren jede `User`-Referenz aus tenant-scoped Models (38 FK-Spalten in 31 Source-Models) und kategorisieren die übrigen Schema-Globals (Entity/EntityGroup/Formula/Onboarding/PromptFlow/RowComment/WorkflowBlock) als deployment-baseline. `models.test.ts` parsed jedes `prisma/schema/*.prisma` und scheitert wenn eine neue FK-Relation eingebaut wird die in keiner der beiden Listen steht — verhindert stille Regressionen.
|
||||
- Commit 2 (`6e0aafa3`) — `globals-runner.ts::collectAndWriteGlobals()`: streamed jede tenant-JSONL, sammelt FK-IDs pro Global, chunked `findManyByIds` (default 500), Anonymizer (gleicher Salt wie tenant-Slice), schreibt `globals/<Model>.jsonl`. Manifest-Schema um optionales `globals: TenantManifestGlobalsEntry[]` erweitert (backward-compat: alte Snapshots haben kein `globals` und werden weiter akzeptiert). `PrismaRunner` extends `Partial<GlobalsPrismaRunner>` mit `findManyByIds(model, ids)`; `createPrismaSnapshotRunner` implementiert es. CLI-Flag `--include-globals=true|false` (default true). Anonymizer-Coverage für `User`: email/firstName/lastName/phone/avatar scrubbed; passwordHash, verifyToken, githubId, googleId, totpSecretEncrypted, totpPendingSecretEncrypted, mfaRecoveryCodeHashes, deletionRequestId nullified. 6 globals-runner-Tests + 3 snapshot-Tests + 2 manifest-Tests + 4 anonymize-Tests.
|
||||
- Commit 3 (`a4cdf6ef`) — Restore replayed globals VOR tenant-scoped slice. Drei Modi:
|
||||
- `upsert` (default): findUnique + create-loop, idempotent, gibt `{created, updated}` zurück
|
||||
- `skip`: createMany({skipDuplicates:true}), legacy-runner-kompatibel
|
||||
- `error`: aborts wenn `updated > 0` (sensitiver Restore in known-empty DB)
|
||||
`PrismaRestoreRunner.upsertManyById?` optional; legacy-fakes fallen automatisch auf `skip`-Semantik zurück mit warn. CLI-Flag `--globals=upsert|skip|error`. `RunRestoreResult.globals[]` mit per-model created/existed/expected counters. 5 restore-Tests + 4 prisma-runner-Tests.
|
||||
- Commit 4 (`27cad89d`) — `round-trip.integration.test.ts` neuer dritter Case: seedet je 1 Row in den 6 zuvor-broken Models (TenantUser, TenantIpAddress, UserRole, Event, Log, SupportTicket) + 2 referenzierte User auf Source. Target hat den Target-Tenant aber KEINE referenzierten User (verified via `target.user.count() === 0` pre-check). Round-Trip führt Globals-Replay zuerst aus, dann tenant-scoped Models — alle 6 reportet `written === expected > 0`, jeder restored TenantUser-FK resolved gegen `target.user.findUnique`. **Live verifiziert** gegen zwei frische `pgvector/pgvector:pg16` Container (Ports 55801/55802) mit allen 138 Migrationen: **365/365 Tests grün**.
|
||||
- Commit 5 (`<docs>`) — Runbook entschärft: `## Restore prerequisites: global rows are NOT in the snapshot` → `## How globals are handled in restore` mit 3-Modi-Tabelle, platform-catalog-Erklärung, Backward-Compat-Note, Anonymization-Block. Troubleshooting um 2 neue Symptome erweitert.
|
||||
- **Coverage**: Nur `User` ist heute in `GLOBAL_REFERENCED_MODELS`. Andere FK-Targets (Entity/Formula/EntityGroup/Onboarding/PromptFlow/RowComment/WorkflowBlock) sind als platform-catalog markiert und bleiben deployment-baseline; per-tenant-Kopie wäre semantisch falsch. Der Test scheitert wenn jemand eine neue tenant-scoped FK in irgendeine andere globale Tabelle einbaut → forciert bewusste Entscheidung.
|
||||
- **Live-Verifikation**: durchgeführt via `TENANT_TEST_DATABASE_URL` + `TENANT_TEST_TARGET_DATABASE_URL` gegen Docker-pgvector-DBs. Alle 365 Tests inkl. der 3 round-trip-Cases grün. Container nach Lauf gestoppt.
|
||||
|
||||
- **2026-05-17 (agent/03-tenant-snapshot):**
|
||||
- Commit 1 — scaffolded `tenant` command group under `packages/cli` (registered in `src/index.ts`).
|
||||
- Commit 2 — enumerated **54 tenant-scoped Prisma models** (`models.ts`); Vitest parses every `prisma/schema/*.prisma` to enforce coverage.
|
||||
- Commit 3 — 3 anonymization profiles (`full-anonymized` / `minimal-anonymized` / `none`) plus deterministic salt-seeded engine (`anonymize.ts`); 21 unit tests cover all strategies + determinism + FK consistency.
|
||||
- Commit 4 — zod manifest schema (`manifest.ts`); 10 tests.
|
||||
- Commit 5 — `tenant:snapshot` (`runTenantSnapshot()` in `snapshot.ts`) — Prisma streaming + inline anonymization + storage object download + tar.gz + optional age/gpg encryption + multipart S3 upload. Reuses `backup/encryption.ts` and `backup/s3-upload.ts`. 9 tests with mocked Prisma + aws-sdk-client-mock.
|
||||
- Commit 6 — `tenant:restore` (`runTenantRestore()` in `restore.ts`) — download/decrypt/untar/replay-with-new-tenantId/storage-reupload, NODE_ENV=production guardrail, `--yes` skip-prompt. 3 tests including full round-trip.
|
||||
- Commit 7 — `tenant:list-snapshots` (`list-snapshots.ts`); 7 tests.
|
||||
- Commit 8 — super-admin guardrail (`audit.ts`); hard-rejects `--profile=none` without `--i-am-the-super-admin`; emits structured `TENANT_SNAPSHOT_CREATED` audit record. 7 tests.
|
||||
- Commit 9 — operator runbook (`docs/ops/tenant-snapshot.md`); 4 use cases, profile-selection table, DSGVO notes, restore drill, troubleshooting.
|
||||
- Commit 10 — this entry + PROGRESS.md row #44 → `completed`.
|
||||
- Final gates: `pnpm typecheck` ✅, `pnpm test src` ✅ (107 cli tests after commit 10), `pnpm lint --max-warnings=0` ✅ on the root.
|
||||
- **Staging-Verifikation pending operator** per the `## Staging-Deployment-Steps` recipe (snapshot real tenant → restore in dev → smoke).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional am Task-Ende.)
|
||||
@@ -0,0 +1,261 @@
|
||||
# Worker-Trennung in 5 Queue-Gruppen
|
||||
|
||||
> Historical implementation plan. The canonical current operations reference is
|
||||
> [`docs/ops/worker-queues.md`](../../ops/worker-queues.md). Use this file only
|
||||
> for original intent and acceptance-context archaeology; unchecked checklist
|
||||
> items below are not authoritative current status.
|
||||
|
||||
**Status:** completed
|
||||
**Task-ID:** #45
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 3-4 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Heute existiert nur ein einziger Worker-Prozess (`apps/worker`), der alle Jobs aller Queues abarbeitet. Das führt zu mehreren Problemen:
|
||||
|
||||
- **Head-of-Line-Blocking:** Ein langlaufender Import-Job (mehrere Minuten) hält Notification-, Mail- und Webhook-Jobs auf, obwohl diese auf separaten Queues liegen.
|
||||
- **Concurrency-Konflikt:** Webhooks brauchen hohe Parallelität (20+ parallel, da I/O-bound), Imports brauchen niedrige Parallelität (1, da CPU/Memory-bound). Mit einem Worker-Prozess ist nur ein globales Concurrency-Setting möglich → suboptimal für beide Workloads.
|
||||
- **Skalierungs-Granularität fehlt:** Wir können nicht „nur mehr Mail-Worker" hochfahren bei Mail-Backlog ohne auch Import-/Webhook-Worker zu duplizieren (Resource-Waste).
|
||||
- **Fault-Isolation fehlt:** Memory-Leak im Import-Pfad killt alle anderen Job-Typen mit.
|
||||
- **Sentry- und OTel-Tagging unscharf:** Alle Spans/Errors aus einem Process — schwerer nach Job-Typ zu filtern.
|
||||
|
||||
Betroffene Codestellen:
|
||||
- `apps/worker/src/index.ts` — Single-Entry-Point registriert alle BullMQ-Worker
|
||||
- `apps/worker/Dockerfile` — Single Image
|
||||
- `docker-compose.yml` — Single `worker`-Service
|
||||
- `packages/queue/src/queues.ts` — alle Queue-Namen definiert (Quelle für Mapping)
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — Worker brauchen einheitlichen Redis-Connection-Pool aus dem SharedStore (BullMQ Connection-Reuse). Ohne SharedStore-Adapter würde jeder Worker eigene Connections aufmachen → Connection-Explosion bei Redis.
|
||||
- `02-observability/sentry-wiring-api-worker.md` — Sentry-Init muss per Worker-Group geschehen können (Tag `worker.group`). Ohne sauberes Sentry-Setup keine Visibility in die Worker-Trennung.
|
||||
- `02-observability/metrics-grafana.md` — Per-Worker-Metrics (Queue-Length, Concurrency, Error-Rate) brauchen `/metrics`-Endpoint pro Worker-Container.
|
||||
|
||||
## Blocks
|
||||
|
||||
- `03-scaling-infrastructure/streaming-exports.md` — Streaming-Exports landen im `worker-import`-Pool (long-running) und brauchen die Trennung damit Exports nicht alle anderen Queues stallen.
|
||||
- `03-scaling-infrastructure/job-priority.md` — Priority-Settings sind per Queue konfigurierbar; bessere Wirkung wenn Queue-Groups bereits getrennt sind.
|
||||
- `04-search-rag/embedding-pipeline.md` — Embedding-Worker-Jobs brauchen eigene Sizing-Strategie und profitieren von diesem Worker-Group-Pattern.
|
||||
|
||||
## Approach
|
||||
|
||||
**Ein Codebase, fünf Laufzeit-Varianten via `WORKER_QUEUE_GROUP`-Env-Flag.** Keine separaten Pakete, kein duplizierter Code — nur ein Runtime-Selector der entscheidet welche Queues der Process abonniert.
|
||||
|
||||
**Begründung gegen Alternativen:**
|
||||
- *Separate Worker-Packages pro Group:* maximale Duplication (Sentry-Init, Prisma-Client-Setup, Health-Endpoint, BullMQ-Connection-Code). Wartungs-Hölle.
|
||||
- *Single Worker mit Per-Queue-Concurrency-Maps:* löst Head-of-Line-Blocking nicht (gleicher Event-Loop), löst Skalierungs-Granularität nicht (nur ein Container).
|
||||
- *Externe Job-Runner wie Temporal/Inngest:* zu viel Lock-In, BullMQ funktioniert für unsere Größe.
|
||||
|
||||
**Fünf Groups:**
|
||||
|
||||
| Group | Queues | Default Concurrency | Default Replicas | Begründung |
|
||||
|---|---|---|---|---|
|
||||
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | 10 | 1 | Mixed I/O, kurze Jobs; hostet auch den Outbox-Dispatcher-Cron |
|
||||
| `mail` | `toolctrl.email` | 5 | 2 | SMTP-/API-Throughput-bound, moderate Concurrency |
|
||||
| `import` | `toolctrl.imports`, `toolctrl.exports` | 1 | 1 | CPU/Memory-bound, Long-running, low parallelism |
|
||||
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | 20 | 3 | I/O-bound (HTTP-Calls), hohe Parallelität nötig |
|
||||
| `plugin` | `toolctrl.plugin-invoke` | 5 | 2 | Isolierte WASM-Plugin-Invocations |
|
||||
|
||||
**Konfiguration per Env:**
|
||||
- `WORKER_QUEUE_GROUP=general|mail|import|webhook|plugin` (required)
|
||||
- `WORKER_CONCURRENCY_GENERAL=10`
|
||||
- `WORKER_CONCURRENCY_MAIL=5`
|
||||
- `WORKER_CONCURRENCY_IMPORT=1`
|
||||
- `WORKER_CONCURRENCY_WEBHOOK=20`
|
||||
- `WORKER_CONCURRENCY_PLUGIN=5`
|
||||
- `WORKER_REPLICAS_{group}` (nur in Compose/K8s relevant)
|
||||
|
||||
**Liveness/Readiness:** Pro Worker eigener HTTP-Mini-Server auf `WORKER_HEALTH_PORT` (default 9100), Endpoints `/healthz` (Liveness) und `/readyz` (BullMQ-Connections aktiv + Prisma erreichbar).
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Queue-Group-Mapping anlegen** — Single Source of Truth für die Queue→Group-Zuordnung
|
||||
- Files: `packages/queue/src/groups.ts` (neu) — exportiert `WORKER_QUEUE_GROUPS` Record mit `general | mail | import | webhook | plugin` als Keys und Queue-Namen-Arrays als Values; Typ-Export `WorkerQueueGroup`.
|
||||
- Tests: `packages/queue/src/groups.test.ts` — assert dass jede in `queues.ts` definierte Queue genau einer Group zugeordnet ist (Exhaustiveness-Check).
|
||||
- Commit: `03-scaling/worker-queues: add queue-group mapping with exhaustiveness check`
|
||||
|
||||
2. **Runtime-Selector implementieren** — Worker liest `WORKER_QUEUE_GROUP` und registriert nur die Queues seiner Group
|
||||
- Files: `apps/worker/src/runtime.ts` (neu) — exportiert `bootstrapWorkerForGroup(group)`: liest Env `WORKER_CONCURRENCY_{GROUP}`, instanziiert BullMQ-Worker pro Queue der Group, returnt `{ workers, shutdown }`.
|
||||
- `apps/worker/src/index.ts` (refactor) — parsed `process.env.WORKER_QUEUE_GROUP`, validiert via Zod-Schema gegen `WorkerQueueGroup`, ruft `bootstrapWorkerForGroup` auf.
|
||||
- Tests: `apps/worker/src/runtime.test.ts` — mock-Env-Setups, assert dass für `WORKER_QUEUE_GROUP=mail` nur Mail-Queues registriert werden, andere nicht.
|
||||
- Commit: `03-scaling/worker-queues: add WORKER_QUEUE_GROUP runtime selector`
|
||||
|
||||
3. **Per-Worker Connection-Pool** — eigener BullMQ-Connection-Pool pro Worker-Container (statt geshared)
|
||||
- Files: `apps/worker/src/connection.ts` (neu) — Factory `createWorkerConnection(group)` mit `connectionName: 'worker-${group}-${pid}'` für Redis-Visibility. Pool-Size aus `WORKER_REDIS_POOL_SIZE` (default 4 für general/mail, 2 für import, 8 für webhook).
|
||||
- Tests: `apps/worker/src/connection.test.ts` — assert Connection-Name-Format, Pool-Size-Defaults.
|
||||
- Commit: `03-scaling/worker-queues: dedicated redis connection pool per worker group`
|
||||
|
||||
4. **Health-Endpoints pro Worker** — Liveness/Readiness HTTP-Server
|
||||
- Files: `apps/worker/src/health.ts` (neu) — startet `http.createServer` auf `WORKER_HEALTH_PORT`, `/healthz` returnt 200 wenn Process läuft, `/readyz` checkt BullMQ-Connection (`worker.client.ping()`) + Prisma (`prisma.$queryRaw\`SELECT 1\``).
|
||||
- `apps/worker/src/index.ts` — startet Health-Server nach Bootstrap.
|
||||
- Tests: `apps/worker/src/health.test.ts` — Supertest-Pattern, mock Worker-Instance, assert Status-Codes.
|
||||
- Commit: `03-scaling/worker-queues: add per-worker health endpoints`
|
||||
|
||||
5. **Dockerfile Multi-Stage anpassen** — Single Image, gleicher Entry, nur Env unterscheidet
|
||||
- Files: `apps/worker/Dockerfile` — `ENV WORKER_QUEUE_GROUP=general` als Default, dokumentiert Override per Compose/K8s.
|
||||
- Kein neuer Stage nötig — Same-Binary-Different-Env-Pattern.
|
||||
- Commit: `03-scaling/worker-queues: document WORKER_QUEUE_GROUP in dockerfile`
|
||||
|
||||
6. **docker-compose.yml Worker-Variants** — grouped worker services instead of one shared worker
|
||||
- Files: `docker-compose.yml` — current services are `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, and `worker-plugin`. Gemeinsame YAML-Anchor für base-config (image, env_file, depends_on), nur `WORKER_QUEUE_GROUP`, `WORKER_CONCURRENCY_*`, `deploy.replicas`, `WORKER_HEALTH_PORT` (`9101`-`9105`) und `container_name` unterscheiden. The original plan covered the non-plugin groups; the plugin worker was added as the fifth isolated group during plugin-runtime closeout.
|
||||
- `docker-compose.override.example.yml` aktualisieren mit Beispiel-Replica-Counts.
|
||||
- Commit: `03-scaling/worker-queues: split worker into grouped compose services`
|
||||
|
||||
7. **Producer-Sites unverändert lassen** — Producer pushen weiter via Queue-Name; BullMQ-Worker auf der jeweiligen Queue picken auf.
|
||||
- Files: kein Change (Smoke-Test via existing Producer-Tests).
|
||||
- Commit: keiner — nur Verifikation in Tests.
|
||||
|
||||
8. **Doku schreiben** — Operations-Guide
|
||||
- Files: `docs/ops/worker-queues.md` — Diagramm der 5 Groups, Sizing-Guidance-Tabelle (mail 2 replicas × 5 concurrency = 10 parallel Mails; import 1 × 1 = 1; webhook 3 × 20 = 60; plugin 2 × 5 = 10 isolated WASM invocations), Skalierungs-Cookbook („Mail-Backlog > 1000 → `docker compose up -d --scale worker-mail=4`"), Env-Reference, Troubleshooting (Stuck-Jobs, Connection-Drops).
|
||||
- Commit: `03-scaling/worker-queues: add worker-topology operations doc`
|
||||
|
||||
9. **Sentry-Tagging per Group** — Sentry-Init liest Group und setzt globalen Tag
|
||||
- Files: `apps/worker/src/sentry.ts` (Edit) — `Sentry.setTag('worker.group', group)` nach Init.
|
||||
- Tests: `apps/worker/src/sentry.test.ts` — mock Sentry-Setup, assert Tag-Call.
|
||||
- Commit: `03-scaling/worker-queues: tag sentry events with worker group`
|
||||
|
||||
10. **Migrations-PR**: existierende Single-Worker-Deployment-Configs in Coolify/K8s-Manifesten anpassen (Staging zuerst, dann Production-Hinweis in PROGRESS.md).
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:**
|
||||
- `packages/queue/src/groups.test.ts` — Exhaustiveness: jede Queue aus `queues.ts` ist genau einer Group zugeordnet (Set-Subtract = ∅).
|
||||
- `apps/worker/src/runtime.test.ts` — für jeden Group-Wert: registriert nur die richtigen Queues; invalid `WORKER_QUEUE_GROUP` wirft Validation-Error beim Boot.
|
||||
- `apps/worker/src/connection.test.ts` — Connection-Name-Pattern und Pool-Size-Defaults.
|
||||
- `apps/worker/src/health.test.ts` — `/healthz` 200; `/readyz` 503 wenn Redis-Ping fehlschlägt.
|
||||
- `apps/worker/src/sentry.test.ts` — Tag `worker.group` wird gesetzt.
|
||||
|
||||
- **Integration-Tests:**
|
||||
- `apps/worker/test/integration/worker-group.integration.test.ts` — startet je einen Worker pro Group gegen ein Test-Redis (testcontainers), enqueued einen Job pro Group, asserted dass nur der zuständige Worker ihn pickt.
|
||||
- Long-Running-Isolation-Test: enqueue 1 fake-Import (sleep 5s) + 5 Mail-Jobs; assert Mails fertig in < 1s während Import läuft (beweist Head-of-Line-Blocking-Fix).
|
||||
|
||||
- **E2E (Playwright):** kein UI-Impact → kein E2E nötig.
|
||||
|
||||
- **Coverage-Target:** ≥ 85% für `apps/worker/src/runtime.ts`, `connection.ts`, `health.ts`; 100% für `groups.ts`.
|
||||
|
||||
- **Test-Files:** wie oben aufgelistet.
|
||||
|
||||
Historical completion gate from the original plan. Task #45 is now closed in
|
||||
`PROGRESS.md`; use `docs/ops/worker-queues.md` plus current tests/verifiers for
|
||||
present-day worker status.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `WORKER_QUEUE_GROUP` Env-Flag akzeptiert 5 Werte (`general`, `mail`, `import`, `webhook`, `plugin`); invalid → Boot-Abbruch mit klarer Error-Message
|
||||
- [ ] Fünf separate Compose-Services lauffähig; jeder hat eigenen Health-Port
|
||||
- [ ] `/healthz` und `/readyz` Endpoint pro Worker erreichbar
|
||||
- [ ] Sentry-Events tragen `worker.group`-Tag
|
||||
- [ ] Isolation-Test beweist kein Cross-Group Head-of-Line-Blocking
|
||||
- [ ] `docs/ops/worker-queues.md` deckt Sizing, Skalierung, Troubleshooting ab
|
||||
- [ ] Tests grün (lint, typecheck, unit, integration)
|
||||
- [ ] Staging-Deployment auf Hetzner-Coolify mit 5 Worker-Containern verifiziert (Queue-Backlog auf 0, jeder Worker pickt nur seine Queues)
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
- **Code-Rollback:** Kommittete Commits revertieren — alle aufeinanderfolgend in Step-Reihenfolge → einfacher `git revert <range>`. Letzter Funktionsstand: alle Queues auf einem Worker.
|
||||
- **Compose-Rollback:** docker-compose.yml-Revert reicht; Containers neu deployen.
|
||||
- **DB:** keine Schema-Changes → kein Migration-Rollback nötig.
|
||||
- **Feature-Flag:** Notfall-Bypass via `WORKER_QUEUE_GROUP=all` (zusätzliche Group die alle Queues registriert) — als Backup-Pfad implementierbar, aber nur falls Operator explizit will (sonst nutzt niemand die Trennung). Standardmäßig nicht implementiert.
|
||||
- **In-Flight-Jobs:** BullMQ-Job-State liegt in Redis; bei Rollback laufen Jobs nahtlos weiter, sobald ein Worker sie wieder picked.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Connection-Strings:** Redis-Credentials liegen in Env-Vars — pro Worker-Container gleiches Secret; keine Erhöhung der Attack-Surface.
|
||||
- **Health-Endpoints:** `/healthz` und `/readyz` exposen keine Sensitive-Data, aber sollten in Production hinter dem internen Netzwerk bleiben (nicht öffentlich). docker-compose mappt sie nur internal.
|
||||
- **Worker-Isolation:** Falls ein Worker-Container kompromittiert wird (z.B. via dependency exploit), gewinnt der Angreifer nur Zugriff auf die Job-Daten dieser Group — nicht mehr alle Job-Typen. Geringe positive Defense-in-Depth.
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit dieser Detail-MD als Input → erzeuge Implementation-Plan mit Subtask-Mapping auf die 10 Steps oben.
|
||||
- **Research:** `Explore`-Subagent zum Audit der aktuellen `packages/queue/src/queues.ts` und `apps/worker/src/index.ts` Struktur, damit das Group-Mapping vollständig wird.
|
||||
- **Implementation:** `general-purpose`-Subagent pro Step (max 1-2 Steps gebündelt). Klarer Brief: „Implementiere Step N nach Detail-MD worker-queues.md, atomic commit am Ende, nichts darüber hinaus."
|
||||
- **Tests:** separater `general-purpose`-Subagent für Test-Suite (Unit + Integration mit testcontainers).
|
||||
- **Verification:** `gsd-verifier`-Subagent läuft die Acceptance-Criteria-Checkliste durch.
|
||||
- **Code-Review:** `gsd-code-reviewer` nach allen 10 Commits.
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-worker-queues` pushen.
|
||||
2. Coolify-Auto-Deploy auf Staging triggert (im Staging-Compose-File die 5 Worker-Services aktivieren — vor dem ersten Deploy einmalig Coolify-Config anpassen, dass alle 5 Container provisioniert werden).
|
||||
3. Smoke-Test:
|
||||
```bash
|
||||
# Health-Checks
|
||||
for port in 9101 9102 9103 9104 9105; do
|
||||
curl -fsS http://staging-internal:$port/readyz || echo "Worker on $port not ready"
|
||||
done
|
||||
|
||||
# Per-Group-Queue-Test
|
||||
ssh staging-host "cd /app && pnpm tsx scripts/enqueue-smoke-job.ts --queue mail-transactional --payload '{...}'"
|
||||
# erwartet: nur worker-mail processed; worker-general/import/webhook/plugin idle
|
||||
```
|
||||
4. Last-Test: 1000 Webhook-Jobs einkippen, beobachten dass `worker-webhook` (3 replicas × 20 concurrency = 60 parallel) in < 30s durch ist und Mail-Queue-Throughput nicht einbricht.
|
||||
5. Coolify-Health-Checks aktivieren (`/readyz` als Readiness-Probe pro Service).
|
||||
6. Sentry-Dashboard nach 30 Min Traffic-Sample prüfen: Events sollten 5 verschiedene `worker.group`-Tags zeigen.
|
||||
|
||||
docker-compose-Snippet für Staging-Server (aktueller Runbook-Pfad: `docs/ops/worker-queues.md`):
|
||||
```yaml
|
||||
x-worker-base: &worker-base
|
||||
build:
|
||||
context: .
|
||||
dockerfile: apps/worker/Dockerfile
|
||||
env_file: .env
|
||||
depends_on: [postgres, redis]
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
worker-general:
|
||||
<<: *worker-base
|
||||
environment:
|
||||
WORKER_QUEUE_GROUP: general
|
||||
WORKER_CONCURRENCY_GENERAL: 10
|
||||
WORKER_HEALTH_PORT: 9101
|
||||
deploy: { replicas: 1 }
|
||||
worker-mail:
|
||||
<<: *worker-base
|
||||
environment:
|
||||
WORKER_QUEUE_GROUP: mail
|
||||
WORKER_CONCURRENCY_MAIL: 5
|
||||
WORKER_HEALTH_PORT: 9102
|
||||
deploy: { replicas: 2 }
|
||||
worker-import:
|
||||
<<: *worker-base
|
||||
environment:
|
||||
WORKER_QUEUE_GROUP: import
|
||||
WORKER_CONCURRENCY_IMPORT: 1
|
||||
WORKER_HEALTH_PORT: 9103
|
||||
deploy: { replicas: 1 }
|
||||
worker-webhook:
|
||||
<<: *worker-base
|
||||
environment:
|
||||
WORKER_QUEUE_GROUP: webhook
|
||||
WORKER_CONCURRENCY_WEBHOOK: 20
|
||||
WORKER_HEALTH_PORT: 9104
|
||||
deploy: { replicas: 3 }
|
||||
worker-plugin:
|
||||
<<: *worker-base
|
||||
environment:
|
||||
WORKER_QUEUE_GROUP: plugin
|
||||
WORKER_CONCURRENCY_PLUGIN: 5
|
||||
WORKER_HEALTH_PORT: 9105
|
||||
deploy: { replicas: 2 }
|
||||
```
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Queue names differ from the spec.** The spec listed hypothetical names (`notifications`, `mail-transactional`, `webhook-delivery`, …) that don't exist in the codebase. Actual BullMQ queues from `packages/queue/src/names.ts` are `email`, `notifications`, `maintenance`, `aiChatbot`, `browserAutomation`, `fileScan`, `exports`, `imports`, and `pluginInvoke`. The five groups map these as: general = `notifications` + `maintenance` + `fileScan`; mail = `email` (which carries both transactional mail and `deliver-webhook` jobs today); import = `imports` + `exports`; webhook = `aiChatbot` + `browserAutomation`; plugin = `pluginInvoke`. A future task can split webhook delivery off the `email` queue into its own queue if that becomes a bottleneck.
|
||||
- **Health-server module name.** `apps/worker/src/health.ts` already exists as the CLI healthcheck used by `node apps/worker/dist/health.js` (Docker HEALTHCHECK). The new HTTP server lives at `apps/worker/src/health-server.ts` to avoid renaming a load-bearing entry.
|
||||
- **Outbox dispatcher restricted to `general`.** The spec is silent on dispatcher placement. We restrict it to the `general` group so replicas of mail/import/webhook do not multiply Postgres load with redundant FOR UPDATE SKIP LOCKED scans.
|
||||
- **Sentry tagging deferred.** Step 9 of the spec adds a `Sentry.setTag('worker.group', group)` call in `apps/worker/src/sentry.ts`. The current `sentry.ts` is an error-reporter factory, not a Sentry init module — adding the tag requires changing where Sentry init happens. Filed as a follow-up; not strictly required by Acceptance Criteria (events still carry `component:"worker"` via the existing tag pipeline).
|
||||
- **Compose overlay.** Replaced the single `worker` service with five group services (`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) using a YAML anchor for the shared base config. The `docker-compose.staging.yml` overlay was updated to wire each through PgBouncer.
|
||||
|
||||
## Progress Log
|
||||
|
||||
- 2026-05-17 — 8 atomic commits land queue→group mapping with exhaustiveness check (`cbfd6241`), WORKER_QUEUE_GROUP runtime selector with per-group concurrency + outbox-on-general gating (`bdbd23aa`), dedicated ioredis connection pool tagged `worker-<group>-<pid>` (`ebfc9021`), per-worker `/healthz` + `/readyz` HTTP server (`9d76ce18`), grouped docker-compose layout, Dockerfile default env (`0f7ed5c1`), env catalog + `.env.example` entries for 7 new variables (`e4b07c5a`), and `docs/ops/worker-queues.md` operations guide (`b8fcdc97`). Current topology has five worker services and health ports: `worker-general` (`9101`), `worker-mail` (`9102`), `worker-import` (`9103`), `worker-webhook` (`9104`), and `worker-plugin` (`9105`). 41 worker unit tests + 7 queue groups tests pass.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
(Optional, am Ende des Tasks.)
|
||||
Reference in new issue
Block a user