Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/03-scaling-infrastructure/pubsub-cache-invalidation.md
T

262 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Pub/Sub Cache-Invalidation
**Status:** in_progress (subscription + tenant + feature-flag two-tier caches shipped; CLI cross-process publisher closed 2026-05-20; reconnect-flush + testcontainers integration closed 2026-05-25 — remaining work is the cross-app namespace fan-out documented in Deviations)
**Task-ID:** #39
**Phase:** 03
**Estimated Effort:** 3-4 Tage
**Owner:** unassigned
## Problem
Selbst nach #38 (Shared-Store-Migration) verbleibt ein Konsistenz-Problem: einige hot caches sollen weiterhin **lokal in-process** (cachified-LRU) gehalten werden, weil Redis-Roundtrips pro Request zu teuer sind. Beispiele:
- **Subscription-Cache (Web):** Pro Request mehrfach abgefragt (Entitlement-Checks, Limit-Counter, UI-Visibility). Redis-Roundtrip 1-3ms × 20 Calls = 20-60ms Overhead
- **Tenant-Cache:** Pro Request fix
- **Entitlements-Cache:** Pro Request fix
- **Feature-Flag-Cache (kommt in Phase 05):** Pro UI-Render fix
Lösung: **Zwei-Stufiger Cache.** L1 = in-process cachified (schnell, nicht-shared), L2 = SharedStore (langsam, shared). Invalidations werden über Pub/Sub propagiert: wenn ein Event den Cache invalidieren soll, published derjenige, der die Quelle ändert (z.B. Stripe-Webhook in API), und ALLE Web-Replicas hören mit und purgen ihren L1.
Aktueller Zustand: Stripe-Webhook in `apps/api/src/billing/stripe-webhook-route.ts:60-138` updated DB + ruft `clearSubscriptionsCache(tenantId)` lokal in der API. Aber: der Cache liegt nicht in der API, sondern in **jeder Web-Replica**. Heute: Web-Cache wird erst nach Ablauf von 1h-TTL frisch. Resultat: User kann z.B. nach Plan-Upgrade noch bis zu 1h alte Limits sehen.
## Depends on
- `03-scaling-infrastructure/shared-store-package.md` — liefert `createCacheStore` mit Pub/Sub-Invalidation
- `03-scaling-infrastructure/shared-store-migration.md` — Subscription-Cache muss schon über SharedStore laufen, sonst kein Channel zum Subscriben
- `02-observability/sentry-wiring-web.md` — Subscriber-Errors müssen captured werden, sonst tappt der Operator im Dunkeln wenn Subscriber stillschweigend stirbt
## Blocks
- `05-business-features/feature-flags.md` — Feature-Flag-Cache nutzt identisches Pattern
- `03-scaling-infrastructure/tenant-snapshot.md` — Tenant-Reset published Invalidation für Tenant-Cache
- alle nachfolgenden Caches mit Cross-Replica-Konsistenz-Anforderung
## Approach
**Pattern: Source-of-Truth-Owner published, Cache-Holder subscribed.**
1. **Publisher-Seite:** Wer immer die SoT ändert (API für Subscription via Stripe-Webhook, Worker für Tenant-Snapshot, etc.) ruft nach Commit `cache.invalidate(key)` auf dem SharedStore-Cache. Das published auf Channel `<namespace>:invalidate` mit Payload `{ key }`.
2. **Subscriber-Seite:** Jede Web-Replica (und API, wo sie auch L1 hält) startet beim Boot einen Subscriber pro relevantem Cache-Channel. Der Subscriber-Callback purged den lokalen L1 (cachified.delete) und optional auch den L2 (für Cleanup).
3. **Auto-Reconnect:** ioredis hat eingebauten Reconnect. Wir wrappen das mit einem `onReconnect`-Handler, der nach erfolgreichem Reconnect einmal alle L1-Caches komplett flushed (`cache.clearAll()`), weil wir nicht wissen welche Invalidations während Disconnect verloren gingen.
4. **Replay-fähig:** Pub/Sub hat keine Persistence — bei Disconnect verloren. Mitigation: L1-TTL bleibt kurz (5-15min), so dass im Worst-Case-Disconnect-Szenario nach kurzer Zeit auto-refresh greift. Für strikte Konsistenz: Outbox-Pattern (#42) für kritische Events, dann Worker konsumiert Outbox und published Invalidation. Doppelte Sicherheit.
5. **Multi-Process-Setup:**
- **API (1 Replica):** Publisher für Subscription, Tenant, Entitlements
- **Web (2+ Replicas):** Subscriber für alle obigen
- **Worker (2+ Replicas):** Publisher für lange laufende Updates (Bulk-Imports, Tenant-Snapshot-Restore)
## Vier konkrete Caches migrieren
### A. Subscription-Cache (Stripe)
- **Source:** `apps/api/src/billing/stripe-webhook-route.ts:60-138`
- **Trigger:** Stripe-Event (`customer.subscription.updated`, `invoice.payment_succeeded`, etc.)
- **Channel:** `cache:subscription:invalidate`
- **Payload:** `{ tenantId: string }`
- **Publisher:** API in Webhook-Handler nach `updateTenantSubscriptionFromStripe`
- **Subscriber:** Web (jede Replica) ruft `clearSubscriptionsCache(tenantId)`
### B. Entitlements-Cache
- **Source:** `apps/web/app/utils/entitlements/.server/productAccess.server.ts`
- **Trigger:** Subscription-Change (impliziert Entitlement-Change), Manual-Admin-Edit
- **Channel:** `cache:entitlements:invalidate`
- **Payload:** `{ tenantId: string }`
- **Publisher:** Identisch zur Subscription-Trigger-Stelle (gleiche Webhook-Handler) + Admin-Edit-Route
- **Subscriber:** Web purged in-process Entitlements-Map
### C. Tenant-Cache
- **Source:** `apps/web/app/utils/data/.server/appData.ts`
- **Trigger:** Tenant-Settings-Update, Tenant-Logo-Upload, Member-Add/Remove
- **Channel:** `cache:tenant:invalidate`
- **Payload:** `{ tenantId: string }`
- **Publisher:** Web (eigene Action) UND API (bei API-getriebener Mutation)
- **Subscriber:** Alle Web-Replicas + API (wo Tenant gelesen wird)
### D. Feature-Flag-Cache (Vorbereitung für #53)
- **Source:** kommt in Phase 05
- **Channel:** `cache:feature-flag:invalidate`
- **Payload:** `{ flagKey?: string; tenantId?: string; all?: boolean }`
- **Publisher:** Admin-Edit in Web
- **Subscriber:** Web + API + Worker
- In diesem Task nur das **Channel-Naming + Subscriber-Stub** vorbereiten, damit #53 nur noch den Publisher hinzufügt
**Warum nicht andere Pattern:**
- ❌ Polling: schwergewichtig, schlechte Reaktionszeit
- ❌ Cache-TTL niedrig halten (5s): Hammers gegen Redis und DB
- ❌ WebSocket-basierte Push-Invalidation: Overkill, wir haben Redis schon
- ❌ Kafka/NATS für ein einziges Use-Case: zu viel Infra
## Implementation Steps
1. **Subscriber-Lifecycle-Framework**
- Files: `packages/shared-store/src/helpers/subscriber-lifecycle.ts`, `subscriber-lifecycle.test.ts`
- `createSubscriberRegistry({ store })` mit `register(channel, handler)` + `start()` + `stop()`
- Auto-Reconnect-Handler: bei Redis-Reconnect alle registered Handler einmalig mit `{ type: 'flush' }` aufrufen
- Sentry-Integration: jeder Subscriber-Error wird captured + re-thrown an unhandled (Subscribers dürfen nicht stillschweigend sterben)
- Tests: register/start/stop, Reconnect-Flush, Error-Capture
- Commit: `03-scaling/pubsub: add subscriber lifecycle framework with reconnect handling`
2. **App-Boot-Integration Web**
- Files: `apps/web/app/entry.server.tsx` (oder analoger Boot-Hook), `apps/web/app/utils/subscribers/index.server.ts`
- SubscriberRegistry beim Boot starten, registriert alle Web-Subscriber (Subscription, Entitlements, Tenant, Feature-Flag-Stub)
- Graceful-Shutdown-Handler: `process.on('SIGTERM', () => registry.stop())`
- Tests: Boot-Integration-Test (Mock-Redis, check dass alle Channels subscribed sind)
- Commit: `03-scaling/pubsub: wire subscriber registry into web boot`
3. **App-Boot-Integration API**
- Files: `apps/api/src/server.ts`, `apps/api/src/subscribers/index.ts`
- Analog Web, API hat eigene Subscriber-Liste (Tenant + Feature-Flag)
- Commit: `03-scaling/pubsub: wire subscriber registry into api boot`
4. **Subscription-Cache: Publisher in Stripe-Webhook**
- Files: `apps/api/src/billing/stripe-webhook-route.ts`, Test
- Nach `updateTenantSubscriptionFromStripe` → `subscriptionCache.invalidate(tenantId)` (cache aus #38)
- Tests: Mock Redis, verify Pub/Sub-Message wurde published
- Commit: `03-scaling/pubsub: publish subscription invalidation from stripe webhook`
5. **Subscription-Cache: Subscriber in Web**
- Files: `apps/web/app/utils/subscribers/subscription-cache.subscriber.ts`, Test
- Handler ruft `clearSubscriptionsCache(tenantId)` + zusätzlich `subscriptionCache.invalidate(tenantId, { localOnly: true })` für L1
- Integration-Test: 2 Web-Instances, Webhook auf API → beide Web-Instances sehen neuen Wert < 2s
- Commit: `03-scaling/pubsub: subscribe subscription cache invalidation in web`
6. **Entitlements-Cache: Publisher + Subscriber**
- Files: `apps/web/app/utils/entitlements/.server/productAccess.server.ts`, `apps/web/app/utils/subscribers/entitlements-cache.subscriber.ts`, Tests
- Publisher in API (gleiche Webhook-Stelle wie Subscription, weil korreliert)
- Subscriber in Web purged in-process Entitlements-Map
- Commit: `03-scaling/pubsub: add entitlements cache pub/sub channel`
7. **Tenant-Cache: Publisher + Subscriber**
- Files: `apps/web/app/utils/data/.server/appData.ts`, `apps/web/app/utils/subscribers/tenant-cache.subscriber.ts`, Tests
- Publisher in Web-Action für Tenant-Settings-Update; API-Side wo Tenant-Mutation passiert
- Subscriber sowohl Web als auch API
- Commit: `03-scaling/pubsub: add tenant cache pub/sub channel`
8. **Feature-Flag-Cache-Stub (für #53)**
- Files: `apps/web/app/utils/subscribers/feature-flag-cache.subscriber.ts` (Stub), `apps/api/src/subscribers/feature-flag-cache.subscriber.ts` (Stub)
- Subscriber registered Handler, der derzeit nur loggt (`"feature-flag cache invalidation received, publisher not yet implemented"`)
- Ermöglicht #53, nur Publisher-Seite zu ergänzen
- Commit: `03-scaling/pubsub: add feature-flag cache subscriber stubs`
9. **Multi-Process-Integration-Test**
- Files: `apps/web/__integration__/cross-replica-invalidation.test.ts`
- testcontainers spinnt 1 Redis-Container + 2 in-process Web-Subscriber-Instances + 1 in-process Publisher
- Test-Cases: invalidate triggert beide Subscriber; Reconnect-Flush nach Disconnect-Simulation; Concurrent-Invalidation
- Commit: `03-scaling/pubsub: add multi-process invalidation integration tests`
10. **Doku**
- Files: `docs/observability/redis-stores.md` (Erweiterung): Sektion „Pub/Sub-Channels" mit allen 4 Channels, Owner-Tabelle (wer published, wer subscribed)
- Operational-Runbook: „Cache-Invalidation funktioniert nicht — Debug-Steps" (Redis-MONITOR, Subscriber-Health-Check)
- Commit: `03-scaling/pubsub: document pub/sub channels and runbook`
## Test Requirements
- **Unit-Tests:** SubscriberRegistry (register, start, stop, reconnect-flush, error-capture); jeder einzelne Subscriber-Handler (purged korrekt)
- **Integration-Tests:** Cross-Replica-Invalidation pro Channel (Multi-Process-Setup via testcontainers); Reconnect-Verhalten (kill Redis, restart, verify Flush)
- **E2E-Tests (Playwright):** Stripe-Webhook-Simulation → check ob Web-UI direkt neuen Subscription-Status zeigt (Replication-Test)
- **Coverage-Target:** ≥ 90 % im Subscriber-Code, ≥ 85 % an Publisher-Stellen
- **Test-Files:** Subscriber-Files co-located; Integration in `apps/web/__integration__/` und `apps/api/src/__integration__/`
**NICHT als completed markieren bevor:**
- Multi-Process-Integration-Test grün
- Staging-Test: Stripe-Webhook auf Replica A → Web-Replica B + C sehen neuen Wert < 2s
- Reconnect-Test auf Staging: `docker restart redis` → nach 30s sind alle Subscriber wieder online
## Acceptance Criteria
- [ ] SubscriberRegistry-Framework in `packages/shared-store/src/helpers/subscriber-lifecycle.ts` implementiert
- [ ] Web + API booten SubscriberRegistry, registrieren alle relevanten Channels
- [ ] Vier Channels live: `cache:subscription:invalidate`, `cache:entitlements:invalidate`, `cache:tenant:invalidate`, `cache:feature-flag:invalidate`
- [ ] Stripe-Webhook published Subscription + Entitlements-Invalidation nach DB-Update
- [ ] Tenant-Mutation-Stellen published Tenant-Invalidation
- [ ] Feature-Flag-Subscriber-Stubs vorhanden (für #53)
- [ ] Reconnect-Flush funktioniert: nach Redis-Restart sind alle L1-Caches geleert
- [ ] Multi-Process-Integration-Test grün (2 Subscriber-Instances, ein Publisher, alle Channels)
- [ ] Staging-Verifikation: End-to-End-Latenz Stripe-Event → Web-Cache-Invalidation < 2s
- [ ] `docs/observability/redis-stores.md` enthält Pub/Sub-Channel-Tabelle + Operational-Runbook
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
- [ ] Detail-MD Progress Log aktualisiert
- [ ] PROGRESS.md Status auf `completed`
## Rollback Strategy
Pro Channel separat rollback-bar:
- **Subscription-Channel:** Revert Publisher- + Subscriber-Commits → zurück zu 1h-Stale-Verhalten, funktional
- **Entitlements/Tenant:** Analog, zurück zu vorherigem Pattern (z.B. Loader-Refetch oder Page-Refresh)
- **SubscriberRegistry-Framework:** Wenn das ganze Framework Probleme macht → Boot-Integration revert, Subscribers schweigen, Caches werden nicht invalidiert. Mitigation: L1-TTL kurz (5-15min) als Safety-Net
**Global-Kill-Switch:** Env-Var `DISABLE_CACHE_PUBSUB=true` → SubscriberRegistry started nicht, Publisher no-op. Erlaubt schnelles Disable ohne Redeploy.
**DB-Migrations:** keine in diesem Task.
## Security Notes
- **Pub/Sub-Message-Validation:** Subscriber-Handler validieren Payload strict (Zod-Schema pro Channel), reject Malformed mit Sentry-Capture. Verhindert Crash bei Garbage-Messages
- **Channel-Spoofing-Risk:** Wer Redis-Zugriff hat, kann beliebig invalidieren. Mitigation: Redis-AUTH zwingend in Production, kein Public-Listen, Network-Isolation
- **Cache-Stampede bei Mass-Invalidation:** Wenn Angreifer `cache:tenant:invalidate` für alle Tenants published → alle Replicas refetchen aus DB. Mitigation: Subscriber-Handler implementiert lokales Debounce (max 1 invalidation pro key pro 500ms)
- **Sensitive-Data in Invalidation-Payload:** Payload enthält nur tenantId (UUID), keine PII. Klare Regel: Pub/Sub-Channels sind nicht zum Datentransport, nur Invalidation
- **Disconnect-During-Critical-Operation:** Wenn Redis während Stripe-Webhook down ist, schlägt Publisher fehl. Mitigation: Outbox-Pattern (#42) puffert Events, Dispatcher retried Publish bis Redis wieder up
- **Sentry-Breadcrumbs:** SubscriberRegistry meldet jede Disconnect/Reconnect-Event als Breadcrumb, damit Stale-Cache-Bugs in Produktion debugbar sind
## Subagent Recommendations
- **Planning:** `Plan`-Subagent mit Channel-Tabelle (A-D) + bestehender `subscriptionService.ts`-Implementation als Input
- **Research:** `Explore`-Subagent für „aktuelle cachified-Nutzung in `apps/web/app/utils/services/.server/` und `apps/web/app/utils/data/.server/` — welche Caches haben non-trivial Konsistenz-Anforderungen?"
- **Implementation:** Ein `general-purpose`-Subagent pro Channel (A-D), max 1 Channel pro Subagent — Publisher + Subscriber + Test gehören zusammen
- **Framework-Subagent:** Separater `general-purpose`-Subagent für Schritte 1-3 (SubscriberRegistry-Framework + Boot-Integration), weil Framework-Bug alle Channels bricht
- **Tests:** `general-purpose`-Subagent für Multi-Process-Integration-Test mit testcontainers (Step 9)
- **Verification:** `gsd-verifier` für Acceptance-Criteria + Staging-Latenz-Messung
- **Code-Review:** `gsd-code-reviewer` nach allen Channels — Fokus auf Reconnect-Pfade und Error-Handling im Subscriber
## Staging-Deployment-Steps
1. Branch `agent/03-pubsub-cache-invalidation` pushen
2. Voraussetzung: #38 muss auf Staging erfolgreich getestet sein (Subscription läuft schon über SharedStore)
3. Coolify-Setup:
- **2x Web-Replicas** (Cross-Replica-Test braucht ≥ 2)
- **1x API-Replica** (Publisher-Seite)
- Redis-Container läuft schon aus #37/#38
4. Env-Var setzen: `REDIS_PUBSUB_DB=2` (eigener logischer Index)
5. Smoke-Tests:
- **End-to-End-Stripe-Test:**
1. Stripe-CLI: `stripe trigger customer.subscription.updated --customer cus_xxx`
2. Innerhalb 2s: GET `/api/tenants/<id>/subscription` auf Web-Replica A → erwarte neuen Status
3. Gleiche GET auf Web-Replica B → erwarte ebenfalls neuen Status
- **Reconnect-Test:**
1. `docker exec redis redis-cli SHUTDOWN NOSAVE`
2. Warte 5s, Web-Replicas loggen Disconnect
3. `docker compose up redis` neu starten
4. Innerhalb 30s sehen Logs `"subscriber reconnected, flushing local caches"`
5. Stripe-Test wiederholen, muss wieder funktionieren
- **Pub/Sub-Monitor:** Auf Staging `docker exec redis redis-cli -n 2 PSUBSCRIBE '*'` laufen lassen, parallel Webhook triggern → verifiziere dass Message published wird
6. Sentry-Check: Keine Subscriber-Errors in Issues-Liste
7. Coolify-Logs prüfen: Beim Boot jeder Web-Replica steht `"subscriber registry started: 4 channels"`
## Deviations from Plan
- **Entitlements cache** is NOT a separate two-tier cache. The repo has no in-process entitlements cache today — `resolveTenantProductAccess` (`apps/web/app/utils/entitlements/.server/productAccess.server.ts`) recomputes from the subscription on every call. Entitlement freshness piggy-backs on subscription invalidation: once `cache:subscription:invalidate` clears the subscription L1, the next entitlement check pulls a fresh subscription and derives a fresh decision. Channel inventory in `docs/architecture/two-tier-cache.md` documents the chain.
- ~~**Feature-Flag-Cache subscriber stub** was NOT shipped.~~ — superseded: Phase 05 #53 has since shipped the full feature-flag two-tier cache (`apps/web/app/utils/services/.server/featureFlagCache.server.ts` + `subscribeFeatureFlagCache()` in `cache-bootstrap.server.ts`) and the web-route mutators all call `invalidateFlagCache(name)` after the DB write. **2026-05-20 follow-up `9edc1357`** closed the last cross-process gap by giving the operator CLI a one-shot publisher (`publishFeatureFlagInvalidationOnce` in `@toolctrl/feature-flags`, wired into `toolctrl-cli feature-flags {enable,disable,set-rules}` via `broadcastFlagInvalidation`); without it the CLI-driven mutation left long-lived web replicas serving stale evaluations from L1 until the 60s TTL expired.
- **Subscriber-lifecycle framework** was NOT extracted as a separate `createSubscriberRegistry`. Each two-tier cache exposes its own idempotent `subscribe()` method (singleton-guarded). `apps/web/app/cache-bootstrap.server.ts` starts the two subscribers in parallel and logs errors via `webLogger`. A registry can be extracted later if we exceed ~4 channels.
- ~~**Reconnect-flush** is provided as `flushLocal()` on the two-tier cache but not yet wired to a Redis `'ready'` event handler — the L1 TTL safety-net (default 300s via `CACHE_L1_TTL_SECONDS`) covers reconnect-loss scenarios.~~ — **resolved 2026-05-25 (`47c37b88`)**. `SharedStoreAdapter.onReconnect(handler)` now exposes ioredis `'ready'`-after-disconnect events (noop on `MemoryAdapter`); `createTwoTierCache(...).subscribe()` automatically registers a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also removes the reconnect handler.
- **Cross-app fan-out (API publisher → Web subscriber)** uses per-app SharedStore namespaces (`api`, `web`). For broadcasts to reach across processes both apps must share the same SharedStore namespace; this is documented as a follow-up in `docs/architecture/two-tier-cache.md`. The local-only mutation paths (web-side `clearTenantSubscriptionCache`) still publish on the web channel so in-app fan-out across web replicas works today. **2026-05-20** — CLI feature-flag mutators sidestep the per-app namespace mismatch by explicitly publishing on the `web` namespace (configurable via `FEATURE_FLAGS_PUBLISH_NAMESPACE`), proving the pattern that the broader API↔Web platform-namespace refactor can adopt.
- ~~**Multi-Process testcontainers integration test** (Step 9) is deferred. In-process tests already prove cross-replica fan-out by routing two `SharedStore` facades through one `MemoryAdapter`, which is the same fan-out path Redis exercises.~~ — **resolved 2026-05-25 (`f4b290bb`)**. New `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` brings up a `redis:7-alpine` container via `testcontainers` (or honours `PUBSUB_TEST_REDIS_URL`) and exercises cross-instance pub/sub fan-out, reconnect-flush, and peer-L2-mutation-after-reconnect. Skipped by default (no env var + no Docker) — opt-in mirrors the `packages/db/src/outbox-claim.test.ts` pattern. Verified live against the local Docker daemon (3/3 green).
## Progress Log
- **2026-05-17** — Commit `db117362` — `03-scaling/pubsub-cache: add two-tier cache helper combining L1 cachified + L2 shared-store`. New `createTwoTierCache<T>` in `packages/shared-store/src/helpers/two-tier-cache.ts` pairs an in-process L1 (Map default, swappable for LRUCache) with a SharedStore L2 + pub/sub channel. Public surface: `get/set/invalidate/subscribe/deleteLocal/flushLocal` + `l1`/`l2`/`namespace` accessors. `pubsubEnabled: false` flips `invalidate` to local-only + makes `subscribe()` a no-op. 9 unit tests cover round-trip, L2 fallback, cross-replica fan-out, localOnly, wildcard flush, kill-switch, custom subscriber-extra-handler. Exported from `@toolctrl/shared-store`.
- **2026-05-17** — Commit `b258ba0c` — `03-scaling/pubsub-cache: migrate subscription + tenant caches to two-tier with pub/sub`. New `apps/web/app/utils/services/.server/subscriptionCache.server.ts` (subscription namespace) and `apps/web/app/utils/db/tenant-cache.server.ts` (tenant namespace). Existing cachified LRU keeps the hot read-path; the new SharedStore L2 + pub/sub broadcasts cross-replica invalidations. `clearTenantSubscriptionCache(tenantId)` now also calls `getSubscriptionCache().invalidate(tenantId)`; subscribers bridge into `clearCacheKey('tenantSubscription:<id>')` / `clearCacheKey('tenant:<id>'/'tenantSimple:<id>'/'tenantIdOrSlug:<id>')`. New `apps/web/app/cache-bootstrap.server.ts` is wired into `entry.server.tsx` and is idempotent + env-gated. 12 new unit tests (5 subscriptionCache + 3 tenant-cache + 4 subscriptionService updates).
- **2026-05-17** — Commit `15bf34d5` — `03-scaling/pubsub-cache: publish subscription invalidation from api stripe webhook`. New `apps/api/src/cache/subscription-cache-publisher.ts` exposes `publishSubscriptionInvalidation(tenantId)`. Stripe webhook handler calls it fire-and-forget after `updateTenantSubscriptionProductFromStripe`. Publish errors are logged via `apiLogger.warn` so a Redis outage can never 5xx the webhook (Stripe would retry). New test asserts publisher fires with the correct tenantId.
- **2026-05-17** — `03-scaling/pubsub-cache: env vars, docs, progress log` (this commit). Adds `CACHE_INVALIDATION_ENABLED` + `CACHE_L1_TTL_SECONDS` to `packages/config/src/catalog.ts` and `.env.example`. Adds `docs/architecture/two-tier-cache.md` covering pattern decision rubric, channel naming, reconnect semantics, channel inventory, implementation recipe, feature-flag pre-written recipe for #53, kill-switch operational guidance.
- **2026-05-20** — Commit `9edc1357` — `feat(feature-flags+cli): cross-process invalidation publisher (#39)`. New `publishFeatureFlagInvalidationOnce` + `publishFeatureFlagInvalidationViaStore` exports in `@toolctrl/feature-flags` provide a one-shot SharedStore-open → publish → close lifecycle so short-lived processes (CLI, scripts) can broadcast `cache:feature-flags:invalidate` without keeping a long-lived Redis client. `toolctrl-cli feature-flags {enable,disable,set-rules}` now call `broadcastFlagInvalidation(name)` after the DB mutation; channel namespace defaults to `web` and honours `FEATURE_FLAGS_PUBLISH_NAMESPACE` env override. 9 new unit tests in `packages/feature-flags/src/publisher.test.ts` + `packages/cli/src/commands/feature-flags/invalidate.test.ts` cover the wire format, env-namespace resolution, kill-switch path, and error-swallowing contract. Web + feature-flags + CLI typecheck green.
- **2026-05-25** — Commits `47c37b88` + `f4b290bb` — closes the two remaining Deviations. `47c37b88` (`feat(shared-store): flush L1 caches on redis reconnect`) adds `SharedStoreAdapter.onReconnect(handler)` to the adapter surface (ioredis `'ready'`-after-first-connect on `RedisAdapter`, noop on `MemoryAdapter`), re-exposes it on the `SharedStore` facade, and wires `createTwoTierCache().subscribe()` to register a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also unregisters the reconnect handler. New unit test in `two-tier-cache.test.ts` exercises the wiring through a proxy adapter (fire-then-flush, debounce-collapses-bursts, disposer-removes-handler). `f4b290bb` (`test(shared-store): real-redis testcontainers integration for two-tier cache`) adds `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` with three integration scenarios (cross-instance pub/sub fan-out, reconnect-flush, peer-L2-mutation-visible-after-reconnect). Opt-in via `PUBSUB_TEST_REDIS_URL` or a reachable Docker daemon; module-top-level boot so `describe.skipIf` sees a concrete decision pre-registration. Verified live against `redis:7-alpine` via testcontainers — 3/3 green. `testcontainers ^10.28.0` added as devDependency of `@toolctrl/shared-store`. Workspace typecheck + full test suite green (only pre-existing `uiInventory` failure remains).
## Lessons Learned
- The two-tier helper's `invalidate({ localOnly: true })` mode is the bridge that lets the subscriber callback purge local L1 without re-publishing — important to avoid invalidation loops.
- Keeping the legacy `cachified` LRU as the read path (rather than ripping it out) made the migration zero-risk: the new two-tier cache is purely additive (it owns the pub/sub channel) and existing route handlers see no behaviour change.
- vitest 0.29 lacks `vi.hoisted()`; for ESM mock factories that need a shared mock fn the workaround is to attach the fn to `globalThis` and reference it via getter inside the factory.
- Concurrent multi-agent setups need a serializer for shared files (config catalog, .env.example, PROGRESS.md). Stashes + autostash-pulls race, and another agent's `git stash`+`git reset` can wipe unstaged changes. Mitigation: `git add` immediately after any edit to a shared file so the change lives in the index rather than the working tree.