Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
+261
@@ -0,0 +1,261 @@
|
||||
# Pub/Sub Cache-Invalidation
|
||||
|
||||
**Status:** in_progress (subscription + tenant + feature-flag two-tier caches shipped; CLI cross-process publisher closed 2026-05-20; reconnect-flush + testcontainers integration closed 2026-05-25 — remaining work is the cross-app namespace fan-out documented in Deviations)
|
||||
**Task-ID:** #39
|
||||
**Phase:** 03
|
||||
**Estimated Effort:** 3-4 Tage
|
||||
**Owner:** unassigned
|
||||
|
||||
## Problem
|
||||
|
||||
Selbst nach #38 (Shared-Store-Migration) verbleibt ein Konsistenz-Problem: einige hot caches sollen weiterhin **lokal in-process** (cachified-LRU) gehalten werden, weil Redis-Roundtrips pro Request zu teuer sind. Beispiele:
|
||||
|
||||
- **Subscription-Cache (Web):** Pro Request mehrfach abgefragt (Entitlement-Checks, Limit-Counter, UI-Visibility). Redis-Roundtrip 1-3ms × 20 Calls = 20-60ms Overhead
|
||||
- **Tenant-Cache:** Pro Request fix
|
||||
- **Entitlements-Cache:** Pro Request fix
|
||||
- **Feature-Flag-Cache (kommt in Phase 05):** Pro UI-Render fix
|
||||
|
||||
Lösung: **Zwei-Stufiger Cache.** L1 = in-process cachified (schnell, nicht-shared), L2 = SharedStore (langsam, shared). Invalidations werden über Pub/Sub propagiert: wenn ein Event den Cache invalidieren soll, published derjenige, der die Quelle ändert (z.B. Stripe-Webhook in API), und ALLE Web-Replicas hören mit und purgen ihren L1.
|
||||
|
||||
Aktueller Zustand: Stripe-Webhook in `apps/api/src/billing/stripe-webhook-route.ts:60-138` updated DB + ruft `clearSubscriptionsCache(tenantId)` lokal in der API. Aber: der Cache liegt nicht in der API, sondern in **jeder Web-Replica**. Heute: Web-Cache wird erst nach Ablauf von 1h-TTL frisch. Resultat: User kann z.B. nach Plan-Upgrade noch bis zu 1h alte Limits sehen.
|
||||
|
||||
## Depends on
|
||||
|
||||
- `03-scaling-infrastructure/shared-store-package.md` — liefert `createCacheStore` mit Pub/Sub-Invalidation
|
||||
- `03-scaling-infrastructure/shared-store-migration.md` — Subscription-Cache muss schon über SharedStore laufen, sonst kein Channel zum Subscriben
|
||||
- `02-observability/sentry-wiring-web.md` — Subscriber-Errors müssen captured werden, sonst tappt der Operator im Dunkeln wenn Subscriber stillschweigend stirbt
|
||||
|
||||
## Blocks
|
||||
|
||||
- `05-business-features/feature-flags.md` — Feature-Flag-Cache nutzt identisches Pattern
|
||||
- `03-scaling-infrastructure/tenant-snapshot.md` — Tenant-Reset published Invalidation für Tenant-Cache
|
||||
- alle nachfolgenden Caches mit Cross-Replica-Konsistenz-Anforderung
|
||||
|
||||
## Approach
|
||||
|
||||
**Pattern: Source-of-Truth-Owner published, Cache-Holder subscribed.**
|
||||
|
||||
1. **Publisher-Seite:** Wer immer die SoT ändert (API für Subscription via Stripe-Webhook, Worker für Tenant-Snapshot, etc.) ruft nach Commit `cache.invalidate(key)` auf dem SharedStore-Cache. Das published auf Channel `<namespace>:invalidate` mit Payload `{ key }`.
|
||||
|
||||
2. **Subscriber-Seite:** Jede Web-Replica (und API, wo sie auch L1 hält) startet beim Boot einen Subscriber pro relevantem Cache-Channel. Der Subscriber-Callback purged den lokalen L1 (cachified.delete) und optional auch den L2 (für Cleanup).
|
||||
|
||||
3. **Auto-Reconnect:** ioredis hat eingebauten Reconnect. Wir wrappen das mit einem `onReconnect`-Handler, der nach erfolgreichem Reconnect einmal alle L1-Caches komplett flushed (`cache.clearAll()`), weil wir nicht wissen welche Invalidations während Disconnect verloren gingen.
|
||||
|
||||
4. **Replay-fähig:** Pub/Sub hat keine Persistence — bei Disconnect verloren. Mitigation: L1-TTL bleibt kurz (5-15min), so dass im Worst-Case-Disconnect-Szenario nach kurzer Zeit auto-refresh greift. Für strikte Konsistenz: Outbox-Pattern (#42) für kritische Events, dann Worker konsumiert Outbox und published Invalidation. Doppelte Sicherheit.
|
||||
|
||||
5. **Multi-Process-Setup:**
|
||||
- **API (1 Replica):** Publisher für Subscription, Tenant, Entitlements
|
||||
- **Web (2+ Replicas):** Subscriber für alle obigen
|
||||
- **Worker (2+ Replicas):** Publisher für lange laufende Updates (Bulk-Imports, Tenant-Snapshot-Restore)
|
||||
|
||||
## Vier konkrete Caches migrieren
|
||||
|
||||
### A. Subscription-Cache (Stripe)
|
||||
- **Source:** `apps/api/src/billing/stripe-webhook-route.ts:60-138`
|
||||
- **Trigger:** Stripe-Event (`customer.subscription.updated`, `invoice.payment_succeeded`, etc.)
|
||||
- **Channel:** `cache:subscription:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** API in Webhook-Handler nach `updateTenantSubscriptionFromStripe`
|
||||
- **Subscriber:** Web (jede Replica) ruft `clearSubscriptionsCache(tenantId)`
|
||||
|
||||
### B. Entitlements-Cache
|
||||
- **Source:** `apps/web/app/utils/entitlements/.server/productAccess.server.ts`
|
||||
- **Trigger:** Subscription-Change (impliziert Entitlement-Change), Manual-Admin-Edit
|
||||
- **Channel:** `cache:entitlements:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** Identisch zur Subscription-Trigger-Stelle (gleiche Webhook-Handler) + Admin-Edit-Route
|
||||
- **Subscriber:** Web purged in-process Entitlements-Map
|
||||
|
||||
### C. Tenant-Cache
|
||||
- **Source:** `apps/web/app/utils/data/.server/appData.ts`
|
||||
- **Trigger:** Tenant-Settings-Update, Tenant-Logo-Upload, Member-Add/Remove
|
||||
- **Channel:** `cache:tenant:invalidate`
|
||||
- **Payload:** `{ tenantId: string }`
|
||||
- **Publisher:** Web (eigene Action) UND API (bei API-getriebener Mutation)
|
||||
- **Subscriber:** Alle Web-Replicas + API (wo Tenant gelesen wird)
|
||||
|
||||
### D. Feature-Flag-Cache (Vorbereitung für #53)
|
||||
- **Source:** kommt in Phase 05
|
||||
- **Channel:** `cache:feature-flag:invalidate`
|
||||
- **Payload:** `{ flagKey?: string; tenantId?: string; all?: boolean }`
|
||||
- **Publisher:** Admin-Edit in Web
|
||||
- **Subscriber:** Web + API + Worker
|
||||
- In diesem Task nur das **Channel-Naming + Subscriber-Stub** vorbereiten, damit #53 nur noch den Publisher hinzufügt
|
||||
|
||||
**Warum nicht andere Pattern:**
|
||||
- ❌ Polling: schwergewichtig, schlechte Reaktionszeit
|
||||
- ❌ Cache-TTL niedrig halten (5s): Hammers gegen Redis und DB
|
||||
- ❌ WebSocket-basierte Push-Invalidation: Overkill, wir haben Redis schon
|
||||
- ❌ Kafka/NATS für ein einziges Use-Case: zu viel Infra
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **Subscriber-Lifecycle-Framework**
|
||||
- Files: `packages/shared-store/src/helpers/subscriber-lifecycle.ts`, `subscriber-lifecycle.test.ts`
|
||||
- `createSubscriberRegistry({ store })` mit `register(channel, handler)` + `start()` + `stop()`
|
||||
- Auto-Reconnect-Handler: bei Redis-Reconnect alle registered Handler einmalig mit `{ type: 'flush' }` aufrufen
|
||||
- Sentry-Integration: jeder Subscriber-Error wird captured + re-thrown an unhandled (Subscribers dürfen nicht stillschweigend sterben)
|
||||
- Tests: register/start/stop, Reconnect-Flush, Error-Capture
|
||||
- Commit: `03-scaling/pubsub: add subscriber lifecycle framework with reconnect handling`
|
||||
|
||||
2. **App-Boot-Integration Web**
|
||||
- Files: `apps/web/app/entry.server.tsx` (oder analoger Boot-Hook), `apps/web/app/utils/subscribers/index.server.ts`
|
||||
- SubscriberRegistry beim Boot starten, registriert alle Web-Subscriber (Subscription, Entitlements, Tenant, Feature-Flag-Stub)
|
||||
- Graceful-Shutdown-Handler: `process.on('SIGTERM', () => registry.stop())`
|
||||
- Tests: Boot-Integration-Test (Mock-Redis, check dass alle Channels subscribed sind)
|
||||
- Commit: `03-scaling/pubsub: wire subscriber registry into web boot`
|
||||
|
||||
3. **App-Boot-Integration API**
|
||||
- Files: `apps/api/src/server.ts`, `apps/api/src/subscribers/index.ts`
|
||||
- Analog Web, API hat eigene Subscriber-Liste (Tenant + Feature-Flag)
|
||||
- Commit: `03-scaling/pubsub: wire subscriber registry into api boot`
|
||||
|
||||
4. **Subscription-Cache: Publisher in Stripe-Webhook**
|
||||
- Files: `apps/api/src/billing/stripe-webhook-route.ts`, Test
|
||||
- Nach `updateTenantSubscriptionFromStripe` → `subscriptionCache.invalidate(tenantId)` (cache aus #38)
|
||||
- Tests: Mock Redis, verify Pub/Sub-Message wurde published
|
||||
- Commit: `03-scaling/pubsub: publish subscription invalidation from stripe webhook`
|
||||
|
||||
5. **Subscription-Cache: Subscriber in Web**
|
||||
- Files: `apps/web/app/utils/subscribers/subscription-cache.subscriber.ts`, Test
|
||||
- Handler ruft `clearSubscriptionsCache(tenantId)` + zusätzlich `subscriptionCache.invalidate(tenantId, { localOnly: true })` für L1
|
||||
- Integration-Test: 2 Web-Instances, Webhook auf API → beide Web-Instances sehen neuen Wert < 2s
|
||||
- Commit: `03-scaling/pubsub: subscribe subscription cache invalidation in web`
|
||||
|
||||
6. **Entitlements-Cache: Publisher + Subscriber**
|
||||
- Files: `apps/web/app/utils/entitlements/.server/productAccess.server.ts`, `apps/web/app/utils/subscribers/entitlements-cache.subscriber.ts`, Tests
|
||||
- Publisher in API (gleiche Webhook-Stelle wie Subscription, weil korreliert)
|
||||
- Subscriber in Web purged in-process Entitlements-Map
|
||||
- Commit: `03-scaling/pubsub: add entitlements cache pub/sub channel`
|
||||
|
||||
7. **Tenant-Cache: Publisher + Subscriber**
|
||||
- Files: `apps/web/app/utils/data/.server/appData.ts`, `apps/web/app/utils/subscribers/tenant-cache.subscriber.ts`, Tests
|
||||
- Publisher in Web-Action für Tenant-Settings-Update; API-Side wo Tenant-Mutation passiert
|
||||
- Subscriber sowohl Web als auch API
|
||||
- Commit: `03-scaling/pubsub: add tenant cache pub/sub channel`
|
||||
|
||||
8. **Feature-Flag-Cache-Stub (für #53)**
|
||||
- Files: `apps/web/app/utils/subscribers/feature-flag-cache.subscriber.ts` (Stub), `apps/api/src/subscribers/feature-flag-cache.subscriber.ts` (Stub)
|
||||
- Subscriber registered Handler, der derzeit nur loggt (`"feature-flag cache invalidation received, publisher not yet implemented"`)
|
||||
- Ermöglicht #53, nur Publisher-Seite zu ergänzen
|
||||
- Commit: `03-scaling/pubsub: add feature-flag cache subscriber stubs`
|
||||
|
||||
9. **Multi-Process-Integration-Test**
|
||||
- Files: `apps/web/__integration__/cross-replica-invalidation.test.ts`
|
||||
- testcontainers spinnt 1 Redis-Container + 2 in-process Web-Subscriber-Instances + 1 in-process Publisher
|
||||
- Test-Cases: invalidate triggert beide Subscriber; Reconnect-Flush nach Disconnect-Simulation; Concurrent-Invalidation
|
||||
- Commit: `03-scaling/pubsub: add multi-process invalidation integration tests`
|
||||
|
||||
10. **Doku**
|
||||
- Files: `docs/observability/redis-stores.md` (Erweiterung): Sektion „Pub/Sub-Channels" mit allen 4 Channels, Owner-Tabelle (wer published, wer subscribed)
|
||||
- Operational-Runbook: „Cache-Invalidation funktioniert nicht — Debug-Steps" (Redis-MONITOR, Subscriber-Health-Check)
|
||||
- Commit: `03-scaling/pubsub: document pub/sub channels and runbook`
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- **Unit-Tests:** SubscriberRegistry (register, start, stop, reconnect-flush, error-capture); jeder einzelne Subscriber-Handler (purged korrekt)
|
||||
- **Integration-Tests:** Cross-Replica-Invalidation pro Channel (Multi-Process-Setup via testcontainers); Reconnect-Verhalten (kill Redis, restart, verify Flush)
|
||||
- **E2E-Tests (Playwright):** Stripe-Webhook-Simulation → check ob Web-UI direkt neuen Subscription-Status zeigt (Replication-Test)
|
||||
- **Coverage-Target:** ≥ 90 % im Subscriber-Code, ≥ 85 % an Publisher-Stellen
|
||||
- **Test-Files:** Subscriber-Files co-located; Integration in `apps/web/__integration__/` und `apps/api/src/__integration__/`
|
||||
|
||||
**NICHT als completed markieren bevor:**
|
||||
- Multi-Process-Integration-Test grün
|
||||
- Staging-Test: Stripe-Webhook auf Replica A → Web-Replica B + C sehen neuen Wert < 2s
|
||||
- Reconnect-Test auf Staging: `docker restart redis` → nach 30s sind alle Subscriber wieder online
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] SubscriberRegistry-Framework in `packages/shared-store/src/helpers/subscriber-lifecycle.ts` implementiert
|
||||
- [ ] Web + API booten SubscriberRegistry, registrieren alle relevanten Channels
|
||||
- [ ] Vier Channels live: `cache:subscription:invalidate`, `cache:entitlements:invalidate`, `cache:tenant:invalidate`, `cache:feature-flag:invalidate`
|
||||
- [ ] Stripe-Webhook published Subscription + Entitlements-Invalidation nach DB-Update
|
||||
- [ ] Tenant-Mutation-Stellen published Tenant-Invalidation
|
||||
- [ ] Feature-Flag-Subscriber-Stubs vorhanden (für #53)
|
||||
- [ ] Reconnect-Flush funktioniert: nach Redis-Restart sind alle L1-Caches geleert
|
||||
- [ ] Multi-Process-Integration-Test grün (2 Subscriber-Instances, ein Publisher, alle Channels)
|
||||
- [ ] Staging-Verifikation: End-to-End-Latenz Stripe-Event → Web-Cache-Invalidation < 2s
|
||||
- [ ] `docs/observability/redis-stores.md` enthält Pub/Sub-Channel-Tabelle + Operational-Runbook
|
||||
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
|
||||
- [ ] Detail-MD Progress Log aktualisiert
|
||||
- [ ] PROGRESS.md Status auf `completed`
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
Pro Channel separat rollback-bar:
|
||||
|
||||
- **Subscription-Channel:** Revert Publisher- + Subscriber-Commits → zurück zu 1h-Stale-Verhalten, funktional
|
||||
- **Entitlements/Tenant:** Analog, zurück zu vorherigem Pattern (z.B. Loader-Refetch oder Page-Refresh)
|
||||
- **SubscriberRegistry-Framework:** Wenn das ganze Framework Probleme macht → Boot-Integration revert, Subscribers schweigen, Caches werden nicht invalidiert. Mitigation: L1-TTL kurz (5-15min) als Safety-Net
|
||||
|
||||
**Global-Kill-Switch:** Env-Var `DISABLE_CACHE_PUBSUB=true` → SubscriberRegistry started nicht, Publisher no-op. Erlaubt schnelles Disable ohne Redeploy.
|
||||
|
||||
**DB-Migrations:** keine in diesem Task.
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **Pub/Sub-Message-Validation:** Subscriber-Handler validieren Payload strict (Zod-Schema pro Channel), reject Malformed mit Sentry-Capture. Verhindert Crash bei Garbage-Messages
|
||||
- **Channel-Spoofing-Risk:** Wer Redis-Zugriff hat, kann beliebig invalidieren. Mitigation: Redis-AUTH zwingend in Production, kein Public-Listen, Network-Isolation
|
||||
- **Cache-Stampede bei Mass-Invalidation:** Wenn Angreifer `cache:tenant:invalidate` für alle Tenants published → alle Replicas refetchen aus DB. Mitigation: Subscriber-Handler implementiert lokales Debounce (max 1 invalidation pro key pro 500ms)
|
||||
- **Sensitive-Data in Invalidation-Payload:** Payload enthält nur tenantId (UUID), keine PII. Klare Regel: Pub/Sub-Channels sind nicht zum Datentransport, nur Invalidation
|
||||
- **Disconnect-During-Critical-Operation:** Wenn Redis während Stripe-Webhook down ist, schlägt Publisher fehl. Mitigation: Outbox-Pattern (#42) puffert Events, Dispatcher retried Publish bis Redis wieder up
|
||||
- **Sentry-Breadcrumbs:** SubscriberRegistry meldet jede Disconnect/Reconnect-Event als Breadcrumb, damit Stale-Cache-Bugs in Produktion debugbar sind
|
||||
|
||||
## Subagent Recommendations
|
||||
|
||||
- **Planning:** `Plan`-Subagent mit Channel-Tabelle (A-D) + bestehender `subscriptionService.ts`-Implementation als Input
|
||||
- **Research:** `Explore`-Subagent für „aktuelle cachified-Nutzung in `apps/web/app/utils/services/.server/` und `apps/web/app/utils/data/.server/` — welche Caches haben non-trivial Konsistenz-Anforderungen?"
|
||||
- **Implementation:** Ein `general-purpose`-Subagent pro Channel (A-D), max 1 Channel pro Subagent — Publisher + Subscriber + Test gehören zusammen
|
||||
- **Framework-Subagent:** Separater `general-purpose`-Subagent für Schritte 1-3 (SubscriberRegistry-Framework + Boot-Integration), weil Framework-Bug alle Channels bricht
|
||||
- **Tests:** `general-purpose`-Subagent für Multi-Process-Integration-Test mit testcontainers (Step 9)
|
||||
- **Verification:** `gsd-verifier` für Acceptance-Criteria + Staging-Latenz-Messung
|
||||
- **Code-Review:** `gsd-code-reviewer` nach allen Channels — Fokus auf Reconnect-Pfade und Error-Handling im Subscriber
|
||||
|
||||
## Staging-Deployment-Steps
|
||||
|
||||
1. Branch `agent/03-pubsub-cache-invalidation` pushen
|
||||
2. Voraussetzung: #38 muss auf Staging erfolgreich getestet sein (Subscription läuft schon über SharedStore)
|
||||
3. Coolify-Setup:
|
||||
- **2x Web-Replicas** (Cross-Replica-Test braucht ≥ 2)
|
||||
- **1x API-Replica** (Publisher-Seite)
|
||||
- Redis-Container läuft schon aus #37/#38
|
||||
4. Env-Var setzen: `REDIS_PUBSUB_DB=2` (eigener logischer Index)
|
||||
5. Smoke-Tests:
|
||||
- **End-to-End-Stripe-Test:**
|
||||
1. Stripe-CLI: `stripe trigger customer.subscription.updated --customer cus_xxx`
|
||||
2. Innerhalb 2s: GET `/api/tenants/<id>/subscription` auf Web-Replica A → erwarte neuen Status
|
||||
3. Gleiche GET auf Web-Replica B → erwarte ebenfalls neuen Status
|
||||
- **Reconnect-Test:**
|
||||
1. `docker exec redis redis-cli SHUTDOWN NOSAVE`
|
||||
2. Warte 5s, Web-Replicas loggen Disconnect
|
||||
3. `docker compose up redis` neu starten
|
||||
4. Innerhalb 30s sehen Logs `"subscriber reconnected, flushing local caches"`
|
||||
5. Stripe-Test wiederholen, muss wieder funktionieren
|
||||
- **Pub/Sub-Monitor:** Auf Staging `docker exec redis redis-cli -n 2 PSUBSCRIBE '*'` laufen lassen, parallel Webhook triggern → verifiziere dass Message published wird
|
||||
6. Sentry-Check: Keine Subscriber-Errors in Issues-Liste
|
||||
7. Coolify-Logs prüfen: Beim Boot jeder Web-Replica steht `"subscriber registry started: 4 channels"`
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- **Entitlements cache** is NOT a separate two-tier cache. The repo has no in-process entitlements cache today — `resolveTenantProductAccess` (`apps/web/app/utils/entitlements/.server/productAccess.server.ts`) recomputes from the subscription on every call. Entitlement freshness piggy-backs on subscription invalidation: once `cache:subscription:invalidate` clears the subscription L1, the next entitlement check pulls a fresh subscription and derives a fresh decision. Channel inventory in `docs/architecture/two-tier-cache.md` documents the chain.
|
||||
- ~~**Feature-Flag-Cache subscriber stub** was NOT shipped.~~ — superseded: Phase 05 #53 has since shipped the full feature-flag two-tier cache (`apps/web/app/utils/services/.server/featureFlagCache.server.ts` + `subscribeFeatureFlagCache()` in `cache-bootstrap.server.ts`) and the web-route mutators all call `invalidateFlagCache(name)` after the DB write. **2026-05-20 follow-up `9edc1357`** closed the last cross-process gap by giving the operator CLI a one-shot publisher (`publishFeatureFlagInvalidationOnce` in `@toolctrl/feature-flags`, wired into `toolctrl-cli feature-flags {enable,disable,set-rules}` via `broadcastFlagInvalidation`); without it the CLI-driven mutation left long-lived web replicas serving stale evaluations from L1 until the 60s TTL expired.
|
||||
- **Subscriber-lifecycle framework** was NOT extracted as a separate `createSubscriberRegistry`. Each two-tier cache exposes its own idempotent `subscribe()` method (singleton-guarded). `apps/web/app/cache-bootstrap.server.ts` starts the two subscribers in parallel and logs errors via `webLogger`. A registry can be extracted later if we exceed ~4 channels.
|
||||
- ~~**Reconnect-flush** is provided as `flushLocal()` on the two-tier cache but not yet wired to a Redis `'ready'` event handler — the L1 TTL safety-net (default 300s via `CACHE_L1_TTL_SECONDS`) covers reconnect-loss scenarios.~~ — **resolved 2026-05-25 (`47c37b88`)**. `SharedStoreAdapter.onReconnect(handler)` now exposes ioredis `'ready'`-after-disconnect events (noop on `MemoryAdapter`); `createTwoTierCache(...).subscribe()` automatically registers a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also removes the reconnect handler.
|
||||
- **Cross-app fan-out (API publisher → Web subscriber)** uses per-app SharedStore namespaces (`api`, `web`). For broadcasts to reach across processes both apps must share the same SharedStore namespace; this is documented as a follow-up in `docs/architecture/two-tier-cache.md`. The local-only mutation paths (web-side `clearTenantSubscriptionCache`) still publish on the web channel so in-app fan-out across web replicas works today. **2026-05-20** — CLI feature-flag mutators sidestep the per-app namespace mismatch by explicitly publishing on the `web` namespace (configurable via `FEATURE_FLAGS_PUBLISH_NAMESPACE`), proving the pattern that the broader API↔Web platform-namespace refactor can adopt.
|
||||
- ~~**Multi-Process testcontainers integration test** (Step 9) is deferred. In-process tests already prove cross-replica fan-out by routing two `SharedStore` facades through one `MemoryAdapter`, which is the same fan-out path Redis exercises.~~ — **resolved 2026-05-25 (`f4b290bb`)**. New `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` brings up a `redis:7-alpine` container via `testcontainers` (or honours `PUBSUB_TEST_REDIS_URL`) and exercises cross-instance pub/sub fan-out, reconnect-flush, and peer-L2-mutation-after-reconnect. Skipped by default (no env var + no Docker) — opt-in mirrors the `packages/db/src/outbox-claim.test.ts` pattern. Verified live against the local Docker daemon (3/3 green).
|
||||
|
||||
## Progress Log
|
||||
|
||||
- **2026-05-17** — Commit `db117362` — `03-scaling/pubsub-cache: add two-tier cache helper combining L1 cachified + L2 shared-store`. New `createTwoTierCache<T>` in `packages/shared-store/src/helpers/two-tier-cache.ts` pairs an in-process L1 (Map default, swappable for LRUCache) with a SharedStore L2 + pub/sub channel. Public surface: `get/set/invalidate/subscribe/deleteLocal/flushLocal` + `l1`/`l2`/`namespace` accessors. `pubsubEnabled: false` flips `invalidate` to local-only + makes `subscribe()` a no-op. 9 unit tests cover round-trip, L2 fallback, cross-replica fan-out, localOnly, wildcard flush, kill-switch, custom subscriber-extra-handler. Exported from `@toolctrl/shared-store`.
|
||||
- **2026-05-17** — Commit `b258ba0c` — `03-scaling/pubsub-cache: migrate subscription + tenant caches to two-tier with pub/sub`. New `apps/web/app/utils/services/.server/subscriptionCache.server.ts` (subscription namespace) and `apps/web/app/utils/db/tenant-cache.server.ts` (tenant namespace). Existing cachified LRU keeps the hot read-path; the new SharedStore L2 + pub/sub broadcasts cross-replica invalidations. `clearTenantSubscriptionCache(tenantId)` now also calls `getSubscriptionCache().invalidate(tenantId)`; subscribers bridge into `clearCacheKey('tenantSubscription:<id>')` / `clearCacheKey('tenant:<id>'/'tenantSimple:<id>'/'tenantIdOrSlug:<id>')`. New `apps/web/app/cache-bootstrap.server.ts` is wired into `entry.server.tsx` and is idempotent + env-gated. 12 new unit tests (5 subscriptionCache + 3 tenant-cache + 4 subscriptionService updates).
|
||||
- **2026-05-17** — Commit `15bf34d5` — `03-scaling/pubsub-cache: publish subscription invalidation from api stripe webhook`. New `apps/api/src/cache/subscription-cache-publisher.ts` exposes `publishSubscriptionInvalidation(tenantId)`. Stripe webhook handler calls it fire-and-forget after `updateTenantSubscriptionProductFromStripe`. Publish errors are logged via `apiLogger.warn` so a Redis outage can never 5xx the webhook (Stripe would retry). New test asserts publisher fires with the correct tenantId.
|
||||
- **2026-05-17** — `03-scaling/pubsub-cache: env vars, docs, progress log` (this commit). Adds `CACHE_INVALIDATION_ENABLED` + `CACHE_L1_TTL_SECONDS` to `packages/config/src/catalog.ts` and `.env.example`. Adds `docs/architecture/two-tier-cache.md` covering pattern decision rubric, channel naming, reconnect semantics, channel inventory, implementation recipe, feature-flag pre-written recipe for #53, kill-switch operational guidance.
|
||||
- **2026-05-20** — Commit `9edc1357` — `feat(feature-flags+cli): cross-process invalidation publisher (#39)`. New `publishFeatureFlagInvalidationOnce` + `publishFeatureFlagInvalidationViaStore` exports in `@toolctrl/feature-flags` provide a one-shot SharedStore-open → publish → close lifecycle so short-lived processes (CLI, scripts) can broadcast `cache:feature-flags:invalidate` without keeping a long-lived Redis client. `toolctrl-cli feature-flags {enable,disable,set-rules}` now call `broadcastFlagInvalidation(name)` after the DB mutation; channel namespace defaults to `web` and honours `FEATURE_FLAGS_PUBLISH_NAMESPACE` env override. 9 new unit tests in `packages/feature-flags/src/publisher.test.ts` + `packages/cli/src/commands/feature-flags/invalidate.test.ts` cover the wire format, env-namespace resolution, kill-switch path, and error-swallowing contract. Web + feature-flags + CLI typecheck green.
|
||||
- **2026-05-25** — Commits `47c37b88` + `f4b290bb` — closes the two remaining Deviations. `47c37b88` (`feat(shared-store): flush L1 caches on redis reconnect`) adds `SharedStoreAdapter.onReconnect(handler)` to the adapter surface (ioredis `'ready'`-after-first-connect on `RedisAdapter`, noop on `MemoryAdapter`), re-exposes it on the `SharedStore` facade, and wires `createTwoTierCache().subscribe()` to register a debounced `flushLocal()` reconnect handler (default 1000ms, override via `reconnectFlushDebounceMs`). The subscribe-disposer also unregisters the reconnect handler. New unit test in `two-tier-cache.test.ts` exercises the wiring through a proxy adapter (fire-then-flush, debounce-collapses-bursts, disposer-removes-handler). `f4b290bb` (`test(shared-store): real-redis testcontainers integration for two-tier cache`) adds `packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts` with three integration scenarios (cross-instance pub/sub fan-out, reconnect-flush, peer-L2-mutation-visible-after-reconnect). Opt-in via `PUBSUB_TEST_REDIS_URL` or a reachable Docker daemon; module-top-level boot so `describe.skipIf` sees a concrete decision pre-registration. Verified live against `redis:7-alpine` via testcontainers — 3/3 green. `testcontainers ^10.28.0` added as devDependency of `@toolctrl/shared-store`. Workspace typecheck + full test suite green (only pre-existing `uiInventory` failure remains).
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
- The two-tier helper's `invalidate({ localOnly: true })` mode is the bridge that lets the subscriber callback purge local L1 without re-publishing — important to avoid invalidation loops.
|
||||
- Keeping the legacy `cachified` LRU as the read path (rather than ripping it out) made the migration zero-risk: the new two-tier cache is purely additive (it owns the pub/sub channel) and existing route handlers see no behaviour change.
|
||||
- vitest 0.29 lacks `vi.hoisted()`; for ESM mock factories that need a shared mock fn the workaround is to attach the fn to `globalThis` and reference it via getter inside the factory.
|
||||
- Concurrent multi-agent setups need a serializer for shared files (config catalog, .env.example, PROGRESS.md). Stashes + autostash-pulls race, and another agent's `git stash`+`git reset` can wipe unstaged changes. Mitigation: `git add` immediately after any edit to a shared file so the change lives in the index rather than the working tree.
|
||||
Reference in new issue
Block a user