Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/03-scaling-infrastructure/pubsub-cache-invalidation.md
T

25 KiB
Raw Blame History

Pub/Sub Cache-Invalidation

Status: in_progress (subscription + tenant + feature-flag two-tier caches shipped; CLI cross-process publisher closed 2026-05-20; reconnect-flush + testcontainers integration closed 2026-05-25 — remaining work is the cross-app namespace fan-out documented in Deviations) Task-ID: #39 Phase: 03 Estimated Effort: 3-4 Tage Owner: unassigned

Problem

Selbst nach #38 (Shared-Store-Migration) verbleibt ein Konsistenz-Problem: einige hot caches sollen weiterhin lokal in-process (cachified-LRU) gehalten werden, weil Redis-Roundtrips pro Request zu teuer sind. Beispiele:

  • Subscription-Cache (Web): Pro Request mehrfach abgefragt (Entitlement-Checks, Limit-Counter, UI-Visibility). Redis-Roundtrip 1-3ms × 20 Calls = 20-60ms Overhead
  • Tenant-Cache: Pro Request fix
  • Entitlements-Cache: Pro Request fix
  • Feature-Flag-Cache (kommt in Phase 05): Pro UI-Render fix

Lösung: Zwei-Stufiger Cache. L1 = in-process cachified (schnell, nicht-shared), L2 = SharedStore (langsam, shared). Invalidations werden über Pub/Sub propagiert: wenn ein Event den Cache invalidieren soll, published derjenige, der die Quelle ändert (z.B. Stripe-Webhook in API), und ALLE Web-Replicas hören mit und purgen ihren L1.

Aktueller Zustand: Stripe-Webhook in apps/api/src/billing/stripe-webhook-route.ts:60-138 updated DB + ruft clearSubscriptionsCache(tenantId) lokal in der API. Aber: der Cache liegt nicht in der API, sondern in jeder Web-Replica. Heute: Web-Cache wird erst nach Ablauf von 1h-TTL frisch. Resultat: User kann z.B. nach Plan-Upgrade noch bis zu 1h alte Limits sehen.

Depends on

  • 03-scaling-infrastructure/shared-store-package.md — liefert createCacheStore mit Pub/Sub-Invalidation
  • 03-scaling-infrastructure/shared-store-migration.md — Subscription-Cache muss schon über SharedStore laufen, sonst kein Channel zum Subscriben
  • 02-observability/sentry-wiring-web.md — Subscriber-Errors müssen captured werden, sonst tappt der Operator im Dunkeln wenn Subscriber stillschweigend stirbt

Blocks

  • 05-business-features/feature-flags.md — Feature-Flag-Cache nutzt identisches Pattern
  • 03-scaling-infrastructure/tenant-snapshot.md — Tenant-Reset published Invalidation für Tenant-Cache
  • alle nachfolgenden Caches mit Cross-Replica-Konsistenz-Anforderung

Approach

Pattern: Source-of-Truth-Owner published, Cache-Holder subscribed.

  1. Publisher-Seite: Wer immer die SoT ändert (API für Subscription via Stripe-Webhook, Worker für Tenant-Snapshot, etc.) ruft nach Commit cache.invalidate(key) auf dem SharedStore-Cache. Das published auf Channel <namespace>:invalidate mit Payload { key }.

  2. Subscriber-Seite: Jede Web-Replica (und API, wo sie auch L1 hält) startet beim Boot einen Subscriber pro relevantem Cache-Channel. Der Subscriber-Callback purged den lokalen L1 (cachified.delete) und optional auch den L2 (für Cleanup).

  3. Auto-Reconnect: ioredis hat eingebauten Reconnect. Wir wrappen das mit einem onReconnect-Handler, der nach erfolgreichem Reconnect einmal alle L1-Caches komplett flushed (cache.clearAll()), weil wir nicht wissen welche Invalidations während Disconnect verloren gingen.

  4. Replay-fähig: Pub/Sub hat keine Persistence — bei Disconnect verloren. Mitigation: L1-TTL bleibt kurz (5-15min), so dass im Worst-Case-Disconnect-Szenario nach kurzer Zeit auto-refresh greift. Für strikte Konsistenz: Outbox-Pattern (#42) für kritische Events, dann Worker konsumiert Outbox und published Invalidation. Doppelte Sicherheit.

  5. Multi-Process-Setup:

    • API (1 Replica): Publisher für Subscription, Tenant, Entitlements
    • Web (2+ Replicas): Subscriber für alle obigen
    • Worker (2+ Replicas): Publisher für lange laufende Updates (Bulk-Imports, Tenant-Snapshot-Restore)

Vier konkrete Caches migrieren

A. Subscription-Cache (Stripe)

  • Source: apps/api/src/billing/stripe-webhook-route.ts:60-138
  • Trigger: Stripe-Event (customer.subscription.updated, invoice.payment_succeeded, etc.)
  • Channel: cache:subscription:invalidate
  • Payload: { tenantId: string }
  • Publisher: API in Webhook-Handler nach updateTenantSubscriptionFromStripe
  • Subscriber: Web (jede Replica) ruft clearSubscriptionsCache(tenantId)

B. Entitlements-Cache

  • Source: apps/web/app/utils/entitlements/.server/productAccess.server.ts
  • Trigger: Subscription-Change (impliziert Entitlement-Change), Manual-Admin-Edit
  • Channel: cache:entitlements:invalidate
  • Payload: { tenantId: string }
  • Publisher: Identisch zur Subscription-Trigger-Stelle (gleiche Webhook-Handler) + Admin-Edit-Route
  • Subscriber: Web purged in-process Entitlements-Map

C. Tenant-Cache

  • Source: apps/web/app/utils/data/.server/appData.ts
  • Trigger: Tenant-Settings-Update, Tenant-Logo-Upload, Member-Add/Remove
  • Channel: cache:tenant:invalidate
  • Payload: { tenantId: string }
  • Publisher: Web (eigene Action) UND API (bei API-getriebener Mutation)
  • Subscriber: Alle Web-Replicas + API (wo Tenant gelesen wird)

D. Feature-Flag-Cache (Vorbereitung für #53)

  • Source: kommt in Phase 05
  • Channel: cache:feature-flag:invalidate
  • Payload: { flagKey?: string; tenantId?: string; all?: boolean }
  • Publisher: Admin-Edit in Web
  • Subscriber: Web + API + Worker
  • In diesem Task nur das Channel-Naming + Subscriber-Stub vorbereiten, damit #53 nur noch den Publisher hinzufügt

Warum nicht andere Pattern:

  • ❌ Polling: schwergewichtig, schlechte Reaktionszeit
  • ❌ Cache-TTL niedrig halten (5s): Hammers gegen Redis und DB
  • ❌ WebSocket-basierte Push-Invalidation: Overkill, wir haben Redis schon
  • ❌ Kafka/NATS für ein einziges Use-Case: zu viel Infra

Implementation Steps

  1. Subscriber-Lifecycle-Framework

    • Files: packages/shared-store/src/helpers/subscriber-lifecycle.ts, subscriber-lifecycle.test.ts
    • createSubscriberRegistry({ store }) mit register(channel, handler) + start() + stop()
    • Auto-Reconnect-Handler: bei Redis-Reconnect alle registered Handler einmalig mit { type: 'flush' } aufrufen
    • Sentry-Integration: jeder Subscriber-Error wird captured + re-thrown an unhandled (Subscribers dürfen nicht stillschweigend sterben)
    • Tests: register/start/stop, Reconnect-Flush, Error-Capture
    • Commit: 03-scaling/pubsub: add subscriber lifecycle framework with reconnect handling
  2. App-Boot-Integration Web

    • Files: apps/web/app/entry.server.tsx (oder analoger Boot-Hook), apps/web/app/utils/subscribers/index.server.ts
    • SubscriberRegistry beim Boot starten, registriert alle Web-Subscriber (Subscription, Entitlements, Tenant, Feature-Flag-Stub)
    • Graceful-Shutdown-Handler: process.on('SIGTERM', () => registry.stop())
    • Tests: Boot-Integration-Test (Mock-Redis, check dass alle Channels subscribed sind)
    • Commit: 03-scaling/pubsub: wire subscriber registry into web boot
  3. App-Boot-Integration API

    • Files: apps/api/src/server.ts, apps/api/src/subscribers/index.ts
    • Analog Web, API hat eigene Subscriber-Liste (Tenant + Feature-Flag)
    • Commit: 03-scaling/pubsub: wire subscriber registry into api boot
  4. Subscription-Cache: Publisher in Stripe-Webhook

    • Files: apps/api/src/billing/stripe-webhook-route.ts, Test
    • Nach updateTenantSubscriptionFromStripe → subscriptionCache.invalidate(tenantId) (cache aus #38)
    • Tests: Mock Redis, verify Pub/Sub-Message wurde published
    • Commit: 03-scaling/pubsub: publish subscription invalidation from stripe webhook
  5. Subscription-Cache: Subscriber in Web

    • Files: apps/web/app/utils/subscribers/subscription-cache.subscriber.ts, Test
    • Handler ruft clearSubscriptionsCache(tenantId) + zusätzlich subscriptionCache.invalidate(tenantId, { localOnly: true }) für L1
    • Integration-Test: 2 Web-Instances, Webhook auf API → beide Web-Instances sehen neuen Wert < 2s
    • Commit: 03-scaling/pubsub: subscribe subscription cache invalidation in web
  6. Entitlements-Cache: Publisher + Subscriber

    • Files: apps/web/app/utils/entitlements/.server/productAccess.server.ts, apps/web/app/utils/subscribers/entitlements-cache.subscriber.ts, Tests
    • Publisher in API (gleiche Webhook-Stelle wie Subscription, weil korreliert)
    • Subscriber in Web purged in-process Entitlements-Map
    • Commit: 03-scaling/pubsub: add entitlements cache pub/sub channel
  7. Tenant-Cache: Publisher + Subscriber

    • Files: apps/web/app/utils/data/.server/appData.ts, apps/web/app/utils/subscribers/tenant-cache.subscriber.ts, Tests
    • Publisher in Web-Action für Tenant-Settings-Update; API-Side wo Tenant-Mutation passiert
    • Subscriber sowohl Web als auch API
    • Commit: 03-scaling/pubsub: add tenant cache pub/sub channel
  8. Feature-Flag-Cache-Stub (für #53)

    • Files: apps/web/app/utils/subscribers/feature-flag-cache.subscriber.ts (Stub), apps/api/src/subscribers/feature-flag-cache.subscriber.ts (Stub)
    • Subscriber registered Handler, der derzeit nur loggt ("feature-flag cache invalidation received, publisher not yet implemented")
    • Ermöglicht #53, nur Publisher-Seite zu ergänzen
    • Commit: 03-scaling/pubsub: add feature-flag cache subscriber stubs
  9. Multi-Process-Integration-Test

    • Files: apps/web/__integration__/cross-replica-invalidation.test.ts
    • testcontainers spinnt 1 Redis-Container + 2 in-process Web-Subscriber-Instances + 1 in-process Publisher
    • Test-Cases: invalidate triggert beide Subscriber; Reconnect-Flush nach Disconnect-Simulation; Concurrent-Invalidation
    • Commit: 03-scaling/pubsub: add multi-process invalidation integration tests
  10. Doku

    • Files: docs/observability/redis-stores.md (Erweiterung): Sektion „Pub/Sub-Channels" mit allen 4 Channels, Owner-Tabelle (wer published, wer subscribed)
    • Operational-Runbook: „Cache-Invalidation funktioniert nicht — Debug-Steps" (Redis-MONITOR, Subscriber-Health-Check)
    • Commit: 03-scaling/pubsub: document pub/sub channels and runbook

Test Requirements

  • Unit-Tests: SubscriberRegistry (register, start, stop, reconnect-flush, error-capture); jeder einzelne Subscriber-Handler (purged korrekt)
  • Integration-Tests: Cross-Replica-Invalidation pro Channel (Multi-Process-Setup via testcontainers); Reconnect-Verhalten (kill Redis, restart, verify Flush)
  • E2E-Tests (Playwright): Stripe-Webhook-Simulation → check ob Web-UI direkt neuen Subscription-Status zeigt (Replication-Test)
  • Coverage-Target: ≥ 90 % im Subscriber-Code, ≥ 85 % an Publisher-Stellen
  • Test-Files: Subscriber-Files co-located; Integration in apps/web/__integration__/ und apps/api/src/__integration__/

NICHT als completed markieren bevor:

  • Multi-Process-Integration-Test grün
  • Staging-Test: Stripe-Webhook auf Replica A → Web-Replica B + C sehen neuen Wert < 2s
  • Reconnect-Test auf Staging: docker restart redis → nach 30s sind alle Subscriber wieder online

Acceptance Criteria

  • SubscriberRegistry-Framework in packages/shared-store/src/helpers/subscriber-lifecycle.ts implementiert
  • Web + API booten SubscriberRegistry, registrieren alle relevanten Channels
  • Vier Channels live: cache:subscription:invalidate, cache:entitlements:invalidate, cache:tenant:invalidate, cache:feature-flag:invalidate
  • Stripe-Webhook published Subscription + Entitlements-Invalidation nach DB-Update
  • Tenant-Mutation-Stellen published Tenant-Invalidation
  • Feature-Flag-Subscriber-Stubs vorhanden (für #53)
  • Reconnect-Flush funktioniert: nach Redis-Restart sind alle L1-Caches geleert
  • Multi-Process-Integration-Test grün (2 Subscriber-Instances, ein Publisher, alle Channels)
  • Staging-Verifikation: End-to-End-Latenz Stripe-Event → Web-Cache-Invalidation < 2s
  • docs/observability/redis-stores.md enthält Pub/Sub-Channel-Tabelle + Operational-Runbook
  • pnpm lint, pnpm typecheck, pnpm test grün
  • Detail-MD Progress Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

Pro Channel separat rollback-bar:

  • Subscription-Channel: Revert Publisher- + Subscriber-Commits → zurück zu 1h-Stale-Verhalten, funktional
  • Entitlements/Tenant: Analog, zurück zu vorherigem Pattern (z.B. Loader-Refetch oder Page-Refresh)
  • SubscriberRegistry-Framework: Wenn das ganze Framework Probleme macht → Boot-Integration revert, Subscribers schweigen, Caches werden nicht invalidiert. Mitigation: L1-TTL kurz (5-15min) als Safety-Net

Global-Kill-Switch: Env-Var DISABLE_CACHE_PUBSUB=true → SubscriberRegistry started nicht, Publisher no-op. Erlaubt schnelles Disable ohne Redeploy.

DB-Migrations: keine in diesem Task.

Security Notes

  • Pub/Sub-Message-Validation: Subscriber-Handler validieren Payload strict (Zod-Schema pro Channel), reject Malformed mit Sentry-Capture. Verhindert Crash bei Garbage-Messages
  • Channel-Spoofing-Risk: Wer Redis-Zugriff hat, kann beliebig invalidieren. Mitigation: Redis-AUTH zwingend in Production, kein Public-Listen, Network-Isolation
  • Cache-Stampede bei Mass-Invalidation: Wenn Angreifer cache:tenant:invalidate für alle Tenants published → alle Replicas refetchen aus DB. Mitigation: Subscriber-Handler implementiert lokales Debounce (max 1 invalidation pro key pro 500ms)
  • Sensitive-Data in Invalidation-Payload: Payload enthält nur tenantId (UUID), keine PII. Klare Regel: Pub/Sub-Channels sind nicht zum Datentransport, nur Invalidation
  • Disconnect-During-Critical-Operation: Wenn Redis während Stripe-Webhook down ist, schlägt Publisher fehl. Mitigation: Outbox-Pattern (#42) puffert Events, Dispatcher retried Publish bis Redis wieder up
  • Sentry-Breadcrumbs: SubscriberRegistry meldet jede Disconnect/Reconnect-Event als Breadcrumb, damit Stale-Cache-Bugs in Produktion debugbar sind

Subagent Recommendations

  • Planning: Plan-Subagent mit Channel-Tabelle (A-D) + bestehender subscriptionService.ts-Implementation als Input
  • Research: Explore-Subagent für „aktuelle cachified-Nutzung in apps/web/app/utils/services/.server/ und apps/web/app/utils/data/.server/ — welche Caches haben non-trivial Konsistenz-Anforderungen?"
  • Implementation: Ein general-purpose-Subagent pro Channel (A-D), max 1 Channel pro Subagent — Publisher + Subscriber + Test gehören zusammen
  • Framework-Subagent: Separater general-purpose-Subagent für Schritte 1-3 (SubscriberRegistry-Framework + Boot-Integration), weil Framework-Bug alle Channels bricht
  • Tests: general-purpose-Subagent für Multi-Process-Integration-Test mit testcontainers (Step 9)
  • Verification: gsd-verifier für Acceptance-Criteria + Staging-Latenz-Messung
  • Code-Review: gsd-code-reviewer nach allen Channels — Fokus auf Reconnect-Pfade und Error-Handling im Subscriber

Staging-Deployment-Steps

  1. Branch agent/03-pubsub-cache-invalidation pushen
  2. Voraussetzung: #38 muss auf Staging erfolgreich getestet sein (Subscription läuft schon über SharedStore)
  3. Coolify-Setup:
    • 2x Web-Replicas (Cross-Replica-Test braucht ≥ 2)
    • 1x API-Replica (Publisher-Seite)
    • Redis-Container läuft schon aus #37/#38
  4. Env-Var setzen: REDIS_PUBSUB_DB=2 (eigener logischer Index)
  5. Smoke-Tests:
    • End-to-End-Stripe-Test:
      1. Stripe-CLI: stripe trigger customer.subscription.updated --customer cus_xxx
      2. Innerhalb 2s: GET /api/tenants/<id>/subscription auf Web-Replica A → erwarte neuen Status
      3. Gleiche GET auf Web-Replica B → erwarte ebenfalls neuen Status
    • Reconnect-Test:
      1. docker exec redis redis-cli SHUTDOWN NOSAVE
      2. Warte 5s, Web-Replicas loggen Disconnect
      3. docker compose up redis neu starten
      4. Innerhalb 30s sehen Logs "subscriber reconnected, flushing local caches"
      5. Stripe-Test wiederholen, muss wieder funktionieren
    • Pub/Sub-Monitor: Auf Staging docker exec redis redis-cli -n 2 PSUBSCRIBE '*' laufen lassen, parallel Webhook triggern → verifiziere dass Message published wird
  6. Sentry-Check: Keine Subscriber-Errors in Issues-Liste
  7. Coolify-Logs prüfen: Beim Boot jeder Web-Replica steht "subscriber registry started: 4 channels"

Deviations from Plan

  • Entitlements cache is NOT a separate two-tier cache. The repo has no in-process entitlements cache today — resolveTenantProductAccess (apps/web/app/utils/entitlements/.server/productAccess.server.ts) recomputes from the subscription on every call. Entitlement freshness piggy-backs on subscription invalidation: once cache:subscription:invalidate clears the subscription L1, the next entitlement check pulls a fresh subscription and derives a fresh decision. Channel inventory in docs/architecture/two-tier-cache.md documents the chain.
  • Feature-Flag-Cache subscriber stub was NOT shipped. — superseded: Phase 05 #53 has since shipped the full feature-flag two-tier cache (apps/web/app/utils/services/.server/featureFlagCache.server.ts + subscribeFeatureFlagCache() in cache-bootstrap.server.ts) and the web-route mutators all call invalidateFlagCache(name) after the DB write. 2026-05-20 follow-up 9edc1357 closed the last cross-process gap by giving the operator CLI a one-shot publisher (publishFeatureFlagInvalidationOnce in @toolctrl/feature-flags, wired into toolctrl-cli feature-flags {enable,disable,set-rules} via broadcastFlagInvalidation); without it the CLI-driven mutation left long-lived web replicas serving stale evaluations from L1 until the 60s TTL expired.
  • Subscriber-lifecycle framework was NOT extracted as a separate createSubscriberRegistry. Each two-tier cache exposes its own idempotent subscribe() method (singleton-guarded). apps/web/app/cache-bootstrap.server.ts starts the two subscribers in parallel and logs errors via webLogger. A registry can be extracted later if we exceed ~4 channels.
  • Reconnect-flush is provided as flushLocal() on the two-tier cache but not yet wired to a Redis 'ready' event handler — the L1 TTL safety-net (default 300s via CACHE_L1_TTL_SECONDS) covers reconnect-loss scenarios. — resolved 2026-05-25 (47c37b88). SharedStoreAdapter.onReconnect(handler) now exposes ioredis 'ready'-after-disconnect events (noop on MemoryAdapter); createTwoTierCache(...).subscribe() automatically registers a debounced flushLocal() reconnect handler (default 1000ms, override via reconnectFlushDebounceMs). The subscribe-disposer also removes the reconnect handler.
  • Cross-app fan-out (API publisher → Web subscriber) uses per-app SharedStore namespaces (api, web). For broadcasts to reach across processes both apps must share the same SharedStore namespace; this is documented as a follow-up in docs/architecture/two-tier-cache.md. The local-only mutation paths (web-side clearTenantSubscriptionCache) still publish on the web channel so in-app fan-out across web replicas works today. 2026-05-20 — CLI feature-flag mutators sidestep the per-app namespace mismatch by explicitly publishing on the web namespace (configurable via FEATURE_FLAGS_PUBLISH_NAMESPACE), proving the pattern that the broader API↔Web platform-namespace refactor can adopt.
  • Multi-Process testcontainers integration test (Step 9) is deferred. In-process tests already prove cross-replica fan-out by routing two SharedStore facades through one MemoryAdapter, which is the same fan-out path Redis exercises. — resolved 2026-05-25 (f4b290bb). New packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts brings up a redis:7-alpine container via testcontainers (or honours PUBSUB_TEST_REDIS_URL) and exercises cross-instance pub/sub fan-out, reconnect-flush, and peer-L2-mutation-after-reconnect. Skipped by default (no env var + no Docker) — opt-in mirrors the packages/db/src/outbox-claim.test.ts pattern. Verified live against the local Docker daemon (3/3 green).

Progress Log

  • 2026-05-17 — Commit db117362 — 03-scaling/pubsub-cache: add two-tier cache helper combining L1 cachified + L2 shared-store. New createTwoTierCache<T> in packages/shared-store/src/helpers/two-tier-cache.ts pairs an in-process L1 (Map default, swappable for LRUCache) with a SharedStore L2 + pub/sub channel. Public surface: get/set/invalidate/subscribe/deleteLocal/flushLocal + l1/l2/namespace accessors. pubsubEnabled: false flips invalidate to local-only + makes subscribe() a no-op. 9 unit tests cover round-trip, L2 fallback, cross-replica fan-out, localOnly, wildcard flush, kill-switch, custom subscriber-extra-handler. Exported from @toolctrl/shared-store.
  • 2026-05-17 — Commit b258ba0c — 03-scaling/pubsub-cache: migrate subscription + tenant caches to two-tier with pub/sub. New apps/web/app/utils/services/.server/subscriptionCache.server.ts (subscription namespace) and apps/web/app/utils/db/tenant-cache.server.ts (tenant namespace). Existing cachified LRU keeps the hot read-path; the new SharedStore L2 + pub/sub broadcasts cross-replica invalidations. clearTenantSubscriptionCache(tenantId) now also calls getSubscriptionCache().invalidate(tenantId); subscribers bridge into clearCacheKey('tenantSubscription:<id>') / clearCacheKey('tenant:<id>'/'tenantSimple:<id>'/'tenantIdOrSlug:<id>'). New apps/web/app/cache-bootstrap.server.ts is wired into entry.server.tsx and is idempotent + env-gated. 12 new unit tests (5 subscriptionCache + 3 tenant-cache + 4 subscriptionService updates).
  • 2026-05-17 — Commit 15bf34d5 — 03-scaling/pubsub-cache: publish subscription invalidation from api stripe webhook. New apps/api/src/cache/subscription-cache-publisher.ts exposes publishSubscriptionInvalidation(tenantId). Stripe webhook handler calls it fire-and-forget after updateTenantSubscriptionProductFromStripe. Publish errors are logged via apiLogger.warn so a Redis outage can never 5xx the webhook (Stripe would retry). New test asserts publisher fires with the correct tenantId.
  • 2026-05-17 — 03-scaling/pubsub-cache: env vars, docs, progress log (this commit). Adds CACHE_INVALIDATION_ENABLED + CACHE_L1_TTL_SECONDS to packages/config/src/catalog.ts and .env.example. Adds docs/architecture/two-tier-cache.md covering pattern decision rubric, channel naming, reconnect semantics, channel inventory, implementation recipe, feature-flag pre-written recipe for #53, kill-switch operational guidance.
  • 2026-05-20 — Commit 9edc1357 — feat(feature-flags+cli): cross-process invalidation publisher (#39). New publishFeatureFlagInvalidationOnce + publishFeatureFlagInvalidationViaStore exports in @toolctrl/feature-flags provide a one-shot SharedStore-open → publish → close lifecycle so short-lived processes (CLI, scripts) can broadcast cache:feature-flags:invalidate without keeping a long-lived Redis client. toolctrl-cli feature-flags {enable,disable,set-rules} now call broadcastFlagInvalidation(name) after the DB mutation; channel namespace defaults to web and honours FEATURE_FLAGS_PUBLISH_NAMESPACE env override. 9 new unit tests in packages/feature-flags/src/publisher.test.ts + packages/cli/src/commands/feature-flags/invalidate.test.ts cover the wire format, env-namespace resolution, kill-switch path, and error-swallowing contract. Web + feature-flags + CLI typecheck green.
  • 2026-05-25 — Commits 47c37b88 + f4b290bb — closes the two remaining Deviations. 47c37b88 (feat(shared-store): flush L1 caches on redis reconnect) adds SharedStoreAdapter.onReconnect(handler) to the adapter surface (ioredis 'ready'-after-first-connect on RedisAdapter, noop on MemoryAdapter), re-exposes it on the SharedStore facade, and wires createTwoTierCache().subscribe() to register a debounced flushLocal() reconnect handler (default 1000ms, override via reconnectFlushDebounceMs). The subscribe-disposer also unregisters the reconnect handler. New unit test in two-tier-cache.test.ts exercises the wiring through a proxy adapter (fire-then-flush, debounce-collapses-bursts, disposer-removes-handler). f4b290bb (test(shared-store): real-redis testcontainers integration for two-tier cache) adds packages/shared-store/src/helpers/two-tier-cache.testcontainers.test.ts with three integration scenarios (cross-instance pub/sub fan-out, reconnect-flush, peer-L2-mutation-visible-after-reconnect). Opt-in via PUBSUB_TEST_REDIS_URL or a reachable Docker daemon; module-top-level boot so describe.skipIf sees a concrete decision pre-registration. Verified live against redis:7-alpine via testcontainers — 3/3 green. testcontainers ^10.28.0 added as devDependency of @toolctrl/shared-store. Workspace typecheck + full test suite green (only pre-existing uiInventory failure remains).

Lessons Learned

  • The two-tier helper's invalidate({ localOnly: true }) mode is the bridge that lets the subscriber callback purge local L1 without re-publishing — important to avoid invalidation loops.
  • Keeping the legacy cachified LRU as the read path (rather than ripping it out) made the migration zero-risk: the new two-tier cache is purely additive (it owns the pub/sub channel) and existing route handlers see no behaviour change.
  • vitest 0.29 lacks vi.hoisted(); for ESM mock factories that need a shared mock fn the workaround is to attach the fn to globalThis and reference it via getter inside the factory.
  • Concurrent multi-agent setups need a serializer for shared files (config catalog, .env.example, PROGRESS.md). Stashes + autostash-pulls race, and another agent's git stash+git reset can wipe unstaged changes. Mitigation: git add immediately after any edit to a shared file so the change lives in the index rather than the working tree.