HiveGrid-Wireframes/hivegrid-cloud/docs/architecture/two-tier-cache.md

Two-Tier Cache Pattern

Status: active Owner: platform-scaling Related: packages/shared-store/README.md, docs/improvement-plan/03-scaling-infrastructure/pubsub-cache-invalidation.md

Why

A SharedStore-only cache pays one Redis round-trip per read (~1-3 ms on a healthy network). Hot paths that hit the same key 10-30 times per request (entitlement checks, limit counters, UI feature gating) accumulate enough overhead that we need an in-process L1 — but an in-process L1 alone goes stale across replicas. The two-tier cache combines both:

Tier	Backing	Latency	Consistency
L1	Map / LRUCache in process	sub-microsecond	per-replica
L2	SharedStore (Redis or memory)	1-3 ms	cross-replica
Pub/Sub	Redis PUBLISH on cache:<ns>:invalidate	< 50 ms typical fan-out	best-effort, see safety net

A mutating call invalidates its local L1+L2 and publishes the key on the namespace channel. Every replica's subscriber receives the message and drops the key from its L1 (and optionally the L2). Reads probe L1 first; on miss, fall back to L2; on L2 miss, refresh from the source-of-truth (typically Postgres) and populate both tiers.

Decision rubric

Use a two-tier cache when all three hold:

1. The key is read more than ~5x per request, or by every request (subscription, tenant, entitlements, feature flag).
2. Source-of-truth mutations are rare relative to reads (Stripe webhook firing seconds-to-minutes apart, admin edits).
3. Stale reads have user-visible consequences (showing the wrong plan limit, denying access after an upgrade).

Otherwise, prefer:

• In-process LRU only when consistency doesn't matter (route-stable computed values, request-scoped caches).
• SharedStore cache only when the read-rate is low and consistency must be strict (idempotency keys, session lookups).
• No cache when the source is already fast (cached Prisma result, in-memory blueprint).

Channel naming

Channels follow shared-store:<store-namespace>:ch:cache:<cache-namespace>:invalidate. Concretely, the web SharedStore (namespace: "web") emits subscription invalidations on shared-store:web:ch:cache:subscription:invalidate. The API SharedStore (namespace: "api") emits on shared-store:api:ch:cache:subscription:invalidate.

For cross-app fan-out (Stripe webhook in API → web replicas), both apps must use the same SharedStore namespace so they share a channel. The current implementation uses per-app namespaces (api, web) but both connect to the same Redis logical DB index (default 1). The actual channel string is namespaced under the writing app's SharedStore prefix. If you need cross-app fan-out, either:

1. Use a dedicated shared namespace (e.g. namespace: "platform") for both apps' two-tier caches, or
2. Have the publishing app PUBLISH the message via a SharedStore configured with the receiving app's namespace explicitly.

The Stripe webhook → subscription pattern in this codebase publishes on shared-store:api:ch:cache:subscription:invalidate. For a production multi-replica web deploy to actually receive that broadcast, configure both apps/api/src/cache/subscription-cache-publisher.ts and apps/web/app/utils/services/.server/subscriptionCache.server.ts to construct their SharedStores with the same namespace (recommended: introduce getCacheSharedStore({ namespace: "cache" }) for both apps). The current implementation broadcasts within the API process and applies the same clearTenantSubscriptionCache semantics on the web side via local mutation-paths that already publish; the cross-process broadcast is the work item documented under Deviations in the detail-md.

Reconnect semantics

ioredis auto-reconnects on connection loss with exponential backoff. While disconnected, the subscriber is offline and any PUBLISH issued by another replica is lost forever (pub/sub has no persistence). Mitigations layered in order:

1. L1 TTL is the floor. Default CACHE_L1_TTL_SECONDS=300 (5 min). Any missed invalidation self-heals within five minutes because the L1 entry expires. Lower the TTL for hotter paths if stale-window tolerance shrinks; raise it for marketing-content-grade caches.
2. Reconnect flush. Each two-tier cache exposes flushLocal() so a higher-level boot hook can cache.flushLocal() on a Redis reconnect event. The minimal implementation is "TTL is enough"; a more aggressive deployment can wire subscriber.on('ready', () => cache.flushLocal()) in app boot.
3. Outbox fallback. For strictly-consistent flows (billing, entitlement enforcement), the source of truth writes an outbox event (see docs/architecture/outbox-pattern.md); the outbox dispatcher consumes the event and re-issues the invalidation. This guarantees at-least-once delivery and survives Redis outages — at the cost of one DB round-trip and ~1 second dispatch latency.

Channel inventory

Channel	Publisher	Subscriber	Payload	Defined in
cache:subscription:invalidate	apps/api/src/billing/stripe-webhook-route.ts (after updateTenantSubscriptionProductFromStripe); apps/web/.../subscriptionService.ts (clearTenantSubscriptionCache, clearAllSubscriptionsCache)	apps/web/app/utils/services/.server/subscriptionCache.server.ts → bridges into clearCacheKey('tenantSubscription:<tenantId>')	tenantId (string) or "*" for wildcard	subscriptionCache.server.ts + subscription-cache-publisher.ts
cache:tenant:invalidate	apps/web/.../tenant-cache.server.ts (publishTenantInvalidation)	apps/web/.../tenant-cache.server.ts → clears tenant:, tenantSimple:, tenantIdOrSlug: cachified keys	tenantId (string) or "*"	tenant-cache.server.ts
cache:entitlements:invalidate	piggy-backs on subscription — entitlements derive from the subscription and recompute on the next read after the subscription L1 is flushed	piggy-backs on subscription	n/a — chained	n/a (chain)
cache:feature-flags:invalidate	packages/feature-flags publishers, including CLI feature-flag enable/disable/rule updates	web feature-flag cache subscribers through the shared two-tier cache bootstrap	{ flagKey?, tenantId?, all? }	packages/feature-flags/src/publisher.ts

Implementing a new two-tier cache

1. Create a <namespace>-cache.server.ts module per app (api/web) that wraps createTwoTierCache({ store: getSharedStore(), namespace, ttlSeconds, pubsubEnabled, logger }).
2. Expose two functions: publish<Namespace>Invalidation(key) (for mutators) and subscribe<Namespace>Cache() (idempotent boot hook).
3. The subscriber callback bridges into the existing local cache representation (clearCacheKey() for cachified-LRU, Map.delete() for ad-hoc maps).
4. Add the boot hook to apps/web/app/cache-bootstrap.server.ts (or equivalent API bootstrap) so the subscriber starts before request traffic.
5. Add publisher call-sites at every mutation that owns the SoT (database write, admin action, scheduled job).
6. Write a co-located test that drives the two-tier helper directly against a MemoryAdapter and asserts the cachified-bridge fires.

Feature-Flag Cache Notes

Feature-flag invalidation is now implemented through the shared two-tier cache publisher path. CLI feature-flag mutations publish to cache:feature-flags:invalidate after the audit write, and web subscribers flush L1 entries immediately instead of waiting for the TTL window. Keep new feature-flag mutators on the same publisher path.

Kill switch

CACHE_INVALIDATION_ENABLED=false (env var) flips both publishers and subscribers into local-only mode. The cache still works — read-through still populates L1+L2 — but invalidate becomes a local-only delete + no PUBLISH, and subscribers do not start. The system degrades to "every replica has its own L1 that expires after CACHE_L1_TTL_SECONDS". Use this switch:

• During a Redis incident to stop publisher errors from spamming Sentry
• During a deployment when subscribers can't keep up with traffic patterns
• During investigation of a stale-cache bug to confirm whether the publisher or subscriber path is at fault

Always re-enable after the incident; the L1-only mode has no replica-consistency floor below the L1 TTL.

Tests

• Unit tests for createTwoTierCache cover round-trip, fallback, cross-replica invalidation, localOnly, wildcard flush, pubsub kill-switch — packages/shared-store/src/helpers/two-tier-cache.test.ts
• Subscription cache integration: apps/web/app/utils/services/.server/subscriptionCache.server.test.ts
• Tenant cache integration: apps/web/app/utils/db/tenant-cache.server.test.ts
• Stripe webhook publish path: apps/api/src/billing/stripe-webhook-route.test.ts (asserts publishSubscriptionInvalidation(tenantId) is called after the DB update)