HiveGrid-Wireframes/hivegrid-cloud/docs/architecture/rw-split.md

Read / Write Split (Prisma $extends)

Phase 03 task #41. Status: code-complete; replica provisioning is operator-owned.

TL;DR

import { db, dbRead } from "~/utils/db.server";

// Default — primary (write target). Use for all writes, auth, read-after-write.
await db.user.update({ where: { id }, data: { lastSeenAt: new Date() } });

// Explicit primary — same client, communicates intent.
import { getWriteDbClient } from "@toolctrl/db/client.server";

// Replica — eventually consistent. Use for listings, dashboards, public reads.
const users = await dbRead.user.findMany({ where: { tenantId } });

dbRead is wired to DATABASE_READ_URL when set. When unset (single-DB dev, self-hosted setups) it returns the same client as db, so the code path is identical and lag = 0.

Convention: when to use which

Use dbRead (replica)	Use db / getWriteDbClient() (primary)
List endpoints / paginated grids	All mutations (create, update, delete, upsert)
Public marketing pages (sites, portals)	Auth-resolution paths (session → user → tenant)
Knowledge-base public reads + search	Stripe webhook handlers reading state they just wrote
Analytics dashboards & aggregations	Outbox dispatcher reading PENDING events (see #42)
Admin tenant / role enumerations (post-auth)	Anything inside a db.$transaction(...) callback
Dashboard counts (count() / groupBy())	Idempotency-key lookups (request-scoped consistency)
Search-result rendering	Read-after-write inside the same request

When in doubt: leave on db (= primary). The replica is an optimisation, not a correctness requirement. Auto-routing is intentionally not implemented — the spec rejects it because it makes read-after-write hazards invisible.

Read-after-write hazard

Streaming replication is eventually consistent:

• Typical lag: < 1 s
• Burst writes (bulk imports, k6 load tests): up to 5–10 s
• A query issued on the replica milliseconds after the write hits the primary will read the pre-write state.

Mitigation:

1. Always use db inside transactions and for writes.
2. Never re-read on dbRead what you just wrote on db in the same request. If you must, route the read through db too.
3. Auth-critical paths (getUserAndTenants, session resolution) stay on db even though they're "reads" — they gate the rest of the request, and a stale read can mis-route a tenant boundary check.

Fallback semantics

getReadDbClient():

export function getReadDbClient(): DbClient {
  const readUrl = process.env.DATABASE_READ_URL;
  if (!readUrl || readUrl.length === 0) {
    return getDbClient(); // single instance, no double connection
  }
  // memoised replica singleton
  if (!global.__dbRead) {
    global.__dbRead = createDbClient({ connectionString: readUrl });
  }
  return global.__dbRead;
}

State of DATABASE_READ_URL	dbRead returns
unset / empty	the write client (db === dbRead)
valid postgres://...	a distinct read-only client

This means CI, local dev, and any deploy without a replica continue to work without code changes. All migrations are zero-risk to roll out.

Telemetry

@toolctrl/db/telemetry.server exposes attachPrismaTelemetry() — a $extends({ query }) wrapper that records every operation with { client: 'read' | 'write', model, operation, durationMs, ok }. Wire it in the app bootstrap by adapting the recorder to your metrics sink (prom-client Histogram / Counter or an OpenTelemetry Meter). The canonical counter shape is prisma.queries.total{client,model,operation}.

Coolify / staging setup (operator-owned)

Replica provisioning is out of scope for the application repo — these steps live with the platform operator.

1. Primary Postgres (Coolify service postgres). postgresql.conf:

   wal_level = replica
   max_wal_senders = 3
   wal_keep_size = 256MB
   

   Then:

   CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD '...';
   CREATE USER read_user WITH ENCRYPTED PASSWORD '...';
   GRANT CONNECT ON DATABASE toolctrl TO read_user;
   GRANT USAGE ON SCHEMA public TO read_user;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_user;
   ALTER DEFAULT PRIVILEGES IN SCHEMA public
     GRANT SELECT ON TABLES TO read_user;
   

2. Replica Postgres (Coolify service postgres-replica, postgres:16-alpine):

   # Init from primary
   until pg_basebackup -h postgres -D /var/lib/postgresql/data -U replicator -P -R; do
     sleep 5
   done
   

   postgresql.replica.conf:

   hot_standby = on
   primary_conninfo = 'host=postgres port=5432 user=replicator password=...'
   

3. App env:

   DATABASE_URL=postgres://app_user:...@pgbouncer:6432/toolctrl?pgbouncer=true&connection_limit=10
   DATABASE_READ_URL=postgres://read_user:...@postgres-replica:5432/toolctrl?connection_limit=10
   

   Optionally front the replica with its own PgBouncer for connection pooling.

4. Verification:

   -- On primary
   SELECT * FROM pg_stat_replication;
   
   -- On replica
   SELECT pg_last_xact_replay_timestamp(), now() - pg_last_xact_replay_timestamp() AS lag;
   

   Grafana panel prisma.queries.total{client="read"} should rise on dashboard / KB / portal traffic.

Rollback

• Per-call-site: git revert <migration-commit> flips that module back to db (primary). No data change.
• Global: unset DATABASE_READ_URL and restart the app. dbRead === db, all traffic hits the primary.
• Replica down: Prisma's read client will throw connection errors. Either unset DATABASE_READ_URL (degraded performance, no errors) or add a circuit-breaker (future work, out of scope here).

Migration recipe (for future modules)

1. Find candidate read-only functions — pure findMany, findUnique, findFirst, count, groupBy, aggregate with no surrounding write.
2. Check the caller graph — is this function called immediately after a write that updates the same rows? If yes, leave on db.
3. Replace db.<model>.<op> with dbRead.<model>.<op> and add a one-line comment explaining the choice.
4. Add dbRead to the file's imports alongside db (keep db for the mutations still in the file).
5. Run pnpm lint --max-warnings=0 and pnpm typecheck.
6. Tests stay green because of the fallback semantics — single-DB CI doesn't see a difference.
7. Commit per-module (03-scaling/rw-split: migrate <module> ...) so a single revert can roll back one module.

What was migrated in #41

• apps/web/app/modules/crm/services/CrmService.ts — user enumeration for sync display.
• apps/web/app/modules/knowledgeBase/db/*.db.server.ts — public KB reads, listings, search, featured-article queries, dashboard counts.
• apps/web/app/modules/analytics/db/analyticsUniqueVisitors.db.server.ts — groupBy aggregations.
• apps/web/app/utils/db/tenants.db.server.ts — admin tenant enumerations (post-auth).
• apps/web/app/modules/sites/db/sites.db.server.ts — read-only site loaders (auth-gating getSiteByTypeAndSiteId left on primary).
• apps/web/app/modules/portals/db/portals.db.server.ts and portalPages.db.server.ts — listings, page lookups, dashboard counts.

appData.ts was not migrated wholesale; only the secondary tenant listings it consumes were moved. The auth-resolution path stays on primary by design.