HiveGrid-Wireframes/hivegrid-cloud/docs/architecture/rw-split.md

# Read / Write Split (Prisma `$extends`)

> Phase 03 task #41. Status: code-complete; replica provisioning is operator-owned.

## TL;DR

```ts
import { db, dbRead } from "~/utils/db.server";

// Default — primary (write target). Use for all writes, auth, read-after-write.
await db.user.update({ where: { id }, data: { lastSeenAt: new Date() } });

// Explicit primary — same client, communicates intent.
import { getWriteDbClient } from "@toolctrl/db/client.server";

// Replica — eventually consistent. Use for listings, dashboards, public reads.
const users = await dbRead.user.findMany({ where: { tenantId } });
```

`dbRead` is wired to `DATABASE_READ_URL` when set. When unset (single-DB dev,
self-hosted setups) it returns the same client as `db`, so the code path is
identical and lag = 0.

## Convention: when to use which

| Use `dbRead` (replica) | Use `db` / `getWriteDbClient()` (primary) |
| ---------------------- | ----------------------------------------- |
| List endpoints / paginated grids | All mutations (`create`, `update`, `delete`, `upsert`) |
| Public marketing pages (sites, portals) | Auth-resolution paths (session → user → tenant) |
| Knowledge-base public reads + search | Stripe webhook handlers reading state they just wrote |
| Analytics dashboards & aggregations | Outbox dispatcher reading `PENDING` events (see #42) |
| Admin tenant / role enumerations (post-auth) | Anything inside a `db.$transaction(...)` callback |
| Dashboard counts (`count()` / `groupBy()`) | Idempotency-key lookups (request-scoped consistency) |
| Search-result rendering | Read-after-write inside the same request |

When in doubt: leave on `db` (= primary). The replica is an optimisation,
not a correctness requirement. **Auto-routing is intentionally not
implemented** — the spec rejects it because it makes read-after-write
hazards invisible.

## Read-after-write hazard

Streaming replication is **eventually consistent**:

- Typical lag: < 1 s
- Burst writes (bulk imports, k6 load tests): up to 5–10 s
- A query issued on the replica milliseconds after the write hits the
  primary will read the **pre-write** state.

**Mitigation:**

1. Always use `db` inside transactions and for writes.
2. Never re-read on `dbRead` what you just wrote on `db` in the same
   request. If you must, route the read through `db` too.
3. Auth-critical paths (`getUserAndTenants`, session resolution) stay on
   `db` even though they're "reads" — they gate the rest of the
   request, and a stale read can mis-route a tenant boundary check.

## Fallback semantics

`getReadDbClient()`:

```ts
export function getReadDbClient(): DbClient {
  const readUrl = process.env.DATABASE_READ_URL;
  if (!readUrl || readUrl.length === 0) {
    return getDbClient(); // single instance, no double connection
  }
  // memoised replica singleton
  if (!global.__dbRead) {
    global.__dbRead = createDbClient({ connectionString: readUrl });
  }
  return global.__dbRead;
}
```

| State of `DATABASE_READ_URL` | `dbRead` returns |
| ---------------------------- | --------------------------------- |
| unset / empty                | the write client (`db === dbRead`) |
| valid `postgres://...`       | a distinct read-only client       |

This means CI, local dev, and any deploy without a replica continue to
work without code changes. All migrations are zero-risk to roll out.

## Telemetry

`@toolctrl/db/telemetry.server` exposes `attachPrismaTelemetry()` — a
`$extends({ query })` wrapper that records every operation with
`{ client: 'read' | 'write', model, operation, durationMs, ok }`. Wire it
in the app bootstrap by adapting the recorder to your metrics sink
(prom-client `Histogram` / `Counter` or an OpenTelemetry `Meter`). The
canonical counter shape is `prisma.queries.total{client,model,operation}`.

## Coolify / staging setup (operator-owned)

Replica provisioning is **out of scope** for the application repo — these
steps live with the platform operator.

1. **Primary Postgres** (Coolify service `postgres`). `postgresql.conf`:

   ```
   wal_level = replica
   max_wal_senders = 3
   wal_keep_size = 256MB
   ```

   Then:

   ```sql
   CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD '...';
   CREATE USER read_user WITH ENCRYPTED PASSWORD '...';
   GRANT CONNECT ON DATABASE toolctrl TO read_user;
   GRANT USAGE ON SCHEMA public TO read_user;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_user;
   ALTER DEFAULT PRIVILEGES IN SCHEMA public
     GRANT SELECT ON TABLES TO read_user;
   ```

2. **Replica Postgres** (Coolify service `postgres-replica`,
   `postgres:16-alpine`):

   ```bash
   # Init from primary
   until pg_basebackup -h postgres -D /var/lib/postgresql/data -U replicator -P -R; do
     sleep 5
   done
   ```

   `postgresql.replica.conf`:

   ```
   hot_standby = on
   primary_conninfo = 'host=postgres port=5432 user=replicator password=...'
   ```

3. **App env**:

   ```sh
   DATABASE_URL=postgres://app_user:...@pgbouncer:6432/toolctrl?pgbouncer=true&connection_limit=10
   DATABASE_READ_URL=postgres://read_user:...@postgres-replica:5432/toolctrl?connection_limit=10
   ```

   Optionally front the replica with its own PgBouncer for connection pooling.

4. **Verification:**

   ```sql
   -- On primary
   SELECT * FROM pg_stat_replication;

   -- On replica
   SELECT pg_last_xact_replay_timestamp(), now() - pg_last_xact_replay_timestamp() AS lag;
   ```

   Grafana panel `prisma.queries.total{client="read"}` should rise on
   dashboard / KB / portal traffic.

## Rollback

- **Per-call-site:** `git revert <migration-commit>` flips that module
  back to `db` (primary). No data change.
- **Global:** unset `DATABASE_READ_URL` and restart the app. `dbRead === db`,
  all traffic hits the primary.
- **Replica down:** Prisma's read client will throw connection errors.
  Either unset `DATABASE_READ_URL` (degraded performance, no errors) or
  add a circuit-breaker (future work, out of scope here).

## Migration recipe (for future modules)

1. **Find candidate read-only functions** — pure `findMany`, `findUnique`,
   `findFirst`, `count`, `groupBy`, `aggregate` with no surrounding write.
2. **Check the caller graph** — is this function called immediately
   after a write that updates the same rows? If yes, leave on `db`.
3. **Replace `db.<model>.<op>` with `dbRead.<model>.<op>`** and add a
   one-line comment explaining the choice.
4. **Add `dbRead` to the file's imports** alongside `db` (keep `db` for
   the mutations still in the file).
5. **Run** `pnpm lint --max-warnings=0` and `pnpm typecheck`.
6. **Tests** stay green because of the fallback semantics — single-DB
   CI doesn't see a difference.
7. **Commit per-module** (`03-scaling/rw-split: migrate <module> ...`)
   so a single revert can roll back one module.

## What was migrated in #41

- `apps/web/app/modules/crm/services/CrmService.ts` — user enumeration
  for sync display.
- `apps/web/app/modules/knowledgeBase/db/*.db.server.ts` — public KB
  reads, listings, search, featured-article queries, dashboard counts.
- `apps/web/app/modules/analytics/db/analyticsUniqueVisitors.db.server.ts`
  — `groupBy` aggregations.
- `apps/web/app/utils/db/tenants.db.server.ts` — admin tenant enumerations
  (post-auth).
- `apps/web/app/modules/sites/db/sites.db.server.ts` — read-only site
  loaders (auth-gating `getSiteByTypeAndSiteId` left on primary).
- `apps/web/app/modules/portals/db/portals.db.server.ts` and
  `portalPages.db.server.ts` — listings, page lookups, dashboard counts.

`appData.ts` was **not** migrated wholesale; only the secondary tenant
listings it consumes were moved. The auth-resolution path stays on
primary by design.