# PgBouncer Operations Runbook PgBouncer multiplexes Prisma's per-process connection pools onto a small set of real Postgres backends. The dev compose stack and the staging Coolify stack both terminate application traffic at `pgbouncer:6432`; only the short-lived deployment migrator receives a separate `DIRECT_URL` path through `postgres:5432`. ## Topology ```text +--------------+ +-------------+ +----------+ | app runtimes | 6432 | pgbouncer | 5432 | postgres | | app / worker +--------->+ transaction +-------->+ 16-alp. | | Prisma client| | pool, 25/db | | | +------+-------+ +-------------+ +-----^----+ | one-shot migrator -- DIRECT_URL (5432) ---------------+ ``` - `DATABASE_URL = postgres://...@pgbouncer:6432/` - PrismaPg uses unnamed statements by default, which is compatible with transaction-mode pooling. - `DATABASE_POOL_MAX=10` caps the explicit node-postgres pool. A URL `connection_limit` parameter does not configure PrismaPg/node-postgres. - `APP_DATABASE_URL`, `PORTAL_DATABASE_URL`, `WORKER_DATABASE_URL`, and the callback-only `AUTHENTIK_ADMIN_JIT_DATABASE_URL` contain distinct RLS-enforced roles and point through PgBouncer. Coolify exposes the JIT URL only to the core-web resource; the portal and other services receive only their normal `DATABASE_URL`. - `DIRECT_URL = postgres://...@postgres:5432/` — supplied only to `start.sh migrate` in a short-lived deployment step. It must never exist on web, API, MCP, Hocuspocus, or worker resources. ## Local development ```bash # Boot Postgres + PgBouncer side by side. The compose service is intentionally # Docker-network-only — no host port mapping for 6432. docker compose up postgres pgbouncer # In another shell: DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \ DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid" \ DATABASE_POOL_MAX=10 \ pnpm dev ``` If you do not have a Postgres service in `docker-compose.yml` yet, point both URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely). `prisma.config.ts` falls back to `DATABASE_URL` when `DIRECT_URL` is unset. ## Coolify (staging) deploy procedure The pooler must run on the same Docker network as Postgres. The current operator-owned deployment source is the Coolify service/network config; do not copy historical staging inventory into a live cutover. 1. **Provision the reviewed roles and generate `userlist.txt`.** Run the idempotent role provisioner from [`docs/database.md`](../database.md) first. Postgres uses `password_encryption=scram-sha-256`, so the role-specific SCRAM hashes then live in `pg_authid.rolpassword`. From the Postgres container: ```bash psql -U postgres -tAc \ "SELECT '\"'||rolname||'\" \"'||rolpassword||'\"' FROM pg_authid WHERE rolname IN ( 'hivegrid_app','hivegrid_portal','hivegrid_worker','hivegrid_authentik_jit','pgb_admin' );" > userlist.txt chmod 600 userlist.txt ``` The query must return all five rows before rollout. `hivegrid_app`, `hivegrid_portal`, and `hivegrid_worker` handle their isolated runtime traffic; `hivegrid_authentik_jit` is the single-function callback capability and `pgb_admin` is reserved for PgBouncer administration and health checks. Drop the file into a Coolify volume mount that the pgbouncer container will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.** 2. **Create the Coolify service.** - Image: `edoburu/pgbouncer:latest@sha256:…` (reviewed manifest digest) - Network: `coolify` (must match Postgres) - Mounts: - `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro` - `userlist.txt` (from step 1) → `/etc/pgbouncer/userlist.txt:ro` - No public port mapping — Traefik should not see 6432. - Healthcheck: `pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin` 3. **Wait for the pooler to be `healthy`.** Verify from inside the network: ```bash docker exec psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;" ``` 4. **Switch runtime env vars.** In Coolify: - core web/API/MCP/Hocuspocus: `DATABASE_URL` uses the `hivegrid_app` role. - customer portal: `DATABASE_URL` uses the `hivegrid_portal` role. - every worker group: `DATABASE_URL` uses the `hivegrid_worker` role. - only core web: `AUTHENTIK_ADMIN_JIT_DATABASE_URL` uses the `hivegrid_authentik_jit` role. Remove every Admin-SSO key from API, MCP, Hocuspocus and worker resources, including inherited groups. - every runtime sets `RLS_RUNTIME_MODE=required` and the matching `DATABASE_RUNTIME_ROLE`. - `DATABASE_POOL_MAX` ist auf jeder Runtime explizit gesetzt. Für den ersten Produktionsrollout gilt je Prozess/Instanz: Core-Web `10`, Customer-Portal `5`; API/MCP/Hocuspocus und Worker folgen ihrem separat freigegebenen Kapazitätsplan. - `DATABASE_POOL_IDLE_TIMEOUT_MS=10000` - `DATABASE_POOL_CONNECTION_TIMEOUT_MS=5000` - remove `DIRECT_URL`, `MIGRATOR_*`, `MIGRATE_ON_BOOT`, `SEED_ON_BOOT`, and all `SEED_*` keys from every runtime resource, including inherited env groups. Use service and database names from the current Coolify project state; do not copy archived staging inventory into a live cutover. 5. **Restart apps last.** Apps must boot only _after_ the pooler reports healthy; otherwise the first Prisma query fails and the container loops. ## Migration runbook (production cutover) 1. Provision PgBouncer in Coolify but leave app env vars unchanged. 2. Verify `SHOW STATS` and `SHOW POOLS` respond. 3. Smoke-test from a one-off container: ```bash docker run --rm --network coolify \ postgres:16-alpine psql \ "postgres://:${PG_PASS}@:6432/" \ -c "SELECT 1;" ``` 4. Configure the migrator credential as a CI secret reachable from the runner; do not copy it to Coolify applications. 5. Run the immutable web image once with `DIRECT_URL` and argument `migrate`. A missing/unreachable credential or failed migration stops the rollout. 6. Roll out the role-specific `DATABASE_URL` and explicit `DATABASE_POOL_*` limits. Coolify will recreate containers — they must reconnect to the pooler. 7. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`. 8. Confirm `prisma migrate status` from the same step-scoped CI credential. ## Tuning knobs | Knob | Default | When to change | | --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pool_mode` | `transaction` | Stay on `transaction` for web traffic. Long-lived advisory locks or `LISTEN/NOTIFY` need `session` mode on a separate logical database. | | `default_pool_size` | `25` | Rule of thumb: `postgres.max_connections / number_of_app_pools / 1.5`. Hetzner Postgres is provisioned with `max_connections=200`, so 25 per logical DB × app stack still leaves headroom. | | `max_client_conn` | `1000` | Raises the upper bound of _client_-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn). | | `reserve_pool_size` | `5` | Burst buffer when the main pool is exhausted. Increase only if `SHOW POOLS` shows recurring `cl_waiting`. | | `server_idle_timeout` | `600` | Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects. | | `query_timeout` | `0` | Leave at `0`. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug. | | `server_reset_query` | `DISCARD ALL` | Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken. | Grafana dashboard lives under `infra/grafana/dashboards/` (provisioned by Phase 02 `#35`). Panels of interest: pool wait time p95, server connection count vs. `default_pool_size`, total clients vs. `max_client_conn`. ## Troubleshooting ### `FATAL: prepared statement "s0" already exists` PgBouncer in transaction pool mode reused a server-side connection that still has a prepared statement cached. HiveGrid's PrismaPg adapter uses unnamed statements by default; a URL `pgbouncer=true` flag is ignored. Check for a custom `statementNameGenerator`, an older non-adapter Prisma client, or another client sharing the same PgBouncer user. ### `error: too many clients already` Postgres ran out of connections. Either: - `default_pool_size` × number-of-app-pools > `max_connections` — lower pool size or raise `max_connections` on Postgres. - `DATABASE_POOL_MAX` is too high or unset for the deployment budget. Set it explicitly (10 by default, or lower) for each application process. ### `cl_waiting > 0` persistently in `SHOW POOLS` Application sustained pool exhaustion. Options: - raise `default_pool_size` (and double-check `max_connections` headroom), - introduce a read replica (see `03-scaling-infrastructure/read-write-split.md`), - inspect slow queries — long-running transactions hold backends. ### Migrations fail with "prepared statement" or DDL errors The step-scoped migrator credential is wrong, missing, or cannot reach Postgres directly. Verify it only inside a one-shot container: ```bash docker run --rm \ -e DIRECT_URL=postgres://...@postgres:5432/db \ migrate ``` ### Healthcheck flapping `pg_isready` returns non-zero whenever PgBouncer cannot reach Postgres. Check Postgres health first; PgBouncer surfaces an empty server pool as unhealthy. ## Rollback To bypass PgBouncer without code changes: 1. In Coolify, change `DATABASE_URL` to point at Postgres directly (port 5432; pool sizing remains in `DATABASE_POOL_*`). 2. Preserve the distinct `hivegrid_app`, `hivegrid_portal`, `hivegrid_worker`, and callback-only `hivegrid_authentik_jit` credentials and restart core web, customer portal, API, MCP, Hocuspocus, and workers. The JIT URL remains core-web-only. 3. Stop the `pgbouncer` Coolify service. The migrator secret remains step-scoped and unchanged; never add it to the runtime resources during rollback.