# PgBouncer Operations Runbook PgBouncer multiplexes Prisma's per-process connection pools onto a small set of real Postgres backends. The dev compose stack and the staging Coolify stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate keeps a separate path through `postgres:5432` (`DIRECT_URL`). Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](../improvement-plan/03-scaling-infrastructure/pgbouncer.md). ## Topology ``` +--------------+ +-------------+ +----------+ | apps (web, | 6432 | pgbouncer | 5432 | postgres | | api, worker) +--------->+ transaction +-------->+ 16-alp. | | Prisma client| | pool, 25/db | | | +------+-------+ +-------------+ +-----^----+ | | | DIRECT_URL (5432) — only for prisma migrate ---+ v prisma migrate / db push / db pull ``` - `DATABASE_URL = postgres://...@pgbouncer:6432/?pgbouncer=true&connection_limit=10` - `pgbouncer=true` tells Prisma to skip prepared-statement caching. - `connection_limit=10` caps Prisma's internal pool — PgBouncer pools again on top, doubling the budget without this would defeat the purpose. - `DIRECT_URL = postgres://...@postgres:5432/` — read by `prisma.config.ts` for `migrate`, `db push`, `db pull`, `db seed`. ## Local development ```bash # Boot Postgres + PgBouncer side by side. The compose service is intentionally # Docker-network-only — no host port mapping for 6432. docker compose up postgres pgbouncer # In another shell: DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \ DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \ pnpm dev ``` If you do not have a Postgres service in `docker-compose.yml` yet, point both URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely). `prisma.config.ts` falls back to `DATABASE_URL` when `DIRECT_URL` is unset. ## Coolify (staging) deploy procedure The pooler must run on the same Docker network as Postgres. The current operator-owned deployment source is the Coolify service/network config; the archived [`STAGING-ACCESS.md`](../improvement-plan/archive/STAGING-ACCESS.md) is historical inventory only and must not override live Coolify state. 1. **Generate `userlist.txt`.** Postgres on staging is initialised with `password_encryption=scram-sha-256`, so the SCRAM hash already lives in `pg_authid.rolpassword`. From the Postgres container: ```bash psql -U postgres -tAc \ "SELECT '\"'||rolname||'\" \"'||rolpassword||'\"' FROM pg_authid WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt chmod 600 userlist.txt ``` Drop the file into a Coolify volume mount that the pgbouncer container will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.** 2. **Create the Coolify service.** - Image: `edoburu/pgbouncer:latest` - Network: `coolify` (must match Postgres) - Mounts: - `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro` - `userlist.txt` (from step 1) → `/etc/pgbouncer/userlist.txt:ro` - No public port mapping — Traefik should not see 6432. - Healthcheck: `pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin` 3. **Wait for the pooler to be `healthy`.** Verify from inside the network: ```bash docker exec psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;" ``` 4. **Switch app env vars.** In Coolify, on `web`, `api`, `worker`: - `DATABASE_URL=postgres://:${PG_PASS}@:6432/?pgbouncer=true&connection_limit=10` - `DIRECT_URL=postgres://:${PG_PASS}@:5432/` Use service and database names from the current Coolify project state; do not copy archived staging inventory into a live cutover. 5. **Restart apps last.** Apps must boot only _after_ the pooler reports healthy; otherwise the first Prisma query fails and the container loops. ## Migration runbook (production cutover) 1. Provision PgBouncer in Coolify but leave app env vars unchanged. 2. Verify `SHOW STATS` and `SHOW POOLS` respond. 3. Smoke-test from a one-off container: ```bash docker run --rm --network coolify \ postgres:16-alpine psql \ "postgres://:${PG_PASS}@:6432/" \ -c "SELECT 1;" ``` 4. Roll out `DIRECT_URL` to all apps first (no behaviour change because `prisma.config.ts` falls back to `DATABASE_URL`). 5. Roll out the new `DATABASE_URL` (with `pgbouncer=true&connection_limit=10`). Coolify will recreate containers — they must reconnect to the pooler. 6. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`. 7. Confirm `prisma migrate status` from a CI container still passes (uses `DIRECT_URL`). ## Tuning knobs | Knob | Default | When to change | | --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pool_mode` | `transaction` | Stay on `transaction` for web traffic. Long-lived advisory locks or `LISTEN/NOTIFY` need `session` mode on a separate logical database. | | `default_pool_size` | `25` | Rule of thumb: `postgres.max_connections / number_of_app_pools / 1.5`. Hetzner Postgres is provisioned with `max_connections=200`, so 25 per logical DB × app stack still leaves headroom. | | `max_client_conn` | `1000` | Raises the upper bound of _client_-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn). | | `reserve_pool_size` | `5` | Burst buffer when the main pool is exhausted. Increase only if `SHOW POOLS` shows recurring `cl_waiting`. | | `server_idle_timeout` | `600` | Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects. | | `query_timeout` | `0` | Leave at `0`. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug. | | `server_reset_query` | `DISCARD ALL` | Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken. | Grafana dashboard lives under `infra/grafana/dashboards/` (provisioned by Phase 02 `#35`). Panels of interest: pool wait time p95, server connection count vs. `default_pool_size`, total clients vs. `max_client_conn`. ## Troubleshooting ### `FATAL: prepared statement "s0" already exists` PgBouncer in transaction pool mode reused a server-side connection that still has the prepared statement cached. Confirm `DATABASE_URL` has `?pgbouncer=true` and Prisma 5+ is in use. Without the flag, Prisma keeps local prepared statements and the next transaction lands on a different backend. ### `error: too many clients already` Postgres ran out of connections. Either: - `default_pool_size` × number-of-app-pools > `max_connections` — lower pool size or raise `max_connections` on Postgres. - Prisma's `connection_limit` is unset, so each app process opens its own default pool (CPU + 1) against PgBouncer. Set `connection_limit=10` (or lower) in the query string of `DATABASE_URL`. ### `cl_waiting > 0` persistently in `SHOW POOLS` Application sustained pool exhaustion. Options: - raise `default_pool_size` (and double-check `max_connections` headroom), - introduce a read replica (see `03-scaling-infrastructure/read-write-split.md`), - inspect slow queries — long-running transactions hold backends. ### Migrations fail with "prepared statement" or DDL errors `DIRECT_URL` is wrong or missing. `prisma migrate` must hit Postgres directly. Verify `prisma.config.ts` reads `DIRECT_URL`: ```bash DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \ DIRECT_URL=postgres://...@postgres:5432/db \ pnpm exec prisma migrate status ``` ### Healthcheck flapping `pg_isready` returns non-zero whenever PgBouncer cannot reach Postgres. Check Postgres health first; PgBouncer surfaces an empty server pool as unhealthy. ## Rollback To bypass PgBouncer without code changes: 1. In Coolify, change `DATABASE_URL` to point at Postgres directly (port 5432, drop `?pgbouncer=true&connection_limit=10`). 2. Restart `web`, `api`, `worker`. 3. Stop the `pgbouncer` Coolify service. `DIRECT_URL` stays valid — it never depended on the pooler.