HiveGrid-Wireframes/hivegrid-cloud/docs/ops/pgbouncer.md

PgBouncer Operations Runbook

PgBouncer multiplexes Prisma's per-process connection pools onto a small set of real Postgres backends. The dev compose stack and the staging Coolify stack both terminate application traffic at pgbouncer:6432; Prisma Migrate keeps a separate path through postgres:5432 (DIRECT_URL).

Related task: docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md.

Topology

+--------------+          +-------------+         +----------+
| apps (web,   |  6432    | pgbouncer   |  5432   | postgres |
| api, worker) +--------->+ transaction +-------->+  16-alp. |
| Prisma client|          | pool, 25/db |         |          |
+------+-------+          +-------------+         +-----^----+
       |                                                |
       | DIRECT_URL (5432) — only for prisma migrate ---+
       v
   prisma migrate / db push / db pull

• DATABASE_URL = postgres://...@pgbouncer:6432/<db>?pgbouncer=true&connection_limit=10
  • pgbouncer=true tells Prisma to skip prepared-statement caching.
  • connection_limit=10 caps Prisma's internal pool — PgBouncer pools again on top, doubling the budget without this would defeat the purpose.
• DIRECT_URL = postgres://...@postgres:5432/<db> — read by prisma.config.ts for migrate, db push, db pull, db seed.

Local development

# Boot Postgres + PgBouncer side by side. The compose service is intentionally
# Docker-network-only — no host port mapping for 6432.
docker compose up postgres pgbouncer

# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \
DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \
  pnpm dev

If you do not have a Postgres service in docker-compose.yml yet, point both URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely). prisma.config.ts falls back to DATABASE_URL when DIRECT_URL is unset.

Coolify (staging) deploy procedure

The pooler must run on the same Docker network as Postgres. The current operator-owned deployment source is the Coolify service/network config; the archived STAGING-ACCESS.md is historical inventory only and must not override live Coolify state.

1. Generate userlist.txt. Postgres on staging is initialised with password_encryption=scram-sha-256, so the SCRAM hash already lives in pg_authid.rolpassword. From the Postgres container:

   psql -U postgres -tAc \
     "SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
        FROM pg_authid
       WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt
   chmod 600 userlist.txt
   

   Drop the file into a Coolify volume mount that the pgbouncer container will see at /etc/pgbouncer/userlist.txt. Never commit a real hash.

2. Create the Coolify service.

   • Image: edoburu/pgbouncer:latest
   • Network: coolify (must match Postgres)
   • Mounts:
     • infra/pgbouncer/pgbouncer.ini → /etc/pgbouncer/pgbouncer.ini:ro
     • userlist.txt (from step 1) → /etc/pgbouncer/userlist.txt:ro
   • No public port mapping — Traefik should not see 6432.
   • Healthcheck: pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin

3. Wait for the pooler to be healthy. Verify from inside the network:

   docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
   

4. Switch app env vars. In Coolify, on web, api, worker:

   • DATABASE_URL=postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>?pgbouncer=true&connection_limit=10
   • DIRECT_URL=postgres://<database-user>:${PG_PASS}@<postgres-service>:5432/<database>

   Use service and database names from the current Coolify project state; do not copy archived staging inventory into a live cutover.

5. Restart apps last. Apps must boot only after the pooler reports healthy; otherwise the first Prisma query fails and the container loops.

Migration runbook (production cutover)

1. Provision PgBouncer in Coolify but leave app env vars unchanged.
2. Verify SHOW STATS and SHOW POOLS respond.
3. Smoke-test from a one-off container:

   docker run --rm --network coolify \
     postgres:16-alpine psql \
     "postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
     -c "SELECT 1;"
   

4. Roll out DIRECT_URL to all apps first (no behaviour change because prisma.config.ts falls back to DATABASE_URL).
5. Roll out the new DATABASE_URL (with pgbouncer=true&connection_limit=10). Coolify will recreate containers — they must reconnect to the pooler.
6. Watch pgbouncer logs for WARNING|ERROR|FATAL.
7. Confirm prisma migrate status from a CI container still passes (uses DIRECT_URL).

Tuning knobs

Knob	Default	When to change
pool_mode	transaction	Stay on transaction for web traffic. Long-lived advisory locks or LISTEN/NOTIFY need session mode on a separate logical database.
default_pool_size	25	Rule of thumb: postgres.max_connections / number_of_app_pools / 1.5. Hetzner Postgres is provisioned with max_connections=200, so 25 per logical DB × app stack still leaves headroom.
max_client_conn	1000	Raises the upper bound of client-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn).
reserve_pool_size	5	Burst buffer when the main pool is exhausted. Increase only if SHOW POOLS shows recurring cl_waiting.
server_idle_timeout	600	Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects.
query_timeout	0	Leave at 0. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug.
server_reset_query	DISCARD ALL	Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken.

Grafana dashboard lives under infra/grafana/dashboards/ (provisioned by Phase 02 #35). Panels of interest: pool wait time p95, server connection count vs. default_pool_size, total clients vs. max_client_conn.

Troubleshooting

FATAL: prepared statement "s0" already exists

PgBouncer in transaction pool mode reused a server-side connection that still has the prepared statement cached. Confirm DATABASE_URL has ?pgbouncer=true and Prisma 5+ is in use. Without the flag, Prisma keeps local prepared statements and the next transaction lands on a different backend.

error: too many clients already

Postgres ran out of connections. Either:

• default_pool_size × number-of-app-pools > max_connections — lower pool size or raise max_connections on Postgres.
• Prisma's connection_limit is unset, so each app process opens its own default pool (CPU + 1) against PgBouncer. Set connection_limit=10 (or lower) in the query string of DATABASE_URL.

cl_waiting > 0 persistently in SHOW POOLS

Application sustained pool exhaustion. Options:

• raise default_pool_size (and double-check max_connections headroom),
• introduce a read replica (see 03-scaling-infrastructure/read-write-split.md),
• inspect slow queries — long-running transactions hold backends.

Migrations fail with "prepared statement" or DDL errors

DIRECT_URL is wrong or missing. prisma migrate must hit Postgres directly. Verify prisma.config.ts reads DIRECT_URL:

DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \
DIRECT_URL=postgres://...@postgres:5432/db \
  pnpm exec prisma migrate status

Healthcheck flapping

pg_isready returns non-zero whenever PgBouncer cannot reach Postgres. Check Postgres health first; PgBouncer surfaces an empty server pool as unhealthy.

Rollback

To bypass PgBouncer without code changes:

1. In Coolify, change DATABASE_URL to point at Postgres directly (port 5432, drop ?pgbouncer=true&connection_limit=10).
2. Restart web, api, worker.
3. Stop the pgbouncer Coolify service.

DIRECT_URL stays valid — it never depended on the pooler.