HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/pgbouncer.md

PgBouncer Operations Runbook

PgBouncer multiplexes Prisma's per-process connection pools onto a small set of real Postgres backends. The dev compose stack and the staging Coolify stack both terminate application traffic at pgbouncer:6432; only the short-lived deployment migrator receives a separate DIRECT_URL path through postgres:5432.

Topology

+--------------+          +-------------+         +----------+
| app runtimes |  6432    | pgbouncer   |  5432   | postgres |
| app / worker +--------->+ transaction +-------->+  16-alp. |
| Prisma client|          | pool, 25/db |         |          |
+------+-------+          +-------------+         +-----^----+
                                                       |
 one-shot migrator -- DIRECT_URL (5432) ---------------+

• DATABASE_URL = postgres://...@pgbouncer:6432/<db>
  • PrismaPg uses unnamed statements by default, which is compatible with transaction-mode pooling.
  • DATABASE_POOL_MAX=10 caps the explicit node-postgres pool. A URL connection_limit parameter does not configure PrismaPg/node-postgres.
• APP_DATABASE_URL, PORTAL_DATABASE_URL, WORKER_DATABASE_URL, and the callback-only AUTHENTIK_ADMIN_JIT_DATABASE_URL contain distinct RLS-enforced roles and point through PgBouncer. Coolify exposes the JIT URL only to the core-web resource; the portal and other services receive only their normal DATABASE_URL.
• DIRECT_URL = postgres://...@postgres:5432/<db> — supplied only to start.sh migrate in a short-lived deployment step. It must never exist on web, API, MCP, Hocuspocus, or worker resources.

Local development

# Boot Postgres + PgBouncer side by side. The compose service is intentionally
# Docker-network-only — no host port mapping for 6432.
docker compose up postgres pgbouncer

# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid" \
DATABASE_POOL_MAX=10 \
  pnpm dev

If you do not have a Postgres service in docker-compose.yml yet, point both URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely). prisma.config.ts falls back to DATABASE_URL when DIRECT_URL is unset.

Coolify (staging) deploy procedure

The pooler must run on the same Docker network as Postgres. The current operator-owned deployment source is the Coolify service/network config; do not copy historical staging inventory into a live cutover.

1. Provision the reviewed roles and generate userlist.txt. Run the idempotent role provisioner from docs/database.md first. Postgres uses password_encryption=scram-sha-256, so the role-specific SCRAM hashes then live in pg_authid.rolpassword. From the Postgres container:

   psql -U postgres -tAc \
     "SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
        FROM pg_authid
       WHERE rolname IN (
         'hivegrid_app','hivegrid_portal','hivegrid_worker','hivegrid_authentik_jit','pgb_admin'
       );" > userlist.txt
   chmod 600 userlist.txt
   

   The query must return all five rows before rollout. hivegrid_app, hivegrid_portal, and hivegrid_worker handle their isolated runtime traffic; hivegrid_authentik_jit is the single-function callback capability and pgb_admin is reserved for PgBouncer administration and health checks. Drop the file into a Coolify volume mount that the pgbouncer container will see at /etc/pgbouncer/userlist.txt. Never commit a real hash.

2. Create the Coolify service.

   • Image: edoburu/pgbouncer:latest@sha256:… (reviewed manifest digest)
   • Network: coolify (must match Postgres)
   • Mounts:
     • infra/pgbouncer/pgbouncer.ini → /etc/pgbouncer/pgbouncer.ini:ro
     • userlist.txt (from step 1) → /etc/pgbouncer/userlist.txt:ro
   • No public port mapping — Traefik should not see 6432.
   • Healthcheck: pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin

3. Wait for the pooler to be healthy. Verify from inside the network:

   docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
   

4. Switch runtime env vars. In Coolify:

   • core web/API/MCP/Hocuspocus: DATABASE_URL uses the hivegrid_app role.
   • customer portal: DATABASE_URL uses the hivegrid_portal role.
   • every worker group: DATABASE_URL uses the hivegrid_worker role.
   • only core web: AUTHENTIK_ADMIN_JIT_DATABASE_URL uses the hivegrid_authentik_jit role. Remove every Admin-SSO key from API, MCP, Hocuspocus and worker resources, including inherited groups.
   • every runtime sets RLS_RUNTIME_MODE=required and the matching DATABASE_RUNTIME_ROLE.
   • DATABASE_POOL_MAX ist auf jeder Runtime explizit gesetzt. Für den ersten Produktionsrollout gilt je Prozess/Instanz: Core-Web 10, Customer-Portal 5; API/MCP/Hocuspocus und Worker folgen ihrem separat freigegebenen Kapazitätsplan.
   • DATABASE_POOL_IDLE_TIMEOUT_MS=10000
   • DATABASE_POOL_CONNECTION_TIMEOUT_MS=5000
   • remove DIRECT_URL, MIGRATOR_*, MIGRATE_ON_BOOT, SEED_ON_BOOT, and all SEED_* keys from every runtime resource, including inherited env groups.

   Use service and database names from the current Coolify project state; do not copy archived staging inventory into a live cutover.

5. Restart apps last. Apps must boot only after the pooler reports healthy; otherwise the first Prisma query fails and the container loops.

Migration runbook (production cutover)

1. Provision PgBouncer in Coolify but leave app env vars unchanged.

2. Verify SHOW STATS and SHOW POOLS respond.

3. Smoke-test from a one-off container:

   docker run --rm --network coolify \
     postgres:16-alpine psql \
     "postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
     -c "SELECT 1;"
   

4. Configure the migrator credential as a CI secret reachable from the runner; do not copy it to Coolify applications.

5. Run the immutable web image once with DIRECT_URL and argument migrate. A missing/unreachable credential or failed migration stops the rollout.

6. Roll out the role-specific DATABASE_URL and explicit DATABASE_POOL_* limits. Coolify will recreate containers — they must reconnect to the pooler.

7. Watch pgbouncer logs for WARNING|ERROR|FATAL.

8. Confirm prisma migrate status from the same step-scoped CI credential.

Tuning knobs

Knob	Default	When to change
pool_mode	transaction	Stay on transaction for web traffic. Long-lived advisory locks or LISTEN/NOTIFY need session mode on a separate logical database.
default_pool_size	25	Rule of thumb: postgres.max_connections / number_of_app_pools / 1.5. Hetzner Postgres is provisioned with max_connections=200, so 25 per logical DB × app stack still leaves headroom.
max_client_conn	1000	Raises the upper bound of client-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn).
reserve_pool_size	5	Burst buffer when the main pool is exhausted. Increase only if SHOW POOLS shows recurring cl_waiting.
server_idle_timeout	600	Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects.
query_timeout	0	Leave at 0. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug.
server_reset_query	DISCARD ALL	Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken.

Grafana dashboard lives under infra/grafana/dashboards/ (provisioned by Phase 02 #35). Panels of interest: pool wait time p95, server connection count vs. default_pool_size, total clients vs. max_client_conn.

Troubleshooting

FATAL: prepared statement "s0" already exists

PgBouncer in transaction pool mode reused a server-side connection that still has a prepared statement cached. HiveGrid's PrismaPg adapter uses unnamed statements by default; a URL pgbouncer=true flag is ignored. Check for a custom statementNameGenerator, an older non-adapter Prisma client, or another client sharing the same PgBouncer user.

error: too many clients already

Postgres ran out of connections. Either:

• default_pool_size × number-of-app-pools > max_connections — lower pool size or raise max_connections on Postgres.
• DATABASE_POOL_MAX is too high or unset for the deployment budget. Set it explicitly (10 by default, or lower) for each application process.

cl_waiting > 0 persistently in SHOW POOLS

Application sustained pool exhaustion. Options:

• raise default_pool_size (and double-check max_connections headroom),
• introduce a read replica (see 03-scaling-infrastructure/read-write-split.md),
• inspect slow queries — long-running transactions hold backends.

Migrations fail with "prepared statement" or DDL errors

The step-scoped migrator credential is wrong, missing, or cannot reach Postgres directly. Verify it only inside a one-shot container:

docker run --rm \
  -e DIRECT_URL=postgres://...@postgres:5432/db \
  <immutable-web-image> migrate

Healthcheck flapping

pg_isready returns non-zero whenever PgBouncer cannot reach Postgres. Check Postgres health first; PgBouncer surfaces an empty server pool as unhealthy.

Rollback

To bypass PgBouncer without code changes:

1. In Coolify, change DATABASE_URL to point at Postgres directly (port 5432; pool sizing remains in DATABASE_POOL_*).
2. Preserve the distinct hivegrid_app, hivegrid_portal, hivegrid_worker, and callback-only hivegrid_authentik_jit credentials and restart core web, customer portal, API, MCP, Hocuspocus, and workers. The JIT URL remains core-web-only.
3. Stop the pgbouncer Coolify service.

The migrator secret remains step-scoped and unchanged; never add it to the runtime resources during rollback.