Files
HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/pgbouncer.md
T

217 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PgBouncer Operations Runbook
PgBouncer multiplexes Prisma's per-process connection pools onto a small set
of real Postgres backends. The dev compose stack and the staging Coolify
stack both terminate application traffic at `pgbouncer:6432`; only the
short-lived deployment migrator receives a separate `DIRECT_URL` path through
`postgres:5432`.
## Topology
```text
+--------------+ +-------------+ +----------+
| app runtimes | 6432 | pgbouncer | 5432 | postgres |
| app / worker +--------->+ transaction +-------->+ 16-alp. |
| Prisma client| | pool, 25/db | | |
+------+-------+ +-------------+ +-----^----+
|
one-shot migrator -- DIRECT_URL (5432) ---------------+
```
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>`
- PrismaPg uses unnamed statements by default, which is compatible with
transaction-mode pooling.
- `DATABASE_POOL_MAX=10` caps the explicit node-postgres pool. A URL
`connection_limit` parameter does not configure PrismaPg/node-postgres.
- `APP_DATABASE_URL`, `PORTAL_DATABASE_URL`, `WORKER_DATABASE_URL`, and the callback-only
`AUTHENTIK_ADMIN_JIT_DATABASE_URL` contain distinct RLS-enforced roles and
point through PgBouncer. Coolify exposes the JIT URL only to the core-web
resource; the portal and other services receive only their normal `DATABASE_URL`.
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — supplied only to
`start.sh migrate` in a short-lived deployment step. It must never exist on
web, API, MCP, Hocuspocus, or worker resources.
## Local development
```bash
# Boot Postgres + PgBouncer side by side. The compose service is intentionally
# Docker-network-only — no host port mapping for 6432.
docker compose up postgres pgbouncer
# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid" \
DATABASE_POOL_MAX=10 \
pnpm dev
```
If you do not have a Postgres service in `docker-compose.yml` yet, point both
URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely).
`prisma.config.ts` falls back to `DATABASE_URL` when `DIRECT_URL` is unset.
## Coolify (staging) deploy procedure
The pooler must run on the same Docker network as Postgres. The current
operator-owned deployment source is the Coolify service/network config; do not
copy historical staging inventory into a live cutover.
1. **Provision the reviewed roles and generate `userlist.txt`.** Run the
idempotent role provisioner from [`docs/database.md`](../database.md) first.
Postgres uses `password_encryption=scram-sha-256`, so the role-specific
SCRAM hashes then live in `pg_authid.rolpassword`. From the Postgres
container:
```bash
psql -U postgres -tAc \
"SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
FROM pg_authid
WHERE rolname IN (
'hivegrid_app','hivegrid_portal','hivegrid_worker','hivegrid_authentik_jit','pgb_admin'
);" > userlist.txt
chmod 600 userlist.txt
```
The query must return all five rows before rollout. `hivegrid_app`,
`hivegrid_portal`, and `hivegrid_worker` handle their isolated runtime traffic;
`hivegrid_authentik_jit` is the single-function callback capability and
`pgb_admin` is reserved for PgBouncer administration and health checks. Drop the file
into a Coolify volume mount that the pgbouncer container will see at
`/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
2. **Create the Coolify service.**
- Image: `edoburu/pgbouncer:latest@sha256:…` (reviewed manifest digest)
- Network: `coolify` (must match Postgres)
- Mounts:
- `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro`
- `userlist.txt` (from step 1) → `/etc/pgbouncer/userlist.txt:ro`
- No public port mapping — Traefik should not see 6432.
- Healthcheck:
`pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin`
3. **Wait for the pooler to be `healthy`.** Verify from inside the network:
```bash
docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
```
4. **Switch runtime env vars.** In Coolify:
- core web/API/MCP/Hocuspocus: `DATABASE_URL` uses the `hivegrid_app` role.
- customer portal: `DATABASE_URL` uses the `hivegrid_portal` role.
- every worker group: `DATABASE_URL` uses the `hivegrid_worker` role.
- only core web: `AUTHENTIK_ADMIN_JIT_DATABASE_URL` uses the
`hivegrid_authentik_jit` role. Remove every Admin-SSO key from API, MCP,
Hocuspocus and worker resources, including inherited groups.
- every runtime sets `RLS_RUNTIME_MODE=required` and the matching
`DATABASE_RUNTIME_ROLE`.
- `DATABASE_POOL_MAX` ist auf jeder Runtime explizit gesetzt. Für den ersten
Produktionsrollout gilt je Prozess/Instanz: Core-Web `10`, Customer-Portal
`5`; API/MCP/Hocuspocus und Worker folgen ihrem separat freigegebenen
Kapazitätsplan.
- `DATABASE_POOL_IDLE_TIMEOUT_MS=10000`
- `DATABASE_POOL_CONNECTION_TIMEOUT_MS=5000`
- remove `DIRECT_URL`, `MIGRATOR_*`, `MIGRATE_ON_BOOT`, `SEED_ON_BOOT`, and
all `SEED_*` keys from every runtime resource, including inherited env groups.
Use service and database names from the current Coolify project state; do
not copy archived staging inventory into a live cutover.
5. **Restart apps last.** Apps must boot only _after_ the pooler reports
healthy; otherwise the first Prisma query fails and the container loops.
## Migration runbook (production cutover)
1. Provision PgBouncer in Coolify but leave app env vars unchanged.
2. Verify `SHOW STATS` and `SHOW POOLS` respond.
3. Smoke-test from a one-off container:
```bash
docker run --rm --network coolify \
postgres:16-alpine psql \
"postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
-c "SELECT 1;"
```
4. Configure the migrator credential as a CI secret reachable from the runner;
do not copy it to Coolify applications.
5. Run the immutable web image once with `DIRECT_URL` and argument `migrate`.
A missing/unreachable credential or failed migration stops the rollout.
6. Roll out the role-specific `DATABASE_URL` and explicit `DATABASE_POOL_*` limits.
Coolify will recreate containers — they must reconnect to the pooler.
7. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
8. Confirm `prisma migrate status` from the same step-scoped CI credential.
## Tuning knobs
| Knob | Default | When to change |
| --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `pool_mode` | `transaction` | Stay on `transaction` for web traffic. Long-lived advisory locks or `LISTEN/NOTIFY` need `session` mode on a separate logical database. |
| `default_pool_size` | `25` | Rule of thumb: `postgres.max_connections / number_of_app_pools / 1.5`. Hetzner Postgres is provisioned with `max_connections=200`, so 25 per logical DB × app stack still leaves headroom. |
| `max_client_conn` | `1000` | Raises the upper bound of _client_-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn). |
| `reserve_pool_size` | `5` | Burst buffer when the main pool is exhausted. Increase only if `SHOW POOLS` shows recurring `cl_waiting`. |
| `server_idle_timeout` | `600` | Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects. |
| `query_timeout` | `0` | Leave at `0`. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug. |
| `server_reset_query` | `DISCARD ALL` | Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken. |
Grafana dashboard lives under
`infra/grafana/dashboards/` (provisioned by Phase 02 `#35`).
Panels of interest: pool wait time p95, server connection count vs.
`default_pool_size`, total clients vs. `max_client_conn`.
## Troubleshooting
### `FATAL: prepared statement "s0" already exists`
PgBouncer in transaction pool mode reused a server-side connection that
still has a prepared statement cached. HiveGrid's PrismaPg adapter uses
unnamed statements by default; a URL `pgbouncer=true` flag is ignored. Check
for a custom `statementNameGenerator`, an older non-adapter Prisma client, or
another client sharing the same PgBouncer user.
### `error: too many clients already`
Postgres ran out of connections. Either:
- `default_pool_size` × number-of-app-pools > `max_connections` — lower
pool size or raise `max_connections` on Postgres.
- `DATABASE_POOL_MAX` is too high or unset for the deployment budget. Set it
explicitly (10 by default, or lower) for each application process.
### `cl_waiting > 0` persistently in `SHOW POOLS`
Application sustained pool exhaustion. Options:
- raise `default_pool_size` (and double-check `max_connections` headroom),
- introduce a read replica (see `03-scaling-infrastructure/read-write-split.md`),
- inspect slow queries — long-running transactions hold backends.
### Migrations fail with "prepared statement" or DDL errors
The step-scoped migrator credential is wrong, missing, or cannot reach
Postgres directly. Verify it only inside a one-shot container:
```bash
docker run --rm \
-e DIRECT_URL=postgres://...@postgres:5432/db \
<immutable-web-image> migrate
```
### Healthcheck flapping
`pg_isready` returns non-zero whenever PgBouncer cannot reach Postgres.
Check Postgres health first; PgBouncer surfaces an empty server pool as
unhealthy.
## Rollback
To bypass PgBouncer without code changes:
1. In Coolify, change `DATABASE_URL` to point at Postgres directly
(port 5432; pool sizing remains in `DATABASE_POOL_*`).
2. Preserve the distinct `hivegrid_app`, `hivegrid_portal`, `hivegrid_worker`, and callback-only
`hivegrid_authentik_jit` credentials and restart core web, customer portal,
API, MCP, Hocuspocus, and workers. The JIT URL remains core-web-only.
3. Stop the `pgbouncer` Coolify service.
The migrator secret remains step-scoped and unchanged; never add it to the
runtime resources during rollback.