Track HiveGrid Cloud planning reference

This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3

No files matched your search

+186
View File
@@ -0,0 +1,186 @@
# PgBouncer Operations Runbook
PgBouncer multiplexes Prisma's per-process connection pools onto a small set
of real Postgres backends. The dev compose stack and the staging Coolify
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](../improvement-plan/03-scaling-infrastructure/pgbouncer.md).
## Topology
```
+--------------+ +-------------+ +----------+
| apps (web, | 6432 | pgbouncer | 5432 | postgres |
| api, worker) +--------->+ transaction +-------->+ 16-alp. |
| Prisma client| | pool, 25/db | | |
+------+-------+ +-------------+ +-----^----+
| |
| DIRECT_URL (5432) — only for prisma migrate ---+
v
prisma migrate / db push / db pull
```
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>?pgbouncer=true&connection_limit=10`
- `pgbouncer=true` tells Prisma to skip prepared-statement caching.
- `connection_limit=10` caps Prisma's internal pool — PgBouncer pools again
on top, doubling the budget without this would defeat the purpose.
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — read by
`prisma.config.ts` for `migrate`, `db push`, `db pull`, `db seed`.
## Local development
```bash
# Boot Postgres + PgBouncer side by side. The compose service is intentionally
# Docker-network-only — no host port mapping for 6432.
docker compose up postgres pgbouncer
# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \
DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \
pnpm dev
```
If you do not have a Postgres service in `docker-compose.yml` yet, point both
URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely).
`prisma.config.ts` falls back to `DATABASE_URL` when `DIRECT_URL` is unset.
## Coolify (staging) deploy procedure
The pooler must run on the same Docker network as Postgres. The current
operator-owned deployment source is the Coolify service/network config; the
archived [`STAGING-ACCESS.md`](../improvement-plan/archive/STAGING-ACCESS.md)
is historical inventory only and must not override live Coolify state.
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
`pg_authid.rolpassword`. From the Postgres container:
```bash
psql -U postgres -tAc \
"SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
FROM pg_authid
WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt
chmod 600 userlist.txt
```
Drop the file into a Coolify volume mount that the pgbouncer container
will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
2. **Create the Coolify service.**
- Image: `edoburu/pgbouncer:latest`
- Network: `coolify` (must match Postgres)
- Mounts:
- `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro`
- `userlist.txt` (from step 1) → `/etc/pgbouncer/userlist.txt:ro`
- No public port mapping — Traefik should not see 6432.
- Healthcheck:
`pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin`
3. **Wait for the pooler to be `healthy`.** Verify from inside the network:
```bash
docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
```
4. **Switch app env vars.** In Coolify, on `web`, `api`, `worker`:
- `DATABASE_URL=postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>?pgbouncer=true&connection_limit=10`
- `DIRECT_URL=postgres://<database-user>:${PG_PASS}@<postgres-service>:5432/<database>`
Use service and database names from the current Coolify project state; do
not copy archived staging inventory into a live cutover.
5. **Restart apps last.** Apps must boot only _after_ the pooler reports
healthy; otherwise the first Prisma query fails and the container loops.
## Migration runbook (production cutover)
1. Provision PgBouncer in Coolify but leave app env vars unchanged.
2. Verify `SHOW STATS` and `SHOW POOLS` respond.
3. Smoke-test from a one-off container:
```bash
docker run --rm --network coolify \
postgres:16-alpine psql \
"postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
-c "SELECT 1;"
```
4. Roll out `DIRECT_URL` to all apps first (no behaviour change because
`prisma.config.ts` falls back to `DATABASE_URL`).
5. Roll out the new `DATABASE_URL` (with `pgbouncer=true&connection_limit=10`).
Coolify will recreate containers — they must reconnect to the pooler.
6. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
7. Confirm `prisma migrate status` from a CI container still passes (uses
`DIRECT_URL`).
## Tuning knobs
| Knob | Default | When to change |
| --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `pool_mode` | `transaction` | Stay on `transaction` for web traffic. Long-lived advisory locks or `LISTEN/NOTIFY` need `session` mode on a separate logical database. |
| `default_pool_size` | `25` | Rule of thumb: `postgres.max_connections / number_of_app_pools / 1.5`. Hetzner Postgres is provisioned with `max_connections=200`, so 25 per logical DB × app stack still leaves headroom. |
| `max_client_conn` | `1000` | Raises the upper bound of _client_-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn). |
| `reserve_pool_size` | `5` | Burst buffer when the main pool is exhausted. Increase only if `SHOW POOLS` shows recurring `cl_waiting`. |
| `server_idle_timeout` | `600` | Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects. |
| `query_timeout` | `0` | Leave at `0`. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug. |
| `server_reset_query` | `DISCARD ALL` | Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken. |
Grafana dashboard lives under
`infra/grafana/dashboards/` (provisioned by Phase 02 `#35`).
Panels of interest: pool wait time p95, server connection count vs.
`default_pool_size`, total clients vs. `max_client_conn`.
## Troubleshooting
### `FATAL: prepared statement "s0" already exists`
PgBouncer in transaction pool mode reused a server-side connection that
still has the prepared statement cached. Confirm `DATABASE_URL` has
`?pgbouncer=true` and Prisma 5+ is in use. Without the flag, Prisma keeps
local prepared statements and the next transaction lands on a different
backend.
### `error: too many clients already`
Postgres ran out of connections. Either:
- `default_pool_size` × number-of-app-pools > `max_connections` — lower
pool size or raise `max_connections` on Postgres.
- Prisma's `connection_limit` is unset, so each app process opens its own
default pool (CPU + 1) against PgBouncer. Set `connection_limit=10` (or
lower) in the query string of `DATABASE_URL`.
### `cl_waiting > 0` persistently in `SHOW POOLS`
Application sustained pool exhaustion. Options:
- raise `default_pool_size` (and double-check `max_connections` headroom),
- introduce a read replica (see `03-scaling-infrastructure/read-write-split.md`),
- inspect slow queries — long-running transactions hold backends.
### Migrations fail with "prepared statement" or DDL errors
`DIRECT_URL` is wrong or missing. `prisma migrate` must hit Postgres
directly. Verify `prisma.config.ts` reads `DIRECT_URL`:
```bash
DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \
DIRECT_URL=postgres://...@postgres:5432/db \
pnpm exec prisma migrate status
```
### Healthcheck flapping
`pg_isready` returns non-zero whenever PgBouncer cannot reach Postgres.
Check Postgres health first; PgBouncer surfaces an empty server pool as
unhealthy.
## Rollback
To bypass PgBouncer without code changes:
1. In Coolify, change `DATABASE_URL` to point at Postgres directly
(port 5432, drop `?pgbouncer=true&connection_limit=10`).
2. Restart `web`, `api`, `worker`.
3. Stop the `pgbouncer` Coolify service.
`DIRECT_URL` stays valid — it never depended on the pooler.