Refresh HiveGrid Cloud reference snapshot

This commit is contained in:
david committed 2026-08-03 15:31:14 +02:00
1 parent 5e99e14772
commit 7f88026f79
5955 files changed
+1177325 -188362

No files matched your search

@@ -2,29 +2,34 @@
PgBouncer multiplexes Prisma's per-process connection pools onto a small set
of real Postgres backends. The dev compose stack and the staging Coolify
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
stack both terminate application traffic at `pgbouncer:6432`; only the
short-lived deployment migrator receives a separate `DIRECT_URL` path through
`postgres:5432`.
## Topology
```
```text
+--------------+ +-------------+ +----------+
| apps (web, | 6432 | pgbouncer | 5432 | postgres |
| api, worker) +--------->+ transaction +-------->+ 16-alp. |
| app runtimes | 6432 | pgbouncer | 5432 | postgres |
| app / worker +--------->+ transaction +-------->+ 16-alp. |
| Prisma client| | pool, 25/db | | |
+------+-------+ +-------------+ +-----^----+
| |
| DIRECT_URL (5432) — only for prisma migrate ---+
v
prisma migrate / db push / db pull
|
one-shot migrator -- DIRECT_URL (5432) ---------------+
```
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>?pgbouncer=true&connection_limit=10`
- `pgbouncer=true` tells Prisma to skip prepared-statement caching.
- `connection_limit=10` caps Prisma's internal pool — PgBouncer pools again
on top, doubling the budget without this would defeat the purpose.
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — read by
`prisma.config.ts` for `migrate`, `db push`, `db pull`, `db seed`.
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>`
- PrismaPg uses unnamed statements by default, which is compatible with
transaction-mode pooling.
- `DATABASE_POOL_MAX=10` caps the explicit node-postgres pool. A URL
`connection_limit` parameter does not configure PrismaPg/node-postgres.
- `APP_DATABASE_URL`, `PORTAL_DATABASE_URL`, `WORKER_DATABASE_URL`, and the callback-only
`AUTHENTIK_ADMIN_JIT_DATABASE_URL` contain distinct RLS-enforced roles and
point through PgBouncer. Coolify exposes the JIT URL only to the core-web
resource; the portal and other services receive only their normal `DATABASE_URL`.
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — supplied only to
`start.sh migrate` in a short-lived deployment step. It must never exist on
web, API, MCP, Hocuspocus, or worker resources.
## Local development
@@ -35,7 +40,8 @@ docker compose up postgres pgbouncer
# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid?pgbouncer=true&connection_limit=10" \
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid" \
DATABASE_POOL_MAX=10 \
pnpm dev
```
@@ -49,23 +55,31 @@ The pooler must run on the same Docker network as Postgres. The current
operator-owned deployment source is the Coolify service/network config; do not
copy historical staging inventory into a live cutover.
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
`pg_authid.rolpassword`. From the Postgres container:
1. **Provision the reviewed roles and generate `userlist.txt`.** Run the
idempotent role provisioner from [`docs/database.md`](../database.md) first.
Postgres uses `password_encryption=scram-sha-256`, so the role-specific
SCRAM hashes then live in `pg_authid.rolpassword`. From the Postgres
container:
```bash
psql -U postgres -tAc \
"SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
FROM pg_authid
WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt
WHERE rolname IN (
'hivegrid_app','hivegrid_portal','hivegrid_worker','hivegrid_authentik_jit','pgb_admin'
);" > userlist.txt
chmod 600 userlist.txt
```
Drop the file into a Coolify volume mount that the pgbouncer container
will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
The query must return all five rows before rollout. `hivegrid_app`,
`hivegrid_portal`, and `hivegrid_worker` handle their isolated runtime traffic;
`hivegrid_authentik_jit` is the single-function callback capability and
`pgb_admin` is reserved for PgBouncer administration and health checks. Drop the file
into a Coolify volume mount that the pgbouncer container will see at
`/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
2. **Create the Coolify service.**
- Image: `edoburu/pgbouncer:latest`
- Image: `edoburu/pgbouncer:latest@sha256:…` (reviewed manifest digest)
- Network: `coolify` (must match Postgres)
- Mounts:
- `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro`
@@ -80,9 +94,23 @@ copy historical staging inventory into a live cutover.
docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
```
4. **Switch app env vars.** In Coolify, on `web`, `api`, `worker`:
- `DATABASE_URL=postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>?pgbouncer=true&connection_limit=10`
- `DIRECT_URL=postgres://<database-user>:${PG_PASS}@<postgres-service>:5432/<database>`
4. **Switch runtime env vars.** In Coolify:
- core web/API/MCP/Hocuspocus: `DATABASE_URL` uses the `hivegrid_app` role.
- customer portal: `DATABASE_URL` uses the `hivegrid_portal` role.
- every worker group: `DATABASE_URL` uses the `hivegrid_worker` role.
- only core web: `AUTHENTIK_ADMIN_JIT_DATABASE_URL` uses the
`hivegrid_authentik_jit` role. Remove every Admin-SSO key from API, MCP,
Hocuspocus and worker resources, including inherited groups.
- every runtime sets `RLS_RUNTIME_MODE=required` and the matching
`DATABASE_RUNTIME_ROLE`.
- `DATABASE_POOL_MAX` ist auf jeder Runtime explizit gesetzt. Für den ersten
Produktionsrollout gilt je Prozess/Instanz: Core-Web `10`, Customer-Portal
`5`; API/MCP/Hocuspocus und Worker folgen ihrem separat freigegebenen
Kapazitätsplan.
- `DATABASE_POOL_IDLE_TIMEOUT_MS=10000`
- `DATABASE_POOL_CONNECTION_TIMEOUT_MS=5000`
- remove `DIRECT_URL`, `MIGRATOR_*`, `MIGRATE_ON_BOOT`, `SEED_ON_BOOT`, and
all `SEED_*` keys from every runtime resource, including inherited env groups.
Use service and database names from the current Coolify project state; do
not copy archived staging inventory into a live cutover.
@@ -95,19 +123,22 @@ copy historical staging inventory into a live cutover.
1. Provision PgBouncer in Coolify but leave app env vars unchanged.
2. Verify `SHOW STATS` and `SHOW POOLS` respond.
3. Smoke-test from a one-off container:
```bash
docker run --rm --network coolify \
postgres:16-alpine psql \
"postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
-c "SELECT 1;"
```
4. Roll out `DIRECT_URL` to all apps first (no behaviour change because
`prisma.config.ts` falls back to `DATABASE_URL`).
5. Roll out the new `DATABASE_URL` (with `pgbouncer=true&connection_limit=10`).
4. Configure the migrator credential as a CI secret reachable from the runner;
do not copy it to Coolify applications.
5. Run the immutable web image once with `DIRECT_URL` and argument `migrate`.
A missing/unreachable credential or failed migration stops the rollout.
6. Roll out the role-specific `DATABASE_URL` and explicit `DATABASE_POOL_*` limits.
Coolify will recreate containers — they must reconnect to the pooler.
6. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
7. Confirm `prisma migrate status` from a CI container still passes (uses
`DIRECT_URL`).
7. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
8. Confirm `prisma migrate status` from the same step-scoped CI credential.
## Tuning knobs
@@ -131,10 +162,10 @@ Panels of interest: pool wait time p95, server connection count vs.
### `FATAL: prepared statement "s0" already exists`
PgBouncer in transaction pool mode reused a server-side connection that
still has the prepared statement cached. Confirm `DATABASE_URL` has
`?pgbouncer=true` and Prisma 5+ is in use. Without the flag, Prisma keeps
local prepared statements and the next transaction lands on a different
backend.
still has a prepared statement cached. HiveGrid's PrismaPg adapter uses
unnamed statements by default; a URL `pgbouncer=true` flag is ignored. Check
for a custom `statementNameGenerator`, an older non-adapter Prisma client, or
another client sharing the same PgBouncer user.
### `error: too many clients already`
@@ -142,9 +173,8 @@ Postgres ran out of connections. Either:
- `default_pool_size` × number-of-app-pools > `max_connections` — lower
pool size or raise `max_connections` on Postgres.
- Prisma's `connection_limit` is unset, so each app process opens its own
default pool (CPU + 1) against PgBouncer. Set `connection_limit=10` (or
lower) in the query string of `DATABASE_URL`.
- `DATABASE_POOL_MAX` is too high or unset for the deployment budget. Set it
explicitly (10 by default, or lower) for each application process.
### `cl_waiting > 0` persistently in `SHOW POOLS`
@@ -156,13 +186,13 @@ Application sustained pool exhaustion. Options:
### Migrations fail with "prepared statement" or DDL errors
`DIRECT_URL` is wrong or missing. `prisma migrate` must hit Postgres
directly. Verify `prisma.config.ts` reads `DIRECT_URL`:
The step-scoped migrator credential is wrong, missing, or cannot reach
Postgres directly. Verify it only inside a one-shot container:
```bash
DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \
DIRECT_URL=postgres://...@postgres:5432/db \
pnpm exec prisma migrate status
docker run --rm \
-e DIRECT_URL=postgres://...@postgres:5432/db \
<immutable-web-image> migrate
```
### Healthcheck flapping
@@ -176,8 +206,11 @@ unhealthy.
To bypass PgBouncer without code changes:
1. In Coolify, change `DATABASE_URL` to point at Postgres directly
(port 5432, drop `?pgbouncer=true&connection_limit=10`).
2. Restart `web`, `api`, `worker`.
(port 5432; pool sizing remains in `DATABASE_POOL_*`).
2. Preserve the distinct `hivegrid_app`, `hivegrid_portal`, `hivegrid_worker`, and callback-only
`hivegrid_authentik_jit` credentials and restart core web, customer portal,
API, MCP, Hocuspocus, and workers. The JIT URL remains core-web-only.
3. Stop the `pgbouncer` Coolify service.
`DIRECT_URL` stays valid — it never depended on the pooler.
The migrator secret remains step-scoped and unchanged; never add it to the
runtime resources during rollback.