153 lines
14 KiB
Markdown
153 lines
14 KiB
Markdown
# Production Database Operations Guide
|
|
|
|
## Purpose
|
|
|
|
This guide is the operator-facing entrypoint for PostgreSQL and Prisma database operations in production-facing environments. It explains the safe command boundaries for checking migration status, deploying migrations, generating Prisma client artifacts, and triaging database failures without exposing credentials or customer data.
|
|
|
|
Use this guide with the existing database CI and local setup references:
|
|
|
|
- `docs/db-ci.md` documents the CI migration-status verifier and redacted status output.
|
|
- `docs/local-dev.md` documents local PostgreSQL and Prisma setup for day-to-day development.
|
|
|
|
## Required environment keys
|
|
|
|
Database operations are driven by key names only. Documentation, issues, CI logs, and runbooks must never include connection strings, database passwords, usernames, hosts, database names, or customer data.
|
|
|
|
Required or relevant keys:
|
|
|
|
- `DATABASE_URL`: Prisma-Verbindung der jeweiligen Runtime. Core-Web/API/MCP/Hocuspocus nutzen `hivegrid_app`, das Customer-Portal nutzt `hivegrid_portal`, Worker nutzen `hivegrid_worker`.
|
|
- `DIRECT_URL`: direkte, ungepoolte Verbindung des kurzlebigen Migrationsprozesses als `hivegrid_migrator`; sie gehört in keine Runtime.
|
|
- `AUTHENTIK_ADMIN_JIT_DATABASE_URL`: separate PgBouncer-Verbindung als `hivegrid_authentik_jit`; ausschließlich für den verifizierten Authentik-Callback im Web-Resource.
|
|
- `HIVEGRID_RLS_PROVISION_DATABASE_URL`: direkte Superuser-Verbindung ausschließlich für die einmalige oder bewusst wiederholte Rollen-Provisionierung.
|
|
- `HIVEGRID_RLS_MIGRATOR_PASSWORD`, `HIVEGRID_RLS_APP_PASSWORD`, `HIVEGRID_RLS_PORTAL_PASSWORD`, `HIVEGRID_RLS_WORKER_PASSWORD`, `HIVEGRID_RLS_AUTHENTIK_JIT_PASSWORD`: fünf unterschiedliche, mindestens 32 Zeichen lange Secrets aus dem freigegebenen Secret-Store.
|
|
|
|
Operators should confirm that the required key names exist in the target secret store, CI setting, or platform environment. Do not paste raw values into tickets, chat, docs, screenshots, or verifier output.
|
|
|
|
## PostgreSQL-Rollen und Provisionierung
|
|
|
|
HiveGrid verwendet sechs getrennte Rollen:
|
|
|
|
| Rolle | Zweck | Sicherheitsvertrag |
|
|
| ---------------------- | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
|
|
| `hivegrid_migrator` | Eigentümer von Datenbank, `public`-Schema und den normalen Anwendungsobjekten; führt Prisma-Migrationen aus | `LOGIN`, kein Superuser, kein `BYPASSRLS`, kein `INHERIT` |
|
|
| `hivegrid_app` | Web-, API-, MCP- und Hocuspocus-Runtime | `LOGIN`, kein Superuser, kein `BYPASSRLS`, kein `INHERIT` |
|
|
| `hivegrid_portal` | Isolierte Customer-Portal-Runtime mit expliziten Portal-/Projektfreigaben | `LOGIN`, kein Superuser, kein `BYPASSRLS`, kein `INHERIT`, keine Mitgliedschaften |
|
|
| `hivegrid_worker` | Worker-Runtime | `LOGIN`, kein Superuser, kein `BYPASSRLS`, kein `INHERIT` |
|
|
| `hivegrid_authentik_jit` | Verifizierter Authentik-Callback im Web; nur JIT-Admin-Capability | `LOGIN`, keine Tabellen-/Sequenzrechte oder Mitgliedschaft |
|
|
| `hivegrid_rls_definer` | Eigentümer der explizit geprüften `SECURITY DEFINER`-Funktionen | `NOLOGIN`, `BYPASSRLS`, sonst keine administrativen Rechte |
|
|
|
|
Die einzige zulässige Mitgliedschaft ist `hivegrid_migrator` als Mitglied von `hivegrid_rls_definer`, mit `SET TRUE`, aber ohne `ADMIN` und ohne vererbte Rechte. App, Portal, Worker und Authentik-JIT dürfen weder Mitglied der Definer-Rolle noch einer anderen Rolle sein.
|
|
|
|
Die Rollen werden vor der ersten Migration und bei einer kontrollierten Passwortrotation idempotent provisioniert:
|
|
|
|
```sh
|
|
pnpm run db:roles:provision
|
|
```
|
|
|
|
Der Befehl verlangt die Provisionierungs-URL und alle fünf Passwort-Keys, gibt keine URL und kein Passwort aus und bricht ohne Superuser-Verbindung ab. Er setzt die Rollenattribute auf den exakten Vertrag zurück, entfernt unerwartete Mitgliedschaften, erzwingt SCRAM-Passwort-Hashes und prüft den Zustand vor dem Commit. Bei einem Fehler wird die Transaktion zurückgerollt.
|
|
|
|
Der Provisionierer überträgt die aktuelle Datenbank, das `public`-Schema und vorhandene, nicht zu einer Extension gehörende Anwendungsobjekte an `hivegrid_migrator`. Deshalb darf er nur gegen die dedizierte HiveGrid-Datenbank und bei bestehenden Installationen erst nach Backup und bestätigtem Restore-Pfad ausgeführt werden. Extension-Objekte behalten bewusst ihren bisherigen Eigentümer.
|
|
|
|
Sicherer Ablauf für eine neue Datenbank:
|
|
|
|
1. Datenbank anlegen und die sechs Rollen mit `pnpm run db:roles:provision` provisionieren.
|
|
2. `DIRECT_URL` ausschließlich im kurzlebigen Migrationsprozess auf `hivegrid_migrator` setzen.
|
|
3. Mit `pnpm run verify:migrator-role` den tatsächlichen Session-User, die nicht privilegierten Rollenattribute und die einzige erlaubte Definer-Mitgliedschaft prüfen.
|
|
4. Das freigegebene, unveränderliche Web-Image einmalig im `migrate`-Modus starten (`/myapp/start.sh migrate`). Dieser Pfad führt Rollenprüfung, Prisma-Migrationen und den globalen Permission-Reconcile als eine fail-closed Deploy-Kette aus.
|
|
5. Runtimes nur mit `hivegrid_app`, `hivegrid_portal` beziehungsweise `hivegrid_worker` starten. Die separate JIT-URL kommt ausschließlich in die Core-Web-Resource; Portal, API, MCP, Hocuspocus, Worker und Migrator erhalten keine Admin-SSO-Secrets.
|
|
|
|
Bei einer bestehenden Datenbank müssen vor Schritt 1 alle laufenden Schreiber angehalten oder in ein freigegebenes Cutover-Fenster gebracht werden. Nach der Migration sind Rollenattribute, Objekt-Ownership und Tenant-A/B-Isolation zu prüfen, bevor der Traffic wieder freigegeben wird.
|
|
|
|
## Local PostgreSQL and Prisma workflow
|
|
|
|
For local development, follow `docs/local-dev.md` first. Start a local PostgreSQL instance, create a disposable local database, set `DATABASE_URL` to a local placeholder value, and keep production credentials out of local files.
|
|
|
|
Local-only Prisma commands may be used while developing schema changes:
|
|
|
|
```sh
|
|
pnpm exec prisma migrate dev
|
|
pnpm exec prisma migrate status
|
|
pnpm exec prisma db push
|
|
pnpm exec prisma generate
|
|
```
|
|
|
|
Use `pnpm exec prisma migrate dev` for normal local migration authoring. Use `pnpm exec prisma migrate status` to inspect local drift. Use `pnpm exec prisma db push` only for deliberate disposable local database bootstraps where migration history is not the source of truth. Refresh generated Prisma client artifacts with `pnpm exec prisma generate` when local schema or dependency changes require it.
|
|
|
|
These local commands are not production deployment commands. They must not be copied into production migration automation.
|
|
|
|
## CI migration-status gate
|
|
|
|
The CI status gate is a read-only verification boundary for migration status:
|
|
|
|
```sh
|
|
pnpm run verify:db-migrate-status
|
|
```
|
|
|
|
This command is documented in `docs/db-ci.md` and routes through the repository-owned `@hivegrid/db` package surface. It checks that Prisma can report migration status for the configured PostgreSQL database and emits bounded, redacted diagnostic fields such as phase, command, exit code, status outcome, and sanitized output excerpts.
|
|
|
|
A passing migration-status gate means the status command completed. It does not deploy migrations, seed data, reset schemas, or prove browser-facing behavior. Do not confuse status checks with deploy migrations.
|
|
|
|
## Production migration deployment
|
|
|
|
Production migration deployment uses the immutable web image's one-shot entrypoint:
|
|
|
|
```sh
|
|
/myapp/start.sh migrate
|
|
```
|
|
|
|
Run this only inside the approved one-shot container for the target environment. `DIRECT_URL` must be the unpooled connection that authenticates directly as `hivegrid_migrator` and must be scoped to that short-lived migration process. A superuser URL, a URL that reaches the role through `SET ROLE`, or any drift in the required role attributes and membership is forbidden. The bare package command `pnpm run db:migrate:deploy` is not an approved production path because it omits the mandatory post-migration permission reconcile.
|
|
|
|
The deployed web image runs `pnpm run verify:migrator-role` from `start.sh migrate` before it exposes the URL to Prisma. The probe fails closed unless both `session_user` and `current_user` are exactly `hivegrid_migrator`, the role is `LOGIN` but not superuser and cannot create databases or roles, inherit, replicate, or bypass RLS, and its only role edge is the constrained membership in `hivegrid_rls_definer`. The entrypoint then applies the migrations and reconciles the global permission catalog before it can exit successfully. None of these steps prints the connection string.
|
|
|
|
The one-shot migration path applies already-authored Prisma migrations; it is not a schema authoring workflow and must not be mixed with local-only commands such as `pnpm exec prisma migrate dev` or `pnpm exec prisma db push`.
|
|
|
|
### Globaler Permission-Reconcile nach Migrationen
|
|
|
|
Der freigegebene Containerpfad `start.sh migrate` führt erst nach einem erfolgreichen `prisma migrate deploy` den gebündelten Reconcile aus. Er gleicht ausschließlich den globalen `Permission`-Katalog, die globalen Defaultrollen (`tenantId IS NULL`) und deren katalogeigene Verknüpfungen ab. Der allein aufgerufene Package-Befehl `pnpm run db:migrate:deploy` enthält diesen Post-Migration-Schritt nicht.
|
|
|
|
Der Reconcile läuft fail-closed in einer kurzen Datenbanktransaktion. `hivegrid_rls_definer` erhält darin nur temporär die erforderlichen DML-Rechte; vor dem Commit werden sie wieder entzogen. `FORCE ROW LEVEL SECURITY` bleibt aktiv. Fehler bei Sicherheitsprüfung, Abgleich oder Daten-Postcondition rollen Daten und ACL-Änderungen dieser Transaktion zurück und stoppen den Deploy.
|
|
|
|
AD-29 schützt tenant-eigene Rollen und alle Rollen mit `isTemplateOrigin = true`: Sie werden im automatischen Deploy niemals angelegt, geändert, gelöscht oder mit Berechtigungen nachgezogen. Ist eine kanonische globale Rolle als Template-Ursprung markiert, bricht der Reconcile vor Rollen-DML fail-closed ab; Tenant-Rollen bleiben unberührt. Änderungen an Bestands-Tenants erfordern eine explizit freigegebene, auditierte Operator-Aktion. Allgemeines Seeding bleibt unter erzwungener RLS verboten; der globale Reconcile ist keine Seed-Freigabe.
|
|
|
|
Before deploying, confirm that the intended migration set has passed review and that the CI migration-status gate is healthy. After deployment, inspect the approved deployment logs and the migration-status verifier output for redacted success or failure signals. If deployment fails, do not paste connection details or table contents into the incident record; capture the command name, exit code, diagnostic phase, and sanitized Prisma excerpt only.
|
|
|
|
## Backup and restore expectations
|
|
|
|
Backups and restores are environment-owned operational responsibilities. Before running production migrations, operators must confirm that the target environment has a current backup or restore point according to the hosting provider or database platform policy.
|
|
|
|
Restore drills should use platform-approved tooling and should document only high-level evidence: environment name, backup timestamp or restore-point identifier, operator, result, and redacted error code if applicable. Never include database dumps, customer rows, credentials, hosts, usernames, passwords, or full connection strings in repository docs or issue trackers.
|
|
|
|
For local testing of destructive schema work, use disposable local databases instead of production data. Synthetic fixtures and redacted samples are acceptable; customer data is not.
|
|
|
|
## Prisma client generation
|
|
|
|
Generated Prisma client artifacts are refreshed through the repository script:
|
|
|
|
```sh
|
|
pnpm run db:generate
|
|
```
|
|
|
|
The root build also generates the database client before building the web runtime. Run `pnpm run db:generate` after schema changes, dependency updates, or deployment preparation when generated types or client artifacts may be stale.
|
|
|
|
Generation does not connect to production data and must not be treated as a migration or data validation step. If generation fails, inspect package install state, Prisma schema validity, and sanitized compiler output before investigating database connectivity.
|
|
|
|
## Troubleshooting and safe observability
|
|
|
|
Use these safe inspection surfaces for database failures:
|
|
|
|
- CI logs from `pnpm run verify:db-migrate-status` with sanitized stdout and stderr excerpts.
|
|
- Approved deployment logs from the immutable web image's `/myapp/start.sh migrate` one-shot path, with secrets redacted.
|
|
- Local command output from `pnpm exec prisma migrate status` against a disposable local PostgreSQL database.
|
|
- Package script names, exit codes, diagnostic phases, and redacted Prisma error codes.
|
|
- `docs/db-ci.md` for the CI status verifier contract and `docs/local-dev.md` for local setup checks.
|
|
|
|
Common failure patterns:
|
|
|
|
- Missing `DATABASE_URL`: confirm that the key exists in the CI secret or platform environment and is scoped only to the database operation that needs it.
|
|
- Connection or authentication failure: confirm the secret store and database service health without copying the value, username, password, host, or database name into logs.
|
|
- Migration drift: run the status gate, inspect the sanitized Prisma status message, and distinguish read-only status output from deploy behavior before deciding whether to invoke the approved `/myapp/start.sh migrate` one-shot path.
|
|
- Stale Prisma client: run `pnpm run db:generate` and re-run the focused check that reported stale generated artifacts.
|
|
- Local schema mismatch: use local-only `pnpm exec prisma migrate status`, `pnpm exec prisma migrate dev`, or a deliberate disposable `pnpm exec prisma db push` according to `docs/local-dev.md`.
|
|
|
|
If any troubleshooting output reveals a connection string, credential, hostname, username, password, database name, or customer data, treat it as a secret leak. Remove the exposed material from the shared surface, rotate affected credentials, preserve only redacted evidence, and fix the logging or documentation path before continuing.
|