129 lines
6.7 KiB
Markdown
129 lines
6.7 KiB
Markdown
# Database Rollback Runbook
|
|
|
|
Practical runbook for rolling back a Prisma migration or seed in the boilerplate's PostgreSQL deployment. Pair every rollback with a forensic note in the relevant incident channel — rollbacks are not silent operations.
|
|
|
|
## Decision tree
|
|
|
|
Use this order before reaching for any destructive command:
|
|
|
|
1. **Migration applied but app fails at boot** — Most likely cause: code does not yet handle a new column, OR new column lacks a backfill. Prefer **forward-fix**: deploy a hotfix patch that handles the schema, then continue.
|
|
2. **Migration breaks data integrity** — Restore from the most recent verified backup. Do not run `prisma migrate resolve --rolled-back` against production data without a backup snapshot.
|
|
3. **Lokaler/historischer Seed erzeugte falsche Zeilen** — Nur in einer wegwerfbaren lokalen Datenbank bereinigen und erneut prüfen. Allgemeines Seeding oder ein Re-Run gegen Staging/Produktion ist unter erzwungener RLS verboten.
|
|
|
|
## Reverting a migration locally
|
|
|
|
The boilerplate's `prisma/schema` ships forward-only Prisma migrations. There is no Prisma equivalent of `down` migrations. To reverse a change locally:
|
|
|
|
```sh
|
|
# 1. Inspect the migration to revert
|
|
ls prisma/schema/migrations
|
|
cat prisma/schema/migrations/<TIMESTAMP>_<name>/migration.sql
|
|
|
|
# 2. Hand-roll the inverse SQL into a NEW migration
|
|
pnpm exec prisma migrate dev --name revert-<original-name>
|
|
# Edit the generated migration.sql to contain the inverse DDL.
|
|
|
|
# 3. Apply
|
|
pnpm exec prisma migrate dev
|
|
```
|
|
|
|
Never delete an applied migration directory from `prisma/schema/migrations/` on a branch other than your own work branch — the migration record on staging/production will go inconsistent.
|
|
|
|
## Reverting a migration on staging or production
|
|
|
|
Prerequisites:
|
|
|
|
- Working PostgreSQL `pg_dump`/`pg_restore` or snapshot on the deployment host.
|
|
- Confirmed shell access to the database (Coolify staging SSH defined in `verify:db-migrate-status:staging`).
|
|
- Approval recorded in the change/incident channel.
|
|
|
|
Procedure:
|
|
|
|
```sh
|
|
# 1. Snapshot first.
|
|
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"
|
|
|
|
# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
|
|
# HIVEGRID_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.
|
|
|
|
# 3. Apply the inverse migration that was prepared and committed beforehand.
|
|
pnpm exec prisma migrate deploy
|
|
|
|
# 4. Verify schema with the same gate CI uses:
|
|
pnpm run verify:db-migrate-status
|
|
|
|
# 5. Bring the application back online; tail logs for at least 5 minutes.
|
|
```
|
|
|
|
If step 3 fails halfway through, `prisma migrate resolve --rolled-back <migration-name>` marks the record as rolled-back so the runtime can re-apply. Never use `--applied` to silently mark a partial migration as done.
|
|
|
|
## Undoing a seed run
|
|
|
|
> **Staging/Produktion:** Allgemeines Seeding ist unter erzwungener RLS nicht freigegeben. Die folgenden historischen Bereinigungshinweise sind keine Erlaubnis, den Produktions-Seed oder `permissions:sync` gegen eine produktionsnahe Datenbank auszuführen.
|
|
|
|
The boilerplate's production seed (`packages/db/dist/production-seed.server.js`) is gated on `NODE_ENV=production` and explicit `SEED_ADMIN_EMAIL` / `SEED_ADMIN_PASSWORD`. It is idempotent for the canonical permission/entitlement rows but may insert demo data when `SEED_DEMO_DATA=true` accidentally leaks into a non-local environment.
|
|
|
|
Undo procedure:
|
|
|
|
```sh
|
|
# 1. Snapshot first (same command as above).
|
|
# 2. Identify rows that should not exist:
|
|
psql "$DATABASE_URL" -c "SELECT id, email, tenant_id FROM \"User\" WHERE email LIKE '%@company.com';"
|
|
psql "$DATABASE_URL" -c "SELECT id, name FROM \"Tenant\" WHERE slug IN ('acme-corp-1','acme-corp-2');"
|
|
|
|
# 3. Delete in dependency order (children first). NEVER use DELETE without a WHERE clause on a production database.
|
|
```
|
|
|
|
Nur gegen eine wegwerfbare lokale Datenbank darf anschließend der kanonische (nicht Demo-)Seed zur Idempotenzprüfung erneut laufen:
|
|
|
|
```sh
|
|
SEED_DEMO_DATA=false NODE_ENV=production node packages/db/dist/production-seed.server.js
|
|
```
|
|
|
|
## Permission/role sync rollback
|
|
|
|
Der produktive Deploy verwendet nicht `pnpm run permissions:sync`, sondern den fail-closed globalen Reconcile in `start.sh migrate`. Schlägt er fehl, werden seine Katalogänderungen und temporären DML-Grants an `hivegrid_rls_definer` gemeinsam zurückgerollt; `FORCE ROW LEVEL SECURITY` bleibt aktiv. Bereits zuvor erfolgreich angewandte Prisma-Migrationen bleiben bestehen.
|
|
|
|
1. Rollout gestoppt lassen und den redigierten Fehlernachweis sichern; weder allgemeinen Seed noch `permissions:sync` starten.
|
|
2. Katalogdefinition oder Reconcile-Code per geprüftem Forward-Fix korrigieren und den vollständigen One-shot-Pfad erneut ausführen.
|
|
3. Hat ein fehlerhafter Reconcile bereits committed, vor jeder Datenkorrektur Backup/Restore-Punkt bestätigen und die Korrektur als explizit freigegebene, auditierte Operator-Aktion ausführen.
|
|
|
|
Tenant-eigene Rollen und Rollen mit `isTemplateOrigin = true` dürfen auch beim Rollback nicht automatisiert verändert oder nachgezogen werden. Manuelles `DELETE FROM "Permission" WHERE name = '…'` bleibt wegen kaskadierender `RolePermission`-Referenzen unzulässig.
|
|
|
|
## Practice drill
|
|
|
|
`scripts/drill-migration-rollback.sh` boots an ephemeral PostgreSQL
|
|
container, replays the full migration chain, runs the production seed
|
|
twice (idempotency probe), and asserts the second `prisma migrate
|
|
deploy` is a no-op. Requires Docker and pnpm.
|
|
|
|
```sh
|
|
./scripts/drill-migration-rollback.sh # full forward + seed cycle
|
|
./scripts/drill-migration-rollback.sh --no-seed # forward chain only
|
|
./scripts/drill-migration-rollback.sh --keep-container # leave PG running for inspection
|
|
```
|
|
|
|
Run it before any release that introduces migrations. A failing drill
|
|
means the migration chain or the seed is not safe to apply against a
|
|
fresh production database.
|
|
|
|
## Verification gates that detect drift
|
|
|
|
```sh
|
|
pnpm run verify:db-migrate-status # local Prisma diff against schema
|
|
pnpm run verify:db-migrate-status:staging # SSH'd staging diff (CI gate)
|
|
pnpm run verify:permission-matrix # role/permission documentation parity
|
|
pnpm run verify:production-seed-artifact # seed artifact compile + idempotency contract
|
|
```
|
|
|
|
A green run of all four after rollback is the signal that the database is back on a known-good baseline.
|
|
|
|
## Incident handoff template
|
|
|
|
When handing off after a rollback, include:
|
|
|
|
- Snapshot file location and SHA.
|
|
- Migration name and direction (forward/reverse) at the start and end of the incident.
|
|
- The four gate outputs above.
|
|
- A link to the post-incident commit that locks in the fix or the deferred follow-up.
|