# Database Rollback Runbook

Practical runbook for rolling back a Prisma migration or seed in the boilerplate's PostgreSQL deployment. Pair every rollback with a forensic note in the relevant incident channel — rollbacks are not silent operations.

## Decision tree

Use this order before reaching for any destructive command:

1. **Migration applied but app fails at boot** — Most likely cause: code does not yet handle a new column, OR new column lacks a backfill. Prefer **forward-fix**: deploy a hotfix patch that handles the schema, then continue.
2. **Migration breaks data integrity** — Restore from the most recent verified backup. Do not run `prisma migrate resolve --rolled-back` against production data without a backup snapshot.
3. **Lokaler/historischer Seed erzeugte falsche Zeilen** — Nur in einer wegwerfbaren lokalen Datenbank bereinigen und erneut prüfen. Allgemeines Seeding oder ein Re-Run gegen Staging/Produktion ist unter erzwungener RLS verboten.

## Reverting a migration locally

The boilerplate's `prisma/schema` ships forward-only Prisma migrations. There is no Prisma equivalent of `down` migrations. To reverse a change locally:

```sh
# 1. Inspect the migration to revert
ls prisma/schema/migrations
cat prisma/schema/migrations/<TIMESTAMP>_<name>/migration.sql

# 2. Hand-roll the inverse SQL into a NEW migration
pnpm exec prisma migrate dev --name revert-<original-name>
# Edit the generated migration.sql to contain the inverse DDL.

# 3. Apply
pnpm exec prisma migrate dev
```

Never delete an applied migration directory from `prisma/schema/migrations/` on a branch other than your own work branch — the migration record on staging/production will go inconsistent.

## Reverting a migration on staging or production

Prerequisites:

- Working PostgreSQL `pg_dump`/`pg_restore` or snapshot on the deployment host.
- Confirmed shell access to the database (Coolify staging SSH defined in `verify:db-migrate-status:staging`).
- Approval recorded in the change/incident channel.

Procedure:

```sh
# 1. Snapshot first.
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"

# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
#    HIVEGRID_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.

# 3. Apply the inverse migration that was prepared and committed beforehand.
pnpm exec prisma migrate deploy

# 4. Verify schema with the same gate CI uses:
pnpm run verify:db-migrate-status

# 5. Bring the application back online; tail logs for at least 5 minutes.
```

If step 3 fails halfway through, `prisma migrate resolve --rolled-back <migration-name>` marks the record as rolled-back so the runtime can re-apply. Never use `--applied` to silently mark a partial migration as done.

## Undoing a seed run

> **Staging/Produktion:** Allgemeines Seeding ist unter erzwungener RLS nicht freigegeben. Die folgenden historischen Bereinigungshinweise sind keine Erlaubnis, den Produktions-Seed oder `permissions:sync` gegen eine produktionsnahe Datenbank auszuführen.

The boilerplate's production seed (`packages/db/dist/production-seed.server.js`) is gated on `NODE_ENV=production` and explicit `SEED_ADMIN_EMAIL` / `SEED_ADMIN_PASSWORD`. It is idempotent for the canonical permission/entitlement rows but may insert demo data when `SEED_DEMO_DATA=true` accidentally leaks into a non-local environment.

Undo procedure:

```sh
# 1. Snapshot first (same command as above).
# 2. Identify rows that should not exist:
psql "$DATABASE_URL" -c "SELECT id, email, tenant_id FROM \"User\" WHERE email LIKE '%@company.com';"
psql "$DATABASE_URL" -c "SELECT id, name FROM \"Tenant\" WHERE slug IN ('acme-corp-1','acme-corp-2');"

# 3. Delete in dependency order (children first). NEVER use DELETE without a WHERE clause on a production database.
```

Nur gegen eine wegwerfbare lokale Datenbank darf anschließend der kanonische (nicht Demo-)Seed zur Idempotenzprüfung erneut laufen:

```sh
SEED_DEMO_DATA=false NODE_ENV=production node packages/db/dist/production-seed.server.js
```

## Permission/role sync rollback

Der produktive Deploy verwendet nicht `pnpm run permissions:sync`, sondern den fail-closed globalen Reconcile in `start.sh migrate`. Schlägt er fehl, werden seine Katalogänderungen und temporären DML-Grants an `hivegrid_rls_definer` gemeinsam zurückgerollt; `FORCE ROW LEVEL SECURITY` bleibt aktiv. Bereits zuvor erfolgreich angewandte Prisma-Migrationen bleiben bestehen.

1. Rollout gestoppt lassen und den redigierten Fehlernachweis sichern; weder allgemeinen Seed noch `permissions:sync` starten.
2. Katalogdefinition oder Reconcile-Code per geprüftem Forward-Fix korrigieren und den vollständigen One-shot-Pfad erneut ausführen.
3. Hat ein fehlerhafter Reconcile bereits committed, vor jeder Datenkorrektur Backup/Restore-Punkt bestätigen und die Korrektur als explizit freigegebene, auditierte Operator-Aktion ausführen.

Tenant-eigene Rollen und Rollen mit `isTemplateOrigin = true` dürfen auch beim Rollback nicht automatisiert verändert oder nachgezogen werden. Manuelles `DELETE FROM "Permission" WHERE name = '…'` bleibt wegen kaskadierender `RolePermission`-Referenzen unzulässig.

## Practice drill

`scripts/drill-migration-rollback.sh` boots an ephemeral PostgreSQL
container, replays the full migration chain, runs the production seed
twice (idempotency probe), and asserts the second `prisma migrate
deploy` is a no-op. Requires Docker and pnpm.

```sh
./scripts/drill-migration-rollback.sh             # full forward + seed cycle
./scripts/drill-migration-rollback.sh --no-seed   # forward chain only
./scripts/drill-migration-rollback.sh --keep-container  # leave PG running for inspection
```

Run it before any release that introduces migrations. A failing drill
means the migration chain or the seed is not safe to apply against a
fresh production database.

## Verification gates that detect drift

```sh
pnpm run verify:db-migrate-status            # local Prisma diff against schema
pnpm run verify:db-migrate-status:staging    # SSH'd staging diff (CI gate)
pnpm run verify:permission-matrix       # role/permission documentation parity
pnpm run verify:production-seed-artifact     # seed artifact compile + idempotency contract
```

A green run of all four after rollback is the signal that the database is back on a known-good baseline.

## Incident handoff template

When handing off after a rollback, include:

- Snapshot file location and SHA.
- Migration name and direction (forward/reverse) at the start and end of the incident.
- The four gate outputs above.
- A link to the post-incident commit that locks in the fix or the deferred follow-up.