Database Rollback Runbook

Practical runbook for rolling back a Prisma migration or seed in the boilerplate's PostgreSQL deployment. Pair every rollback with a forensic note in the relevant incident channel — rollbacks are not silent operations.

Decision tree

Use this order before reaching for any destructive command:

Migration applied but app fails at boot — Most likely cause: code does not yet handle a new column, OR new column lacks a backfill. Prefer forward-fix: deploy a hotfix patch that handles the schema, then continue.
Migration breaks data integrity — Restore from the most recent verified backup. Do not run prisma migrate resolve --rolled-back against production data without a backup snapshot.
Lokaler/historischer Seed erzeugte falsche Zeilen — Nur in einer wegwerfbaren lokalen Datenbank bereinigen und erneut prüfen. Allgemeines Seeding oder ein Re-Run gegen Staging/Produktion ist unter erzwungener RLS verboten.

Reverting a migration locally

The boilerplate's prisma/schema ships forward-only Prisma migrations. There is no Prisma equivalent of down migrations. To reverse a change locally:

# 1. Inspect the migration to revert
ls prisma/schema/migrations
cat prisma/schema/migrations/<TIMESTAMP>_<name>/migration.sql

# 2. Hand-roll the inverse SQL into a NEW migration
pnpm exec prisma migrate dev --name revert-<original-name>
# Edit the generated migration.sql to contain the inverse DDL.

# 3. Apply
pnpm exec prisma migrate dev

Never delete an applied migration directory from prisma/schema/migrations/ on a branch other than your own work branch — the migration record on staging/production will go inconsistent.

Reverting a migration on staging or production

Prerequisites:

Working PostgreSQL pg_dump/pg_restore or snapshot on the deployment host.
Confirmed shell access to the database (Coolify staging SSH defined in verify:db-migrate-status:staging).
Approval recorded in the change/incident channel.

Procedure:

# 1. Snapshot first.
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"

# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
#    HIVEGRID_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.

# 3. Apply the inverse migration that was prepared and committed beforehand.
pnpm exec prisma migrate deploy

# 4. Verify schema with the same gate CI uses:
pnpm run verify:db-migrate-status

# 5. Bring the application back online; tail logs for at least 5 minutes.

If step 3 fails halfway through, prisma migrate resolve --rolled-back <migration-name> marks the record as rolled-back so the runtime can re-apply. Never use --applied to silently mark a partial migration as done.

Undoing a seed run

Staging/Produktion: Allgemeines Seeding ist unter erzwungener RLS nicht freigegeben. Die folgenden historischen Bereinigungshinweise sind keine Erlaubnis, den Produktions-Seed oder permissions:sync gegen eine produktionsnahe Datenbank auszuführen.

The boilerplate's production seed (packages/db/dist/production-seed.server.js) is gated on NODE_ENV=production and explicit SEED_ADMIN_EMAIL / SEED_ADMIN_PASSWORD. It is idempotent for the canonical permission/entitlement rows but may insert demo data when SEED_DEMO_DATA=true accidentally leaks into a non-local environment.

Undo procedure:

# 1. Snapshot first (same command as above).
# 2. Identify rows that should not exist:
psql "$DATABASE_URL" -c "SELECT id, email, tenant_id FROM \"User\" WHERE email LIKE '%@company.com';"
psql "$DATABASE_URL" -c "SELECT id, name FROM \"Tenant\" WHERE slug IN ('acme-corp-1','acme-corp-2');"

# 3. Delete in dependency order (children first). NEVER use DELETE without a WHERE clause on a production database.

Nur gegen eine wegwerfbare lokale Datenbank darf anschließend der kanonische (nicht Demo-)Seed zur Idempotenzprüfung erneut laufen:

SEED_DEMO_DATA=false NODE_ENV=production node packages/db/dist/production-seed.server.js

Permission/role sync rollback

Der produktive Deploy verwendet nicht pnpm run permissions:sync, sondern den fail-closed globalen Reconcile in start.sh migrate. Schlägt er fehl, werden seine Katalogänderungen und temporären DML-Grants an hivegrid_rls_definer gemeinsam zurückgerollt; FORCE ROW LEVEL SECURITY bleibt aktiv. Bereits zuvor erfolgreich angewandte Prisma-Migrationen bleiben bestehen.

Rollout gestoppt lassen und den redigierten Fehlernachweis sichern; weder allgemeinen Seed noch permissions:sync starten.
Katalogdefinition oder Reconcile-Code per geprüftem Forward-Fix korrigieren und den vollständigen One-shot-Pfad erneut ausführen.
Hat ein fehlerhafter Reconcile bereits committed, vor jeder Datenkorrektur Backup/Restore-Punkt bestätigen und die Korrektur als explizit freigegebene, auditierte Operator-Aktion ausführen.

Tenant-eigene Rollen und Rollen mit isTemplateOrigin = true dürfen auch beim Rollback nicht automatisiert verändert oder nachgezogen werden. Manuelles DELETE FROM "Permission" WHERE name = '…' bleibt wegen kaskadierender RolePermission-Referenzen unzulässig.

Practice drill

scripts/drill-migration-rollback.sh boots an ephemeral PostgreSQL container, replays the full migration chain, runs the production seed twice (idempotency probe), and asserts the second prisma migrate deploy is a no-op. Requires Docker and pnpm.

./scripts/drill-migration-rollback.sh             # full forward + seed cycle
./scripts/drill-migration-rollback.sh --no-seed   # forward chain only
./scripts/drill-migration-rollback.sh --keep-container  # leave PG running for inspection

Run it before any release that introduces migrations. A failing drill means the migration chain or the seed is not safe to apply against a fresh production database.

Verification gates that detect drift

pnpm run verify:db-migrate-status            # local Prisma diff against schema
pnpm run verify:db-migrate-status:staging    # SSH'd staging diff (CI gate)
pnpm run verify:permission-matrix       # role/permission documentation parity
pnpm run verify:production-seed-artifact     # seed artifact compile + idempotency contract

A green run of all four after rollback is the signal that the database is back on a known-good baseline.

Incident handoff template

When handing off after a rollback, include:

Snapshot file location and SHA.
Migration name and direction (forward/reverse) at the start and end of the incident.
The four gate outputs above.
A link to the post-incident commit that locks in the fix or the deferred follow-up.

6.7 KiB Raw Blame History