Database Rollback Runbook

Practical runbook for rolling back a Prisma migration or seed in the boilerplate's PostgreSQL deployment. Pair every rollback with a forensic note in the relevant incident channel — rollbacks are not silent operations.

Decision tree

Use this order before reaching for any destructive command:

Migration applied but app fails at boot — Most likely cause: code does not yet handle a new column, OR new column lacks a backfill. Prefer forward-fix: deploy a hotfix patch that handles the schema, then continue.
Migration breaks data integrity — Restore from the most recent verified backup. Do not run prisma migrate resolve --rolled-back against production data without a backup snapshot.
Seed produced wrong rows — Idempotent seeds (the boilerplate's permissions/entitlements seeds are idempotent by design) tolerate re-runs after manual cleanup. Non-idempotent seed sections should never run in production.

Reverting a migration locally

The boilerplate's prisma/schema ships forward-only Prisma migrations. There is no Prisma equivalent of down migrations. To reverse a change locally:

# 1. Inspect the migration to revert
ls prisma/schema/migrations
cat prisma/schema/migrations/<TIMESTAMP>_<name>/migration.sql

# 2. Hand-roll the inverse SQL into a NEW migration
pnpm exec prisma migrate dev --name revert-<original-name>
# Edit the generated migration.sql to contain the inverse DDL.

# 3. Apply
pnpm exec prisma migrate dev

Never delete an applied migration directory from prisma/schema/migrations/ on a branch other than your own work branch — the migration record on staging/production will go inconsistent.

Reverting a migration on staging or production

Prerequisites:

Working PostgreSQL pg_dump/pg_restore or snapshot on the deployment host.
Confirmed shell access to the database (Coolify staging SSH defined in verify:db-migrate-status:staging).
Approval recorded in the change/incident channel.

Procedure:

# 1. Snapshot first.
pg_dump --format=custom --no-owner --file=pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump "$DATABASE_URL"

# 2. Take the application offline (Coolify -> scale to 0 / put the service in maintenance).
#    TOOLCTRL_API_MAINTENANCE_MODE=1 is available for the API surface; the web surface relies on the deployment platform's maintenance toggle.

# 3. Apply the inverse migration that was prepared and committed beforehand.
pnpm exec prisma migrate deploy

# 4. Verify schema with the same gate CI uses:
pnpm run verify:db-migrate-status

# 5. Bring the application back online; tail logs for at least 5 minutes.

If step 3 fails halfway through, prisma migrate resolve --rolled-back <migration-name> marks the record as rolled-back so the runtime can re-apply. Never use --applied to silently mark a partial migration as done.

Undoing a seed run

The boilerplate's production seed (packages/db/dist/production-seed.server.js) is gated on NODE_ENV=production and explicit SEED_ADMIN_EMAIL / SEED_ADMIN_PASSWORD. It is idempotent for the canonical permission/entitlement rows but may insert demo data when SEED_DEMO_DATA=true accidentally leaks into a non-local environment.

Undo procedure:

# 1. Snapshot first (same command as above).
# 2. Identify rows that should not exist:
psql "$DATABASE_URL" -c "SELECT id, email, tenant_id FROM \"User\" WHERE email LIKE '%@company.com';"
psql "$DATABASE_URL" -c "SELECT id, name FROM \"Tenant\" WHERE slug IN ('acme-corp-1','acme-corp-2');"

# 3. Delete in dependency order (children first). NEVER use DELETE without a WHERE clause on a production database.

After cleanup, re-run the canonical (non-demo) seed to confirm idempotency:

SEED_DEMO_DATA=false NODE_ENV=production node packages/db/dist/production-seed.server.js

Permission/role sync rollback

pnpm run permissions:sync is idempotent. To undo an unwanted permission rename:

Revert the offending commit in packages/db/src/permissions-seed.server.ts.
Re-run pnpm run permissions:sync against staging.
Run verify:permission-matrix to assert documentation parity.

Permission rows themselves are immutable identifiers; the seed only adds/updates. Manual DELETE FROM "Permission" WHERE name = '…' is risky because foreign-key references in RolePermission will cascade.

Practice drill

scripts/drill-migration-rollback.sh boots an ephemeral PostgreSQL container, replays the full migration chain, runs the production seed twice (idempotency probe), and asserts the second prisma migrate deploy is a no-op. Requires Docker and pnpm.

./scripts/drill-migration-rollback.sh             # full forward + seed cycle
./scripts/drill-migration-rollback.sh --no-seed   # forward chain only
./scripts/drill-migration-rollback.sh --keep-container  # leave PG running for inspection

Run it before any release that introduces migrations. A failing drill means the migration chain or the seed is not safe to apply against a fresh production database.

Verification gates that detect drift

pnpm run verify:db-migrate-status            # local Prisma diff against schema
pnpm run verify:db-migrate-status:staging    # SSH'd staging diff (CI gate)
pnpm run verify:permission-matrix       # role/permission documentation parity
pnpm run verify:production-seed-artifact     # seed artifact compile + idempotency contract

A green run of all four after rollback is the signal that the database is back on a known-good baseline.

Incident handoff template

When handing off after a rollback, include:

Snapshot file location and SHA.
Migration name and direction (forward/reverse) at the start and end of the incident.
The four gate outputs above.
A link to the post-incident commit that locks in the fix or the deferred follow-up.

5.7 KiB Raw Blame History