# Tenant Snapshot — Operations Runbook `tenant:snapshot` and `tenant:restore` ship inside `packages/cli` and let operators clone a single tenant's Postgres rows + S3 storage objects into a PII-anonymized archive, then replay that archive into another environment. This runbook is the operator-facing companion to the tenant snapshot implementation in `packages/cli`. --- ## When to use it | Use case | Profile | Encryption | Restore target | | --- | --- | --- | --- | | **Bug repro** ("Customer X hit a workflow bug") | `full-anonymized` | `age` | local dev box | | **QA fixture** (representative Pro-tier tenant) | `full-anonymized` | `age` | CI seed | | **Sales demo** (realistic but anonymized) | `minimal-anonymized` | `age` | staging | | **Tenant migration** to a dedicated instance | `none` (super-admin only) | `age` (mandatory) | the destination environment | The default profile is `full-anonymized`. Never use `none` for any path that leaves your production network. --- ## Anonymization profiles Three profiles ship today. Picking one is a tradeoff between fidelity and privacy risk. The full mapping table lives in `packages/cli/src/commands/tenant/anonymize.ts`; the highlights: | Field type | `full-anonymized` | `minimal-anonymized` | `none` | | --- | --- | --- | --- | | Email | `user-@anonymized.test` | `user-@anonymized.test` | (untouched) | | First / Last name | random from a curated pool | **kept** (display names usable in QA UI) | (untouched) | | Phone number | `null` | `null` | (untouched) | | IP address | `127.0.0.` | `127.0.0.` | (untouched) | | ZIP code | first two digits kept, rest reseeded | first two digits kept, rest reseeded | (untouched) | | Legacy-Provider customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) | | Webhook / SSO / domain URL | `https://anonymized.test/<8 hex>` | same as full | (untouched) | | Secrets, API keys, refresh tokens, payloads | `[REDACTED]` | `[REDACTED]` | (untouched) | | Avatar URL | `https://anonymized.test/avatar/<8 hex>.png` | same as full | (untouched) | **Determinism.** All anonymizers are seeded with `sha256(salt + original)` where `salt` is a fresh 32-byte random value generated per snapshot. Two rows that referenced the same original email get the same anonymized email — FK integrity inside a snapshot is preserved. The salt itself is **not** stored anywhere; only its `sha256` lives in the manifest as a reproducibility marker. --- ## CLI usage ### Create a snapshot (default profile, age-encrypted, upload to S3) ```bash hivegrid-cli tenant snapshot \ --tenant-id ten_abc123 \ --output s3://hivegrid-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age ``` The CLI requires `BACKUP_AGE_PUBLIC_KEY` (recipient public key) plus `BACKUP_S3_*` credentials. See `docs/ops/backup-cli.md` for the env-var catalog — the snapshot command reuses the same encryption layer. ### Restore into a dev box ```bash NODE_ENV=development hivegrid-cli tenant restore \ --tenant-id ten_devclone \ --source s3://hivegrid-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age ``` The command refuses to run under `NODE_ENV=production` unless the operator passes `--allow-production-restore`. The confirmation prompt can be silenced with `--yes` for non-interactive pipelines. ### List existing snapshots ```bash hivegrid-cli tenant list-snapshots --bucket hivegrid-snapshots ``` ### Super-admin escape hatch (`--profile=none`) Raw, un-anonymized exports require an explicit confirmation flag: ```bash hivegrid-cli tenant snapshot \ --tenant-id ten_abc123 \ --profile none \ --i-am-the-super-admin \ --output s3://hivegrid-snapshots/migration/ten_abc123.tar.gz.age ``` Without `--i-am-the-super-admin`, the command exits non-zero with a clear error. Every snapshot — including the `none` profile — emits a structured `[audit] TENANT_SNAPSHOT_CREATED` line to stdout. Operators are expected to redirect that into their central audit log; set `auditFilePath` when calling `runTenantSnapshot()` programmatically. --- ## DSGVO / GDPR notes - **`full-anonymized` is the only profile that may leave EU production infrastructure.** The other profiles ship un-anonymized display names (`minimal`) or full PII (`none`). - Snapshots are themselves sensitive (business workflows, schema layout, org structure). Always set `BACKUP_ENCRYPTION=age` so the artifact can only be decrypted by the operator who holds the private identity file. - The salt is discarded after snapshot creation; only the `sha256(salt)` travels in the manifest. There is no way to reverse-engineer the original PII out of an anonymized row given just the snapshot file. - If a tenant invokes their right-to-be-forgotten on the **source** environment after a snapshot exists, the snapshot still contains the anonymized projection (which by design contains no resolvable PII). For `none` snapshots, the operator must manually destroy the archive. --- ## Restore drill (recommended cadence: monthly) 1. Snapshot a representative tenant from staging: ```bash hivegrid-cli tenant snapshot --tenant-id \ --profile full-anonymized \ --output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age ``` 2. Copy to a dev box and restore into a fresh database: ```bash NODE_ENV=development DATABASE_URL=postgres://... \ hivegrid-cli tenant restore \ --tenant-id ten_drill_$(date +%s) \ --source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \ --yes ``` 3. Smoke test in the dev UI: login as a restored user, view a workflow, confirm subscription state shows as expected. 4. Record the drill in the change log. --- ## How globals are handled in restore Several tenant-scoped models carry foreign keys into **global** tables (today only `User`). Earlier snapshots only emitted tenant-scoped rows, so restoring into a truly empty target database failed for those models with `ForeignKeyConstraintViolation`. Closed by Task G — the extractor now harvests every referenced global id from the tenant slice, fetches the rows, anonymizes them with the same per-snapshot salt, and writes them to `globals/.jsonl` inside the archive. Models that previously broke (now restore clean against an empty target via the globals replay): | Model | Foreign key into | | ----------------- | ---------------- | | `TenantUser` | `User` | | `TenantIpAddress` | `User` | | `UserRole` | `User` | | `Event` | `User` | | `Log` | `User` | | `SupportTicket` | `User` | The full list of `User`-referencing tenant-scoped models is machine-enumerated in `packages/cli/src/commands/tenant/models.ts::GLOBAL_REFERENCED_MODELS` and asserted complete by `models.test.ts` against every prisma schema relation. ### Restore modes for globals (`--globals=`) The restore command accepts one of three strategies (default `upsert`): | Mode | Behaviour | When to use | | -------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------- | | `upsert` | For each id, `findUnique` → if present, leave in place; else `create`. Idempotent. | Default. Merging a snapshot into a non-empty dev/staging. | | `skip` | `createMany({skipDuplicates:true})`. Faster but silent on non-id unique conflicts. | Bulk restores where pre-existing rows are expected. | | `error` | Same as upsert but aborts if any id already exists on target. | Sensitive restores into a verifiably-empty DB. | ```bash # Default — overlay snapshot rows onto whatever already exists. hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz # Sensitive restore: refuse to proceed if any global is already there. hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \ --globals=error ``` ### Platform-catalog globals A second category of globals — `Entity`, `EntityGroup`, `Formula`, `Onboarding`, `PromptFlow`, `RowComment`, `WorkflowBlock` — is referenced by tenant-scoped models but **intentionally NOT** in the snapshot. These are seeded once per deployment via migrations/setup scripts; copying them per-tenant would be wrong. If a tenant slice references a platform-catalog row that does not exist on the target, the restore fails for that specific row and you must align the target deployment's seed first. The list is asserted by `models.test.ts::PLATFORM_CATALOG_GLOBALS`. ### Backward compatibility Snapshots taken before this extractor landed have no `globals` array in their manifest. The restore detects that, logs a warning, and falls back to the pre-Task-G behaviour where the operator must pre-seed referenced rows manually. Use `--include-globals=false` on `tenant:snapshot` to produce a legacy-shaped archive intentionally. ### Anonymization of global rows `User` rows in the globals slice are scrubbed by the same anonymizer that handles tenant rows. The `full-anonymized` profile rewrites `email`, `firstName`, `lastName`, `phone` and `avatar`, and nulls credentials (`passwordHash`, `verifyToken`, `githubId`, `googleId`, `totpSecretEncrypted`, `totpPendingSecretEncrypted`, `mfaRecoveryCodeHashes`, `deletionRequestId`) so the restored user cannot be logged into via the original credentials. The `minimal-anonymized` profile keeps display names but still nulls credentials. See `packages/cli/src/commands/tenant/anonymize.ts::FULL_ANONYMIZED_PROFILE` for the source of truth. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `Refusing to run --profile=none without --i-am-the-super-admin` | Guardrail working as intended. | Re-run with the flag if you are a super-admin and have a documented reason. Otherwise pick a different profile. | | `Invalid --output: expected s3:///` | Output target malformed. | Use either a local path or a fully-qualified `s3://bucket/key/...` URL. | | `Refusing to run tenant:restore in NODE_ENV=production` | Production restore guardrail. | Set `NODE_ENV` to a non-production value, or pass `--allow-production-restore` if you genuinely want to overwrite production. | | `Row-count mismatch in N models` warning during restore | Target DB rejected some rows (FK or unique index conflict with pre-existing data). | Pick a fresh tenant id, or pre-clean the target DB. | | `manifest.json: missing or invalid` | Archive truncated / wrong encryption key. | Verify download size against the S3 ETag; rerun with the correct `BACKUP_AGE_IDENTITY_FILE`. | | Storage bucket mismatch (objects landed under wrong prefix) | `--storage-bucket` or `STORAGE_BUCKET` env mismatched between source and target. | Set the target storage bucket explicitly before re-running restore. | | `tenant:restore aborted (--globals=error): had N pre-existing row(s)` | `--globals=error` saw an id already on target. | Re-run with `--globals=upsert` (default) to overlay or `--globals=skip` to drop conflicts. | | `Manifest carries no globals slice (legacy snapshot)` warning | Snapshot was taken before Task G landed (or with `--include-globals=false`). | Either take a fresh snapshot, or pre-seed referenced globals on target before restoring. | --- ## Internals - Tenant-scoped models enumerated: `packages/cli/src/commands/tenant/models.ts` (`TENANT_SCOPED_MODELS`). - Global-snapshot models enumerated: same file (`GLOBAL_REFERENCED_MODELS`); platform-catalog globals on `PLATFORM_CATALOG_GLOBALS`. - Globals extractor: `globals-runner.ts::collectAndWriteGlobals()`. - Profiles: `packages/cli/src/commands/tenant/anonymize.ts`. - Manifest schema: `packages/cli/src/commands/tenant/manifest.ts`. - Snapshot entrypoint: `runTenantSnapshot()` in `snapshot.ts`. - Restore entrypoint: `runTenantRestore()` in `restore.ts`. - Audit entry: `writeSnapshotAudit()` in `audit.ts`.