254 lines
12 KiB
Markdown
254 lines
12 KiB
Markdown
# Tenant Snapshot — Operations Runbook
|
|
|
|
`tenant:snapshot` and `tenant:restore` ship inside `packages/cli` and let
|
|
operators clone a single tenant's Postgres rows + S3 storage objects into a
|
|
PII-anonymized archive, then replay that archive into another environment.
|
|
|
|
This runbook is the operator-facing companion to the implementation spec at
|
|
`docs/improvement-plan/03-scaling-infrastructure/tenant-snapshot.md`.
|
|
|
|
---
|
|
|
|
## When to use it
|
|
|
|
| Use case | Profile | Encryption | Restore target |
|
|
| --- | --- | --- | --- |
|
|
| **Bug repro** ("Customer X hit a workflow bug") | `full-anonymized` | `age` | local dev box |
|
|
| **QA fixture** (representative Pro-tier tenant) | `full-anonymized` | `age` | CI seed |
|
|
| **Sales demo** (realistic but anonymized) | `minimal-anonymized` | `age` | staging |
|
|
| **Tenant migration** to a dedicated instance | `none` (super-admin only) | `age` (mandatory) | the destination environment |
|
|
|
|
The default profile is `full-anonymized`. Never use `none` for any path that
|
|
leaves your production network.
|
|
|
|
---
|
|
|
|
## Anonymization profiles
|
|
|
|
Three profiles ship today. Picking one is a tradeoff between fidelity and
|
|
privacy risk. The full mapping table lives in
|
|
`packages/cli/src/commands/tenant/anonymize.ts`; the highlights:
|
|
|
|
| Field type | `full-anonymized` | `minimal-anonymized` | `none` |
|
|
| --- | --- | --- | --- |
|
|
| Email | `user-<hash>@anonymized.test` | `user-<hash>@anonymized.test` | (untouched) |
|
|
| First / Last name | random from a curated pool | **kept** (display names usable in QA UI) | (untouched) |
|
|
| Phone number | `null` | `null` | (untouched) |
|
|
| IP address | `127.0.0.<seeded>` | `127.0.0.<seeded>` | (untouched) |
|
|
| ZIP code | first two digits kept, rest reseeded | first two digits kept, rest reseeded | (untouched) |
|
|
| Stripe customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
|
|
| Webhook / SSO / domain URL | `https://anonymized.test/<8 hex>` | same as full | (untouched) |
|
|
| Secrets, API keys, refresh tokens, payloads | `[REDACTED]` | `[REDACTED]` | (untouched) |
|
|
| Avatar URL | `https://anonymized.test/avatar/<8 hex>.png` | same as full | (untouched) |
|
|
|
|
**Determinism.** All anonymizers are seeded with `sha256(salt + original)`
|
|
where `salt` is a fresh 32-byte random value generated per snapshot. Two rows
|
|
that referenced the same original email get the same anonymized email — FK
|
|
integrity inside a snapshot is preserved. The salt itself is **not** stored
|
|
anywhere; only its `sha256` lives in the manifest as a reproducibility marker.
|
|
|
|
---
|
|
|
|
## CLI usage
|
|
|
|
### Create a snapshot (default profile, age-encrypted, upload to S3)
|
|
|
|
```bash
|
|
toolctrl-cli tenant snapshot \
|
|
--tenant-id ten_abc123 \
|
|
--output s3://toolctrl-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
|
|
```
|
|
|
|
The CLI requires `BACKUP_AGE_PUBLIC_KEY` (recipient public key) plus
|
|
`BACKUP_S3_*` credentials. See `docs/ops/backup-cli.md` for the env-var
|
|
catalog — the snapshot command reuses the same encryption layer.
|
|
|
|
### Restore into a dev box
|
|
|
|
```bash
|
|
NODE_ENV=development toolctrl-cli tenant restore \
|
|
--tenant-id ten_devclone \
|
|
--source s3://toolctrl-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
|
|
```
|
|
|
|
The command refuses to run under `NODE_ENV=production` unless the operator
|
|
passes `--allow-production-restore`. The confirmation prompt can be silenced
|
|
with `--yes` for non-interactive pipelines.
|
|
|
|
### List existing snapshots
|
|
|
|
```bash
|
|
toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
|
|
```
|
|
|
|
### Super-admin escape hatch (`--profile=none`)
|
|
|
|
Raw, un-anonymized exports require an explicit confirmation flag:
|
|
|
|
```bash
|
|
toolctrl-cli tenant snapshot \
|
|
--tenant-id ten_abc123 \
|
|
--profile none \
|
|
--i-am-the-super-admin \
|
|
--output s3://toolctrl-snapshots/migration/ten_abc123.tar.gz.age
|
|
```
|
|
|
|
Without `--i-am-the-super-admin`, the command exits non-zero with a clear
|
|
error. Every snapshot — including the `none` profile — emits a structured
|
|
`[audit] TENANT_SNAPSHOT_CREATED` line to stdout. Operators are expected to
|
|
redirect that into their central audit log; set `auditFilePath` when calling
|
|
`runTenantSnapshot()` programmatically.
|
|
|
|
---
|
|
|
|
## DSGVO / GDPR notes
|
|
|
|
- **`full-anonymized` is the only profile that may leave EU production
|
|
infrastructure.** The other profiles ship un-anonymized display names
|
|
(`minimal`) or full PII (`none`).
|
|
- Snapshots are themselves sensitive (business workflows, schema layout,
|
|
org structure). Always set `BACKUP_ENCRYPTION=age` so the artifact can
|
|
only be decrypted by the operator who holds the private identity file.
|
|
- The salt is discarded after snapshot creation; only the `sha256(salt)`
|
|
travels in the manifest. There is no way to reverse-engineer the
|
|
original PII out of an anonymized row given just the snapshot file.
|
|
- If a tenant invokes their right-to-be-forgotten on the **source**
|
|
environment after a snapshot exists, the snapshot still contains the
|
|
anonymized projection (which by design contains no resolvable PII). For
|
|
`none` snapshots, the operator must manually destroy the archive.
|
|
|
|
---
|
|
|
|
## Restore drill (recommended cadence: monthly)
|
|
|
|
1. Snapshot a representative tenant from staging:
|
|
```bash
|
|
toolctrl-cli tenant snapshot --tenant-id <staging-tenant> \
|
|
--profile full-anonymized \
|
|
--output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age
|
|
```
|
|
2. Copy to a dev box and restore into a fresh database:
|
|
```bash
|
|
NODE_ENV=development DATABASE_URL=postgres://... \
|
|
toolctrl-cli tenant restore \
|
|
--tenant-id ten_drill_$(date +%s) \
|
|
--source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \
|
|
--yes
|
|
```
|
|
3. Smoke test in the dev UI: login as a restored user, view a workflow,
|
|
confirm subscription state shows as expected.
|
|
4. Record the drill in the change log.
|
|
|
|
---
|
|
|
|
## How globals are handled in restore
|
|
|
|
Several tenant-scoped models carry foreign keys into **global** tables
|
|
(today only `User`). Earlier snapshots only emitted tenant-scoped rows,
|
|
so restoring into a truly empty target database failed for those
|
|
models with `ForeignKeyConstraintViolation`. Closed by Task G — the
|
|
extractor now harvests every referenced global id from the tenant
|
|
slice, fetches the rows, anonymizes them with the same per-snapshot
|
|
salt, and writes them to `globals/<Model>.jsonl` inside the archive.
|
|
|
|
Models that previously broke (now restore clean against an empty
|
|
target via the globals replay):
|
|
|
|
| Model | Foreign key into |
|
|
| ----------------- | ---------------- |
|
|
| `TenantUser` | `User` |
|
|
| `TenantIpAddress` | `User` |
|
|
| `UserRole` | `User` |
|
|
| `Event` | `User` |
|
|
| `Log` | `User` |
|
|
| `SupportTicket` | `User` |
|
|
|
|
The full list of `User`-referencing tenant-scoped models is
|
|
machine-enumerated in
|
|
`packages/cli/src/commands/tenant/models.ts::GLOBAL_REFERENCED_MODELS`
|
|
and asserted complete by `models.test.ts` against every prisma
|
|
schema relation.
|
|
|
|
### Restore modes for globals (`--globals=<mode>`)
|
|
|
|
The restore command accepts one of three strategies (default
|
|
`upsert`):
|
|
|
|
| Mode | Behaviour | When to use |
|
|
| -------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
|
| `upsert` | For each id, `findUnique` → if present, leave in place; else `create`. Idempotent. | Default. Merging a snapshot into a non-empty dev/staging. |
|
|
| `skip` | `createMany({skipDuplicates:true})`. Faster but silent on non-id unique conflicts. | Bulk restores where pre-existing rows are expected. |
|
|
| `error` | Same as upsert but aborts if any id already exists on target. | Sensitive restores into a verifiably-empty DB. |
|
|
|
|
```bash
|
|
# Default — overlay snapshot rows onto whatever already exists.
|
|
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
|
|
|
|
# Sensitive restore: refuse to proceed if any global is already there.
|
|
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
|
|
--globals=error
|
|
```
|
|
|
|
### Platform-catalog globals
|
|
|
|
A second category of globals — `Entity`, `EntityGroup`, `Formula`,
|
|
`Onboarding`, `PromptFlow`, `RowComment`, `WorkflowBlock` — is
|
|
referenced by tenant-scoped models but **intentionally NOT** in the
|
|
snapshot. These are seeded once per deployment via migrations/setup
|
|
scripts; copying them per-tenant would be wrong. If a tenant slice
|
|
references a platform-catalog row that does not exist on the target,
|
|
the restore fails for that specific row and you must align the target
|
|
deployment's seed first. The list is asserted by
|
|
`models.test.ts::PLATFORM_CATALOG_GLOBALS`.
|
|
|
|
### Backward compatibility
|
|
|
|
Snapshots taken before this extractor landed have no `globals` array
|
|
in their manifest. The restore detects that, logs a warning, and
|
|
falls back to the pre-Task-G behaviour where the operator must
|
|
pre-seed referenced rows manually. Use `--include-globals=false` on
|
|
`tenant:snapshot` to produce a legacy-shaped archive intentionally.
|
|
|
|
### Anonymization of global rows
|
|
|
|
`User` rows in the globals slice are scrubbed by the same anonymizer
|
|
that handles tenant rows. The `full-anonymized` profile rewrites
|
|
`email`, `firstName`, `lastName`, `phone` and `avatar`, and nulls
|
|
credentials (`passwordHash`, `verifyToken`, `githubId`, `googleId`,
|
|
`totpSecretEncrypted`, `totpPendingSecretEncrypted`,
|
|
`mfaRecoveryCodeHashes`, `deletionRequestId`) so the restored user
|
|
cannot be logged into via the original credentials. The
|
|
`minimal-anonymized` profile keeps display names but still nulls
|
|
credentials. See
|
|
`packages/cli/src/commands/tenant/anonymize.ts::FULL_ANONYMIZED_PROFILE`
|
|
for the source of truth.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Cause | Fix |
|
|
| --- | --- | --- |
|
|
| `Refusing to run --profile=none without --i-am-the-super-admin` | Guardrail working as intended. | Re-run with the flag if you are a super-admin and have a documented reason. Otherwise pick a different profile. |
|
|
| `Invalid --output: expected s3://<bucket>/<key>` | Output target malformed. | Use either a local path or a fully-qualified `s3://bucket/key/...` URL. |
|
|
| `Refusing to run tenant:restore in NODE_ENV=production` | Production restore guardrail. | Set `NODE_ENV` to a non-production value, or pass `--allow-production-restore` if you genuinely want to overwrite production. |
|
|
| `Row-count mismatch in N models` warning during restore | Target DB rejected some rows (FK or unique index conflict with pre-existing data). | Pick a fresh tenant id, or pre-clean the target DB. |
|
|
| `manifest.json: missing or invalid` | Archive truncated / wrong encryption key. | Verify download size against the S3 ETag; rerun with the correct `BACKUP_AGE_IDENTITY_FILE`. |
|
|
| Storage bucket mismatch (objects landed under wrong prefix) | `--storage-bucket` or `STORAGE_BUCKET` env mismatched between source and target. | Set the target storage bucket explicitly before re-running restore. |
|
|
| `tenant:restore aborted (--globals=error): <Model> had N pre-existing row(s)` | `--globals=error` saw an id already on target. | Re-run with `--globals=upsert` (default) to overlay or `--globals=skip` to drop conflicts. |
|
|
| `Manifest carries no globals slice (legacy snapshot)` warning | Snapshot was taken before Task G landed (or with `--include-globals=false`). | Either take a fresh snapshot, or pre-seed referenced globals on target before restoring. |
|
|
|
|
---
|
|
|
|
## Internals
|
|
|
|
- Tenant-scoped models enumerated: `packages/cli/src/commands/tenant/models.ts`
|
|
(`TENANT_SCOPED_MODELS`).
|
|
- Global-snapshot models enumerated: same file
|
|
(`GLOBAL_REFERENCED_MODELS`); platform-catalog globals on
|
|
`PLATFORM_CATALOG_GLOBALS`.
|
|
- Globals extractor: `globals-runner.ts::collectAndWriteGlobals()`.
|
|
- Profiles: `packages/cli/src/commands/tenant/anonymize.ts`.
|
|
- Manifest schema: `packages/cli/src/commands/tenant/manifest.ts`.
|
|
- Snapshot entrypoint: `runTenantSnapshot()` in `snapshot.ts`.
|
|
- Restore entrypoint: `runTenantRestore()` in `restore.ts`.
|
|
- Audit entry: `writeSnapshotAudit()` in `audit.ts`.
|