HiveGrid-Wireframes/hivegrid-cloud/docs/ops/tenant-snapshot.md

Tenant Snapshot — Operations Runbook

tenant:snapshot and tenant:restore ship inside packages/cli and let operators clone a single tenant's Postgres rows + S3 storage objects into a PII-anonymized archive, then replay that archive into another environment.

This runbook is the operator-facing companion to the implementation spec at docs/improvement-plan/03-scaling-infrastructure/tenant-snapshot.md.

---

When to use it

Use case	Profile	Encryption	Restore target
Bug repro ("Customer X hit a workflow bug")	full-anonymized	age	local dev box
QA fixture (representative Pro-tier tenant)	full-anonymized	age	CI seed
Sales demo (realistic but anonymized)	minimal-anonymized	age	staging
Tenant migration to a dedicated instance	none (super-admin only)	age (mandatory)	the destination environment

The default profile is full-anonymized. Never use none for any path that leaves your production network.

---

Anonymization profiles

Three profiles ship today. Picking one is a tradeoff between fidelity and privacy risk. The full mapping table lives in packages/cli/src/commands/tenant/anonymize.ts; the highlights:

Field type	full-anonymized	minimal-anonymized	none
Email	user-<hash>@anonymized.test	user-<hash>@anonymized.test	(untouched)
First / Last name	random from a curated pool	kept (display names usable in QA UI)	(untouched)
Phone number	null	null	(untouched)
IP address	127.0.0.<seeded>	127.0.0.<seeded>	(untouched)
ZIP code	first two digits kept, rest reseeded	first two digits kept, rest reseeded	(untouched)
Stripe customer / subscription / payment-method id	cus_anon<8 hex> / sub_anon<8 hex> / pm_anon<8 hex>	same as full	(untouched)
Webhook / SSO / domain URL	https://anonymized.test/<8 hex>	same as full	(untouched)
Secrets, API keys, refresh tokens, payloads	[REDACTED]	[REDACTED]	(untouched)
Avatar URL	https://anonymized.test/avatar/<8 hex>.png	same as full	(untouched)

Determinism. All anonymizers are seeded with sha256(salt + original) where salt is a fresh 32-byte random value generated per snapshot. Two rows that referenced the same original email get the same anonymized email — FK integrity inside a snapshot is preserved. The salt itself is not stored anywhere; only its sha256 lives in the manifest as a reproducibility marker.

---

CLI usage

Create a snapshot (default profile, age-encrypted, upload to S3)

toolctrl-cli tenant snapshot \
  --tenant-id ten_abc123 \
  --output s3://toolctrl-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age

The CLI requires BACKUP_AGE_PUBLIC_KEY (recipient public key) plus BACKUP_S3_* credentials. See docs/ops/backup-cli.md for the env-var catalog — the snapshot command reuses the same encryption layer.

Restore into a dev box

NODE_ENV=development toolctrl-cli tenant restore \
  --tenant-id ten_devclone \
  --source s3://toolctrl-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age

The command refuses to run under NODE_ENV=production unless the operator passes --allow-production-restore. The confirmation prompt can be silenced with --yes for non-interactive pipelines.

List existing snapshots

toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots

Super-admin escape hatch (--profile=none)

Raw, un-anonymized exports require an explicit confirmation flag:

toolctrl-cli tenant snapshot \
  --tenant-id ten_abc123 \
  --profile none \
  --i-am-the-super-admin \
  --output s3://toolctrl-snapshots/migration/ten_abc123.tar.gz.age

Without --i-am-the-super-admin, the command exits non-zero with a clear error. Every snapshot — including the none profile — emits a structured [audit] TENANT_SNAPSHOT_CREATED line to stdout. Operators are expected to redirect that into their central audit log; set auditFilePath when calling runTenantSnapshot() programmatically.

---

DSGVO / GDPR notes

• full-anonymized is the only profile that may leave EU production infrastructure. The other profiles ship un-anonymized display names (minimal) or full PII (none).
• Snapshots are themselves sensitive (business workflows, schema layout, org structure). Always set BACKUP_ENCRYPTION=age so the artifact can only be decrypted by the operator who holds the private identity file.
• The salt is discarded after snapshot creation; only the sha256(salt) travels in the manifest. There is no way to reverse-engineer the original PII out of an anonymized row given just the snapshot file.
• If a tenant invokes their right-to-be-forgotten on the source environment after a snapshot exists, the snapshot still contains the anonymized projection (which by design contains no resolvable PII). For none snapshots, the operator must manually destroy the archive.

---

Restore drill (recommended cadence: monthly)

1. Snapshot a representative tenant from staging:

   toolctrl-cli tenant snapshot --tenant-id <staging-tenant> \
     --profile full-anonymized \
     --output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age
   

2. Copy to a dev box and restore into a fresh database:

   NODE_ENV=development DATABASE_URL=postgres://... \
   toolctrl-cli tenant restore \
     --tenant-id ten_drill_$(date +%s) \
     --source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \
     --yes
   

3. Smoke test in the dev UI: login as a restored user, view a workflow, confirm subscription state shows as expected.
4. Record the drill in the change log.

---

How globals are handled in restore

Several tenant-scoped models carry foreign keys into global tables (today only User). Earlier snapshots only emitted tenant-scoped rows, so restoring into a truly empty target database failed for those models with ForeignKeyConstraintViolation. Closed by Task G — the extractor now harvests every referenced global id from the tenant slice, fetches the rows, anonymizes them with the same per-snapshot salt, and writes them to globals/<Model>.jsonl inside the archive.

Models that previously broke (now restore clean against an empty target via the globals replay):

Model	Foreign key into
TenantUser	User
TenantIpAddress	User
UserRole	User
Event	User
Log	User
SupportTicket	User

The full list of User-referencing tenant-scoped models is machine-enumerated in packages/cli/src/commands/tenant/models.ts::GLOBAL_REFERENCED_MODELS and asserted complete by models.test.ts against every prisma schema relation.

Restore modes for globals (--globals=<mode>)

The restore command accepts one of three strategies (default upsert):

Mode	Behaviour	When to use
upsert	For each id, findUnique → if present, leave in place; else create. Idempotent.	Default. Merging a snapshot into a non-empty dev/staging.
skip	createMany({skipDuplicates:true}). Faster but silent on non-id unique conflicts.	Bulk restores where pre-existing rows are expected.
error	Same as upsert but aborts if any id already exists on target.	Sensitive restores into a verifiably-empty DB.

# Default — overlay snapshot rows onto whatever already exists.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz

# Sensitive restore: refuse to proceed if any global is already there.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
  --globals=error

Platform-catalog globals

A second category of globals — Entity, EntityGroup, Formula, Onboarding, PromptFlow, RowComment, WorkflowBlock — is referenced by tenant-scoped models but intentionally NOT in the snapshot. These are seeded once per deployment via migrations/setup scripts; copying them per-tenant would be wrong. If a tenant slice references a platform-catalog row that does not exist on the target, the restore fails for that specific row and you must align the target deployment's seed first. The list is asserted by models.test.ts::PLATFORM_CATALOG_GLOBALS.

Backward compatibility

Snapshots taken before this extractor landed have no globals array in their manifest. The restore detects that, logs a warning, and falls back to the pre-Task-G behaviour where the operator must pre-seed referenced rows manually. Use --include-globals=false on tenant:snapshot to produce a legacy-shaped archive intentionally.

Anonymization of global rows

User rows in the globals slice are scrubbed by the same anonymizer that handles tenant rows. The full-anonymized profile rewrites email, firstName, lastName, phone and avatar, and nulls credentials (passwordHash, verifyToken, githubId, googleId, totpSecretEncrypted, totpPendingSecretEncrypted, mfaRecoveryCodeHashes, deletionRequestId) so the restored user cannot be logged into via the original credentials. The minimal-anonymized profile keeps display names but still nulls credentials. See packages/cli/src/commands/tenant/anonymize.ts::FULL_ANONYMIZED_PROFILE for the source of truth.

Troubleshooting

Symptom	Cause	Fix
Refusing to run --profile=none without --i-am-the-super-admin	Guardrail working as intended.	Re-run with the flag if you are a super-admin and have a documented reason. Otherwise pick a different profile.
Invalid --output: expected s3://<bucket>/<key>	Output target malformed.	Use either a local path or a fully-qualified s3://bucket/key/... URL.
Refusing to run tenant:restore in NODE_ENV=production	Production restore guardrail.	Set NODE_ENV to a non-production value, or pass --allow-production-restore if you genuinely want to overwrite production.
Row-count mismatch in N models warning during restore	Target DB rejected some rows (FK or unique index conflict with pre-existing data).	Pick a fresh tenant id, or pre-clean the target DB.
manifest.json: missing or invalid	Archive truncated / wrong encryption key.	Verify download size against the S3 ETag; rerun with the correct BACKUP_AGE_IDENTITY_FILE.
Storage bucket mismatch (objects landed under wrong prefix)	--storage-bucket or STORAGE_BUCKET env mismatched between source and target.	Set the target storage bucket explicitly before re-running restore.
tenant:restore aborted (--globals=error): <Model> had N pre-existing row(s)	--globals=error saw an id already on target.	Re-run with --globals=upsert (default) to overlay or --globals=skip to drop conflicts.
Manifest carries no globals slice (legacy snapshot) warning	Snapshot was taken before Task G landed (or with --include-globals=false).	Either take a fresh snapshot, or pre-seed referenced globals on target before restoring.

---

Internals

• Tenant-scoped models enumerated: packages/cli/src/commands/tenant/models.ts (TENANT_SCOPED_MODELS).
• Global-snapshot models enumerated: same file (GLOBAL_REFERENCED_MODELS); platform-catalog globals on PLATFORM_CATALOG_GLOBALS.
• Globals extractor: globals-runner.ts::collectAndWriteGlobals().
• Profiles: packages/cli/src/commands/tenant/anonymize.ts.
• Manifest schema: packages/cli/src/commands/tenant/manifest.ts.
• Snapshot entrypoint: runTenantSnapshot() in snapshot.ts.
• Restore entrypoint: runTenantRestore() in restore.ts.
• Audit entry: writeSnapshotAudit() in audit.ts.