# Backup CLI Runbook Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and is callable as `toolctrl-cli backup ` once the workspace is installed. ## Quick reference | Command | Purpose | | ------------------ | ------------------------------------------------------------- | | `backup:create` | Dump Postgres (+ Storage, + ClickHouse) and upload to S3. | | `backup:list` | List backups in the bucket with classification + encryption. | | `backup:rotate` | Apply daily/weekly/monthly retention. | | `backup:restore` | Download, decrypt, extract; pg_restore against a target DSN. | | `backup:verify` | Restore into a `verify_` schema and run sanity checks. | ## 1. Bucket + IAM setup The backup bucket is intentionally separate from the application storage bucket so a compromised app credential cannot pivot to the backups. ```bash # MinIO example (staging) mc mb staging-minio/toolctrl-backups mc anonymous set none staging-minio/toolctrl-backups # AWS S3 example aws s3api create-bucket --bucket toolctrl-backups \ --create-bucket-configuration LocationConstraint=us-east-1 aws s3api put-bucket-encryption --bucket toolctrl-backups \ --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}' aws s3api put-public-access-block --bucket toolctrl-backups \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true ``` Create a dedicated IAM user / access key pair with `PutObject`, `GetObject`, `ListBucket`, `DeleteObject` on the backup bucket only. Use those values for `BACKUP_S3_ACCESS_KEY_ID` + `BACKUP_S3_SECRET_ACCESS_KEY` (do **not** re-use the app `STORAGE_*` credentials). For long-term retention, enable Object Lock with COMPLIANCE mode and / or MFA delete so a rotated IAM key cannot purge history. ## 2. Encryption keys The default encryption method is [age](https://age-encryption.org). ```bash # Generate a key pair on a trusted workstation age-keygen -o ~/.age-toolctrl-backup.key # Output: # Public key: age1example... # Identity (private key) is at ~/.age-toolctrl-backup.key # Set the public key in the deployment env echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env ``` **Store the identity file (private key) somewhere the running app does not have access to** — a password manager, a tamper-evident envelope, a hardware token. The public key alone is what the create flow needs; the private key only appears in restore / verify environments via `BACKUP_AGE_IDENTITY_FILE`. If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and `BACKUP_GPG_RECIPIENT=` instead. > **Warning:** Lose the identity file and the backups are unrecoverable. The > CLI cannot bypass age encryption. ## 3. Environment template ```env # Backup bucket BACKUP_S3_ENDPOINT=https://s3.your-provider.com BACKUP_S3_REGION=us-east-1 BACKUP_S3_BUCKET=toolctrl-backups BACKUP_S3_ACCESS_KEY_ID=AKIA... BACKUP_S3_SECRET_ACCESS_KEY=... BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style # Encryption (age default) BACKUP_ENCRYPTION=age BACKUP_AGE_PUBLIC_KEY=age1example... # Restore-side only: # BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key # Retention BACKUP_RETENTION_DAILY=7 BACKUP_RETENTION_WEEKLY=4 BACKUP_RETENTION_MONTHLY=12 # Optional BACKUP_INCLUDE_CLICKHOUSE=false ``` Postgres source is picked up from `DIRECT_URL` (preferred — bypasses the PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback. ## 4. Daily cron ```cron # /etc/cron.d/toolctrl-backup SHELL=/bin/sh PATH=/usr/local/bin:/usr/bin:/bin MAILTO=ops@example.com 0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1 ``` Example wrapper: ```bash #!/usr/bin/env bash set -euo pipefail cd /opt/toolctrl node ./packages/cli/bin/toolctrl-cli.mjs backup create node ./packages/cli/bin/toolctrl-cli.mjs backup rotate ``` For docker-compose deployments: ```yaml backup-cron: image: alpine:latest volumes: - ./scripts/cron/backup-daily.sh:/script.sh:ro command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"] depends_on: [postgres] env_file: .env ``` Wire failures into your alerting channel (Sentry, Slack, PagerDuty) — an untested backup is no backup, but an unalerted failed backup is the same risk. ## 5. Disaster recovery drill Run **at least monthly**: ```bash # 1. Pick yesterday's backup toolctrl-cli backup list # 2. Verify in-place (creates verify_, drops it after) toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age # 3. End-to-end DR drill (restore into a scratch host) DATABASE_URL=postgres://dr-test/restored \ BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \ toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes ``` Document the drill outcome in the incident channel or your internal ops ledger. If the drill becomes a repository-maintained runbook, add the Markdown file first and link it from this section. ## 6. Troubleshooting | Symptom | Likely cause | Fix | | ------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `pg_dump exited with code 1` | `DIRECT_URL` missing or pgbouncer in the connection. | Set `DIRECT_URL` to the port-5432 DSN, not the pooler. | | `BACKUP_AGE_PUBLIC_KEY is required` | `BACKUP_ENCRYPTION=age` but no public key in env. | Generate via `age-keygen` and set the value. | | `age: error: failed to read header: parsing recipient` | Identity file does not match the public key. | Decrypt with the matching identity; never delete the key store. | | `clickhouse-client: command not found` | `BACKUP_INCLUDE_CLICKHOUSE=true` but no client on PATH. | Either install the client or set the flag to `false`. | | `backup:verify` reports "no tables present" | Restore ran against an empty DSN. | Confirm `DATABASE_URL` points to a non-empty DB and that pg_restore succeeded above. | ## 7. Sizing Backup size grows ≈ `DB dump size` + `Σ(storage object sizes)` (no compression on objects, only on the dump). Track via Prometheus `s3_object_size_bytes` and alert when daily delta is below historic minimum (a silent-fail symptom).