Backup CLI Runbook

Operator guide for @toolctrl/cli backup:*. The CLI ships in packages/cli and is callable as toolctrl-cli backup <subcommand> once the workspace is installed.

Quick reference

Command	Purpose
`backup:create`	Dump Postgres (+ Storage, + ClickHouse) and upload to S3.
`backup:list`	List backups in the bucket with classification + encryption.
`backup:rotate`	Apply daily/weekly/monthly retention.
`backup:restore`	Download, decrypt, extract; pg_restore against a target DSN.
`backup:verify`	Restore into a `verify_<uuid>` schema and run sanity checks.

1. Bucket + IAM setup

The backup bucket is intentionally separate from the application storage bucket so a compromised app credential cannot pivot to the backups.

# MinIO example (staging)
mc mb staging-minio/toolctrl-backups
mc anonymous set none staging-minio/toolctrl-backups

# AWS S3 example
aws s3api create-bucket --bucket toolctrl-backups \
  --create-bucket-configuration LocationConstraint=us-east-1
aws s3api put-bucket-encryption --bucket toolctrl-backups \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-public-access-block --bucket toolctrl-backups \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Create a dedicated IAM user / access key pair with PutObject, GetObject, ListBucket, DeleteObject on the backup bucket only. Use those values for BACKUP_S3_ACCESS_KEY_ID + BACKUP_S3_SECRET_ACCESS_KEY (do not re-use the app STORAGE_* credentials).

For long-term retention, enable Object Lock with COMPLIANCE mode and / or MFA delete so a rotated IAM key cannot purge history.

2. Encryption keys

The default encryption method is age.

# Generate a key pair on a trusted workstation
age-keygen -o ~/.age-toolctrl-backup.key
# Output:
# Public key: age1example...
# Identity (private key) is at ~/.age-toolctrl-backup.key

# Set the public key in the deployment env
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env

Store the identity file (private key) somewhere the running app does not have access to — a password manager, a tamper-evident envelope, a hardware token. The public key alone is what the create flow needs; the private key only appears in restore / verify environments via BACKUP_AGE_IDENTITY_FILE.

If the operator already has a GPG fleet, set BACKUP_ENCRYPTION=gpg and BACKUP_GPG_RECIPIENT=<key id> instead.

Warning: Lose the identity file and the backups are unrecoverable. The CLI cannot bypass age encryption.

3. Environment template

# Backup bucket
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
BACKUP_S3_REGION=us-east-1
BACKUP_S3_BUCKET=toolctrl-backups
BACKUP_S3_ACCESS_KEY_ID=AKIA...
BACKUP_S3_SECRET_ACCESS_KEY=...
BACKUP_S3_FORCE_PATH_STYLE=true   # MinIO/R2; AWS uses host-style

# Encryption (age default)
BACKUP_ENCRYPTION=age
BACKUP_AGE_PUBLIC_KEY=age1example...
# Restore-side only:
# BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key

# Retention
BACKUP_RETENTION_DAILY=7
BACKUP_RETENTION_WEEKLY=4
BACKUP_RETENTION_MONTHLY=12

# Optional
BACKUP_INCLUDE_CLICKHOUSE=false

Postgres source is picked up from DIRECT_URL (preferred — bypasses the PgBouncer pool so prepared statements work) or DATABASE_URL as a fallback.

4. Daily cron

# /etc/cron.d/toolctrl-backup
SHELL=/bin/sh
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=ops@example.com
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1

Example wrapper:

#!/usr/bin/env bash
set -euo pipefail
cd /opt/toolctrl
node ./packages/cli/bin/toolctrl-cli.mjs backup create
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate

For docker-compose deployments:

backup-cron:
  image: alpine:latest
  volumes:
    - ./scripts/cron/backup-daily.sh:/script.sh:ro
  command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"]
  depends_on: [postgres]
  env_file: .env

Wire failures into your alerting channel (Sentry, Slack, PagerDuty) — an untested backup is no backup, but an unalerted failed backup is the same risk.

5. Disaster recovery drill

Run at least monthly:

# 1. Pick yesterday's backup
toolctrl-cli backup list

# 2. Verify in-place (creates verify_<uuid>, drops it after)
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age

# 3. End-to-end DR drill (restore into a scratch host)
DATABASE_URL=postgres://dr-test/restored \
  BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
  toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes

Document the drill outcome in the incident channel or your internal ops ledger. If the drill becomes a repository-maintained runbook, add the Markdown file first and link it from this section.

6. Troubleshooting

Symptom	Likely cause	Fix
`pg_dump exited with code 1`	`DIRECT_URL` missing or pgbouncer in the connection.	Set `DIRECT_URL` to the port-5432 DSN, not the pooler.
`BACKUP_AGE_PUBLIC_KEY is required`	`BACKUP_ENCRYPTION=age` but no public key in env.	Generate via `age-keygen` and set the value.
`age: error: failed to read header: parsing recipient`	Identity file does not match the public key.	Decrypt with the matching identity; never delete the key store.
`clickhouse-client: command not found`	`BACKUP_INCLUDE_CLICKHOUSE=true` but no client on PATH.	Either install the client or set the flag to `false`.
`backup:verify` reports "no tables present"	Restore ran against an empty DSN.	Confirm `DATABASE_URL` points to a non-empty DB and that pg_restore succeeded above.

7. Sizing

Backup size grows ≈ DB dump size + Σ(storage object sizes) (no compression on objects, only on the dump). Track via Prometheus s3_object_size_bytes and alert when daily delta is below historic minimum (a silent-fail symptom).

6.8 KiB Raw Blame History