174 lines
6.8 KiB
Markdown
174 lines
6.8 KiB
Markdown
# Backup CLI Runbook
|
|
|
|
Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and
|
|
is callable as `toolctrl-cli backup <subcommand>` once the workspace is
|
|
installed.
|
|
|
|
## Quick reference
|
|
|
|
| Command | Purpose |
|
|
| ------------------ | ------------------------------------------------------------- |
|
|
| `backup:create` | Dump Postgres (+ Storage, + ClickHouse) and upload to S3. |
|
|
| `backup:list` | List backups in the bucket with classification + encryption. |
|
|
| `backup:rotate` | Apply daily/weekly/monthly retention. |
|
|
| `backup:restore` | Download, decrypt, extract; pg_restore against a target DSN. |
|
|
| `backup:verify` | Restore into a `verify_<uuid>` schema and run sanity checks. |
|
|
|
|
## 1. Bucket + IAM setup
|
|
|
|
The backup bucket is intentionally separate from the application storage
|
|
bucket so a compromised app credential cannot pivot to the backups.
|
|
|
|
```bash
|
|
# MinIO example (staging)
|
|
mc mb staging-minio/toolctrl-backups
|
|
mc anonymous set none staging-minio/toolctrl-backups
|
|
|
|
# AWS S3 example
|
|
aws s3api create-bucket --bucket toolctrl-backups \
|
|
--create-bucket-configuration LocationConstraint=us-east-1
|
|
aws s3api put-bucket-encryption --bucket toolctrl-backups \
|
|
--server-side-encryption-configuration \
|
|
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
|
|
aws s3api put-public-access-block --bucket toolctrl-backups \
|
|
--public-access-block-configuration \
|
|
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
|
|
```
|
|
|
|
Create a dedicated IAM user / access key pair with `PutObject`,
|
|
`GetObject`, `ListBucket`, `DeleteObject` on the backup bucket only. Use those
|
|
values for `BACKUP_S3_ACCESS_KEY_ID` + `BACKUP_S3_SECRET_ACCESS_KEY` (do **not**
|
|
re-use the app `STORAGE_*` credentials).
|
|
|
|
For long-term retention, enable Object Lock with COMPLIANCE mode and / or MFA
|
|
delete so a rotated IAM key cannot purge history.
|
|
|
|
## 2. Encryption keys
|
|
|
|
The default encryption method is [age](https://age-encryption.org).
|
|
|
|
```bash
|
|
# Generate a key pair on a trusted workstation
|
|
age-keygen -o ~/.age-toolctrl-backup.key
|
|
# Output:
|
|
# Public key: age1example...
|
|
# Identity (private key) is at ~/.age-toolctrl-backup.key
|
|
|
|
# Set the public key in the deployment env
|
|
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env
|
|
```
|
|
|
|
**Store the identity file (private key) somewhere the running app does not have
|
|
access to** — a password manager, a tamper-evident envelope, a hardware token.
|
|
The public key alone is what the create flow needs; the private key only
|
|
appears in restore / verify environments via `BACKUP_AGE_IDENTITY_FILE`.
|
|
|
|
If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and
|
|
`BACKUP_GPG_RECIPIENT=<key id>` instead.
|
|
|
|
> **Warning:** Lose the identity file and the backups are unrecoverable. The
|
|
> CLI cannot bypass age encryption.
|
|
|
|
## 3. Environment template
|
|
|
|
```env
|
|
# Backup bucket
|
|
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
|
|
BACKUP_S3_REGION=us-east-1
|
|
BACKUP_S3_BUCKET=toolctrl-backups
|
|
BACKUP_S3_ACCESS_KEY_ID=AKIA...
|
|
BACKUP_S3_SECRET_ACCESS_KEY=...
|
|
BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style
|
|
|
|
# Encryption (age default)
|
|
BACKUP_ENCRYPTION=age
|
|
BACKUP_AGE_PUBLIC_KEY=age1example...
|
|
# Restore-side only:
|
|
# BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key
|
|
|
|
# Retention
|
|
BACKUP_RETENTION_DAILY=7
|
|
BACKUP_RETENTION_WEEKLY=4
|
|
BACKUP_RETENTION_MONTHLY=12
|
|
|
|
# Optional
|
|
BACKUP_INCLUDE_CLICKHOUSE=false
|
|
```
|
|
|
|
Postgres source is picked up from `DIRECT_URL` (preferred — bypasses the
|
|
PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback.
|
|
|
|
## 4. Daily cron
|
|
|
|
```cron
|
|
# /etc/cron.d/toolctrl-backup
|
|
SHELL=/bin/sh
|
|
PATH=/usr/local/bin:/usr/bin:/bin
|
|
MAILTO=ops@example.com
|
|
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1
|
|
```
|
|
|
|
Example wrapper:
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
cd /opt/toolctrl
|
|
node ./packages/cli/bin/toolctrl-cli.mjs backup create
|
|
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate
|
|
```
|
|
|
|
For docker-compose deployments:
|
|
|
|
```yaml
|
|
backup-cron:
|
|
image: alpine:latest
|
|
volumes:
|
|
- ./scripts/cron/backup-daily.sh:/script.sh:ro
|
|
command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"]
|
|
depends_on: [postgres]
|
|
env_file: .env
|
|
```
|
|
|
|
Wire failures into your alerting channel (Sentry, Slack, PagerDuty) — an
|
|
untested backup is no backup, but an unalerted failed backup is the same
|
|
risk.
|
|
|
|
## 5. Disaster recovery drill
|
|
|
|
Run **at least monthly**:
|
|
|
|
```bash
|
|
# 1. Pick yesterday's backup
|
|
toolctrl-cli backup list
|
|
|
|
# 2. Verify in-place (creates verify_<uuid>, drops it after)
|
|
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
|
|
|
|
# 3. End-to-end DR drill (restore into a scratch host)
|
|
DATABASE_URL=postgres://dr-test/restored \
|
|
BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
|
|
toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
|
|
```
|
|
|
|
Document the drill outcome in the incident channel or your internal ops ledger.
|
|
If the drill becomes a repository-maintained runbook, add the Markdown file
|
|
first and link it from this section.
|
|
|
|
## 6. Troubleshooting
|
|
|
|
| Symptom | Likely cause | Fix |
|
|
| ------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
| `pg_dump exited with code 1` | `DIRECT_URL` missing or pgbouncer in the connection. | Set `DIRECT_URL` to the port-5432 DSN, not the pooler. |
|
|
| `BACKUP_AGE_PUBLIC_KEY is required` | `BACKUP_ENCRYPTION=age` but no public key in env. | Generate via `age-keygen` and set the value. |
|
|
| `age: error: failed to read header: parsing recipient` | Identity file does not match the public key. | Decrypt with the matching identity; never delete the key store. |
|
|
| `clickhouse-client: command not found` | `BACKUP_INCLUDE_CLICKHOUSE=true` but no client on PATH. | Either install the client or set the flag to `false`. |
|
|
| `backup:verify` reports "no tables present" | Restore ran against an empty DSN. | Confirm `DATABASE_URL` points to a non-empty DB and that pg_restore succeeded above. |
|
|
|
|
## 7. Sizing
|
|
|
|
Backup size grows ≈ `DB dump size` + `Σ(storage object sizes)` (no
|
|
compression on objects, only on the dump). Track via Prometheus
|
|
`s3_object_size_bytes` and alert when daily delta is below historic minimum
|
|
(a silent-fail symptom).
|