Track HiveGrid Cloud planning reference
This commit is contained in:
david committed
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,173 @@
|
||||
# Backup CLI Runbook
|
||||
|
||||
Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and
|
||||
is callable as `toolctrl-cli backup <subcommand>` once the workspace is
|
||||
installed.
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Command | Purpose |
|
||||
| ------------------ | ------------------------------------------------------------- |
|
||||
| `backup:create` | Dump Postgres (+ Storage, + ClickHouse) and upload to S3. |
|
||||
| `backup:list` | List backups in the bucket with classification + encryption. |
|
||||
| `backup:rotate` | Apply daily/weekly/monthly retention. |
|
||||
| `backup:restore` | Download, decrypt, extract; pg_restore against a target DSN. |
|
||||
| `backup:verify` | Restore into a `verify_<uuid>` schema and run sanity checks. |
|
||||
|
||||
## 1. Bucket + IAM setup
|
||||
|
||||
The backup bucket is intentionally separate from the application storage
|
||||
bucket so a compromised app credential cannot pivot to the backups.
|
||||
|
||||
```bash
|
||||
# MinIO example (staging)
|
||||
mc mb staging-minio/toolctrl-backups
|
||||
mc anonymous set none staging-minio/toolctrl-backups
|
||||
|
||||
# AWS S3 example
|
||||
aws s3api create-bucket --bucket toolctrl-backups \
|
||||
--create-bucket-configuration LocationConstraint=us-east-1
|
||||
aws s3api put-bucket-encryption --bucket toolctrl-backups \
|
||||
--server-side-encryption-configuration \
|
||||
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
|
||||
aws s3api put-public-access-block --bucket toolctrl-backups \
|
||||
--public-access-block-configuration \
|
||||
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
|
||||
```
|
||||
|
||||
Create a dedicated IAM user / access key pair with `PutObject`,
|
||||
`GetObject`, `ListBucket`, `DeleteObject` on the backup bucket only. Use those
|
||||
values for `BACKUP_S3_ACCESS_KEY_ID` + `BACKUP_S3_SECRET_ACCESS_KEY` (do **not**
|
||||
re-use the app `STORAGE_*` credentials).
|
||||
|
||||
For long-term retention, enable Object Lock with COMPLIANCE mode and / or MFA
|
||||
delete so a rotated IAM key cannot purge history.
|
||||
|
||||
## 2. Encryption keys
|
||||
|
||||
The default encryption method is [age](https://age-encryption.org).
|
||||
|
||||
```bash
|
||||
# Generate a key pair on a trusted workstation
|
||||
age-keygen -o ~/.age-toolctrl-backup.key
|
||||
# Output:
|
||||
# Public key: age1example...
|
||||
# Identity (private key) is at ~/.age-toolctrl-backup.key
|
||||
|
||||
# Set the public key in the deployment env
|
||||
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env
|
||||
```
|
||||
|
||||
**Store the identity file (private key) somewhere the running app does not have
|
||||
access to** — a password manager, a tamper-evident envelope, a hardware token.
|
||||
The public key alone is what the create flow needs; the private key only
|
||||
appears in restore / verify environments via `BACKUP_AGE_IDENTITY_FILE`.
|
||||
|
||||
If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and
|
||||
`BACKUP_GPG_RECIPIENT=<key id>` instead.
|
||||
|
||||
> **Warning:** Lose the identity file and the backups are unrecoverable. The
|
||||
> CLI cannot bypass age encryption.
|
||||
|
||||
## 3. Environment template
|
||||
|
||||
```env
|
||||
# Backup bucket
|
||||
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
|
||||
BACKUP_S3_REGION=us-east-1
|
||||
BACKUP_S3_BUCKET=toolctrl-backups
|
||||
BACKUP_S3_ACCESS_KEY_ID=AKIA...
|
||||
BACKUP_S3_SECRET_ACCESS_KEY=...
|
||||
BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style
|
||||
|
||||
# Encryption (age default)
|
||||
BACKUP_ENCRYPTION=age
|
||||
BACKUP_AGE_PUBLIC_KEY=age1example...
|
||||
# Restore-side only:
|
||||
# BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key
|
||||
|
||||
# Retention
|
||||
BACKUP_RETENTION_DAILY=7
|
||||
BACKUP_RETENTION_WEEKLY=4
|
||||
BACKUP_RETENTION_MONTHLY=12
|
||||
|
||||
# Optional
|
||||
BACKUP_INCLUDE_CLICKHOUSE=false
|
||||
```
|
||||
|
||||
Postgres source is picked up from `DIRECT_URL` (preferred — bypasses the
|
||||
PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback.
|
||||
|
||||
## 4. Daily cron
|
||||
|
||||
```cron
|
||||
# /etc/cron.d/toolctrl-backup
|
||||
SHELL=/bin/sh
|
||||
PATH=/usr/local/bin:/usr/bin:/bin
|
||||
MAILTO=ops@example.com
|
||||
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1
|
||||
```
|
||||
|
||||
Example wrapper:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd /opt/toolctrl
|
||||
node ./packages/cli/bin/toolctrl-cli.mjs backup create
|
||||
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate
|
||||
```
|
||||
|
||||
For docker-compose deployments:
|
||||
|
||||
```yaml
|
||||
backup-cron:
|
||||
image: alpine:latest
|
||||
volumes:
|
||||
- ./scripts/cron/backup-daily.sh:/script.sh:ro
|
||||
command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"]
|
||||
depends_on: [postgres]
|
||||
env_file: .env
|
||||
```
|
||||
|
||||
Wire failures into your alerting channel (Sentry, Slack, PagerDuty) — an
|
||||
untested backup is no backup, but an unalerted failed backup is the same
|
||||
risk.
|
||||
|
||||
## 5. Disaster recovery drill
|
||||
|
||||
Run **at least monthly**:
|
||||
|
||||
```bash
|
||||
# 1. Pick yesterday's backup
|
||||
toolctrl-cli backup list
|
||||
|
||||
# 2. Verify in-place (creates verify_<uuid>, drops it after)
|
||||
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
|
||||
|
||||
# 3. End-to-end DR drill (restore into a scratch host)
|
||||
DATABASE_URL=postgres://dr-test/restored \
|
||||
BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
|
||||
toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
|
||||
```
|
||||
|
||||
Document the drill outcome in the incident channel or your internal ops ledger.
|
||||
If the drill becomes a repository-maintained runbook, add the Markdown file
|
||||
first and link it from this section.
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
| ------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
||||
| `pg_dump exited with code 1` | `DIRECT_URL` missing or pgbouncer in the connection. | Set `DIRECT_URL` to the port-5432 DSN, not the pooler. |
|
||||
| `BACKUP_AGE_PUBLIC_KEY is required` | `BACKUP_ENCRYPTION=age` but no public key in env. | Generate via `age-keygen` and set the value. |
|
||||
| `age: error: failed to read header: parsing recipient` | Identity file does not match the public key. | Decrypt with the matching identity; never delete the key store. |
|
||||
| `clickhouse-client: command not found` | `BACKUP_INCLUDE_CLICKHOUSE=true` but no client on PATH. | Either install the client or set the flag to `false`. |
|
||||
| `backup:verify` reports "no tables present" | Restore ran against an empty DSN. | Confirm `DATABASE_URL` points to a non-empty DB and that pg_restore succeeded above. |
|
||||
|
||||
## 7. Sizing
|
||||
|
||||
Backup size grows ≈ `DB dump size` + `Σ(storage object sizes)` (no
|
||||
compression on objects, only on the dump). Track via Prometheus
|
||||
`s3_object_size_bytes` and alert when daily delta is below historic minimum
|
||||
(a silent-fail symptom).
|
||||
Reference in new issue
Block a user