Track HiveGrid Cloud planning reference

2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
@@ -0,0 +1,389 @@
+# Self-Hosted Sentry — Operations Runbook
+
+This runbook covers the **self-hosted Sentry deployment** that lives in
+`infra/sentry/`. It is the operational companion to
+`docs/improvement-plan/02-observability/sentry-selfhost-setup.md` (the
+planning artefact) and assumes you have current operator-approved SSH access to
+the `boilerplate-staging` host. The archived
+`docs/improvement-plan/archive/STAGING-ACCESS.md` file is historical inventory
+only; prefer live Coolify/Hetzner access records when they differ.
+
+For the lightweight alternative (GlitchTip, no Performance/Replay/OTLP), see
+[`sentry-glitchtip-alternative.md`](./sentry-glitchtip-alternative.md).
+
+## At a glance
+
+| Item | Value |
+| --- | --- |
+| Upstream | `getsentry/self-hosted` tag **`25.12.1`** |
+| Public hostname | `https://sentry.boiler.saas-ctrl.com` |
+| Host | `boilerplate-staging` (`46.225.5.93`, Hetzner Nürnberg) |
+| Orchestrator | Coolify-managed Docker Compose, Traefik v3.6 in front |
+| Persistent volumes | `/data/coolify/services/sentry/{postgres,clickhouse,kafka,zookeeper,redis,symbolicator}-data` |
+| OTLP endpoint | `https://sentry.boiler.saas-ctrl.com/api/{project_id}/integration/otlp/v1/traces` |
+| Backup target | `s3://${SENTRY_S3_BACKUP_BUCKET}/sentry/YYYY-MM-DD/` (encrypted) |
+
+## 1. First-time deploy
+
+1. **Provision DNS.** Add an `A` record for
+   `sentry.boiler.saas-ctrl.com` pointing at `46.225.5.93`. TLS is
+   issued automatically by Traefik via the `letsencrypt` ACME resolver on
+   first request.
+2. **Fill secrets.** SSH to the host, then:
+
+   ```bash
+   cd /data/coolify/services/sentry
+   git clone git@git.tool-ctrl.de:toolctrl-labs/boilerplate.git .
+   cd infra/sentry
+   cp .env.example .env
+   # Edit .env — fill SENTRY_ADMIN_EMAIL, SENTRY_ADMIN_PASSWORD,
+   #             SENTRY_SECRET_KEY (generate with `python -c "import secrets; print(secrets.token_urlsafe(50))"`),
+   #             SENTRY_S3_BACKUP_*.
+   chmod 0600 .env
+   ```
+
+3. **Run the installer.** From `infra/sentry/`:
+
+   ```bash
+   ./install.sh
+   ```
+
+   The wrapper clones the pinned upstream tag, applies our overlays,
+   sources `.env`, and runs the upstream installer in non-interactive
+   mode (`--skip-user-prompt --no-report-self-hosted-issues`). Expect
+   **5–10 minutes** on first run because Snuba migrations take a while
+   on a cold ClickHouse.
+
+4. **Verify.**
+   - Web UI is reachable: `curl -I https://sentry.boiler.saas-ctrl.com`
+     returns `200`.
+   - Admin login works with the credentials from `.env`.
+   - `/auth/register/` returns **404** (public signup disabled).
+   - OTLP endpoint accepts an empty payload:
+     ```bash
+     curl -X POST "https://sentry.boiler.saas-ctrl.com/api/<PROJECT_ID>/integration/otlp/v1/traces" \
+       -H "Content-Type: application/json" \
+       -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=<PUBLIC_KEY>" \
+       -d '{"resourceSpans":[]}'
+     # Expect: 202
+     ```
+
+5. **Create the three Sentry projects** in the Sentry UI:
+   - `boilerplate-api`
+   - `boilerplate-worker`
+   - `boilerplate-web`
+
+   Copy their DSNs into `infra/sentry/.env` (`SENTRY_DSN_API`,
+   `SENTRY_DSN_WORKER`, `SENTRY_DSN_WEB`) and into the Coolify secret
+   vault for the respective app containers. The five Coolify worker apps
+   (`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`,
+   `worker-plugin`) intentionally share the server-side
+   `boilerplate-worker` project and `SENTRY_DSN_WORKER` unless the operator
+   explicitly creates per-group Sentry projects later. Per-project
+   allowed-domains list is filled in for the `boilerplate-web` project only
+   (API + Worker DSNs are server-side, no browser exposure).
+
+6. **Enable 2FA** on the admin account immediately (User Settings →
+   Security → Two-Factor → TOTP). The Hardening Checklist below treats
+   admin 2FA as **mandatory**.
+
+## 2. Coolify integration
+
+Coolify on this host runs as a **target** (the controller dashboard is on
+a separate host; verify against live operator access records, not the archived
+STAGING-ACCESS inventory). After the first manual
+`install.sh` run:
+
+- In Coolify, register the `infra/sentry/upstream/self-hosted/` Compose
+  stack as a managed service. Coolify will not re-pull `infra/sentry/` on
+  every push to `main` (the directory of overlays is read by
+  `install.sh`, not by Coolify directly).
+- Trigger `install.sh` again whenever overlays change. Suggested
+  pattern: add a `post-deployment` hook on the Coolify service that
+  runs `cd /data/coolify/services/sentry/infra/sentry && ./install.sh`
+  whenever the parent repo changes.
+
+## 3. Patch upgrade (`25.12.x → 25.12.y`)
+
+1. Read upstream release notes at
+   <https://github.com/getsentry/self-hosted/releases>.
+2. Bump `UPSTREAM_TAG` in `infra/sentry/install.sh` AND update
+   `infra/sentry/UPSTREAM.md`.
+3. If upstream renamed any `sentry.conf.py` setting we override, mirror
+   the change in `infra/sentry/sentry/sentry.conf.py`.
+4. Commit and push to `main`. Coolify auto-deploys; SSH to the host and
+   run `./install.sh` (it is idempotent and runs only the new
+   migrations).
+5. Smoke-test (Section 1, step 4).
+
+## 4. Minor upgrade (`25.x.* → 25.x+1.*`)
+
+Same procedure as Section 3, but:
+
+- Run the **backup container manually first** (`docker compose run --rm
+  backup /srv/backup/backup.sh`) and verify the dump landed in S3.
+- Read **every** intermediate release note (Sentry historically slips
+  Snuba-table migrations into minor releases).
+- After the installer completes, manually re-run the `cleanup` Django
+  command to defragment ClickHouse:
+  ```bash
+  docker compose exec web sentry cleanup --days 90
+  ```
+
+## 5. Major upgrade (`v25 → v26`)
+
+**This is a disaster-recovery-grade operation. Schedule a maintenance
+window.**
+
+1. **Pre-flight backup.** Run `docker compose run --rm backup
+   /srv/backup/backup.sh` AND `docker compose down`. Take a host-level
+   tar of `/data/coolify/services/sentry/{postgres,clickhouse}-data` and
+   copy it off-box.
+2. **Read every changelog** between the source and target tag. Pay
+   particular attention to Kafka consumer renames, Snuba table schema
+   changes, and Symbolicator API breaks.
+3. **Bump `UPSTREAM_TAG`** in `install.sh` + `UPSTREAM.md`. Update our
+   overlays for any renamed config keys.
+4. **Run install.sh on a clean clone first** (e.g.
+   `/data/coolify/services/sentry-test/`) pointed at a restored copy of
+   the production volumes. Verify the migration completes cleanly and
+   the UI loads.
+5. **Switch over.** Stop the old stack, repoint Traefik to the new
+   stack (Compose label rename), bring up the new stack with the live
+   volumes.
+6. **Rollback path.** If the new stack misbehaves, `docker compose
+   down`, restore the pre-flight volume tar over
+   `/data/coolify/services/sentry/`, redeploy the OLD `UPSTREAM_TAG` via
+   `install.sh`.
+
+## 6. Event cleanup / disk pressure
+
+B-008 is closed as an operator alerting item: Coolify disk-pressure alerting is
+enabled, so future disk-full risk is monitored rather than tracked as an active
+code blocker. The remaining responsibilities in this section are
+operator-owned Sentry/Postgres retention and backup hygiene; repository changes
+are not expected unless this runbook or the backup scripts drift.
+
+Sentry's `cleanup` command is idempotent and safe to run any time:
+
+```bash
+docker compose exec web sentry cleanup --days 90
+```
+
+Run weekly via crontab (or as a Coolify scheduled task) when retention
+should drop below 90 days. ClickHouse compacts partitions automatically
+afterwards but it can take 24h to reclaim disk.
+
+To inspect current disk usage:
+
+```bash
+docker compose exec clickhouse \
+  clickhouse-client --query "SELECT database, formatReadableSize(sum(bytes)) AS size \
+                             FROM system.parts WHERE active GROUP BY database ORDER BY sum(bytes) DESC"
+```
+
+## 7. Backup + restore drill
+
+Backups are **continuous and encrypted** (see `infra/sentry/backup/`).
+Restore drill quarterly:
+
+1. Spin up an empty Sentry stack on a scratch host.
+2. Pull the latest dump from S3:
+   ```bash
+   aws s3 cp --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" \
+     "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/postgres-*.dump.gpg" .
+   ```
+3. Decrypt: `gpg --decrypt --output postgres.dump postgres-*.dump.gpg`
+4. Restore: `pg_restore --clean --if-exists --no-owner --no-privileges -d postgres postgres.dump`
+5. Repeat for ClickHouse (`gpg --decrypt | tar -xz`, then
+   `clickhouse-client --query "INSERT INTO db.table FORMAT Native"` per
+   file).
+6. Boot the stack and verify the last day of events is visible in the
+   UI.
+
+## 8. Monitoring health
+
+Out-of-the-box checks (no external monitoring required):
+
+| Check | Command |
+| --- | --- |
+| Web up | `curl -I https://sentry.boiler.saas-ctrl.com` |
+| Relay healthy | `curl https://sentry.boiler.saas-ctrl.com/api/relay/healthcheck/ready/` |
+| Postgres reachable | `docker compose exec postgres pg_isready` |
+| ClickHouse reachable | `docker compose exec clickhouse clickhouse-client --query "SELECT 1"` |
+| Kafka topics | `docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --list` |
+| Workers running | `docker compose ps worker cron snuba-consumer` |
+| Backup heartbeat | `docker compose exec backup cat /srv/backup/logs/heartbeat.txt` |
+| Last backup landed | `aws s3 ls --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/"` |
+
+For continuous monitoring, point Coolify's Uptime/Heartbeat plugin (or
+Healthchecks.io) at the `relay/healthcheck/ready/` URL above.
+
+## 9. Volume sizing guidance
+
+With the staging host's 32 GB RAM / 75 GB disk:
+
+| Component | Disk (90 d retention, small team) | RAM |
+| --- | --- | --- |
+| Postgres | 2–5 GB | 1–2 GB |
+| ClickHouse | 20–40 GB | 4–8 GB |
+| Kafka | 5–10 GB | 1–2 GB |
+| Symbolicator cache | 2–5 GB | 0.5–1 GB |
+| Redis | <500 MB | <500 MB |
+| Sentry worker pool | n/a | 2–4 GB |
+
+If ClickHouse usage approaches 60 GB, lower retention via `sentry
+cleanup --days 30` and reduce `--days` further if needed.
+
+## Hardening Checklist
+
+The checklist below is **mandatory** before declaring this deployment
+production-ready. Each item maps to a setting we ship; verifying them is
+a one-time operator pass plus a quarterly review.
+
+- [ ] **Public signup disabled.** `sentry.conf.py` sets
+      `SENTRY_FEATURES['auth:register'] = False`. Verify:
+      `curl -I https://sentry.boiler.saas-ctrl.com/auth/register/` →
+      `404`.
+- [ ] **Admin 2FA enabled.** In the UI: User Settings → Security → 2FA
+      (TOTP). Recovery codes printed and stored offline.
+- [ ] **Default admin password rotated.** The bootstrap password from
+      `.env` is replaced via UI after first login. The `.env` value
+      becomes a recovery secret only; rotate again every 90 days.
+- [ ] **Relay rate-limit active.** `relay/config.yml` caps each project
+      at 100 events/sec. Verify by sending a burst and watching for
+      `429` responses.
+- [ ] **Per-project allowed-domains list filled.** Project Settings →
+      Client Keys → Allowed Domains. Set for `boilerplate-web` to:
+      `https://boiler.saas-ctrl.com`,
+      `https://www.boiler.saas-ctrl.com`,
+      `https://admin.boiler.saas-ctrl.com`. Leave **empty** for the
+      `boilerplate-api` and `boilerplate-worker` projects (server-side
+      DSNs only).
+- [ ] **Postgres / ClickHouse / Kafka / Redis NOT public.**
+      `docker-compose.override.yml` uses `ports: !reset []` to strip
+      host-port bindings inherited from upstream. Verify:
+      `ss -tlnp | grep -E '5432|9000|9092|6379'` on the host shows
+      **no** matches.
+- [ ] **Backups encrypted.** `backup.sh` uses GPG (recipient-key if
+      `SENTRY_BACKUP_GPG_RECIPIENT` is set, AES256 symmetric with
+      `SENTRY_SECRET_KEY` otherwise). Verify: `aws s3 cp` of a recent
+      dump returns a file with header bytes `8c 0d` (GPG-armored) or
+      `85 02` (binary GPG).
+- [ ] **Symbolicator: no external sym sources.** `sentry.conf.py` sets
+      `SYMBOLICATOR_SYMSOURCES = []`. Verify: `docker compose exec
+      symbolicator cat /etc/symbolicator/config.yml | grep sources`
+      shows no Microsoft / Mozilla / Apple entries.
+- [ ] **SENTRY_SECRET_KEY stored offline.** Lost key = unreadable
+      backups. Copy from Coolify vault to a hardware-backed password
+      manager (1Password, BitWarden Premium with TOTP, YubiKey-backed
+      pass).
+- [ ] **Traefik dashboard port 8080 firewalled.** See
+      `docs/improvement-plan/archive/STAGING-ACCESS.md` open item; close at Hetzner Cloud Firewall
+      level or front it with basic-auth.
+- [ ] **`.env` permissions ≤ 0600.** `ls -l infra/sentry/.env`. The
+      installer enforces 0600 when it copies the file into the upstream
+      tree.
+
+Re-run this checklist (a) after every patch upgrade, (b) at least once
+per quarter, (c) before opening Sentry to additional users.
+
+## CI Secrets
+
+The CI pipelines in `.github/workflows/_sentry-release.yml` and
+`.gitea/workflows/_sentry-release.yml` need three Sentry secrets to tag
+releases, upload source-maps, correlate commits, and mark deploys. The
+main CI build uploads `apps/web/build/client` as the
+`web-client-sourcemaps` workflow artifact; the reusable web release job
+downloads that artifact before calling `scripts/ci/sentry-release.sh`.
+The release job must fail if the source-map directory is absent, so a
+fresh checkout cannot silently publish a web release without artifacts.
+
+The web Vite build can also upload source-maps directly when
+`SENTRY_AUTH_TOKEN` is intentionally present in that build environment.
+The default CI path keeps the token scoped to the reusable release job
+and passes source-maps through the workflow artifact instead.
+
+### 1. Mint an Auth-Token in the Sentry UI
+
+1. Sign in at <https://sentry.boiler.saas-ctrl.com> as the admin user.
+2. Open **Settings → Account → API → Auth Tokens** (this is a
+   user-scoped token; Sentry deliberately does not expose org-scoped
+   integration tokens in single-organisation mode).
+3. Click **Create New Token**, give it a descriptive name (e.g.
+   `ci-release-token-2026-05`), and select the **single** scope
+   `project:releases`. **Do not** select `org:admin`, `org:write`, or
+   `member:admin`.
+4. Copy the token immediately — Sentry only shows it once.
+
+### 2. Store secrets in GitHub Actions
+
+Navigate to **Settings → Secrets and variables → Actions** in the
+GitHub mirror (`github.com/<org>/boilerplate`) and add three repository
+secrets:
+
+| Name | Value |
+| --- | --- |
+| `SENTRY_AUTH_TOKEN` | The token minted in step 1 |
+| `SENTRY_ORG` | `sentry` (the auto-generated slug from single-org install) |
+| `SENTRY_URL` | `https://sentry.boiler.saas-ctrl.com` |
+
+Once configured, `_sentry-release.yml` is invoked once per project
+(api/worker/web) at the end of every CI run via `workflow_call`.
+Only the web invocation passes `sourcemaps-artifact:
+web-client-sourcemaps`; api and worker releases pass `sourcemaps-path:
+none`.
+
+### 3. Store secrets in Gitea Actions
+
+In Gitea (`git.tool-ctrl.de/toolctrl-labs/boilerplate`) open
+**Repository Settings → Actions → Secrets** and create the same three
+secrets (`SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_URL`) with
+identical values. Gitea's secret scoping rules mirror GitHub's, so the
+reusable workflow at `.gitea/workflows/_sentry-release.yml` (and the
+composite-action fallback at `.gitea/actions/sentry-release/action.yml`
+for runner versions < 1.21) will pick them up automatically.
+
+The deploy workflow (`.gitea/workflows/deploy.yml`) reads the same
+secrets in its **Mark Sentry deploy** step; if the secrets are absent
+the step skips silently so existing deploy paths continue to function
+during the rollout window.
+
+### 4. Rotation
+
+- **Rotate every 6 months.** Add a recurring calendar entry for the
+  operator on-call.
+- Rotation procedure:
+  1. Mint a new token (step 1) with the same `project:releases` scope.
+  2. Update `SENTRY_AUTH_TOKEN` in **both** GitHub and Gitea secret
+     stores within the same maintenance window.
+  3. Wait for one CI run on each system to confirm the new token works
+     (look for a fresh release in the Sentry UI).
+  4. Revoke the old token via the Sentry UI (Settings → Account →
+     API → Auth Tokens → trash icon).
+- **Next scheduled rotation: 2026-11-17.** Update this date after each
+  rotation.
+
+### 5. Local development
+
+Releases are a CI-only concern. Local developers do **not** need
+`SENTRY_AUTH_TOKEN`; the web Vite build only uploads source-maps when
+the token is present, and the api/worker bootstrap modules never call
+the Releases API.
+
+If you want to test the helper locally, mint a personal token with
+`project:releases` scope and run:
+
+```bash
+SENTRY_AUTH_TOKEN=<token> \
+SENTRY_ORG=sentry \
+SENTRY_URL=https://sentry.boiler.saas-ctrl.com \
+bash scripts/ci/sentry-release.sh \
+  --project boilerplate-api \
+  --environment preview \
+  --release "$(git rev-parse HEAD)" \
+  --sourcemaps none
+```
+
+Delete the test release afterwards with
+`sentry-cli releases delete <release>`.