HiveGrid-Wireframes/hivegrid-cloud/docs/observability/sentry-selfhost.md

# Self-Hosted Sentry — Operations Runbook

This runbook covers the **self-hosted Sentry deployment** that lives in
`infra/sentry/`. It is the operational companion to
`docs/improvement-plan/02-observability/sentry-selfhost-setup.md` (the
planning artefact) and assumes you have current operator-approved SSH access to
the `boilerplate-staging` host. The archived
`docs/improvement-plan/archive/STAGING-ACCESS.md` file is historical inventory
only; prefer live Coolify/Hetzner access records when they differ.

For the lightweight alternative (GlitchTip, no Performance/Replay/OTLP), see
[`sentry-glitchtip-alternative.md`](./sentry-glitchtip-alternative.md).

## At a glance

| Item | Value |
| --- | --- |
| Upstream | `getsentry/self-hosted` tag **`25.12.1`** |
| Public hostname | `https://sentry.boiler.saas-ctrl.com` |
| Host | `boilerplate-staging` (`46.225.5.93`, Hetzner Nürnberg) |
| Orchestrator | Coolify-managed Docker Compose, Traefik v3.6 in front |
| Persistent volumes | `/data/coolify/services/sentry/{postgres,clickhouse,kafka,zookeeper,redis,symbolicator}-data` |
| OTLP endpoint | `https://sentry.boiler.saas-ctrl.com/api/{project_id}/integration/otlp/v1/traces` |
| Backup target | `s3://${SENTRY_S3_BACKUP_BUCKET}/sentry/YYYY-MM-DD/` (encrypted) |

## 1. First-time deploy

1. **Provision DNS.** Add an `A` record for
   `sentry.boiler.saas-ctrl.com` pointing at `46.225.5.93`. TLS is
   issued automatically by Traefik via the `letsencrypt` ACME resolver on
   first request.
2. **Fill secrets.** SSH to the host, then:

   ```bash
   cd /data/coolify/services/sentry
   git clone git@git.tool-ctrl.de:toolctrl-labs/boilerplate.git .
   cd infra/sentry
   cp .env.example .env
   # Edit .env — fill SENTRY_ADMIN_EMAIL, SENTRY_ADMIN_PASSWORD,
   #             SENTRY_SECRET_KEY (generate with `python -c "import secrets; print(secrets.token_urlsafe(50))"`),
   #             SENTRY_S3_BACKUP_*.
   chmod 0600 .env
   ```

3. **Run the installer.** From `infra/sentry/`:

   ```bash
   ./install.sh
   ```

   The wrapper clones the pinned upstream tag, applies our overlays,
   sources `.env`, and runs the upstream installer in non-interactive
   mode (`--skip-user-prompt --no-report-self-hosted-issues`). Expect
   **5–10 minutes** on first run because Snuba migrations take a while
   on a cold ClickHouse.

4. **Verify.**
   - Web UI is reachable: `curl -I https://sentry.boiler.saas-ctrl.com`
     returns `200`.
   - Admin login works with the credentials from `.env`.
   - `/auth/register/` returns **404** (public signup disabled).
   - OTLP endpoint accepts an empty payload:
     ```bash
     curl -X POST "https://sentry.boiler.saas-ctrl.com/api/<PROJECT_ID>/integration/otlp/v1/traces" \
       -H "Content-Type: application/json" \
       -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=<PUBLIC_KEY>" \
       -d '{"resourceSpans":[]}'
     # Expect: 202
     ```

5. **Create the three Sentry projects** in the Sentry UI:
   - `boilerplate-api`
   - `boilerplate-worker`
   - `boilerplate-web`

   Copy their DSNs into `infra/sentry/.env` (`SENTRY_DSN_API`,
   `SENTRY_DSN_WORKER`, `SENTRY_DSN_WEB`) and into the Coolify secret
   vault for the respective app containers. The five Coolify worker apps
   (`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`,
   `worker-plugin`) intentionally share the server-side
   `boilerplate-worker` project and `SENTRY_DSN_WORKER` unless the operator
   explicitly creates per-group Sentry projects later. Per-project
   allowed-domains list is filled in for the `boilerplate-web` project only
   (API + Worker DSNs are server-side, no browser exposure).

6. **Enable 2FA** on the admin account immediately (User Settings →
   Security → Two-Factor → TOTP). The Hardening Checklist below treats
   admin 2FA as **mandatory**.

## 2. Coolify integration

Coolify on this host runs as a **target** (the controller dashboard is on
a separate host; verify against live operator access records, not the archived
STAGING-ACCESS inventory). After the first manual
`install.sh` run:

- In Coolify, register the `infra/sentry/upstream/self-hosted/` Compose
  stack as a managed service. Coolify will not re-pull `infra/sentry/` on
  every push to `main` (the directory of overlays is read by
  `install.sh`, not by Coolify directly).
- Trigger `install.sh` again whenever overlays change. Suggested
  pattern: add a `post-deployment` hook on the Coolify service that
  runs `cd /data/coolify/services/sentry/infra/sentry && ./install.sh`
  whenever the parent repo changes.

## 3. Patch upgrade (`25.12.x → 25.12.y`)

1. Read upstream release notes at
   <https://github.com/getsentry/self-hosted/releases>.
2. Bump `UPSTREAM_TAG` in `infra/sentry/install.sh` AND update
   `infra/sentry/UPSTREAM.md`.
3. If upstream renamed any `sentry.conf.py` setting we override, mirror
   the change in `infra/sentry/sentry/sentry.conf.py`.
4. Commit and push to `main`. Coolify auto-deploys; SSH to the host and
   run `./install.sh` (it is idempotent and runs only the new
   migrations).
5. Smoke-test (Section 1, step 4).

## 4. Minor upgrade (`25.x.* → 25.x+1.*`)

Same procedure as Section 3, but:

- Run the **backup container manually first** (`docker compose run --rm
  backup /srv/backup/backup.sh`) and verify the dump landed in S3.
- Read **every** intermediate release note (Sentry historically slips
  Snuba-table migrations into minor releases).
- After the installer completes, manually re-run the `cleanup` Django
  command to defragment ClickHouse:
  ```bash
  docker compose exec web sentry cleanup --days 90
  ```

## 5. Major upgrade (`v25 → v26`)

**This is a disaster-recovery-grade operation. Schedule a maintenance
window.**

1. **Pre-flight backup.** Run `docker compose run --rm backup
   /srv/backup/backup.sh` AND `docker compose down`. Take a host-level
   tar of `/data/coolify/services/sentry/{postgres,clickhouse}-data` and
   copy it off-box.
2. **Read every changelog** between the source and target tag. Pay
   particular attention to Kafka consumer renames, Snuba table schema
   changes, and Symbolicator API breaks.
3. **Bump `UPSTREAM_TAG`** in `install.sh` + `UPSTREAM.md`. Update our
   overlays for any renamed config keys.
4. **Run install.sh on a clean clone first** (e.g.
   `/data/coolify/services/sentry-test/`) pointed at a restored copy of
   the production volumes. Verify the migration completes cleanly and
   the UI loads.
5. **Switch over.** Stop the old stack, repoint Traefik to the new
   stack (Compose label rename), bring up the new stack with the live
   volumes.
6. **Rollback path.** If the new stack misbehaves, `docker compose
   down`, restore the pre-flight volume tar over
   `/data/coolify/services/sentry/`, redeploy the OLD `UPSTREAM_TAG` via
   `install.sh`.

## 6. Event cleanup / disk pressure

B-008 is closed as an operator alerting item: Coolify disk-pressure alerting is
enabled, so future disk-full risk is monitored rather than tracked as an active
code blocker. The remaining responsibilities in this section are
operator-owned Sentry/Postgres retention and backup hygiene; repository changes
are not expected unless this runbook or the backup scripts drift.

Sentry's `cleanup` command is idempotent and safe to run any time:

```bash
docker compose exec web sentry cleanup --days 90
```

Run weekly via crontab (or as a Coolify scheduled task) when retention
should drop below 90 days. ClickHouse compacts partitions automatically
afterwards but it can take 24h to reclaim disk.

To inspect current disk usage:

```bash
docker compose exec clickhouse \
  clickhouse-client --query "SELECT database, formatReadableSize(sum(bytes)) AS size \
                             FROM system.parts WHERE active GROUP BY database ORDER BY sum(bytes) DESC"
```

## 7. Backup + restore drill

Backups are **continuous and encrypted** (see `infra/sentry/backup/`).
Restore drill quarterly:

1. Spin up an empty Sentry stack on a scratch host.
2. Pull the latest dump from S3:
   ```bash
   aws s3 cp --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" \
     "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/postgres-*.dump.gpg" .
   ```
3. Decrypt: `gpg --decrypt --output postgres.dump postgres-*.dump.gpg`
4. Restore: `pg_restore --clean --if-exists --no-owner --no-privileges -d postgres postgres.dump`
5. Repeat for ClickHouse (`gpg --decrypt | tar -xz`, then
   `clickhouse-client --query "INSERT INTO db.table FORMAT Native"` per
   file).
6. Boot the stack and verify the last day of events is visible in the
   UI.

## 8. Monitoring health

Out-of-the-box checks (no external monitoring required):

| Check | Command |
| --- | --- |
| Web up | `curl -I https://sentry.boiler.saas-ctrl.com` |
| Relay healthy | `curl https://sentry.boiler.saas-ctrl.com/api/relay/healthcheck/ready/` |
| Postgres reachable | `docker compose exec postgres pg_isready` |
| ClickHouse reachable | `docker compose exec clickhouse clickhouse-client --query "SELECT 1"` |
| Kafka topics | `docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --list` |
| Workers running | `docker compose ps worker cron snuba-consumer` |
| Backup heartbeat | `docker compose exec backup cat /srv/backup/logs/heartbeat.txt` |
| Last backup landed | `aws s3 ls --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/"` |

For continuous monitoring, point Coolify's Uptime/Heartbeat plugin (or
Healthchecks.io) at the `relay/healthcheck/ready/` URL above.

## 9. Volume sizing guidance

With the staging host's 32 GB RAM / 75 GB disk:

| Component | Disk (90 d retention, small team) | RAM |
| --- | --- | --- |
| Postgres | 2–5 GB | 1–2 GB |
| ClickHouse | 20–40 GB | 4–8 GB |
| Kafka | 5–10 GB | 1–2 GB |
| Symbolicator cache | 2–5 GB | 0.5–1 GB |
| Redis | <500 MB | <500 MB |
| Sentry worker pool | n/a | 2–4 GB |

If ClickHouse usage approaches 60 GB, lower retention via `sentry
cleanup --days 30` and reduce `--days` further if needed.

## Hardening Checklist

The checklist below is **mandatory** before declaring this deployment
production-ready. Each item maps to a setting we ship; verifying them is
a one-time operator pass plus a quarterly review.

- [ ] **Public signup disabled.** `sentry.conf.py` sets
      `SENTRY_FEATURES['auth:register'] = False`. Verify:
      `curl -I https://sentry.boiler.saas-ctrl.com/auth/register/` →
      `404`.
- [ ] **Admin 2FA enabled.** In the UI: User Settings → Security → 2FA
      (TOTP). Recovery codes printed and stored offline.
- [ ] **Default admin password rotated.** The bootstrap password from
      `.env` is replaced via UI after first login. The `.env` value
      becomes a recovery secret only; rotate again every 90 days.
- [ ] **Relay rate-limit active.** `relay/config.yml` caps each project
      at 100 events/sec. Verify by sending a burst and watching for
      `429` responses.
- [ ] **Per-project allowed-domains list filled.** Project Settings →
      Client Keys → Allowed Domains. Set for `boilerplate-web` to:
      `https://boiler.saas-ctrl.com`,
      `https://www.boiler.saas-ctrl.com`,
      `https://admin.boiler.saas-ctrl.com`. Leave **empty** for the
      `boilerplate-api` and `boilerplate-worker` projects (server-side
      DSNs only).
- [ ] **Postgres / ClickHouse / Kafka / Redis NOT public.**
      `docker-compose.override.yml` uses `ports: !reset []` to strip
      host-port bindings inherited from upstream. Verify:
      `ss -tlnp | grep -E '5432|9000|9092|6379'` on the host shows
      **no** matches.
- [ ] **Backups encrypted.** `backup.sh` uses GPG (recipient-key if
      `SENTRY_BACKUP_GPG_RECIPIENT` is set, AES256 symmetric with
      `SENTRY_SECRET_KEY` otherwise). Verify: `aws s3 cp` of a recent
      dump returns a file with header bytes `8c 0d` (GPG-armored) or
      `85 02` (binary GPG).
- [ ] **Symbolicator: no external sym sources.** `sentry.conf.py` sets
      `SYMBOLICATOR_SYMSOURCES = []`. Verify: `docker compose exec
      symbolicator cat /etc/symbolicator/config.yml | grep sources`
      shows no Microsoft / Mozilla / Apple entries.
- [ ] **SENTRY_SECRET_KEY stored offline.** Lost key = unreadable
      backups. Copy from Coolify vault to a hardware-backed password
      manager (1Password, BitWarden Premium with TOTP, YubiKey-backed
      pass).
- [ ] **Traefik dashboard port 8080 firewalled.** See
      `docs/improvement-plan/archive/STAGING-ACCESS.md` open item; close at Hetzner Cloud Firewall
      level or front it with basic-auth.
- [ ] **`.env` permissions ≤ 0600.** `ls -l infra/sentry/.env`. The
      installer enforces 0600 when it copies the file into the upstream
      tree.

Re-run this checklist (a) after every patch upgrade, (b) at least once
per quarter, (c) before opening Sentry to additional users.

## CI Secrets

The CI pipelines in `.github/workflows/_sentry-release.yml` and
`.gitea/workflows/_sentry-release.yml` need three Sentry secrets to tag
releases, upload source-maps, correlate commits, and mark deploys. The
main CI build uploads `apps/web/build/client` as the
`web-client-sourcemaps` workflow artifact; the reusable web release job
downloads that artifact before calling `scripts/ci/sentry-release.sh`.
The release job must fail if the source-map directory is absent, so a
fresh checkout cannot silently publish a web release without artifacts.

The web Vite build can also upload source-maps directly when
`SENTRY_AUTH_TOKEN` is intentionally present in that build environment.
The default CI path keeps the token scoped to the reusable release job
and passes source-maps through the workflow artifact instead.

### 1. Mint an Auth-Token in the Sentry UI

1. Sign in at <https://sentry.boiler.saas-ctrl.com> as the admin user.
2. Open **Settings → Account → API → Auth Tokens** (this is a
   user-scoped token; Sentry deliberately does not expose org-scoped
   integration tokens in single-organisation mode).
3. Click **Create New Token**, give it a descriptive name (e.g.
   `ci-release-token-2026-05`), and select the **single** scope
   `project:releases`. **Do not** select `org:admin`, `org:write`, or
   `member:admin`.
4. Copy the token immediately — Sentry only shows it once.

### 2. Store secrets in GitHub Actions

Navigate to **Settings → Secrets and variables → Actions** in the
GitHub mirror (`github.com/<org>/boilerplate`) and add three repository
secrets:

| Name | Value |
| --- | --- |
| `SENTRY_AUTH_TOKEN` | The token minted in step 1 |
| `SENTRY_ORG` | `sentry` (the auto-generated slug from single-org install) |
| `SENTRY_URL` | `https://sentry.boiler.saas-ctrl.com` |

Once configured, `_sentry-release.yml` is invoked once per project
(api/worker/web) at the end of every CI run via `workflow_call`.
Only the web invocation passes `sourcemaps-artifact:
web-client-sourcemaps`; api and worker releases pass `sourcemaps-path:
none`.

### 3. Store secrets in Gitea Actions

In Gitea (`git.tool-ctrl.de/toolctrl-labs/boilerplate`) open
**Repository Settings → Actions → Secrets** and create the same three
secrets (`SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_URL`) with
identical values. Gitea's secret scoping rules mirror GitHub's, so the
reusable workflow at `.gitea/workflows/_sentry-release.yml` (and the
composite-action fallback at `.gitea/actions/sentry-release/action.yml`
for runner versions < 1.21) will pick them up automatically.

The deploy workflow (`.gitea/workflows/deploy.yml`) reads the same
secrets in its **Mark Sentry deploy** step; if the secrets are absent
the step skips silently so existing deploy paths continue to function
during the rollout window.

### 4. Rotation

- **Rotate every 6 months.** Add a recurring calendar entry for the
  operator on-call.
- Rotation procedure:
  1. Mint a new token (step 1) with the same `project:releases` scope.
  2. Update `SENTRY_AUTH_TOKEN` in **both** GitHub and Gitea secret
     stores within the same maintenance window.
  3. Wait for one CI run on each system to confirm the new token works
     (look for a fresh release in the Sentry UI).
  4. Revoke the old token via the Sentry UI (Settings → Account →
     API → Auth Tokens → trash icon).
- **Next scheduled rotation: 2026-11-17.** Update this date after each
  rotation.

### 5. Local development

Releases are a CI-only concern. Local developers do **not** need
`SENTRY_AUTH_TOKEN`; the web Vite build only uploads source-maps when
the token is present, and the api/worker bootstrap modules never call
the Releases API.

If you want to test the helper locally, mint a personal token with
`project:releases` scope and run:

```bash
SENTRY_AUTH_TOKEN=<token> \
SENTRY_ORG=sentry \
SENTRY_URL=https://sentry.boiler.saas-ctrl.com \
bash scripts/ci/sentry-release.sh \
  --project boilerplate-api \
  --environment preview \
  --release "$(git rev-parse HEAD)" \
  --sourcemaps none
```

Delete the test release afterwards with
`sentry-cli releases delete <release>`.