toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e341c5938ac25ff069732732835837de3d39d5e1
HiveGrid-Wireframes/hivegrid-cloud/docs/observability/sentry-selfhost.md
T

17 KiB
Raw Blame History

Self-Hosted Sentry — Operations Runbook

This runbook covers the self-hosted Sentry deployment that lives in infra/sentry/. It is the operational companion to docs/improvement-plan/02-observability/sentry-selfhost-setup.md (the planning artefact) and assumes you have current operator-approved SSH access to the boilerplate-staging host. The archived docs/improvement-plan/archive/STAGING-ACCESS.md file is historical inventory only; prefer live Coolify/Hetzner access records when they differ.

For the lightweight alternative (GlitchTip, no Performance/Replay/OTLP), see sentry-glitchtip-alternative.md.

At a glance

Item Value
Upstream getsentry/self-hosted tag 25.12.1
Public hostname https://sentry.boiler.saas-ctrl.com
Host boilerplate-staging (46.225.5.93, Hetzner Nürnberg)
Orchestrator Coolify-managed Docker Compose, Traefik v3.6 in front
Persistent volumes /data/coolify/services/sentry/{postgres,clickhouse,kafka,zookeeper,redis,symbolicator}-data
OTLP endpoint https://sentry.boiler.saas-ctrl.com/api/{project_id}/integration/otlp/v1/traces
Backup target s3://${SENTRY_S3_BACKUP_BUCKET}/sentry/YYYY-MM-DD/ (encrypted)

1. First-time deploy

  1. Provision DNS. Add an A record for sentry.boiler.saas-ctrl.com pointing at 46.225.5.93. TLS is issued automatically by Traefik via the letsencrypt ACME resolver on first request.

  2. Fill secrets. SSH to the host, then:

    cd /data/coolify/services/sentry
git clone git@git.tool-ctrl.de:toolctrl-labs/boilerplate.git .
cd infra/sentry
cp .env.example .env
# Edit .env — fill SENTRY_ADMIN_EMAIL, SENTRY_ADMIN_PASSWORD,
#             SENTRY_SECRET_KEY (generate with `python -c "import secrets; print(secrets.token_urlsafe(50))"`),
#             SENTRY_S3_BACKUP_*.
chmod 0600 .env

  3. Run the installer. From infra/sentry/:

    ./install.sh

    The wrapper clones the pinned upstream tag, applies our overlays, sources .env, and runs the upstream installer in non-interactive mode (--skip-user-prompt --no-report-self-hosted-issues). Expect 5–10 minutes on first run because Snuba migrations take a while on a cold ClickHouse.

  4. Verify.

    • Web UI is reachable: curl -I https://sentry.boiler.saas-ctrl.com returns 200.
    • Admin login works with the credentials from .env.
    • /auth/register/ returns 404 (public signup disabled).
    • OTLP endpoint accepts an empty payload: 
      curl -X POST "https://sentry.boiler.saas-ctrl.com/api/<PROJECT_ID>/integration/otlp/v1/traces" \
  -H "Content-Type: application/json" \
  -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=<PUBLIC_KEY>" \
  -d '{"resourceSpans":[]}'
# Expect: 202

  5. Create the three Sentry projects in the Sentry UI:

    • boilerplate-api
    • boilerplate-worker
    • boilerplate-web

    Copy their DSNs into infra/sentry/.env (SENTRY_DSN_API, SENTRY_DSN_WORKER, SENTRY_DSN_WEB) and into the Coolify secret vault for the respective app containers. The five Coolify worker apps (worker-general, worker-mail, worker-import, worker-webhook, worker-plugin) intentionally share the server-side boilerplate-worker project and SENTRY_DSN_WORKER unless the operator explicitly creates per-group Sentry projects later. Per-project allowed-domains list is filled in for the boilerplate-web project only (API + Worker DSNs are server-side, no browser exposure).

  6. Enable 2FA on the admin account immediately (User Settings → Security → Two-Factor → TOTP). The Hardening Checklist below treats admin 2FA as mandatory.

2. Coolify integration

Coolify on this host runs as a target (the controller dashboard is on a separate host; verify against live operator access records, not the archived STAGING-ACCESS inventory). After the first manual install.sh run:

  • In Coolify, register the infra/sentry/upstream/self-hosted/ Compose stack as a managed service. Coolify will not re-pull infra/sentry/ on every push to main (the directory of overlays is read by install.sh, not by Coolify directly).
  • Trigger install.sh again whenever overlays change. Suggested pattern: add a post-deployment hook on the Coolify service that runs cd /data/coolify/services/sentry/infra/sentry && ./install.sh whenever the parent repo changes.

3. Patch upgrade (25.12.x → 25.12.y)

  1. Read upstream release notes at https://github.com/getsentry/self-hosted/releases.
  2. Bump UPSTREAM_TAG in infra/sentry/install.sh AND update infra/sentry/UPSTREAM.md.
  3. If upstream renamed any sentry.conf.py setting we override, mirror the change in infra/sentry/sentry/sentry.conf.py.
  4. Commit and push to main. Coolify auto-deploys; SSH to the host and run ./install.sh (it is idempotent and runs only the new migrations).
  5. Smoke-test (Section 1, step 4).

4. Minor upgrade (25.x.* → 25.x+1.*)

Same procedure as Section 3, but:

  • Run the backup container manually first (docker compose run --rm backup /srv/backup/backup.sh) and verify the dump landed in S3.
  • Read every intermediate release note (Sentry historically slips Snuba-table migrations into minor releases).
  • After the installer completes, manually re-run the cleanup Django command to defragment ClickHouse: 
    docker compose exec web sentry cleanup --days 90

5. Major upgrade (v25 → v26)

This is a disaster-recovery-grade operation. Schedule a maintenance window.

  1. Pre-flight backup. Run docker compose run --rm backup /srv/backup/backup.sh AND docker compose down. Take a host-level tar of /data/coolify/services/sentry/{postgres,clickhouse}-data and copy it off-box.
  2. Read every changelog between the source and target tag. Pay particular attention to Kafka consumer renames, Snuba table schema changes, and Symbolicator API breaks.
  3. Bump UPSTREAM_TAG in install.sh + UPSTREAM.md. Update our overlays for any renamed config keys.
  4. Run install.sh on a clean clone first (e.g. /data/coolify/services/sentry-test/) pointed at a restored copy of the production volumes. Verify the migration completes cleanly and the UI loads.
  5. Switch over. Stop the old stack, repoint Traefik to the new stack (Compose label rename), bring up the new stack with the live volumes.
  6. Rollback path. If the new stack misbehaves, docker compose down, restore the pre-flight volume tar over /data/coolify/services/sentry/, redeploy the OLD UPSTREAM_TAG via install.sh.

6. Event cleanup / disk pressure

B-008 is closed as an operator alerting item: Coolify disk-pressure alerting is enabled, so future disk-full risk is monitored rather than tracked as an active code blocker. The remaining responsibilities in this section are operator-owned Sentry/Postgres retention and backup hygiene; repository changes are not expected unless this runbook or the backup scripts drift.

Sentry's cleanup command is idempotent and safe to run any time:

docker compose exec web sentry cleanup --days 90

Run weekly via crontab (or as a Coolify scheduled task) when retention should drop below 90 days. ClickHouse compacts partitions automatically afterwards but it can take 24h to reclaim disk.

To inspect current disk usage:

docker compose exec clickhouse \
  clickhouse-client --query "SELECT database, formatReadableSize(sum(bytes)) AS size \
                             FROM system.parts WHERE active GROUP BY database ORDER BY sum(bytes) DESC"

7. Backup + restore drill

Backups are continuous and encrypted (see infra/sentry/backup/). Restore drill quarterly:

  1. Spin up an empty Sentry stack on a scratch host.
  2. Pull the latest dump from S3: 
    aws s3 cp --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" \
  "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/postgres-*.dump.gpg" .
  3. Decrypt: gpg --decrypt --output postgres.dump postgres-*.dump.gpg
  4. Restore: pg_restore --clean --if-exists --no-owner --no-privileges -d postgres postgres.dump
  5. Repeat for ClickHouse (gpg --decrypt | tar -xz, then clickhouse-client --query "INSERT INTO db.table FORMAT Native" per file).
  6. Boot the stack and verify the last day of events is visible in the UI.

8. Monitoring health

Out-of-the-box checks (no external monitoring required):

Check Command
Web up curl -I https://sentry.boiler.saas-ctrl.com
Relay healthy curl https://sentry.boiler.saas-ctrl.com/api/relay/healthcheck/ready/
Postgres reachable docker compose exec postgres pg_isready
ClickHouse reachable docker compose exec clickhouse clickhouse-client --query "SELECT 1"
Kafka topics docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --list
Workers running docker compose ps worker cron snuba-consumer
Backup heartbeat docker compose exec backup cat /srv/backup/logs/heartbeat.txt
Last backup landed aws s3 ls --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/"

For continuous monitoring, point Coolify's Uptime/Heartbeat plugin (or Healthchecks.io) at the relay/healthcheck/ready/ URL above.

9. Volume sizing guidance

With the staging host's 32 GB RAM / 75 GB disk:

Component Disk (90 d retention, small team) RAM
Postgres 2–5 GB 1–2 GB
ClickHouse 20–40 GB 4–8 GB
Kafka 5–10 GB 1–2 GB
Symbolicator cache 2–5 GB 0.5–1 GB
Redis <500 MB <500 MB
Sentry worker pool n/a 2–4 GB

If ClickHouse usage approaches 60 GB, lower retention via sentry cleanup --days 30 and reduce --days further if needed.

Hardening Checklist

The checklist below is mandatory before declaring this deployment production-ready. Each item maps to a setting we ship; verifying them is a one-time operator pass plus a quarterly review.

  • Public signup disabled. sentry.conf.py sets SENTRY_FEATURES['auth:register'] = False. Verify: curl -I https://sentry.boiler.saas-ctrl.com/auth/register/ → 404.
  • Admin 2FA enabled. In the UI: User Settings → Security → 2FA (TOTP). Recovery codes printed and stored offline.
  • Default admin password rotated. The bootstrap password from .env is replaced via UI after first login. The .env value becomes a recovery secret only; rotate again every 90 days.
  • Relay rate-limit active. relay/config.yml caps each project at 100 events/sec. Verify by sending a burst and watching for 429 responses.
  • Per-project allowed-domains list filled. Project Settings → Client Keys → Allowed Domains. Set for boilerplate-web to: https://boiler.saas-ctrl.com, https://www.boiler.saas-ctrl.com, https://admin.boiler.saas-ctrl.com. Leave empty for the boilerplate-api and boilerplate-worker projects (server-side DSNs only).
  • Postgres / ClickHouse / Kafka / Redis NOT public. docker-compose.override.yml uses ports: !reset [] to strip host-port bindings inherited from upstream. Verify: ss -tlnp | grep -E '5432|9000|9092|6379' on the host shows no matches.
  • Backups encrypted. backup.sh uses GPG (recipient-key if SENTRY_BACKUP_GPG_RECIPIENT is set, AES256 symmetric with SENTRY_SECRET_KEY otherwise). Verify: aws s3 cp of a recent dump returns a file with header bytes 8c 0d (GPG-armored) or 85 02 (binary GPG).
  • Symbolicator: no external sym sources. sentry.conf.py sets SYMBOLICATOR_SYMSOURCES = []. Verify: docker compose exec symbolicator cat /etc/symbolicator/config.yml | grep sources shows no Microsoft / Mozilla / Apple entries.
  • SENTRY_SECRET_KEY stored offline. Lost key = unreadable backups. Copy from Coolify vault to a hardware-backed password manager (1Password, BitWarden Premium with TOTP, YubiKey-backed pass).
  • Traefik dashboard port 8080 firewalled. See docs/improvement-plan/archive/STAGING-ACCESS.md open item; close at Hetzner Cloud Firewall level or front it with basic-auth.
  • .env permissions ≤ 0600. ls -l infra/sentry/.env. The installer enforces 0600 when it copies the file into the upstream tree.

Re-run this checklist (a) after every patch upgrade, (b) at least once per quarter, (c) before opening Sentry to additional users.

CI Secrets

The CI pipelines in .github/workflows/_sentry-release.yml and .gitea/workflows/_sentry-release.yml need three Sentry secrets to tag releases, upload source-maps, correlate commits, and mark deploys. The main CI build uploads apps/web/build/client as the web-client-sourcemaps workflow artifact; the reusable web release job downloads that artifact before calling scripts/ci/sentry-release.sh. The release job must fail if the source-map directory is absent, so a fresh checkout cannot silently publish a web release without artifacts.

The web Vite build can also upload source-maps directly when SENTRY_AUTH_TOKEN is intentionally present in that build environment. The default CI path keeps the token scoped to the reusable release job and passes source-maps through the workflow artifact instead.

1. Mint an Auth-Token in the Sentry UI

  1. Sign in at https://sentry.boiler.saas-ctrl.com as the admin user.
  2. Open Settings → Account → API → Auth Tokens (this is a user-scoped token; Sentry deliberately does not expose org-scoped integration tokens in single-organisation mode).
  3. Click Create New Token, give it a descriptive name (e.g. ci-release-token-2026-05), and select the single scope project:releases. Do not select org:admin, org:write, or member:admin.
  4. Copy the token immediately — Sentry only shows it once.

2. Store secrets in GitHub Actions

Navigate to Settings → Secrets and variables → Actions in the GitHub mirror (github.com/<org>/boilerplate) and add three repository secrets:

Name Value
SENTRY_AUTH_TOKEN The token minted in step 1
SENTRY_ORG sentry (the auto-generated slug from single-org install)
SENTRY_URL https://sentry.boiler.saas-ctrl.com

Once configured, _sentry-release.yml is invoked once per project (api/worker/web) at the end of every CI run via workflow_call. Only the web invocation passes sourcemaps-artifact: web-client-sourcemaps; api and worker releases pass sourcemaps-path: none.

3. Store secrets in Gitea Actions

In Gitea (git.tool-ctrl.de/toolctrl-labs/boilerplate) open Repository Settings → Actions → Secrets and create the same three secrets (SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_URL) with identical values. Gitea's secret scoping rules mirror GitHub's, so the reusable workflow at .gitea/workflows/_sentry-release.yml (and the composite-action fallback at .gitea/actions/sentry-release/action.yml for runner versions < 1.21) will pick them up automatically.

The deploy workflow (.gitea/workflows/deploy.yml) reads the same secrets in its Mark Sentry deploy step; if the secrets are absent the step skips silently so existing deploy paths continue to function during the rollout window.

4. Rotation

  • Rotate every 6 months. Add a recurring calendar entry for the operator on-call.
  • Rotation procedure:
    1. Mint a new token (step 1) with the same project:releases scope.
    2. Update SENTRY_AUTH_TOKEN in both GitHub and Gitea secret stores within the same maintenance window.
    3. Wait for one CI run on each system to confirm the new token works (look for a fresh release in the Sentry UI).
    4. Revoke the old token via the Sentry UI (Settings → Account → API → Auth Tokens → trash icon).
  • Next scheduled rotation: 2026-11-17. Update this date after each rotation.

5. Local development

Releases are a CI-only concern. Local developers do not need SENTRY_AUTH_TOKEN; the web Vite build only uploads source-maps when the token is present, and the api/worker bootstrap modules never call the Releases API.

If you want to test the helper locally, mint a personal token with project:releases scope and run:

SENTRY_AUTH_TOKEN=<token> \
SENTRY_ORG=sentry \
SENTRY_URL=https://sentry.boiler.saas-ctrl.com \
bash scripts/ci/sentry-release.sh \
  --project boilerplate-api \
  --environment preview \
  --release "$(git rev-parse HEAD)" \
  --sourcemaps none

Delete the test release afterwards with sentry-cli releases delete <release>.

Reference in new issue View Git Blame Copy Permalink