Track HiveGrid Cloud planning reference

This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3

No files matched your search

+52
View File
@@ -0,0 +1,52 @@
# infra/sentry/.env.example
#
# Copy to `.env` and fill before running ./install.sh. Never commit the real
# .env — it is git-ignored (see .gitignore). Use Coolify's secret vault to
# manage the production values; this file is only the template.
# -----------------------------------------------------------------------------
# Public URL — must match the Traefik-routed hostname in docker-compose.override.yml
# -----------------------------------------------------------------------------
SENTRY_BASE_URL=https://sentry.boiler.saas-ctrl.com
# -----------------------------------------------------------------------------
# Admin / superuser bootstrap
# -----------------------------------------------------------------------------
# install.sh seeds this user on first run via `sentry createuser --superuser`.
# Enable 2FA in the UI immediately after first login (Hardening Checklist).
SENTRY_ADMIN_EMAIL=ops@example.com
SENTRY_ADMIN_PASSWORD=replace-me-strong-password
# -----------------------------------------------------------------------------
# Sentry secret key
# -----------------------------------------------------------------------------
# Generate ONCE with: python -c "import secrets; print(secrets.token_urlsafe(50))"
# Losing this key invalidates all session cookies AND makes encrypted backups
# unreadable. Store it both in Coolify secrets AND in an offline password vault.
SENTRY_SECRET_KEY=replace-me-with-50-chars-of-urlsafe-entropy
# -----------------------------------------------------------------------------
# Project DSNs (filled AFTER first deploy — read from the Sentry UI)
# -----------------------------------------------------------------------------
# Three projects are created post-install: API, Worker, Web. Their DSNs feed
# into the apps' own .env via Coolify (see Phase 02 tasks #30 / #31).
SENTRY_DSN_API=
SENTRY_DSN_WORKER=
SENTRY_DSN_WEB=
# -----------------------------------------------------------------------------
# S3 backup target (consumed by infra/sentry/backup/)
# -----------------------------------------------------------------------------
# Endpoint format examples:
# AWS: https://s3.eu-central-1.amazonaws.com
# Hetzner: https://nbg1.your-objectstorage.com
# MinIO local: https://minio.boiler.saas-ctrl.com
SENTRY_S3_BACKUP_ENDPOINT=https://nbg1.your-objectstorage.com
SENTRY_S3_BACKUP_BUCKET=boilerplate-sentry-backups
SENTRY_S3_BACKUP_ACCESS_KEY=replace-me
SENTRY_S3_BACKUP_SECRET_KEY=replace-me
# Optional: GPG recipient fingerprint to encrypt the tarballs with before upload.
# If left empty, backup.sh falls back to AES256 with the SENTRY_SECRET_KEY as
# symmetric passphrase (still better than plaintext, but rotate carefully).
SENTRY_BACKUP_GPG_RECIPIENT=
+21
View File
@@ -0,0 +1,21 @@
# Runtime data — never commit
data/
postgres-data/
clickhouse-data/
kafka-data/
zookeeper-data/
redis-data/
symbolicator-data/
relay-data/
# Secrets
*.env
!.env.example
secrets/
# Upstream snapshot is cloned at deploy time, never tracked
upstream/self-hosted/
# Local helper artifacts
*.log
.DS_Store
+68
View File
@@ -0,0 +1,68 @@
# `infra/sentry` — Self-Hosted Sentry Stack
This directory contains the artifacts to deploy a **self-hosted Sentry** instance
on the staging Hetzner host (`boilerplate-staging`, see
`docs/improvement-plan/archive/STAGING-ACCESS.md`) via the existing Coolify-managed
Docker-Compose runtime.
## Purpose
- Aggregate **server-side errors** from the API + Worker containers
- Aggregate **browser errors** (incl. Replay + Performance) from the Web app
- Ingest **OpenTelemetry traces** via the Relay OTLP-HTTP endpoint
(consumed in Phase 02 task #33)
- Provide a DSGVO-compliant, EU-hosted alternative to SaaS Sentry
The companion light-weight alternative (GlitchTip) is documented in
`docs/observability/sentry-glitchtip-alternative.md` and is **not** the default —
it lacks full Performance / Replay / OTLP support.
## Prerequisites
- Hetzner host with **≥ 16 GB RAM, ≥ 4 vCPU, ≥ 50 GB free disk**
(Clickhouse + Kafka + Symbolicator are RAM-hungry; the staging host now has
32 GB RAM after the recent upgrade)
- Docker Engine ≥ 24 + Docker Compose v2 (provided by Coolify)
- Traefik v3.6 reverse-proxy from `coolify-proxy` (Let's Encrypt ACME via
`letsencrypt` cert resolver)
- DNS-record `sentry.boiler.saas-ctrl.com` → `46.225.5.93`
## Layout
```
infra/sentry/
├── README.md # this file
├── .gitignore # excludes runtime volumes + secrets
├── UPSTREAM.md # pinned getsentry/self-hosted version + clone strategy
├── upstream/ # populated at deploy time by install.sh (NOT committed)
│ └── .gitkeep
├── docker-compose.override.yml # overlay merged on top of upstream compose
├── sentry/sentry.conf.py # python-config overlay (single-org, no-signup, OTLP)
├── relay/config.yml # Relay OTLP-ingest config
├── traefik.labels.yml # Compose-fragment with Traefik labels
├── install.sh # wrapper that clones upstream + applies overlays + installs
├── .env.example # template for SENTRY_* secrets
└── backup/ # daily backup container
├── Dockerfile
├── backup.sh
└── crontab
```
## Deploy
See `docs/observability/sentry-selfhost.md` (full deploy + upgrade + backup
runbook). High-level:
1. Copy `.env.example` → `.env`, fill secrets (Coolify-Vault).
2. Run `./install.sh` (clones upstream at pinned tag, applies overlays,
installs).
3. In Coolify, attach this stack to the `sentry.boiler.saas-ctrl.com` domain.
4. Smoke-test per `## Test Requirements` in
`docs/improvement-plan/02-observability/sentry-selfhost-setup.md`.
## Versioning
- Upstream pin: `getsentry/self-hosted` tag **`25.12.1`** (see `UPSTREAM.md` for
rationale — skipping the buggy `25.12.0` migration release; staying on the
v25.x LTS-ish line until we manually qualify v26.x).
- GlitchTip alternative pin: `glitchtip/glitchtip:v6.1.6`.
+88
View File
@@ -0,0 +1,88 @@
# Upstream Pin: `getsentry/self-hosted`
## Pinned Version
| Field | Value |
| --- | --- |
| Upstream repo | <https://github.com/getsentry/self-hosted> |
| Pinned git tag | **`25.12.1`** |
| Release line | `v25.x` (LTS-ish; we treat the v25.x line as our long-term track until v26.x is qualified) |
| Why not `25.12.0` | `25.12.0` shipped a known migration bug in the `snuba-api` migration step that aborts the installer mid-way; `25.12.1` is the hotfix. |
| Why not `26.x` | v26.x is the current major and introduces breaking schema changes (Snuba consumers + ClickHouse-table renames). We intentionally trail one major behind to land changes after they have been hot-fixed in the wild. |
| Image-tag mapping | All Sentry component images (`sentry`, `snuba`, `relay`, `symbolicator`, etc.) pull the `25.12.1` tags as defined in the upstream Compose. |
| Companion GlitchTip pin | `glitchtip/glitchtip:v6.1.6` (see `docs/observability/sentry-glitchtip-alternative.md`) — used only when the full Sentry stack is too heavy. |
## Why we do NOT vendor the upstream snapshot
The upstream `getsentry/self-hosted` repository at `25.12.1` contains **~70
services** in its base Compose plus a non-trivial install script that
bootstraps Postgres/ClickHouse/Kafka migrations. Vendoring the whole tree
would mean:
- ~3 MB of YAML + Python config we do not own
- Every minor upgrade becomes a giant merge-conflict
- The license/attribution surface grows substantially
- Coolify's git-pull-on-deploy gets slower
Instead, we **reference** the upstream repo at the pinned ref and clone it
at deploy time inside our own wrapper. Our repository only contains the
**overlay** we want on top of upstream:
- `docker-compose.override.yml` — Compose merge-overlay (network, volumes,
Traefik labels, env-passthrough)
- `sentry/sentry.conf.py` — Python config overlay (single-org, no-signup,
OTLP, no external symbol sources)
- `relay/config.yml` — Relay config with OTLP ingest enabled
- `backup/` — sidecar container that backs up Postgres + ClickHouse to S3
- `install.sh` — wrapper that clones upstream + applies overlays + invokes
the upstream installer
## Clone strategy (what `install.sh` does)
The wrapper performs the equivalent of:
```bash
# Clone exactly the pinned tag into infra/sentry/upstream/self-hosted/,
# shallow + single-branch to keep the working tree lean.
git clone \
--branch 25.12.1 \
--depth 1 \
--single-branch \
https://github.com/getsentry/self-hosted.git \
infra/sentry/upstream/self-hosted
```
After cloning, the wrapper:
1. Copies `infra/sentry/docker-compose.override.yml` →
`infra/sentry/upstream/self-hosted/docker-compose.override.yml`
2. Copies `infra/sentry/sentry/sentry.conf.py` →
`infra/sentry/upstream/self-hosted/sentry/sentry.conf.py` (overwrites the
upstream default; ours is a strict superset)
3. Copies `infra/sentry/relay/config.yml` →
`infra/sentry/upstream/self-hosted/relay/config.yml`
4. Sources `.env` from the repo root of `infra/sentry/`
5. Executes the upstream installer in non-interactive mode:
`./install.sh --skip-user-prompt --no-report-self-hosted-issues`
The `infra/sentry/upstream/` directory is **git-ignored** (see
`.gitignore`), so the cloned tree never lands in our history. The empty
placeholder `upstream/.gitkeep` exists so the directory is present in the
repo for a fresh checkout but stays empty until the first `install.sh` run.
## Update procedure
To upgrade to a newer `25.x.y` patch:
1. Update the tag in this file (`UPSTREAM.md`) and in `install.sh`.
2. Read upstream release notes for any `sentry.conf.py` migrations and
sync our overlay if upstream renamed/removed any setting we override.
3. On staging, run `./install.sh` again. The upstream installer is
idempotent: it detects the existing volumes, runs only the new
migrations, then restarts services.
4. Smoke-test (web-UI, event-ingest, OTLP-ingest, backup container).
5. Commit the bumped tag + overlay deltas as one commit.
For the eventual `25.x → 26.x` jump see the runbook in
`docs/observability/sentry-selfhost.md` — that one is a manual
disaster-recovery-grade operation, not a `./install.sh` re-run.
@@ -0,0 +1,53 @@
# infra/sentry/backup/Dockerfile
#
# Sidecar container that backs up the self-hosted Sentry stack daily:
# - pg_dump --format=custom of the sentry Postgres
# - clickhouse-backup snapshot uploaded directly to S3
# - both artifacts encrypted with GPG (or AES256 fallback) before upload
#
# Triggered by an internal cron (see crontab) so the container stays running
# in the background. Re-uses the upstream Postgres + ClickHouse client tools
# at versions matching the Sentry stack (Postgres 14, ClickHouse 23.8).
FROM alpine:3.20
# --- Runtime deps ------------------------------------------------------------
# postgresql-client -> pg_dump
# clickhouse-client -> clickhouse-client
# aws-cli -> S3 upload (works with Hetzner Object Storage + MinIO via --endpoint-url)
# rclone -> alternative S3 backend (handy for MinIO-compat that AWS-CLI v2 dislikes)
# gnupg -> GPG encryption of dumps
# tini -> proper signal-handling as PID 1
# bash, coreutils -> POSIX-strict scripts need real coreutils
# tzdata -> consistent UTC timestamps
# busybox-extras (cron) -> crond
RUN apk add --no-cache \
postgresql16-client \
clickhouse-client \
aws-cli \
rclone \
gnupg \
tini \
bash \
coreutils \
tzdata \
busybox-extras \
curl \
&& rm -rf /var/cache/apk/*
ENV TZ=UTC \
BACKUP_HOME=/srv/backup \
PATH=/srv/backup:$PATH
WORKDIR ${BACKUP_HOME}
COPY backup.sh ${BACKUP_HOME}/backup.sh
COPY crontab /etc/crontabs/root
RUN chmod 0755 ${BACKUP_HOME}/backup.sh \
&& mkdir -p ${BACKUP_HOME}/logs ${BACKUP_HOME}/work
# Run cron in the foreground; tini handles SIGTERM cleanly so docker compose
# down doesn't strand a half-finished dump.
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["crond", "-f", "-L", "/dev/stdout", "-l", "2"]
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env bash
# infra/sentry/backup/backup.sh
#
# Daily backup of the self-hosted Sentry stack. Triggered by crond at 03:30 UTC
# (see crontab). Idempotent — re-running on the same day overwrites the same
# S3 prefix.
#
# Env (sourced from the Compose service `environment` block which itself reads
# from infra/sentry/.env):
# SENTRY_S3_BACKUP_ENDPOINT S3 endpoint URL (Hetzner / MinIO / AWS)
# SENTRY_S3_BACKUP_BUCKET target bucket name
# SENTRY_S3_BACKUP_ACCESS_KEY S3 access key
# SENTRY_S3_BACKUP_SECRET_KEY S3 secret key
# SENTRY_SECRET_KEY fallback symmetric passphrase for GPG
# SENTRY_BACKUP_GPG_RECIPIENT optional GPG recipient fingerprint
#
# SENTRY_PG_HOST default "postgres" (upstream service name)
# SENTRY_PG_PORT default 5432
# SENTRY_PG_USER default "postgres"
# SENTRY_PG_DB default "postgres"
# SENTRY_PG_PASSWORD from PGPASSWORD or env
#
# SENTRY_CH_HOST default "clickhouse"
# SENTRY_CH_PORT default 9000
# SENTRY_CH_USER default "default"
# SENTRY_CH_PASSWORD optional
set -Eeuo pipefail
shopt -s inherit_errexit 2>/dev/null || true
# -----------------------------------------------------------------------------
# Defaults + logging
# -----------------------------------------------------------------------------
: "${SENTRY_PG_HOST:=postgres}"
: "${SENTRY_PG_PORT:=5432}"
: "${SENTRY_PG_USER:=postgres}"
: "${SENTRY_PG_DB:=postgres}"
: "${SENTRY_CH_HOST:=clickhouse}"
: "${SENTRY_CH_PORT:=9000}"
: "${SENTRY_CH_USER:=default}"
: "${SENTRY_CH_PASSWORD:=}"
: "${SENTRY_S3_BACKUP_ENDPOINT:?SENTRY_S3_BACKUP_ENDPOINT is required}"
: "${SENTRY_S3_BACKUP_BUCKET:?SENTRY_S3_BACKUP_BUCKET is required}"
: "${SENTRY_S3_BACKUP_ACCESS_KEY:?SENTRY_S3_BACKUP_ACCESS_KEY is required}"
: "${SENTRY_S3_BACKUP_SECRET_KEY:?SENTRY_S3_BACKUP_SECRET_KEY is required}"
TODAY="$(date -u +%Y-%m-%d)"
NOW="$(date -u +%Y-%m-%dT%H-%M-%SZ)"
WORK="${BACKUP_HOME:-/srv/backup}/work/${TODAY}"
mkdir -p "${WORK}"
log() { printf "[backup %s] %s\n" "${NOW}" "$*"; }
cleanup() { rm -rf "${WORK}"; }
trap cleanup EXIT
# -----------------------------------------------------------------------------
# Step 1: Postgres dump
# -----------------------------------------------------------------------------
log "dumping Postgres (${SENTRY_PG_HOST}/${SENTRY_PG_DB})"
PG_DUMP_FILE="${WORK}/postgres-${NOW}.dump"
export PGPASSWORD="${SENTRY_PG_PASSWORD:-${PGPASSWORD:-}}"
pg_dump \
--host "${SENTRY_PG_HOST}" \
--port "${SENTRY_PG_PORT}" \
--username "${SENTRY_PG_USER}" \
--dbname "${SENTRY_PG_DB}" \
--format=custom \
--compress=9 \
--no-owner \
--no-privileges \
--file "${PG_DUMP_FILE}"
log "postgres dump size: $(du -h "${PG_DUMP_FILE}" | cut -f1)"
# -----------------------------------------------------------------------------
# Step 2: ClickHouse snapshot
# -----------------------------------------------------------------------------
log "snapshotting ClickHouse (${SENTRY_CH_HOST})"
CH_DUMP_FILE="${WORK}/clickhouse-${NOW}.tar.gz"
# Use clickhouse-client to BACKUP all sentry tables to /var/lib/clickhouse/backup,
# then archive the directory. The `BACKUP ... TO Disk('backups', ...)` syntax
# is preferred but requires backup-disk config; we fall back to per-database
# SELECT...INTO OUTFILE for portability.
CH_TMP="${WORK}/clickhouse"
mkdir -p "${CH_TMP}"
DATABASES="$(clickhouse-client \
--host "${SENTRY_CH_HOST}" \
--port "${SENTRY_CH_PORT}" \
--user "${SENTRY_CH_USER}" \
${SENTRY_CH_PASSWORD:+--password "${SENTRY_CH_PASSWORD}"} \
--query="SHOW DATABASES" | grep -Ev '^(system|INFORMATION_SCHEMA|information_schema|default)$' || true)"
for db in ${DATABASES}; do
log " -> dumping clickhouse db: ${db}"
TABLES="$(clickhouse-client \
--host "${SENTRY_CH_HOST}" \
--port "${SENTRY_CH_PORT}" \
--user "${SENTRY_CH_USER}" \
${SENTRY_CH_PASSWORD:+--password "${SENTRY_CH_PASSWORD}"} \
--query="SHOW TABLES FROM ${db}")"
mkdir -p "${CH_TMP}/${db}"
for tbl in ${TABLES}; do
clickhouse-client \
--host "${SENTRY_CH_HOST}" \
--port "${SENTRY_CH_PORT}" \
--user "${SENTRY_CH_USER}" \
${SENTRY_CH_PASSWORD:+--password "${SENTRY_CH_PASSWORD}"} \
--query="SELECT * FROM ${db}.${tbl} FORMAT Native" \
> "${CH_TMP}/${db}/${tbl}.native" 2>/dev/null || true
done
done
tar -czf "${CH_DUMP_FILE}" -C "${WORK}" clickhouse
log "clickhouse archive size: $(du -h "${CH_DUMP_FILE}" | cut -f1)"
# -----------------------------------------------------------------------------
# Step 3: encrypt
# -----------------------------------------------------------------------------
encrypt() {
local src="$1"
local dst="${src}.gpg"
if [[ -n "${SENTRY_BACKUP_GPG_RECIPIENT:-}" ]]; then
log "encrypting ${src##*/} to GPG recipient ${SENTRY_BACKUP_GPG_RECIPIENT}"
gpg --batch --yes --trust-model always --recipient "${SENTRY_BACKUP_GPG_RECIPIENT}" \
--output "${dst}" --encrypt "${src}"
else
log "encrypting ${src##*/} symmetrically (AES256) with SENTRY_SECRET_KEY"
gpg --batch --yes --symmetric --cipher-algo AES256 \
--passphrase "${SENTRY_SECRET_KEY:?SENTRY_SECRET_KEY missing for symmetric fallback}" \
--output "${dst}" "${src}"
fi
rm -f "${src}"
printf "%s" "${dst}"
}
PG_ENCRYPTED="$(encrypt "${PG_DUMP_FILE}")"
CH_ENCRYPTED="$(encrypt "${CH_DUMP_FILE}")"
# -----------------------------------------------------------------------------
# Step 4: upload to S3
# -----------------------------------------------------------------------------
export AWS_ACCESS_KEY_ID="${SENTRY_S3_BACKUP_ACCESS_KEY}"
export AWS_SECRET_ACCESS_KEY="${SENTRY_S3_BACKUP_SECRET_KEY}"
export AWS_EC2_METADATA_DISABLED=true
S3_PREFIX="s3://${SENTRY_S3_BACKUP_BUCKET}/sentry/${TODAY}"
log "uploading to ${S3_PREFIX}/"
aws s3 cp --endpoint-url "${SENTRY_S3_BACKUP_ENDPOINT}" "${PG_ENCRYPTED}" "${S3_PREFIX}/$(basename "${PG_ENCRYPTED}")"
aws s3 cp --endpoint-url "${SENTRY_S3_BACKUP_ENDPOINT}" "${CH_ENCRYPTED}" "${S3_PREFIX}/$(basename "${CH_ENCRYPTED}")"
log "backup complete for ${TODAY}"
@@ -0,0 +1,13 @@
# infra/sentry/backup/crontab
#
# Daily Sentry backup. Times are UTC (TZ set in the Dockerfile).
# Format: m h dom mon dow command
#
# 03:30 UTC = quiet hour for both EU/US active windows. After Sentry's own
# nightly cleanup job (default 01:00) and before any morning maintenance.
30 3 * * * /srv/backup/backup.sh >> /srv/backup/logs/backup-$(date -u +\%Y\%m\%d).log 2>&1
# Heartbeat — write a tiny "alive" marker hourly so operators can confirm
# crond is actually running inside the container. Keeps the last 24h of files.
0 * * * * date -u +%Y-%m-%dT%H:%M:%SZ > /srv/backup/logs/heartbeat.txt
@@ -0,0 +1,118 @@
# Compose overlay for the upstream getsentry/self-hosted stack pinned at
# 25.12.1. This file is copied into `upstream/self-hosted/` by `install.sh`
# and merged on top of the upstream `docker-compose.yml`.
#
# Scope of overrides:
# 1. Pin all volumes onto Coolify's host-mount path (`/data/coolify/services/sentry/...`)
# so backups + log-rotation use the same disk plane as every other Coolify stack.
# 2. Attach the `nginx` (HTTP entry) service to Coolify's external `coolify` network
# and add Traefik labels (see traefik.labels.yml for the canonical label set).
# 3. Enforce SSL-aware behavior (TLS terminated by Traefik in front of `nginx`).
# 4. Pass through the project-specific env-vars from `.env`.
# 5. Keep Postgres / ClickHouse / Kafka / Redis on the internal network ONLY —
# we never expose them with a `ports:` mapping (see Security Notes in detail-MD).
#
# Anything not mentioned here inherits from upstream. Do NOT redeclare full services;
# Compose merge handles partials cleanly.
version: "3.8"
# Coolify-managed external network. Created by Coolify itself; we just join it.
networks:
default:
# Internal Sentry-only network; upstream calls this `default` already.
name: sentry-net
coolify:
external: true
services:
# ----- HTTP entry point: Traefik-routed, no host port-mapping -----
nginx:
networks:
- default
- coolify
# Remove any host port-publish from upstream — Traefik handles ingress.
ports: !reset []
environment:
SENTRY_USE_SSL: "1"
labels:
# Canonical Traefik labels live in traefik.labels.yml. Coolify also injects
# its own labels; the ones below are the project-specific routing rules.
- "traefik.enable=true"
- "traefik.docker.network=coolify"
- "traefik.http.routers.sentry.rule=Host(`sentry.boiler.saas-ctrl.com`)"
- "traefik.http.routers.sentry.entrypoints=https"
- "traefik.http.routers.sentry.tls=true"
- "traefik.http.routers.sentry.tls.certresolver=letsencrypt"
- "traefik.http.services.sentry.loadbalancer.server.port=80"
# HTTP -> HTTPS redirect
- "traefik.http.routers.sentry-http.rule=Host(`sentry.boiler.saas-ctrl.com`)"
- "traefik.http.routers.sentry-http.entrypoints=http"
- "traefik.http.routers.sentry-http.middlewares=sentry-https-redirect"
- "traefik.http.middlewares.sentry-https-redirect.redirectscheme.scheme=https"
# ----- Sentry web (Django) -----
web:
environment:
SENTRY_USE_SSL: "1"
SENTRY_SINGLE_ORGANIZATION: "true"
SENTRY_ADMIN_EMAIL: ${SENTRY_ADMIN_EMAIL:?must be set in .env}
# Mount our overlay sentry.conf.py on top of the upstream default so the
# hardened settings (no public signup, OTLP feature flag, empty symbol
# sources) are loaded at process start.
volumes:
- ./sentry/sentry.conf.py:/etc/sentry/sentry.conf.py:ro
# ----- Workers + cron share the same image; same overlay applies -----
worker:
environment:
SENTRY_USE_SSL: "1"
SENTRY_SINGLE_ORGANIZATION: "true"
volumes:
- ./sentry/sentry.conf.py:/etc/sentry/sentry.conf.py:ro
cron:
environment:
SENTRY_USE_SSL: "1"
SENTRY_SINGLE_ORGANIZATION: "true"
volumes:
- ./sentry/sentry.conf.py:/etc/sentry/sentry.conf.py:ro
# ----- Relay: enable OTLP via custom config.yml overlay -----
relay:
volumes:
- ./relay/config.yml:/etc/relay/config.yml:ro
# ----- Postgres: pin onto Coolify-disk volume, NO host port -----
postgres:
ports: !reset []
volumes:
- /data/coolify/services/sentry/postgres-data:/var/lib/postgresql/data
# ----- ClickHouse: pin onto Coolify-disk volume, NO host port -----
clickhouse:
ports: !reset []
volumes:
- /data/coolify/services/sentry/clickhouse-data:/var/lib/clickhouse
# ----- Kafka + Zookeeper: internal only -----
kafka:
ports: !reset []
volumes:
- /data/coolify/services/sentry/kafka-data:/var/lib/kafka/data
zookeeper:
ports: !reset []
volumes:
- /data/coolify/services/sentry/zookeeper-data:/var/lib/zookeeper/data
# ----- Redis: internal only -----
redis:
ports: !reset []
volumes:
- /data/coolify/services/sentry/redis-data:/data
# ----- Symbolicator: no external sym sources (see sentry.conf.py) -----
symbolicator:
volumes:
- /data/coolify/services/sentry/symbolicator-data:/data
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# infra/sentry/install.sh
#
# Wrapper around the upstream getsentry/self-hosted installer pinned at
# 25.12.1 (see UPSTREAM.md). Run this script on the staging Hetzner host
# from the repository root or from `infra/sentry/`:
#
# cd infra/sentry && ./install.sh
#
# The script is idempotent — re-running it picks up new overlay changes and
# re-applies migrations. Volumes under /data/coolify/services/sentry/ are
# preserved across runs.
set -Eeuo pipefail
shopt -s inherit_errexit 2>/dev/null || true
# -----------------------------------------------------------------------------
# Config
# -----------------------------------------------------------------------------
UPSTREAM_REPO="https://github.com/getsentry/self-hosted.git"
UPSTREAM_TAG="25.12.1"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
UPSTREAM_DIR="${SCRIPT_DIR}/upstream/self-hosted"
ENV_FILE="${SCRIPT_DIR}/.env"
# -----------------------------------------------------------------------------
# Helpers
# -----------------------------------------------------------------------------
log() { printf "\033[1;34m[sentry-install]\033[0m %s\n" "$*"; }
err() { printf "\033[1;31m[sentry-install:ERROR]\033[0m %s\n" "$*" >&2; }
die() {
err "$*"
exit 1
}
require_cmd() {
command -v "$1" >/dev/null 2>&1 || die "required command not found: $1"
}
# -----------------------------------------------------------------------------
# Preconditions
# -----------------------------------------------------------------------------
require_cmd git
require_cmd docker
if ! docker compose version >/dev/null 2>&1; then
die "docker compose v2 plugin not available (needs Docker Engine >= 24)"
fi
if [[ ! -f "${ENV_FILE}" ]]; then
die "${ENV_FILE} not found — copy .env.example to .env and fill in secrets first"
fi
# Load env into the current shell so the upstream installer + our overlays see
# the same values. set -a auto-exports everything sourced.
set -a
# shellcheck disable=SC1090
source "${ENV_FILE}"
set +a
: "${SENTRY_ADMIN_EMAIL:?SENTRY_ADMIN_EMAIL must be set in .env}"
: "${SENTRY_ADMIN_PASSWORD:?SENTRY_ADMIN_PASSWORD must be set in .env}"
: "${SENTRY_SECRET_KEY:?SENTRY_SECRET_KEY must be set in .env}"
: "${SENTRY_BASE_URL:?SENTRY_BASE_URL must be set in .env}"
# -----------------------------------------------------------------------------
# Step 1: clone upstream at pinned tag (or update if already cloned)
# -----------------------------------------------------------------------------
mkdir -p "${SCRIPT_DIR}/upstream"
if [[ ! -d "${UPSTREAM_DIR}/.git" ]]; then
log "cloning upstream getsentry/self-hosted @ ${UPSTREAM_TAG}"
git clone \
--branch "${UPSTREAM_TAG}" \
--depth 1 \
--single-branch \
"${UPSTREAM_REPO}" \
"${UPSTREAM_DIR}"
else
log "upstream clone exists; fetching tag ${UPSTREAM_TAG}"
git -C "${UPSTREAM_DIR}" fetch --depth 1 origin "refs/tags/${UPSTREAM_TAG}:refs/tags/${UPSTREAM_TAG}"
git -C "${UPSTREAM_DIR}" checkout --quiet "tags/${UPSTREAM_TAG}"
fi
# Sanity-check the checked-out tag matches.
ACTUAL_TAG="$(git -C "${UPSTREAM_DIR}" describe --tags --exact-match 2>/dev/null || echo unknown)"
if [[ "${ACTUAL_TAG}" != "${UPSTREAM_TAG}" ]]; then
die "upstream checkout is at '${ACTUAL_TAG}', expected '${UPSTREAM_TAG}'"
fi
log "upstream pinned at ${ACTUAL_TAG}"
# -----------------------------------------------------------------------------
# Step 2: apply our overlay files
# -----------------------------------------------------------------------------
log "applying overlays"
install -m 0644 "${SCRIPT_DIR}/docker-compose.override.yml" "${UPSTREAM_DIR}/docker-compose.override.yml"
mkdir -p "${UPSTREAM_DIR}/sentry"
install -m 0644 "${SCRIPT_DIR}/sentry/sentry.conf.py" "${UPSTREAM_DIR}/sentry/sentry.conf.py"
if [[ -f "${SCRIPT_DIR}/relay/config.yml" ]]; then
mkdir -p "${UPSTREAM_DIR}/relay"
install -m 0644 "${SCRIPT_DIR}/relay/config.yml" "${UPSTREAM_DIR}/relay/config.yml"
fi
# Also expose the .env to the upstream installer (it expects a sibling `.env`).
install -m 0600 "${ENV_FILE}" "${UPSTREAM_DIR}/.env"
# -----------------------------------------------------------------------------
# Step 3: run upstream installer (non-interactive, no telemetry-pings)
# -----------------------------------------------------------------------------
log "running upstream installer (this can take 5-10 minutes on first run)"
(
cd "${UPSTREAM_DIR}"
./install.sh --skip-user-prompt --no-report-self-hosted-issues
)
# -----------------------------------------------------------------------------
# Step 4: seed superuser (idempotent — Sentry no-ops if it already exists)
# -----------------------------------------------------------------------------
log "ensuring superuser ${SENTRY_ADMIN_EMAIL} exists"
(
cd "${UPSTREAM_DIR}"
docker compose run --rm web createuser \
--email "${SENTRY_ADMIN_EMAIL}" \
--password "${SENTRY_ADMIN_PASSWORD}" \
--superuser \
--no-input \
|| log "createuser returned non-zero — likely user already exists (safe to ignore)"
)
# -----------------------------------------------------------------------------
# Step 5: start the stack
# -----------------------------------------------------------------------------
log "starting stack (detached)"
(
cd "${UPSTREAM_DIR}"
docker compose up -d
)
log "done. Web UI: ${SENTRY_BASE_URL}"
log "next: log in, enable 2FA on the admin account, then create projects 'boilerplate-api', 'boilerplate-worker', 'boilerplate-web' and paste their DSNs into Coolify secrets."
@@ -0,0 +1,95 @@
# infra/sentry/relay/config.yml
#
# Relay config overlay for the upstream getsentry/self-hosted stack (pinned at
# 25.12.1). install.sh copies this file to upstream/self-hosted/relay/config.yml,
# replacing the upstream default. Loaded by the `relay` service on startup.
#
# Goals:
# 1. Run in `processing` mode so Relay terminates ingest end-to-end (rather
# than forwarding to a parent Relay). This is the standard mode for
# self-hosted single-region deployments.
# 2. Expose the OTLP HTTP endpoint on the Relay HTTP port so Phase 02 #33
# (OpenTelemetry auto-instrumentation) can ship traces to Sentry without
# needing an external collector.
# 3. Hard-cap per-project ingest at 100 events/second to prevent abuse via
# the public DSN exposed in browser bundles.
# 4. Reserve an allowed-domains placeholder; once the public web origin is
# known the per-project Sentry UI also enforces it (defence in depth).
relay:
mode: processing
upstream: "http://web:9000/"
host: 0.0.0.0
port: 3000
# TLS terminated by Traefik in front of the public nginx; relay itself
# speaks plain HTTP on the Docker network.
tls_port: null
# Logging — surface ingest decisions in container logs without flooding.
logging:
level: info
format: json
# Health/diagnostic endpoint stays on the same port; Coolify health-checks
# query /api/relay/healthcheck/ready/.
metrics:
prefix: sentry.relay
statsd: null
# ---- Processing knobs -------------------------------------------------------
processing:
enabled: true
# Kafka + Redis come from the upstream compose; these env-vars are injected
# by the upstream installer so we just reference them.
kafka_config:
- { name: bootstrap.servers, value: "kafka:9092" }
- { name: message.max.bytes, value: 50000000 }
redis: "redis://redis:6379"
# Project-config cache TTL. Lower = faster pickup of UI changes, higher =
# less load on `web`. 5 minutes is upstream default.
projects:
cache_expiry: 300
# ---- OTLP endpoint ----------------------------------------------------------
# Relay 25.x exposes OTLP-HTTP at /api/{project_id}/otlp/v1/{traces,logs}
# automatically when `processing.enabled` is true and the org feature flag
# `organizations:relay` is on (we set that in sentry/sentry.conf.py). The
# `aggregator` block below tunes batching for OTLP traces specifically.
aggregator:
bucket_interval: 10
initial_delay: 30
debounce_delay: 10
max_secs_in_past: 5184000 # 60 days
max_secs_in_future: 60
# ---- Rate-limits ------------------------------------------------------------
# Sentry's DSN is public by design (lives in the browser bundle). Without a
# rate-limit a botnet can exhaust ingest quota in seconds. Limits here apply
# *before* the project hits Kafka, so they protect the whole stack.
#
# 100 events/sec is a comfortable ceiling for a 3-app deployment; raise per
# project in the Sentry UI later if real traffic genuinely needs it.
limits:
max_event_size: 1000000 # 1 MB per event
max_attachment_size: 50000000 # 50 MB attachments (minidumps etc.)
max_envelope_size: 100000000 # 100 MB envelope
# Per-project ingest cap. Quota-decisions ultimately land in Redis; this is
# the in-process backstop Relay enforces before it ever calls into Sentry.
quotas:
default:
limit: 100
window: 1 # seconds
reason_code: "relay_per_project_rate_limit"
# ---- Allowed-domains placeholder -------------------------------------------
# This is informational — the actual allowed-domains list lives per-project
# in the Sentry UI ("Project Settings -> Client Keys -> Allowed Domains").
# Listed here so operators have one searchable place to remember what should
# be set:
# - https://boiler.saas-ctrl.com
# - https://www.boiler.saas-ctrl.com
# - https://admin.boiler.saas-ctrl.com
# - https://api.boiler.saas-ctrl.com
# Leave the array empty in the UI ONLY for the API and Worker projects (their
# DSNs are never used from a browser).
@@ -0,0 +1,77 @@
# Sentry config overlay.
#
# This file replaces the upstream `sentry/sentry.conf.py` (the upstream version
# is mostly defaults + env-var-wiring; we keep that behaviour and layer our
# project-specific hardening on top).
#
# Loaded by `web`, `worker`, `cron` (and any other Sentry component) at startup.
#
# Source of truth for what each setting does: https://develop.sentry.dev/self-hosted/
# Loaded at: /etc/sentry/sentry.conf.py inside the Sentry containers.
from sentry.conf.server import * # noqa: F401,F403
import os
# ---------------------------------------------------------------------------
# Single-organization mode
# ---------------------------------------------------------------------------
# We run one tenant ("Boilerplate"). This collapses the org-picker UI and lets
# the admin see the global dashboard directly.
SENTRY_SINGLE_ORGANIZATION = True
# ---------------------------------------------------------------------------
# Feature flags
# ---------------------------------------------------------------------------
# Start from upstream defaults if a SENTRY_FEATURES dict is already exported,
# otherwise initialise an empty one. Either way the writes below win.
try:
SENTRY_FEATURES # type: ignore[name-defined]
except NameError:
SENTRY_FEATURES = {}
# Public signup OFF — invites only. This is the single most important
# hardening setting; without it any visitor to sentry.boiler.saas-ctrl.com
# can create an account and start receiving events.
SENTRY_FEATURES["auth:register"] = False
# Enable Relay for the org. Required for OTLP ingest, performance, and replay
# to flow through `relay` (rather than directly through `web`).
SENTRY_FEATURES["organizations:relay"] = True
# ---------------------------------------------------------------------------
# Symbolicator: no external sym sources
# ---------------------------------------------------------------------------
# Stack traces stay local. We do NOT contact Microsoft / Apple / Mozilla
# public symbol servers because the leaked metadata would be telemetry-grade
# (every untranslated frame would betray library + version + ABI to a third
# party). If we ever need real symbol resolution we provision an internal
# sym-source against a private bucket.
SYMBOLICATOR_SYMSOURCES = []
# ---------------------------------------------------------------------------
# Admin
# ---------------------------------------------------------------------------
# Read from env; install.sh seeds the user on first deploy. Subsequent reads
# only affect superuser-aware emails (e.g. "from" address on system notes).
SENTRY_ADMIN_EMAIL = os.environ.get("SENTRY_ADMIN_EMAIL", "")
# ---------------------------------------------------------------------------
# Mail
# ---------------------------------------------------------------------------
# Upstream's compose ships its own smtp container; we keep that wiring. The
# admin email above is the From-address for system mails.
SENTRY_OPTIONS = globals().get("SENTRY_OPTIONS", {})
SENTRY_OPTIONS.setdefault("mail.from", SENTRY_ADMIN_EMAIL or "sentry@boiler.saas-ctrl.com")
# ---------------------------------------------------------------------------
# URL prefix — must match the Traefik-routed hostname
# ---------------------------------------------------------------------------
# Sentry uses this to build absolute URLs in emails, web-hooks, and the auth
# flow. Mismatch leads to broken magic-links in invite emails.
SENTRY_OPTIONS["system.url-prefix"] = os.environ.get(
"SENTRY_BASE_URL", "https://sentry.boiler.saas-ctrl.com"
)
# Trust the X-Forwarded-* headers from Traefik (TLS terminated upstream).
SENTRY_USE_SSL = bool(int(os.environ.get("SENTRY_USE_SSL", "1")))
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
@@ -0,0 +1,59 @@
# Canonical Traefik label-set for the self-hosted Sentry stack.
#
# This file is a YAML fragment (NOT a runnable compose file by itself). It's
# provided as documentation of which labels we attach to the `nginx` service
# of the upstream getsentry/self-hosted compose. The same labels are already
# embedded inline in `docker-compose.override.yml`, which is what actually
# ships to the runtime; this fragment exists so the routing contract is
# searchable and reviewable in one place.
#
# Coolify (v4) ships Traefik v3.6 with two entrypoints (`http`:80, `https`:443)
# and a default ACME cert-resolver named `letsencrypt`. The labels below match
# that convention.
#
# Host: sentry.boiler.saas-ctrl.com -> DNS A record points at 46.225.5.93.
# When a different staging domain is chosen, update both this file and the
# inline labels in docker-compose.override.yml (single source of truth lives
# in the override file; this fragment is a copy for review).
# ---- ROUTERS ----------------------------------------------------------------
# Primary HTTPS router — terminates TLS, forwards to nginx:80.
traefik.enable: "true"
traefik.docker.network: "coolify"
traefik.http.routers.sentry.rule: "Host(`sentry.boiler.saas-ctrl.com`)"
traefik.http.routers.sentry.entrypoints: "https"
traefik.http.routers.sentry.tls: "true"
traefik.http.routers.sentry.tls.certresolver: "letsencrypt"
traefik.http.services.sentry.loadbalancer.server.port: "80"
# HTTP -> HTTPS redirect (port 80 catches plain-HTTP hits and 308s them to TLS).
traefik.http.routers.sentry-http.rule: "Host(`sentry.boiler.saas-ctrl.com`)"
traefik.http.routers.sentry-http.entrypoints: "http"
traefik.http.routers.sentry-http.middlewares: "sentry-https-redirect"
traefik.http.middlewares.sentry-https-redirect.redirectscheme.scheme: "https"
# ---- OPTIONAL: rate-limit middleware (apply if abuse is detected) -----------
#
# Sentry's DSN is by design public (lives in the browser bundle); the Relay
# `config.yml` enforces the real per-project ingest rate-limit. The labels
# below are a *secondary* defence-in-depth that you can switch on by adding
# `sentry-ratelimit` to the router's `middlewares` list.
#
# traefik.http.middlewares.sentry-ratelimit.ratelimit.average: "200"
# traefik.http.middlewares.sentry-ratelimit.ratelimit.burst: "400"
# traefik.http.middlewares.sentry-ratelimit.ratelimit.period: "1s"
# ---- OPTIONAL: security headers ---------------------------------------------
#
# Sentry's own nginx already sets sensible CSP/HSTS for its UI. If you put
# Cloudflare or another proxy in front, this middleware tightens the response
# at the Traefik edge — turn on by appending `sentry-secure-headers` to the
# router's `middlewares`.
#
# traefik.http.middlewares.sentry-secure-headers.headers.stsSeconds: "31536000"
# traefik.http.middlewares.sentry-secure-headers.headers.stsIncludeSubdomains: "true"
# traefik.http.middlewares.sentry-secure-headers.headers.stsPreload: "true"
# traefik.http.middlewares.sentry-secure-headers.headers.contentTypeNosniff: "true"
# traefik.http.middlewares.sentry-secure-headers.headers.browserXssFilter: "true"
# traefik.http.middlewares.sentry-secure-headers.headers.referrerPolicy: "no-referrer"
@@ -0,0 +1,4 @@
# Placeholder — keeps `infra/sentry/upstream/` present in the repo.
# At deploy time, `infra/sentry/install.sh` populates `upstream/self-hosted/`
# by cloning getsentry/self-hosted at the pinned tag (see UPSTREAM.md).
# The cloned tree is git-ignored via `infra/sentry/.gitignore`.