toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track HiveGrid Cloud planning reference

Browse Source
This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

hivegrid-cloud/docs/observability/console-migration-inventory.md
+24
View File
@@ -0,0 +1,24 @@
# Console-Call Migration Inventory
> Generated by `scripts/observability/scan-console.ts` on 2026-05-29T06:20:36.301Z.
> Re-run with: `pnpm exec tsx scripts/observability/scan-console.ts`
**Total `console.*` hits in server-code paths: 0**
**Unique files: 0**
Server-paths scanned:
- `apps/api/src/**`
- `apps/worker/src/**`
- `apps/web/app/**/*.server.ts(x)` and `apps/web/app/modules/*/services/**` and `apps/web/app/utils/**`
- `packages/*/src/**`
Excluded:
- `**/*.test.ts(x)`, `**/*.spec.ts(x)`
- Lines after `// eslint-disable-next-line no-console`
- `dist/`, `build/`, `coverage/`, `node_modules/`, `.react-router/`, `generated/`
## All clear
No `console.*` calls remain in server-code paths. The ESLint rule `no-console: error` enforces this going forward.
hivegrid-cloud/docs/observability/logging.md
+194
View File
@@ -0,0 +1,194 @@
# Logging Conventions
> Owner: Phase-02 Observability. Last update: 2026-05-17 (Task #34).
This document is the source of truth for **how to log in this codebase**.
Phase-02 Task #34 migrated every server-side `console.*` call to Pino
and locked the rule in via ESLint. Read this before adding new log
statements.
## TL;DR
- Use the Pino logger appropriate to your runtime — never `console.*`.
- Pino argument order: **object first, message string second**.
- Never log raw PII (email, password, token, full prompt body, full
webhook payload, Stripe card data).
- Errors go in the `err` field so Pino's built-in serializer kicks in.
## Which logger?
| Runtime | Logger | Location |
|---|---|---|
| React Router web (SSR / loaders / actions / `.server.ts`) | `webLogger` (or `createScopedLogger("scope")`) | `apps/web/app/utils/logger.server.ts` |
| Hono API service | `apiLogger` | `apps/api/src/logger.ts` |
| BullMQ worker | `workerLogger` | `apps/worker/src/logger.ts` |
| Package code (`packages/*/src/**`) | `createChildLogger(defaultLogger, { component: "<pkg-name>", scope: "<feature>" })` from `@toolctrl/logger` | `packages/logger/src/index.ts` |
For larger files / modules, scope your logger:
```ts
// apps/web/app/modules/users/services/AuthService.ts
import { createScopedLogger } from "~/utils/logger.server";
const authLogger = createScopedLogger("auth");
```
The scope name appears as a `scope: "auth"` field on every record, so
Loki/Grafana queries like `{service="web"} | json | scope="auth"` work
without having to grep through unrelated messages.
## Pino argument order
```ts
// ❌ VORHER (console.* style — backwards)
console.log("User signed in", { userId, tenantId });
console.error("Failed to send email", error);
// ✅ NACHHER (Pino style — object first, msg second)
authLogger.info({ userId, tenantId, op: "user.signin" }, "User signed in");
authLogger.error({ err: error, userId, op: "user.signin" }, "Failed to send email");
```
The `err` key is special: Pino's built-in serializer flattens it to
`{ type, message, stack }` and auto-redacts `error.config.headers.authorization`.
If you log the error under any other key, the serializer does NOT run.
## Log levels
| Level | Use for |
|---|---|
| `fatal` | Process is about to exit / data was lost. Never reached normally. |
| `error` | An operation failed and we cannot recover. Send to Sentry. |
| `warn` | Unexpected condition, but operation continued (fallback used, missing config). |
| `info` | Significant business event (subscription created, user signed in, seed run started). |
| `debug` | Verbose diagnostic for development. Hidden in prod by default. |
| `trace` | Per-iteration / hot-path debug. Almost never used. |
Production default level is `info` (env: `LOG_LEVEL=info`). Bump to
`debug` per service with `LOG_LEVEL=debug` if you need to capture a
specific flow.
## Forbidden fields
Never log these — even if the user "doesn't think they matter":
- Raw `email` — use `emailFingerprint(email)` from the seed/auth utils
if you need correlation.
- Raw `password`, `passwordHash`, `apiKey`, `secret`, `token`,
`sessionToken`, `bearer`.
- AI **prompt bodies** — only log `{ promptHash, promptLength, model, tenantId }`.
See `apps/web/app/modules/promptBuilder/services/.server/PromptBuilderService.ts`
`hashPrompt()` for the SHA-256 helper.
- Stripe **webhook bodies** — log only `{ eventType, eventId }`.
- ConvertKit / CRM API **response bodies** (contain subscriber email).
- Request **headers** containing `authorization`, `cookie`, `set-cookie`.
`packages/logger/src/index.ts` defines a defensive **redact list** at
the Pino instance level — fields named `password`, `token`, `secret`,
`apiKey`, `authorization`, `cookie` (plus their `*.<field>` variants)
are replaced with `[Redacted]` even if a developer forgets. Do not
rely on this as your primary defence — redaction does NOT cascade
into nested objects with non-matching key names.
## Pretty vs JSON
| Environment | Mode | Why |
|---|---|---|
| Dev (`NODE_ENV !== "production"`) | Pretty-printed (auto-detect via `pino-pretty` if installed) | Readable in your terminal. |
| Prod / Staging | JSON (default) | Coolify → Loki / Grafana / Sentry-Breadcrumb pipeline consumes JSON. Pretty-print would break ingestion. |
Override with `LOG_PRETTY=1` if you need pretty in prod (only for a
short debug session — never permanent).
## ESLint enforcement
`no-console: 'error'` is active for:
- `**/*.server.ts` and `**/*.server.tsx`
- `apps/api/src/**/*.ts`, `apps/worker/src/**/*.ts`
- `packages/*/src/**/*.ts`
- `apps/web/app/utils/**/*.ts`
- `apps/web/app/modules/**/services/**/*.ts`
- `apps/web/app/modules/**/db/**/*.ts`
Override `'off'` for: `**/*.test.ts(x)`, `**/*.spec.ts(x)`, `scripts/**`,
and client-bundle files (`apps/web/app/components/**/*.tsx`,
`apps/web/app/routes/**/*.tsx` without `.server.` suffix).
The lefthook pre-commit hook runs `scripts/eslint-staged.mjs` which
fails the commit on any new `console.*` in server code. Verified
2026-05-17 by injecting `console.log` into `apps/api/src/test-tmp.ts`
and observing the hook block the commit.
## Migration inventory script
```bash
pnpm exec tsx scripts/observability/scan-console.ts
# generates docs/observability/console-migration-inventory.md
# exit 1 with --check if any live console.* remain in server paths
```
Wire this into CI eventually (already runs locally; CI run is tracked
in Task #36 follow-ups).
## Examples
### Standard info event
```ts
authLogger.info(
{ userId, tenantId, op: "user.signin", method: "password" },
"User signed in",
);
```
### Error with err serializer
```ts
try {
await sendEmail(...);
} catch (error) {
emailLogger.error(
{ err: error, recipientFingerprint: emailFingerprint(to), op: "email.send" },
"Failed to send email",
);
}
```
### AI prompt (PII-safe)
```ts
promptLogger.info(
{
op: "executeTemplate.createChatCompletion",
templateId,
model,
promptHash: hashPrompt(prompt), // SHA-256, first 16 hex chars
promptLength: prompt.length,
tenantId,
},
"Creating AI chat completion",
);
```
### Stripe / webhook (PII-safe)
```ts
pricingLogger.info(
{
op: "stripe.webhook.received",
eventType: event.type,
eventId: event.id,
livemode: event.livemode,
},
"Stripe webhook received",
);
```
## See also
- `docs/improvement-plan/02-observability/logging-migration.md` — full
task spec (Task #34) including PII rationale.
- `packages/logger/src/index.ts` — Pino setup + redact paths.
- `scripts/observability/scan-console.ts` — inventory script.
- `docs/observability/console-migration-inventory.md` — auto-generated
per-file inventory (0 hits after Phase-02 #34).
hivegrid-cloud/docs/observability/metrics.md
+150
View File
@@ -0,0 +1,150 @@
# Metrics + Grafana
Prometheus + Grafana stack for the boilerplate. Scrapes `/metrics` from
the three application processes and renders five dashboards.
## Architecture
```
+----------+ scrape +-------------+ query +---------+
| api | <--------------------- | | <------------------ | |
| worker | <--------------------- | Prometheus | | Grafana |
| web | <--------------------- | | | |
+----------+ +-------------+ +---------+
^ |
| v
/status tsdb (30d)
/metrics
```
- All three apps are instrumented via [`@toolctrl/metrics`](../../packages/metrics).
- `/status` is public, JSON, 200 / 503 — suitable for Uptime Kuma.
- `/metrics` is bearer-token protected, returns Prom text-format.
- Both endpoints are gated by `METRICS_ENABLED=true`; default off in dev.
## Metric catalog
### Infrastructure (all services)
| Metric | Type | Labels | Question it answers |
|---|---|---|---|
| `http_requests_total` | counter | method, route, status | Request volume + status distribution. |
| `http_request_duration_seconds_bucket` | histogram | method, route, status | API latency percentiles. |
| `prisma_query_duration_seconds_bucket` | histogram | model, action | DB hotspot detection. |
| `redis_command_duration_seconds_bucket` | histogram | command | Redis cache hit / slow-cmd diagnosis. |
| `process_*` | counter / gauge | — | Node.js process health (default `prom-client`). |
### Worker queue
| Metric | Type | Labels | Question |
|---|---|---|---|
| `queue_jobs_active` | gauge | queue | Concurrency in-flight. |
| `queue_jobs_waiting` | gauge | queue | Backlog depth. |
| `queue_jobs_completed_total` | counter | queue, job_name | Throughput. |
| `queue_jobs_failed_total` | counter | queue, job_name, error_type | Failure-type breakdown. |
| `queue_job_duration_seconds_bucket` | histogram | queue, job_name | Job-duration heatmap. |
### Web SSR
| Metric | Type | Labels | Question |
|---|---|---|---|
| `web_ssr_loader_duration_seconds_bucket` | histogram | route, status | Loader-level latency for the top routes. |
Status enum: `ok | error | redirect`. Apply via `withMetrics(routeId, loader)`
from `apps/web/app/lib/metrics/ssr.ts`. Currently wrapped: index/app-root,
app-tenant, admin-root, dashboard, login, logout, forgot-password,
new-account, invitation.
### Business
| Metric | Type | Labels | Question |
|---|---|---|---|
| `active_tenants` | gauge | — | Tenant growth. |
| `active_users_24h` | gauge | — | DAU. |
| `ai_tokens_consumed_total` | counter | provider, model, tenant_bucket | AI spend allocation. |
| `stripe_webhook_events_total` | counter | event_type, status | Stripe traffic + spoof attempts. |
| `mail_sent_total` | counter | provider, status | Email reliability. |
| `webhook_delivery_attempts_total` | counter | status | Outbound webhook health (success / retry / dlq / rejected). |
| `csrf_token_rotation_total` | counter | — | Auth churn. |
| `subscription_active` | gauge | plan_id | Plan mix. |
## Cardinality rules
> Prometheus dies before you hit a million series. Pre-empt it.
1. **Never use raw user / request input as a label value.** Always use a
pattern (route pattern, error name, status code) or a bucket.
2. **Cap unknown sets.** `ai_tokens_consumed_total.tenant_bucket` is a
process-local LRU of the first 100 tenants seen + `"other"` overflow
(see `apps/web/app/lib/metrics/ai-cost.ts`). When you add a new
metric with a similar shape, apply the same pattern.
3. **No PII labels.** Tenant IDs are opaque slugs, never email / name.
4. **No env-var spill.** Default prom-client process metrics never leak
`process.env`; verify the same for custom metrics in code review.
## Adding a new metric
1. Declare it in `packages/metrics/src/defaults.ts`. Pick the smallest
useful label set.
2. Write a unit test in `packages/metrics/src/defaults.test.ts` that
touches the metric and asserts the rendered text contains the name.
3. Add an instrumentation call at the business-logic site. Wrap the
call in `try { … } catch { /* never crash */ }`.
4. Add a Grafana panel under `infra/grafana/dashboards/` and commit.
5. Update this catalog.
## Deploy on Hetzner staging
The Prometheus and Grafana stacks are intended for the **same Hetzner
observability box that hosts Sentry** (`/opt/observability/` directory
convention). The compose files in `infra/prometheus/` and
`infra/grafana/` can be deployed as Coolify "Docker Compose" services
or as plain `docker compose up -d` units.
Per-step instructions live in
[`infra/prometheus/README.md`](../../infra/prometheus/README.md).
### Coolify vs `/opt/observability/`
Operator decision — both work. Coolify is easier to wire to Traefik for
the `prometheus.boiler.saas-ctrl.com` and `grafana.boiler.saas-ctrl.com`
subdomains. Plain `docker compose` is lighter but requires a manual
reverse-proxy entry. **Default recommendation**: Coolify-managed
Service, same as Sentry.
## Dashboard-update workflow
1. Edit panels in the Grafana UI.
2. Top-right menu → **Share → Export → Save to file** (uncheck "Export
for sharing externally" so UIDs stay stable).
3. Drop the JSON into `infra/grafana/dashboards/<name>.json`.
4. Commit. Provisioning reloads automatically every 30s.
## Smoke check
```sh
# /status — should be 200 OK
curl -fsS https://api.boiler.saas-ctrl.com/status | jq
# /metrics without auth — should be 401
curl -i https://api.boiler.saas-ctrl.com/metrics | head -1
# /metrics with auth — should be 200 + prom text
curl -fsS -H "Authorization: Bearer $METRICS_TOKEN" \
https://api.boiler.saas-ctrl.com/metrics | head -20
```
`scripts/observability/smoke-metrics.sh` automates this against any
base host (defaults to staging).
## Rollback
- **Disable everything**: `METRICS_ENABLED=false`. All three apps stop
exposing `/metrics` (404) within one restart. Counters keep working
in-process; only the surface goes away.
- **Cardinality emergency** (Prometheus OOM): the most likely culprit
is `ai_tokens_consumed_total`. The safety valve is already there
(tenant_bucket caps at 101). If a NEW metric explodes, revert the
commit that added it and reload Prometheus.
- **Dashboard rollback**: `git revert` the JSON file and the file
provider replaces it within 30 s.
hivegrid-cloud/docs/observability/performance-baseline.md
+85
View File
@@ -0,0 +1,85 @@
# Performance Baseline — Phase 02 Quickwins (Task #36)
This document tracks the expected and observed effect of the 8 quickwin
optimizations shipped under task #36
(`docs/improvement-plan/02-observability/performance-quickwins.md`).
## Status
- **Code-level changes:** shipped to `main` 2026-05-17 (8 atomic commits +
this baseline doc).
- **`/metrics` endpoint status:** code shipped for API and worker
runtimes. The API exposes token-protected `GET /metrics` when
`METRICS_ENABLED=true`; worker admin servers expose token-protected
`/metrics` when enabled and keep health probes on a separate port.
- **Live `/metrics` baseline:** operator-owned. The endpoint code exists,
but the live before/after numbers still require an environment with
`METRICS_ENABLED=true`, `METRICS_TOKEN`, scrape access and a controlled
load run.
## Per-item expected gains (code-level reasoning)
| Item | File | Mechanism | Expected gain |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. N+1 in admin users list | `apps/web/app/routes/admin/accounts/users.tsx` + `apps/web/app/utils/db/logs.db.server.ts` | Replace N `findFirst` calls with 1 `groupBy` + 1 `findMany` | DB queries on the admin users-list loader: `O(N)` → `O(1)` (101→2 for a 100-user page). Largest practical win on this route. |
| 2. Duplicate tenant fetch | `apps/web/app/utils/session.server.ts` + `apps/web/app/routes/admin/accounts/users.tsx` + `apps/web/app/routes/oauth/github/callback.tsx` | `setLoggedUser` now returns `defaultTenant`; callers reuse it instead of fetching again | One fewer `getTenant()` round-trip per sign-in / impersonation start (≈ 1 DB query/login). |
| 3. S3 client + signer caching | `apps/api/src/storage.ts` | Memoize S3 storage adapter, signed-URL adapter and per-day SigV4 signing key by SHA256-of-config | Eliminates per-call adapter construction + HMAC chain (4 HMAC-SHA256 ops) on every signed URL. Per-request CPU win in hot upload/download paths. |
| 4. clearSubscriptionsCache thrash | `apps/web/app/utils/services/.server/subscriptionService.ts` + `apps/web/app/utils/db/subscriptions/tenantSubscriptionProducts.db.server.ts` | New `clearTenantSubscriptionCache(tenantId)` replaces a global wipe in `cancelTenantSubscriptionProduct` | Subscription-change webhooks no longer evict every tenant's cached subscription. Tenant N=1000 → only 1 eviction per event instead of 1000. |
| 5. CSRF header-first check | `apps/web/app/utils/session.server.ts` | Already-resolved in commit `83a57f1c`; regression test added | No additional perf change in #36; documents the historical fix and now-enforced contract: 0 body-clones when the header carries the token. |
| 6. Recovery-code rejection sampling | `apps/web/app/utils/auth/mfa/recovery-codes.server.ts` | Drop `while(set.size < n)` loop; generate batch + max 2 retries | Generates 10 codes in one batch (≥99.999% of calls); throws on systemic failure instead of looping forever. Bounds wall-clock generation cost. |
| 7. UUID collision loop | `apps/api/src/auth/session/session-store.ts` | `while (sessions.has(sessionId))` → single insert; throw on collision | 1 ID generation per `createSession` instead of unbounded retry. UUID-v4 has 122 bits of entropy, so a collision is a systemic error to surface, not mask. |
| 8. CSRF token size | `apps/web/app/utils/session.server.ts` | `randomBytes(100).toString("base64")` → `randomBytes(32).toString("hex")` | Token shrinks ~136 chars → 64 chars. Saves ~70 bytes/cookie and ~70 bytes/response per CSRF-bearing endpoint. Aggregate response-body reduction across the cluster. |
## Test coverage added
- `apps/web/app/utils/db/logs.db.server.test.ts` — `getLastUserLogsByUserIds`
uses exactly 1 `groupBy` + 1 `findMany`.
- `apps/web/app/utils/session.server.test.ts` — `setLoggedUser` returns
`defaultTenant`; `validateCSRFToken` never clones the request body when
the header carries the token; `generateCSRFToken` emits 64 hex chars.
- `apps/api/src/storage.test.ts` — repeated config hits the adapter cache;
bucket change misses; secret rotation misses.
- `apps/web/app/utils/services/.server/subscriptionService.test.ts` —
`clearTenantSubscriptionCache` does not enumerate other tenants;
`clearAllSubscriptionsCache` invalidates all tenant entries.
- `apps/web/app/utils/auth/mfa/recovery-codes.server.test.ts` — generation
completes under a strict wall-clock bound (regression guard against
unbounded rejection sampling).
- `apps/api/src/auth/session/session-store.test.ts` — UUID collision now
throws instead of silently retrying.
## Live baseline (operator-owned)
Populate the table below by running `oha -n 5000 -c 50` against staging
twice for a comparable before/after window and scraping API/worker
`/metrics` snapshots with `Authorization: Bearer <METRICS_TOKEN>`.
The code path is shipped; the live baseline values are operational
evidence and should be recorded only from the target environment.
| Metric | Before | After | Delta |
| ----------------------------------------------- | --------------- | ----- | ----- |
| p50 latency, GET `/admin/users` | TBD | TBD | TBD |
| p95 latency, GET `/admin/users` | TBD | TBD | TBD |
| Prisma query count / request (admin users list) | TBD | TBD | TBD |
| p95 latency, signed-URL endpoint | TBD | TBD | TBD |
| p95 latency, login flow | TBD | TBD | TBD |
| Cache evictions per Stripe webhook | TBD | TBD | TBD |
| CSRF cookie size (bytes) | ~136 hex equiv. | 64 | -53% |
## Notes
- All eight items shipped as **atomic** commits; any single item can be
reverted in isolation.
- Item 5 was already in tree from `83a57f1c fix: enforce csrf on settings
mutations`. It is recorded here for completeness and now carries an
explicit no-body-read assertion.
- Path corrections were necessary on items 2 and 6 — see the per-item
Progress Log entries in
`docs/improvement-plan/02-observability/performance-quickwins.md`.
## Footer
- Date: 2026-05-17
- Owner: Phase-02 quickwins agent
- Live-test hardware: operator-owned (fill with the target environment
used for the controlled load run)
hivegrid-cloud/docs/observability/sentry-glitchtip-alternative.md
+194
View File
@@ -0,0 +1,194 @@
# Light Alternative: GlitchTip
This document describes **GlitchTip** as a much lighter alternative to the
self-hosted Sentry stack in `infra/sentry/`. GlitchTip is **NOT** the
default observability backend — it lacks Performance/Tracing, Session
Replay, and OTLP ingest. Use it only when one of the following applies:
- The Sentry host is severely RAM-constrained (< 8 GB available)
- A throw-away preview deployment needs an error sink in < 5 minutes
- You want a backend on the cheapest possible Hetzner instance for a
side-project that only cares about error aggregation
For the full Sentry stack, see
[`sentry-selfhost.md`](./sentry-selfhost.md).
## At a glance
| Item | GlitchTip | Self-Hosted Sentry |
| --- | --- | --- |
| Error events | yes | yes |
| Performance / tracing | no | yes |
| Session replay | no | yes |
| Profiling | no | yes |
| OTLP ingest | no | yes (via Relay) |
| Sentry-SDK compatibility | yes (errors only) | yes (all features) |
| Container count | 4 | ~10 |
| RAM | 1–2 GB | 12–16 GB |
| Disk (90 d) | 5–10 GB | 40–60 GB |
| Upstream image pin | `glitchtip/glitchtip:v6.1.6` | `getsentry/self-hosted` @ `25.12.1` |
## SDK compatibility note
GlitchTip implements a subset of the Sentry HTTP ingest API — **errors only**.
The official Sentry SDKs (`@sentry/node`, `@sentry/react`, etc.) work
without modification for plain error reporting:
```ts
Sentry.init({
dsn: process.env.GLITCHTIP_DSN, // looks like a Sentry DSN
});
Sentry.captureException(new Error("hello"));
```
**Do not** enable `tracesSampleRate`, `replaysSessionSampleRate`, or
`profilesSampleRate` — those payloads are rejected. If you call
`Sentry.startTransaction(...)`, GlitchTip silently drops the span data;
nothing breaks, but you also do not see traces.
For OpenTelemetry ingest (Phase 02 task #33) GlitchTip is **not** a
viable backend.
## Compose (light, 4-container)
The following Compose file runs on Coolify exactly like the Sentry stack
but with 4 services instead of 10. Save under
`infra/glitchtip/docker-compose.yml` (NOT committed today; this is the
reference document so we can stand it up quickly if needed).
```yaml
version: "3.8"
networks:
default:
name: glitchtip-net
coolify:
external: true
services:
postgres:
image: postgres:15-alpine
environment:
POSTGRES_DB: ${GLITCHTIP_DB:-glitchtip}
POSTGRES_USER: ${GLITCHTIP_DB_USER:-glitchtip}
POSTGRES_PASSWORD: ${GLITCHTIP_DB_PASSWORD:?must be set in .env}
volumes:
- /data/coolify/services/glitchtip/postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "${GLITCHTIP_DB_USER:-glitchtip}"]
interval: 10s
retries: 6
valkey:
# GlitchTip 6.x uses Valkey (Redis fork) as cache + broker. Pin to v7
# to match Coolify's existing Redis container ABI.
image: valkey/valkey:7-alpine
volumes:
- /data/coolify/services/glitchtip/valkey-data:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 10s
retries: 6
web:
image: glitchtip/glitchtip:v6.1.6
depends_on:
postgres:
condition: service_healthy
valkey:
condition: service_healthy
environment:
DATABASE_URL: postgres://${GLITCHTIP_DB_USER:-glitchtip}:${GLITCHTIP_DB_PASSWORD}@postgres:5432/${GLITCHTIP_DB:-glitchtip}
REDIS_URL: redis://valkey:6379/0
SECRET_KEY: ${GLITCHTIP_SECRET_KEY:?generate with python -c "import secrets; print(secrets.token_urlsafe(50))"}
GLITCHTIP_DOMAIN: ${GLITCHTIP_BASE_URL:-https://glitchtip.boiler.saas-ctrl.com}
DEFAULT_FROM_EMAIL: ${GLITCHTIP_FROM_EMAIL:-glitchtip@boiler.saas-ctrl.com}
ENABLE_USER_REGISTRATION: "false" # invite-only, matches Sentry hardening
ENABLE_ORGANIZATION_CREATION: "false"
CELERY_WORKER_AUTOSCALE: "1,3"
volumes:
- /data/coolify/services/glitchtip/uploads:/code/uploads
networks:
- default
- coolify
labels:
- "traefik.enable=true"
- "traefik.docker.network=coolify"
- "traefik.http.routers.glitchtip.rule=Host(`glitchtip.boiler.saas-ctrl.com`)"
- "traefik.http.routers.glitchtip.entrypoints=https"
- "traefik.http.routers.glitchtip.tls=true"
- "traefik.http.routers.glitchtip.tls.certresolver=letsencrypt"
- "traefik.http.services.glitchtip.loadbalancer.server.port=8000"
worker:
image: glitchtip/glitchtip:v6.1.6
command: ./bin/run-celery-with-beat.sh
depends_on:
postgres:
condition: service_healthy
valkey:
condition: service_healthy
environment:
DATABASE_URL: postgres://${GLITCHTIP_DB_USER:-glitchtip}:${GLITCHTIP_DB_PASSWORD}@postgres:5432/${GLITCHTIP_DB:-glitchtip}
REDIS_URL: redis://valkey:6379/0
SECRET_KEY: ${GLITCHTIP_SECRET_KEY}
GLITCHTIP_DOMAIN: ${GLITCHTIP_BASE_URL:-https://glitchtip.boiler.saas-ctrl.com}
CELERY_WORKER_AUTOSCALE: "1,3"
volumes:
- /data/coolify/services/glitchtip/uploads:/code/uploads
```
Note: this Compose includes an implicit `migrate` step. The `web`
service of the GlitchTip image runs Django migrations automatically on
startup; no separate `migrate` command is required. If you want an
explicit one-shot, override with `command: ./manage.py migrate`.
## `.env` template
```dotenv
GLITCHTIP_BASE_URL=https://glitchtip.boiler.saas-ctrl.com
# Postgres
GLITCHTIP_DB=glitchtip
GLITCHTIP_DB_USER=glitchtip
GLITCHTIP_DB_PASSWORD=replace-me-strong
# Django secret — generate with `python -c "import secrets; print(secrets.token_urlsafe(50))"`
GLITCHTIP_SECRET_KEY=replace-me-50-chars-urlsafe
# Optional: outbound mail. GlitchTip needs this for invitation flows
# even though we disable self-signup.
GLITCHTIP_FROM_EMAIL=glitchtip@boiler.saas-ctrl.com
EMAIL_URL=smtp://smtp.mailjet.net:587/?_user=...&_password=...
```
## Bootstrap
```bash
# 1. DNS: glitchtip.boiler.saas-ctrl.com -> 46.225.5.93
# 2. Fill .env, then:
docker compose up -d
# 3. First-superuser:
docker compose exec web ./manage.py createsuperuser
# 4. Log in, create projects, copy DSNs into apps' Coolify secrets.
```
## When to choose which
| Scenario | Recommendation |
| --- | --- |
| Default production observability | **Sentry** self-hosted |
| OpenTelemetry traces required | **Sentry** (GlitchTip cannot ingest OTLP) |
| Replay / Performance / Profiling required | **Sentry** |
| Throw-away preview environment | GlitchTip |
| < 8 GB RAM available | GlitchTip |
| EU-only data residency on a budget VPS | GlitchTip (both work, but GlitchTip's resource footprint is dramatically smaller) |
## Migration paths
- **Sentry → GlitchTip:** swap the DSN in app `.env`s. Historical events
are not migrated (different schema). Decommission the Sentry stack
after a 7-day overlap window.
- **GlitchTip → Sentry:** same swap, opposite direction. Historical
GlitchTip events also do not migrate. Sentry can ingest the GlitchTip
DSN format directly.
hivegrid-cloud/docs/observability/sentry-replay-masking.md
+97
View File
@@ -0,0 +1,97 @@
# Sentry Replay — PII Masking Conventions (Web)
This document describes how the boilerplate masks personally identifiable information (PII) in Sentry Session Replay videos so that DSGVO / GDPR obligations are met by default.
The defaults are configured in `apps/web/app/sentry.client.ts`:
```ts
Sentry.replayIntegration({
maskAllInputs: true,
maskAllText: false,
blockAllMedia: false,
});
```
**Why not `maskAllText: true`?** Masking every text node makes the replay video useless for reproduction — buttons, headings, navigation labels all disappear. We instead opt-in to masking on the small set of nodes that actually contain PII.
**Why not `blockAllMedia: true`?** Blocking all `<img>` / `<video>` removes brand assets and UI screenshots that are needed for triage. Operators can flip this per-environment via `VITE_SENTRY_REPLAY_BLOCK_ALL_MEDIA` if their UI legitimately displays user-uploaded media on every page.
## How to mark a region as sensitive
Three options are available, in increasing strictness:
| Marker | Effect |
| ---------------------------- | -------------------------------------------------------------------------------------- |
| `className="sentry-mask"` | Text content is replaced with `*` placeholders in the replay. |
| `data-sentry-mask` attribute | Same as above; useful where you cannot edit the className (e.g. third-party widgets). |
| `className="sentry-block"` | The entire DOM subtree is replaced with a placeholder rectangle — nothing is recorded. |
Inputs are masked automatically thanks to `maskAllInputs: true`, but the `sentry-block` marker is the safer choice for forms that contain free-text PII (chat composer, profile editor, etc.) because it also covers labels and helper text adjacent to the inputs.
To debug masking locally without sending data to Sentry, add `data-sentry-mask-debug="1"` to the `<html>` element. The helper CSS in `apps/web/app/styles/sentry-masking.css` then renders a dashed outline around every marked region.
## Required annotations (must-do list)
The following components/routes MUST be annotated by the time the staging-smoke test runs:
### Authentication & Account
- `apps/web/app/routes/login.tsx` — password input → `sentry-mask`
- `apps/web/app/routes/register.tsx` — password input → `sentry-mask`
- `apps/web/app/routes/forgot-password.tsx` — email input → `sentry-mask`
- `apps/web/app/routes/reset.tsx` — password inputs → `sentry-mask`
- `apps/web/app/routes/auth.totp.tsx` — TOTP code input → `sentry-mask`
- `apps/web/app/routes/auth.magic-link.tsx` — email + token field → `sentry-mask`
- `apps/web/app/modules/users/components/UserProfileSettings.tsx` — email, name, phone → `sentry-mask`
- `apps/web/app/modules/users/components/UserProfileSettings.tsx` — password and account-deletion forms → `sentry-block`
### Billing & Payment
- Any Stripe Elements iframe — automatically masked by Stripe (no action needed).
- Custom payment forms (e.g. invoice address) → `sentry-block`.
- Saved-card list display → `sentry-mask`.
### Tenant data (consider per-feature)
- Customer/contact detail views (`crm/*`) → `sentry-block` for `email`, `phone`, `address` columns.
- Helpdesk threads (`help-desk/*`) — body content of customer messages → `sentry-block`.
- Email-marketing composer body → `sentry-block`.
- File preview/download → already covered by `blockAllMedia` if enabled per-environment.
### Admin surface
- User-management table — email/phone columns → `sentry-mask`.
- Tenant-admin impersonation banner (`apps/web/app/components/impersonation-banner.tsx`) — both email props → `sentry-mask`.
- Audit-log viewer (`apps/web/app/routes/admin/audit-trails.tsx`) — `metadata` cells → `sentry-mask`.
## Network recording
Sentry Replay can also record network responses. We keep this **off by default** because API responses can contain JSON with PII (tenant data, user attributes). Operators may turn it on per-issue inside Sentry when reproduction without network detail is impossible. To enable globally:
```ts
Sentry.replayIntegration({
// ...,
networkDetailAllowUrls: [], // explicitly allow specific paths only
networkRequestHeaders: ["X-Request-Id"],
networkResponseHeaders: ["X-Response-Source"],
});
```
Never add `*` to `networkDetailAllowUrls`.
## Verifying masking on staging
1. Trigger a real user error on staging (e.g. submit a form with an invalid value to a route that throws).
2. In Sentry UI → Replays → open the captured session.
3. Step through the timeline and check:
- All password fields show only `*` placeholders.
- Email/profile fields in marked regions are redacted.
- `<img>` tags load if `blockAllMedia` was left `false`.
- Network tab does not contain JSON response bodies (only headers + status codes).
4. If any sensitive value is visible, file a `BLOCKERS.md` entry referencing this document and add the missing annotation in a follow-up commit.
## Cross-references
- Task spec: `docs/improvement-plan/02-observability/sentry-wiring-web.md`
- Server-side PII strategy: `apps/web/app/sentry.server.ts` (`requestDataIntegration` with `ip: false`)
- Query-string scrubbing: `apps/web/app/sentry.client.ts#scrubSensitiveQueryParams`
hivegrid-cloud/docs/observability/sentry-selfhost.md
+389
View File
@@ -0,0 +1,389 @@
# Self-Hosted Sentry — Operations Runbook
This runbook covers the **self-hosted Sentry deployment** that lives in
`infra/sentry/`. It is the operational companion to
`docs/improvement-plan/02-observability/sentry-selfhost-setup.md` (the
planning artefact) and assumes you have current operator-approved SSH access to
the `boilerplate-staging` host. The archived
`docs/improvement-plan/archive/STAGING-ACCESS.md` file is historical inventory
only; prefer live Coolify/Hetzner access records when they differ.
For the lightweight alternative (GlitchTip, no Performance/Replay/OTLP), see
[`sentry-glitchtip-alternative.md`](./sentry-glitchtip-alternative.md).
## At a glance
| Item | Value |
| --- | --- |
| Upstream | `getsentry/self-hosted` tag **`25.12.1`** |
| Public hostname | `https://sentry.boiler.saas-ctrl.com` |
| Host | `boilerplate-staging` (`46.225.5.93`, Hetzner Nürnberg) |
| Orchestrator | Coolify-managed Docker Compose, Traefik v3.6 in front |
| Persistent volumes | `/data/coolify/services/sentry/{postgres,clickhouse,kafka,zookeeper,redis,symbolicator}-data` |
| OTLP endpoint | `https://sentry.boiler.saas-ctrl.com/api/{project_id}/integration/otlp/v1/traces` |
| Backup target | `s3://${SENTRY_S3_BACKUP_BUCKET}/sentry/YYYY-MM-DD/` (encrypted) |
## 1. First-time deploy
1. **Provision DNS.** Add an `A` record for
`sentry.boiler.saas-ctrl.com` pointing at `46.225.5.93`. TLS is
issued automatically by Traefik via the `letsencrypt` ACME resolver on
first request.
2. **Fill secrets.** SSH to the host, then:
```bash
cd /data/coolify/services/sentry
git clone git@git.tool-ctrl.de:toolctrl-labs/boilerplate.git .
cd infra/sentry
cp .env.example .env
# Edit .env — fill SENTRY_ADMIN_EMAIL, SENTRY_ADMIN_PASSWORD,
# SENTRY_SECRET_KEY (generate with `python -c "import secrets; print(secrets.token_urlsafe(50))"`),
# SENTRY_S3_BACKUP_*.
chmod 0600 .env
```
3. **Run the installer.** From `infra/sentry/`:
```bash
./install.sh
```
The wrapper clones the pinned upstream tag, applies our overlays,
sources `.env`, and runs the upstream installer in non-interactive
mode (`--skip-user-prompt --no-report-self-hosted-issues`). Expect
**5–10 minutes** on first run because Snuba migrations take a while
on a cold ClickHouse.
4. **Verify.**
- Web UI is reachable: `curl -I https://sentry.boiler.saas-ctrl.com`
returns `200`.
- Admin login works with the credentials from `.env`.
- `/auth/register/` returns **404** (public signup disabled).
- OTLP endpoint accepts an empty payload:
```bash
curl -X POST "https://sentry.boiler.saas-ctrl.com/api/<PROJECT_ID>/integration/otlp/v1/traces" \
-H "Content-Type: application/json" \
-H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=<PUBLIC_KEY>" \
-d '{"resourceSpans":[]}'
# Expect: 202
```
5. **Create the three Sentry projects** in the Sentry UI:
- `boilerplate-api`
- `boilerplate-worker`
- `boilerplate-web`
Copy their DSNs into `infra/sentry/.env` (`SENTRY_DSN_API`,
`SENTRY_DSN_WORKER`, `SENTRY_DSN_WEB`) and into the Coolify secret
vault for the respective app containers. The five Coolify worker apps
(`worker-general`, `worker-mail`, `worker-import`, `worker-webhook`,
`worker-plugin`) intentionally share the server-side
`boilerplate-worker` project and `SENTRY_DSN_WORKER` unless the operator
explicitly creates per-group Sentry projects later. Per-project
allowed-domains list is filled in for the `boilerplate-web` project only
(API + Worker DSNs are server-side, no browser exposure).
6. **Enable 2FA** on the admin account immediately (User Settings →
Security → Two-Factor → TOTP). The Hardening Checklist below treats
admin 2FA as **mandatory**.
## 2. Coolify integration
Coolify on this host runs as a **target** (the controller dashboard is on
a separate host; verify against live operator access records, not the archived
STAGING-ACCESS inventory). After the first manual
`install.sh` run:
- In Coolify, register the `infra/sentry/upstream/self-hosted/` Compose
stack as a managed service. Coolify will not re-pull `infra/sentry/` on
every push to `main` (the directory of overlays is read by
`install.sh`, not by Coolify directly).
- Trigger `install.sh` again whenever overlays change. Suggested
pattern: add a `post-deployment` hook on the Coolify service that
runs `cd /data/coolify/services/sentry/infra/sentry && ./install.sh`
whenever the parent repo changes.
## 3. Patch upgrade (`25.12.x → 25.12.y`)
1. Read upstream release notes at
<https://github.com/getsentry/self-hosted/releases>.
2. Bump `UPSTREAM_TAG` in `infra/sentry/install.sh` AND update
`infra/sentry/UPSTREAM.md`.
3. If upstream renamed any `sentry.conf.py` setting we override, mirror
the change in `infra/sentry/sentry/sentry.conf.py`.
4. Commit and push to `main`. Coolify auto-deploys; SSH to the host and
run `./install.sh` (it is idempotent and runs only the new
migrations).
5. Smoke-test (Section 1, step 4).
## 4. Minor upgrade (`25.x.* → 25.x+1.*`)
Same procedure as Section 3, but:
- Run the **backup container manually first** (`docker compose run --rm
backup /srv/backup/backup.sh`) and verify the dump landed in S3.
- Read **every** intermediate release note (Sentry historically slips
Snuba-table migrations into minor releases).
- After the installer completes, manually re-run the `cleanup` Django
command to defragment ClickHouse:
```bash
docker compose exec web sentry cleanup --days 90
```
## 5. Major upgrade (`v25 → v26`)
**This is a disaster-recovery-grade operation. Schedule a maintenance
window.**
1. **Pre-flight backup.** Run `docker compose run --rm backup
/srv/backup/backup.sh` AND `docker compose down`. Take a host-level
tar of `/data/coolify/services/sentry/{postgres,clickhouse}-data` and
copy it off-box.
2. **Read every changelog** between the source and target tag. Pay
particular attention to Kafka consumer renames, Snuba table schema
changes, and Symbolicator API breaks.
3. **Bump `UPSTREAM_TAG`** in `install.sh` + `UPSTREAM.md`. Update our
overlays for any renamed config keys.
4. **Run install.sh on a clean clone first** (e.g.
`/data/coolify/services/sentry-test/`) pointed at a restored copy of
the production volumes. Verify the migration completes cleanly and
the UI loads.
5. **Switch over.** Stop the old stack, repoint Traefik to the new
stack (Compose label rename), bring up the new stack with the live
volumes.
6. **Rollback path.** If the new stack misbehaves, `docker compose
down`, restore the pre-flight volume tar over
`/data/coolify/services/sentry/`, redeploy the OLD `UPSTREAM_TAG` via
`install.sh`.
## 6. Event cleanup / disk pressure
B-008 is closed as an operator alerting item: Coolify disk-pressure alerting is
enabled, so future disk-full risk is monitored rather than tracked as an active
code blocker. The remaining responsibilities in this section are
operator-owned Sentry/Postgres retention and backup hygiene; repository changes
are not expected unless this runbook or the backup scripts drift.
Sentry's `cleanup` command is idempotent and safe to run any time:
```bash
docker compose exec web sentry cleanup --days 90
```
Run weekly via crontab (or as a Coolify scheduled task) when retention
should drop below 90 days. ClickHouse compacts partitions automatically
afterwards but it can take 24h to reclaim disk.
To inspect current disk usage:
```bash
docker compose exec clickhouse \
clickhouse-client --query "SELECT database, formatReadableSize(sum(bytes)) AS size \
FROM system.parts WHERE active GROUP BY database ORDER BY sum(bytes) DESC"
```
## 7. Backup + restore drill
Backups are **continuous and encrypted** (see `infra/sentry/backup/`).
Restore drill quarterly:
1. Spin up an empty Sentry stack on a scratch host.
2. Pull the latest dump from S3:
```bash
aws s3 cp --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" \
"s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/postgres-*.dump.gpg" .
```
3. Decrypt: `gpg --decrypt --output postgres.dump postgres-*.dump.gpg`
4. Restore: `pg_restore --clean --if-exists --no-owner --no-privileges -d postgres postgres.dump`
5. Repeat for ClickHouse (`gpg --decrypt | tar -xz`, then
`clickhouse-client --query "INSERT INTO db.table FORMAT Native"` per
file).
6. Boot the stack and verify the last day of events is visible in the
UI.
## 8. Monitoring health
Out-of-the-box checks (no external monitoring required):
| Check | Command |
| --- | --- |
| Web up | `curl -I https://sentry.boiler.saas-ctrl.com` |
| Relay healthy | `curl https://sentry.boiler.saas-ctrl.com/api/relay/healthcheck/ready/` |
| Postgres reachable | `docker compose exec postgres pg_isready` |
| ClickHouse reachable | `docker compose exec clickhouse clickhouse-client --query "SELECT 1"` |
| Kafka topics | `docker compose exec kafka kafka-topics --bootstrap-server kafka:9092 --list` |
| Workers running | `docker compose ps worker cron snuba-consumer` |
| Backup heartbeat | `docker compose exec backup cat /srv/backup/logs/heartbeat.txt` |
| Last backup landed | `aws s3 ls --endpoint-url "$SENTRY_S3_BACKUP_ENDPOINT" "s3://$SENTRY_S3_BACKUP_BUCKET/sentry/$(date -u +%Y-%m-%d)/"` |
For continuous monitoring, point Coolify's Uptime/Heartbeat plugin (or
Healthchecks.io) at the `relay/healthcheck/ready/` URL above.
## 9. Volume sizing guidance
With the staging host's 32 GB RAM / 75 GB disk:
| Component | Disk (90 d retention, small team) | RAM |
| --- | --- | --- |
| Postgres | 2–5 GB | 1–2 GB |
| ClickHouse | 20–40 GB | 4–8 GB |
| Kafka | 5–10 GB | 1–2 GB |
| Symbolicator cache | 2–5 GB | 0.5–1 GB |
| Redis | <500 MB | <500 MB |
| Sentry worker pool | n/a | 2–4 GB |
If ClickHouse usage approaches 60 GB, lower retention via `sentry
cleanup --days 30` and reduce `--days` further if needed.
## Hardening Checklist
The checklist below is **mandatory** before declaring this deployment
production-ready. Each item maps to a setting we ship; verifying them is
a one-time operator pass plus a quarterly review.
- [ ] **Public signup disabled.** `sentry.conf.py` sets
`SENTRY_FEATURES['auth:register'] = False`. Verify:
`curl -I https://sentry.boiler.saas-ctrl.com/auth/register/` →
`404`.
- [ ] **Admin 2FA enabled.** In the UI: User Settings → Security → 2FA
(TOTP). Recovery codes printed and stored offline.
- [ ] **Default admin password rotated.** The bootstrap password from
`.env` is replaced via UI after first login. The `.env` value
becomes a recovery secret only; rotate again every 90 days.
- [ ] **Relay rate-limit active.** `relay/config.yml` caps each project
at 100 events/sec. Verify by sending a burst and watching for
`429` responses.
- [ ] **Per-project allowed-domains list filled.** Project Settings →
Client Keys → Allowed Domains. Set for `boilerplate-web` to:
`https://boiler.saas-ctrl.com`,
`https://www.boiler.saas-ctrl.com`,
`https://admin.boiler.saas-ctrl.com`. Leave **empty** for the
`boilerplate-api` and `boilerplate-worker` projects (server-side
DSNs only).
- [ ] **Postgres / ClickHouse / Kafka / Redis NOT public.**
`docker-compose.override.yml` uses `ports: !reset []` to strip
host-port bindings inherited from upstream. Verify:
`ss -tlnp | grep -E '5432|9000|9092|6379'` on the host shows
**no** matches.
- [ ] **Backups encrypted.** `backup.sh` uses GPG (recipient-key if
`SENTRY_BACKUP_GPG_RECIPIENT` is set, AES256 symmetric with
`SENTRY_SECRET_KEY` otherwise). Verify: `aws s3 cp` of a recent
dump returns a file with header bytes `8c 0d` (GPG-armored) or
`85 02` (binary GPG).
- [ ] **Symbolicator: no external sym sources.** `sentry.conf.py` sets
`SYMBOLICATOR_SYMSOURCES = []`. Verify: `docker compose exec
symbolicator cat /etc/symbolicator/config.yml | grep sources`
shows no Microsoft / Mozilla / Apple entries.
- [ ] **SENTRY_SECRET_KEY stored offline.** Lost key = unreadable
backups. Copy from Coolify vault to a hardware-backed password
manager (1Password, BitWarden Premium with TOTP, YubiKey-backed
pass).
- [ ] **Traefik dashboard port 8080 firewalled.** See
`docs/improvement-plan/archive/STAGING-ACCESS.md` open item; close at Hetzner Cloud Firewall
level or front it with basic-auth.
- [ ] **`.env` permissions ≤ 0600.** `ls -l infra/sentry/.env`. The
installer enforces 0600 when it copies the file into the upstream
tree.
Re-run this checklist (a) after every patch upgrade, (b) at least once
per quarter, (c) before opening Sentry to additional users.
## CI Secrets
The CI pipelines in `.github/workflows/_sentry-release.yml` and
`.gitea/workflows/_sentry-release.yml` need three Sentry secrets to tag
releases, upload source-maps, correlate commits, and mark deploys. The
main CI build uploads `apps/web/build/client` as the
`web-client-sourcemaps` workflow artifact; the reusable web release job
downloads that artifact before calling `scripts/ci/sentry-release.sh`.
The release job must fail if the source-map directory is absent, so a
fresh checkout cannot silently publish a web release without artifacts.
The web Vite build can also upload source-maps directly when
`SENTRY_AUTH_TOKEN` is intentionally present in that build environment.
The default CI path keeps the token scoped to the reusable release job
and passes source-maps through the workflow artifact instead.
### 1. Mint an Auth-Token in the Sentry UI
1. Sign in at <https://sentry.boiler.saas-ctrl.com> as the admin user.
2. Open **Settings → Account → API → Auth Tokens** (this is a
user-scoped token; Sentry deliberately does not expose org-scoped
integration tokens in single-organisation mode).
3. Click **Create New Token**, give it a descriptive name (e.g.
`ci-release-token-2026-05`), and select the **single** scope
`project:releases`. **Do not** select `org:admin`, `org:write`, or
`member:admin`.
4. Copy the token immediately — Sentry only shows it once.
### 2. Store secrets in GitHub Actions
Navigate to **Settings → Secrets and variables → Actions** in the
GitHub mirror (`github.com/<org>/boilerplate`) and add three repository
secrets:
| Name | Value |
| --- | --- |
| `SENTRY_AUTH_TOKEN` | The token minted in step 1 |
| `SENTRY_ORG` | `sentry` (the auto-generated slug from single-org install) |
| `SENTRY_URL` | `https://sentry.boiler.saas-ctrl.com` |
Once configured, `_sentry-release.yml` is invoked once per project
(api/worker/web) at the end of every CI run via `workflow_call`.
Only the web invocation passes `sourcemaps-artifact:
web-client-sourcemaps`; api and worker releases pass `sourcemaps-path:
none`.
### 3. Store secrets in Gitea Actions
In Gitea (`git.tool-ctrl.de/toolctrl-labs/boilerplate`) open
**Repository Settings → Actions → Secrets** and create the same three
secrets (`SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_URL`) with
identical values. Gitea's secret scoping rules mirror GitHub's, so the
reusable workflow at `.gitea/workflows/_sentry-release.yml` (and the
composite-action fallback at `.gitea/actions/sentry-release/action.yml`
for runner versions < 1.21) will pick them up automatically.
The deploy workflow (`.gitea/workflows/deploy.yml`) reads the same
secrets in its **Mark Sentry deploy** step; if the secrets are absent
the step skips silently so existing deploy paths continue to function
during the rollout window.
### 4. Rotation
- **Rotate every 6 months.** Add a recurring calendar entry for the
operator on-call.
- Rotation procedure:
1. Mint a new token (step 1) with the same `project:releases` scope.
2. Update `SENTRY_AUTH_TOKEN` in **both** GitHub and Gitea secret
stores within the same maintenance window.
3. Wait for one CI run on each system to confirm the new token works
(look for a fresh release in the Sentry UI).
4. Revoke the old token via the Sentry UI (Settings → Account →
API → Auth Tokens → trash icon).
- **Next scheduled rotation: 2026-11-17.** Update this date after each
rotation.
### 5. Local development
Releases are a CI-only concern. Local developers do **not** need
`SENTRY_AUTH_TOKEN`; the web Vite build only uploads source-maps when
the token is present, and the api/worker bootstrap modules never call
the Releases API.
If you want to test the helper locally, mint a personal token with
`project:releases` scope and run:
```bash
SENTRY_AUTH_TOKEN=<token> \
SENTRY_ORG=sentry \
SENTRY_URL=https://sentry.boiler.saas-ctrl.com \
bash scripts/ci/sentry-release.sh \
--project boilerplate-api \
--environment preview \
--release "$(git rev-parse HEAD)" \
--sourcemaps none
```
Delete the test release afterwards with
`sentry-cli releases delete <release>`.