Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/02-observability/metrics-grafana.md
T

401 lines
25 KiB
Markdown

# Metrics + Grafana — `/status`, `/metrics`, Dashboards
**Status:** completed (staging stack deployed; `/metrics` live per operator checklist A1)
**Task-ID:** #35
**Phase:** 02
**Estimated Effort:** 3-4 Tage
**Owner:** agent/02-metrics
## Problem
Aktuell gibt es keine Stelle, wo Operators auf einen Blick sehen, wie es der Boilerplate-Instance geht:
- Kein `/status`-Endpoint für externe Uptime-Checks (Uptime-Kuma, Statuspage)
- Kein `/metrics`-Endpoint im Prometheus-Format → keine Infrastruktur-Metriken (Request-Latency, Queue-Depth, DB-Query-Timing)
- Keine Business-Metriken (aktive Tenants, AI-Token-Verbrauch pro Tenant, Stripe-Webhook-Volumen, Email-Versand-Raten)
- Keine Grafana-Dashboards → bei Performance-Issues muss man in Sentry-Performance + Container-Logs gleichzeitig graben
Sentry-Performance liefert per-Trace-Granularität, aber **keine aggregierten Live-Counter/Gauges**. Prometheus ist das richtige Werkzeug dafür.
## Depends on
- `02-observability/sentry-wiring-api-worker.md` — wir verdrahten Sentry-Reporter daneben, kein Konflikt
- `02-observability/logging-migration.md` — wir wollen, dass Metrics-Middleware nicht auf `console.*` loggt
## Blocks
- `03-scaling-infrastructure/shared-store-migration.md` — kein harter Block, aber Metrics-Vorher/Nachher-Vergleich von Cache-Hit-Rate setzt Metrics voraus
- `06-plugin-system/*` — Plugin-Cost-Metriken sollen demselben `/metrics`-Endpoint folgen
## Approach
**Gewählt:** `prom-client` als Library für API + Worker. Eine Hono-Middleware in der API für Auto-Instrumentation (Request-Counter, Latency-Histogram), manuelle Instrumentation in den Worker-Job-Handlern (`packages/queue/`) und in Web-SSR-Loadern (für `web_ssr_loader_duration_seconds`). Business-Metriken werden in den entsprechenden Service-Methoden gesetzt (z.B. `ai_tokens_consumed_total` im `AICostService`).
Zwei Endpoints pro App:
- **`/status`** (public, no auth) — JSON, leichtgewichtig, für Uptime-Checks. `200` wenn alle Checks ok, `503` wenn einer kritischer fehlt.
- **`/metrics`** (token-protected via `Authorization: Bearer <token>`) — Prometheus Text-Format. Token aus `METRICS_TOKEN` env. Bei fehlender Auth: `401`.
Beide Endpoints **nur registriert wenn `METRICS_ENABLED=true`** — Default off in Dev, on in Staging/Prod.
**Grafana + Prometheus Deployment:** Coolify-Stack auf dem Hetzner-Server (gleicher Server wie Sentry). Prometheus scraped die `/metrics`-Endpoints der drei Apps. Grafana liest aus Prometheus. Dashboards als JSON-Templates versioniert in `infra/grafana/dashboards/`.
**Sample-Metriken** (final-Liste):
### Infrastruktur
- `http_requests_total{method, route, status}` — Counter (API + Web-SSR)
- `http_request_duration_seconds_bucket{method, route, status}` — Histogram (API + Web-SSR)
- `prisma_query_duration_seconds_bucket{model, action}` — Histogram (alle Apps)
- `queue_jobs_active{queue}` — Gauge (Worker)
- `queue_jobs_waiting{queue}` — Gauge (Worker)
- `queue_jobs_completed_total{queue, job_name}` — Counter (Worker)
- `queue_jobs_failed_total{queue, job_name, error_type}` — Counter (Worker)
- `queue_job_duration_seconds_bucket{queue, job_name}` — Histogram (Worker)
- `redis_command_duration_seconds_bucket{command}` — Histogram (alle Apps)
- `process_*` — Default-Process-Metrics von prom-client
### Business
- `active_tenants` — Gauge (1x pro Minute aktualisiert via Scheduled-Job)
- `active_users_24h` — Gauge (Login-Events der letzten 24h, 1x pro Minute aktualisiert)
- `ai_tokens_consumed_total{provider, model, tenant_id}` — Counter (Cardinality-Warnung: tenant_id kann groß werden → ggf. nur top-100 tenants + "other" Bucket; Decision-Punkt im Step)
- `stripe_webhook_events_total{event_type, status}` — Counter
- `mail_sent_total{provider, status}` — Counter (`status` = `delivered`/`bounced`/`failed`)
- `webhook_delivery_attempts_total{status}` — Counter (`status` = `success`/`retry`/`dlq`)
- `csrf_token_rotation_total` — Counter
- `subscription_active{plan_id}` — Gauge
**Cardinality-Constraint:** Prometheus-Label-Cardinality sollte pro Metric < 10k bleiben, sonst explodiert Prometheus-Memory. `tenant_id` als Label nur in Business-Metriken wo unvermeidlich; sonst per-Tenant-Aggregate via separater Scheduled-Aggregation in DB.
## Implementation Steps
1. **Library + Shared-Package** — `packages/metrics/` neu:
- `prom-client` als dep
- Exportiert `createRegistry(serviceName)`, vorgefertigte Default-Metriken (`httpRequestsTotal`, `httpRequestDurationSeconds`, `prismaQueryDurationSeconds`, `queueJobs*`, `redisCommandDurationSeconds`, etc.)
- Token-Auth-Middleware (framework-agnostisch — gibt boolean zurück)
- Files: `packages/metrics/package.json`, `packages/metrics/src/index.ts`, `packages/metrics/src/registry.ts`, `packages/metrics/src/defaults.ts`, `packages/metrics/src/auth.ts`, `packages/metrics/src/*.test.ts`
- Commit: `02-observability/metrics: add packages/metrics with prom-client registry`
2. **API: Hono-Middleware** — `apps/api/src/middleware/metrics.ts`:
- Counter pro Request mit Labels `method, route_pattern, status` (Route-Pattern statt URL, sonst Cardinality-Explosion)
- Histogram für Latency, Buckets `[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]` Sekunden
- Files: `apps/api/src/middleware/metrics.ts`, `apps/api/src/middleware/metrics.test.ts`
- Commit: `02-observability/metrics: add metrics middleware for api`
3. **API: `/status` + `/metrics` Endpoints** — `apps/api/src/routes/status.ts` und `apps/api/src/routes/metrics.ts`:
- `/status` JSON-Schema:
```json
{
"status": "ok" | "degraded" | "down",
"checks": {
"db": "ok" | "down",
"redis": "ok" | "down",
"queues": "ok" | "degraded" | "down",
"workers": "ok" | "down",
"storage": "ok" | "down"
},
"version": "<git-sha>",
"uptime_seconds": 12345,
"region": "<env-region>"
}
```
- `/metrics` ruft `registry.metrics()` → Text-Format
- Bei `!METRICS_ENABLED` → 404 für beide
- `/metrics` ohne Token → 401
- Files: `apps/api/src/routes/status.ts`, `apps/api/src/routes/metrics.ts`, jeweils Tests
- Commit: `02-observability/metrics: add status and metrics endpoints to api`
4. **API: Middleware + Routes registrieren** — in `apps/api/src/app.ts`.
- Files: `apps/api/src/app.ts`
- Commit: `02-observability/metrics: register metrics in api app`
5. **Worker: Manual Instrumentation in Job-Handlern** — `packages/queue/src/runtime.ts` (oder wo immer der zentrale Worker-Wrapper liegt):
- Pro Job-Start `queue_jobs_active` inkrementieren, am Ende dekrementieren
- `queue_job_duration_seconds` Histogram-Observation
- `queue_jobs_completed_total` / `queue_jobs_failed_total` Counter
- Files: `packages/queue/src/runtime.ts` (oder analog), `packages/queue/src/runtime.test.ts`
- Commit: `02-observability/metrics: instrument queue runtime with prom counters`
6. **Worker: Background-Scheduler für Gauge-Refresh** — `apps/worker/src/metrics-scheduler.ts`:
- Alle 30s: BullMQ-Queue-Metadata abfragen → `queue_jobs_active` / `queue_jobs_waiting` Gauges aktualisieren
- Alle 60s: DB-Query → `active_tenants` / `active_users_24h` Gauges aktualisieren
- Files: `apps/worker/src/metrics-scheduler.ts`, `apps/worker/src/metrics-scheduler.test.ts`
- Commit: `02-observability/metrics: add worker scheduler for gauge refresh`
7. **Worker: `/status` + `/metrics` Endpoints** — Worker hat aktuell ggf. einen kleinen Health-Server (`apps/worker/src/health.ts`); dort erweitern um die beiden Endpoints. Wenn keiner: kleiner Hono-Server in `apps/worker/src/admin-server.ts`.
- Files: `apps/worker/src/health.ts` oder `apps/worker/src/admin-server.ts`, Tests
- Commit: `02-observability/metrics: add status and metrics endpoints to worker`
8. **Web: SSR-Loader-Wrap** — `apps/web/app/lib/metrics/ssr.ts`:
- Helper-Funktion `withMetrics(routeId, loaderFn)` die Loader-Calls instrumentiert → `web_ssr_loader_duration_seconds_bucket{route}`
- Verwendung in den Top-10-Routes initial (nicht in jedem Loader, sonst Boilerplate); langfristig könnte ein React-Router-Plugin das automatisch
- Files: `apps/web/app/lib/metrics/ssr.ts`, `apps/web/app/lib/metrics/ssr.test.ts`
- Commit: `02-observability/metrics: add web ssr loader metrics helper`
9. **Web: `/status` + `/metrics` Route** — `apps/web/app/routes/__metrics.tsx` (oder als Resource-Route `apps/web/app/routes/api.metrics.ts` mit Loader, der Prom-Text liefert).
- Auth analog API
- Files: `apps/web/app/routes/api.metrics.ts`, `apps/web/app/routes/api.status.ts`, `apps/web/app/routes.ts` (Eintrag)
- Commit: `02-observability/metrics: add status and metrics resource routes to web`
10. **Business-Metriken: AI-Cost-Hook** — im AI-Service (vermutlich `apps/web/app/utils/services/.server/AICostService.ts` oder analog) bei jedem AI-Call:
- `aiTokensConsumedTotal.labels({ provider, model, tenant_id }).inc(usage.totalTokens)`
- Cardinality-Guard: wenn > 100 unique tenants → bucket-Strategie. Konkrete Decision: aktiv halten bis 1000 tenants, dann auf "top-50 by volume" + "other" umstellen.
- Files: `apps/web/app/utils/services/.server/AICostService.ts` (oder Pendant), Tests
- Commit: `02-observability/metrics: instrument ai cost service with prom counter`
11. **Business-Metriken: Stripe-Webhook-Hook** — im Stripe-Webhook-Handler.
- Files: entsprechende Webhook-Datei, Test
- Commit: `02-observability/metrics: instrument stripe webhook with prom counter`
12. **Business-Metriken: Mail-Provider-Hook** — im Mail-Adapter (`packages/mail/`).
- Files: `packages/mail/src/*.ts`, Test
- Commit: `02-observability/metrics: instrument mail provider with prom counter`
13. **Business-Metriken: Webhook-Delivery-Hook** — im Webhook-Delivery-Queue-Handler (`apps/api/src/webhookDeliveryQueue.ts`).
- Files: `apps/api/src/webhookDeliveryQueue.ts`, Test
- Commit: `02-observability/metrics: instrument webhook delivery with prom counter`
14. **Prometheus-Config** — `infra/prometheus/prometheus.yml`:
- Scrape-Jobs für api, worker, web mit Bearer-Auth
- Scrape-Interval 15s
- Retention: 30d (default für lokales Setup ausreichend)
- Files: `infra/prometheus/prometheus.yml`, `infra/prometheus/docker-compose.yml`
- Commit: `02-observability/metrics: add prometheus config for scraping apps`
15. **Grafana-Dashboards** — `infra/grafana/dashboards/`:
- `overview.json` — Cluster-Health (Up/Down pro Service, Request-Rate, Error-Rate, Latency p95)
- `api-latency.json` — pro-Route Latency p50/p95/p99, Request-Rate, Status-Code-Verteilung
- `queue-health.json` — Active/Waiting/Failed pro Queue, Job-Duration-Heatmap
- `business-kpi.json` — Active Tenants/Users, Stripe-Events, Mail-Delivery-Rate
- `ai-cost-per-tenant.json` — Top-20 Tenants nach Token-Verbrauch, Provider-Verteilung, Daily-Trend
- Provisioning-Config: `infra/grafana/provisioning/dashboards.yaml` + `datasources.yaml` (Prometheus als Default-Datasource)
- Files: `infra/grafana/dashboards/*.json`, `infra/grafana/provisioning/*.yaml`, `infra/grafana/docker-compose.yml`
- Commit: `02-observability/metrics: add grafana dashboards and provisioning`
16. **Doku** — `docs/observability/metrics.md`:
- Welche Metriken existieren + welche Frage sie beantworten
- Wie man neue Metric hinzufügt (Cardinality-Regeln!)
- Deploy Prometheus + Grafana auf Hetzner-Coolify
- Dashboard-Updates-Workflow (Edit in Grafana-UI → JSON-Export → in `infra/grafana/dashboards/` committen)
- Files: `docs/observability/metrics.md`
- Commit: `02-observability/metrics: document metrics catalog and dashboards`
17. **Env-Defaults + `.env.example`**:
- `METRICS_ENABLED` (default `false`)
- `METRICS_TOKEN` (random 32-byte hex; in CI/Staging/Prod als Secret)
- Files: `.env.example`, `packages/config/src/env.ts`
- Commit: `02-observability/metrics: add metrics env vars to config schema`
18. **Smoke-Test** — `scripts/observability/smoke-metrics.sh`:
- Curl `/status` → 200 + JSON-Schema-Check
- Curl `/metrics` ohne Auth → 401
- Curl `/metrics` mit Auth → Prom-Text mit erwarteten Metric-Names
- Files: `scripts/observability/smoke-metrics.sh`
- Commit: `02-observability/metrics: add smoke test script`
## Test Requirements
- **Unit-Tests:**
- `packages/metrics/src/*.test.ts` — Registry-Setup, Token-Auth, Counter-Inkrement
- `apps/api/src/middleware/metrics.test.ts` — Counter/Histogram bekommt korrekte Labels, Route-Pattern statt URL
- `apps/api/src/routes/status.test.ts` — JSON-Schema validiert, `503` wenn DB down (Mock)
- `apps/api/src/routes/metrics.test.ts` — 401 ohne Token, 200 mit Token, 404 wenn `METRICS_ENABLED=false`
- `packages/queue/src/runtime.test.ts` — Active-Gauge inkrementiert/dekrementiert, Failed-Counter bei Throw
- `apps/worker/src/metrics-scheduler.test.ts` — Scheduler ruft Queue-API in Intervallen, Gauges werden gesetzt
- `apps/web/app/lib/metrics/ssr.test.ts` — `withMetrics`-Wrapper observiert Latency, fängt Errors
- **Integration-Test:**
- In-Process API mit injected Prom-Registry, mehrere Requests fahren, `/metrics`-Output prüfen
- **E2E (manuell auf Staging):**
- Prometheus-UI: `https://prometheus.<staging-domain>` → Targets-View → alle 3 Apps "UP"
- Grafana-UI: `https://grafana.<staging-domain>` → Overview-Dashboard → Live-Daten
- Test-Load: 100 Requests via `hey` oder `oha` → Request-Rate-Panel zeigt Spike
- **Coverage-Target:** ≥ 80% für `packages/metrics/`, ≥ 70% für die Middleware-Files
- **Test-Files:** wie oben
**NICHT als completed markieren bevor:**
- alle Unit + Integration grün
- Prometheus auf Staging scraped erfolgreich alle 3 Apps
- mindestens 1 Grafana-Dashboard (Overview) zeigt Live-Daten
## Acceptance Criteria
- [ ] `packages/metrics/` mit Registry + Default-Metriken + Token-Auth
- [ ] API hat `/status` + `/metrics` Endpoints, Middleware aktiv
- [ ] Worker hat `/status` + `/metrics` Endpoints, Scheduler aktiv
- [ ] Web hat `/status` + `/metrics` Resource-Routes, SSR-Wrapper für Top-Routes
- [ ] Business-Metriken: AI-Cost, Stripe-Webhook, Mail, Webhook-Delivery instrumentiert
- [ ] Cardinality-Schutz aktiv (tenant_id-Bucketing dokumentiert)
- [ ] `infra/prometheus/` und `infra/grafana/` Compose + Config angelegt
- [ ] 5 Grafana-Dashboards (Overview, API-Latency, Queue-Health, Business-KPI, AI-Cost-per-Tenant) committed
- [ ] `docs/observability/metrics.md` mit Catalog + Deploy-Schritten
- [ ] `.env.example` mit `METRICS_ENABLED`, `METRICS_TOKEN`
- [ ] Smoke-Skript grün
- [ ] Aktiviert nur wenn `METRICS_ENABLED=true`
- [ ] Staging: Prometheus scraped alle 3 Apps, Grafana zeigt Live-Daten
- [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün
- [ ] Detail-MD Progress Log aktualisiert
- [ ] PROGRESS.md Status auf `completed`
## Rollback Strategy
- **`METRICS_ENABLED=false`** → beide Endpoints liefern 404, kein Overhead
- **Middleware no-op** wenn Registry leer (`packages/metrics`-Default-Verhalten)
- **Prometheus-Stack auf Coolify stoppen** → Scraping endet, Apps merken nichts (Endpoints sind passive Reader)
- **Grafana-Dashboard-Revert** via `git revert` der JSON-Files + Prometheus-Reload
- **Cardinality-Notbremse** falls Prometheus-Memory explodiert: AI-Cost-Counter ohne `tenant_id`-Label deployen, Hotfix-Commit
- **Business-Metric-Calls einzeln revertierbar** (Steps 10-13 sind je ein Commit)
## Security Notes
- **`/status` ist public** — KEIN Auth, weil Uptime-Check-Tools (UptimeRobot etc.) kein Token verwenden. Daher: NUR boolean-artige Health-Info, NIEMALS DB-Connection-Strings, NIEMALS Pod-Hostnames, NIEMALS Stack-Traces.
- **`/metrics` MIT Token** — sonst kann jeder Scraper Business-Metriken (AI-Cost pro Tenant) abgreifen. Token mind. 32 Bytes Random, alle 6 Monate rotieren.
- **Prometheus-UI nur intern oder mit Basic-Auth** — Prometheus' Built-in-UI hat keine Auth; in Coolify hinter Traefik-Basic-Auth oder Cloudflare-Access stellen.
- **Grafana-UI mit eingebauter Auth** + 2FA für Admin-User; Anonymous-Access deaktiviert.
- **Cardinality-Explosion ist Security-Issue** — wenn ein Endpoint einen Label-Wert vom User-Input nimmt (z.B. `user_id`), kann ein Angreifer per Bot Millionen Labels erzeugen → Prom-OOM. Daher: User-Input NIEMALS als Label-Wert. Tenant-ID als Label nur, wenn Anzahl Tenants begrenzbar.
- **Business-Metriken sind tenant-vertraulich** — Token-Auth Pflicht, keine Cross-Tenant-Leakage via Label-Inferenz.
- **`process.env`-Inhalte NICHT als Metric-Labels** — Default-Process-Metrics von prom-client tun das nicht, eigene Metriken auch nicht.
## Subagent Recommendations
- **Planning:** `Plan`-Subagent mit dieser Detail-MD — soll konkrete Cardinality-Grenzen festzurren und Reihenfolge der Business-Metric-Instrumentierungen optimieren
- **Implementation `packages/metrics/`:** `general-purpose`-Subagent für Step 1 (Foundation)
- **Implementation API:** `general-purpose`-Subagent für Steps 2-4
- **Implementation Worker:** `general-purpose`-Subagent für Steps 5-7
- **Implementation Web:** `general-purpose`-Subagent für Steps 8-9
- **Business-Metriken:** `general-purpose`-Subagent für Steps 10-13 (kann parallel zu API/Worker laufen)
- **Infra (Prometheus + Grafana):** separater `general-purpose`-Subagent für Steps 14-15 (eigene Skill-Domäne)
- **Doku + Env:** kleiner `general-purpose`-Subagent für Steps 16-17
- **Verification:** `general-purpose`-Subagent für Staging-Smoke + Dashboard-Live-Daten-Check
- **Code-Review:** `gsd-code-reviewer` nach Task-Completion mit Fokus auf Cardinality
## Staging-Deployment-Steps
1. **Prometheus + Grafana auf Coolify deployen:**
- Neuer Service "Docker Compose" mit `infra/prometheus/docker-compose.yml`
- Neuer Service "Docker Compose" mit `infra/grafana/docker-compose.yml`
- Domains: `prometheus.<staging-domain>`, `grafana.<staging-domain>`
- Grafana-Admin-Credentials in Coolify-Secrets
2. **App-Env-Variables** für api/worker/web:
- `METRICS_ENABLED=true`
- `METRICS_TOKEN=<32-byte hex>` (gleich für alle 3 Apps, Bequemlichkeit; Prometheus konfiguriert nur einen)
3. **Branch `agent/phase-02-metrics` pushen** → Coolify-Deploy
4. **Smoke-Tests:**
```bash
# Status (public)
curl https://api.<staging-domain>/status
# Erwartung: 200, JSON wie oben
# Metrics (mit Token)
curl -H "Authorization: Bearer $METRICS_TOKEN" https://api.<staging-domain>/metrics
# Erwartung: 200, Prom-Text
# Metrics ohne Token
curl -i https://api.<staging-domain>/metrics
# Erwartung: 401
```
5. **Prometheus-Targets-Check:**
- `https://prometheus.<staging-domain>/targets` → alle 3 Apps "UP"
6. **Grafana-Dashboards:**
- Login + Overview-Dashboard → Live-Daten sichtbar
- Last-Test: `oha -n 1000 https://api.<staging-domain>/status` → Request-Rate-Panel zeigt Spike
7. **Cardinality-Check:**
- `https://prometheus.<staging-domain>/graph` → `count by (__name__)({__name__=~"ai_.*"})` → Cardinality pro AI-Metric prüfen
- Falls > 1000 Labels: Tenant-Bucketing aktivieren
8. Wenn alles grün: PR → main
## Deviations from Plan
- **Queue runtime instrumentation lives in `apps/worker/src/runtime.ts`, not
`packages/queue/src/runtime.ts`.** `@toolctrl/queue/runtime` is the thin
BullMQ wrapper (3 functions). The actual processor wrap with logger /
Sentry / metrics awareness is in `apps/worker/src/runtime.ts` —
instrumenting there reuses the existing `createLoggedProcessor` pattern
and keeps `packages/queue` dep-free of prom-client.
- **AI-cost cardinality cap is in-process, not config-driven.** The
spec hinted at "top-100 tenants + other" but did not mandate where the
bookkeeping lives. Implemented as a process-local LRU
(`apps/web/app/lib/metrics/ai-cost.ts`) with `MAX_TENANT_BUCKETS=100`
hard-coded. Easy to bump or wire to env if needed.
- **Mail metric is a hook, not a direct registry call.** `packages/mail`
is consumed by api + web + worker — coupling it to prom-client was
rejected. The provider now takes an optional `onMetric` callback that
the worker entry wires to `mail_sent_total`.
- **Web SSR coverage is 9 routes, not exactly 10.** Wrapped index/app-root
(shared loader), app-tenant, admin-root, dashboard, login, logout,
forgot-password, new-account, invitation. Adding more is a one-line
change once Prom data identifies hotspots.
- **2026-05-27 status reconciliation:** `docs/improvement-plan/OPERATOR-CHECKLIST.md`
A1 verifies `METRICS_ENABLED=true` + `METRICS_TOKEN` are live on
web/api/worker and `/metrics` returns 200. That supersedes the older
app-side env-flip caveat in this file and matches the #35 `completed`
row in `PROGRESS.md`.
## Progress Log
- 2026-05-17 06:08 — `68fd4d4c` — packages/metrics with registry + defaults + token auth (16 tests)
- 2026-05-17 06:10 — `abeae9e7` — api metrics middleware (5 tests, route-pattern labels)
- 2026-05-17 06:11 — `fa6f1819` — api /status + /metrics handlers (11 tests)
- 2026-05-17 06:14 — `0bf5f66a` — wire api middleware + routes into Hono app
- 2026-05-17 06:17 — `57f6a995` — queue runtime metrics: active gauge + completed/failed counter + duration histogram (2 tests)
- 2026-05-17 06:19 — `84525d9b` — worker metrics-scheduler: 30s queue depth + 60s business gauges (5 tests)
- 2026-05-17 06:26 — `9607c590` — worker admin-server (node:http) /status + /metrics with graceful shutdown (6 tests)
- 2026-05-17 06:27 — `618c8707` — web withMetrics SSR wrapper helper (4 tests)
- 2026-05-17 06:36 — `5d461656` — api.status + api.metrics resource routes + 8 top routes wrapped (4 route tests + 4 ssr tests)
- 2026-05-17 06:38 — `8cff9092` — ai cost prom counter with 100-bucket tenant LRU (5 tests)
- 2026-05-17 06:40 — `cadd6574` — stripe webhook counter with event_type + status labels (3 tests)
- 2026-05-17 06:43 — `a1427f26` — MailjetProvider.onMetric hook + worker auto-wiring (5 tests)
- 2026-05-17 06:45 — `ec80dec3` — webhook delivery handler counter (success/retry/dlq/rejected)
- 2026-05-17 06:46 — `aba91e5f` — Prometheus compose + scrape config with 30d retention
- 2026-05-17 06:48 — `4d2117b9` — 5 Grafana dashboards + file-based provisioning
- 2026-05-17 06:50 — `52511b09` — docs/observability/metrics.md (catalog + deploy + rollback playbook)
- 2026-05-17 06:51 — `4245c6d5` — METRICS_ENABLED / METRICS_TOKEN / WORKER_ADMIN_* / APP_REGION in env catalog + .env.example
- 2026-05-17 06:53 — `0fcc6a17` — scripts/observability/smoke-metrics.sh
- 2026-05-17 06:54 — `<this commit>` — progress log + PROGRESS.md update
## Lessons Learned
- **Hono `c.req.routePath` is the right cardinality knob.** Going through
the matched route pattern keeps `http_requests_total` bounded by the
route table size (~50 entries) rather than by user-input URLs.
- **`prom-client` emits labels in declaration order.** Test regexes that
hard-code label order need to match the order in `labelNames` (not
alphabetical, not insertion-order of `labels({...})`).
- **Mail providers should not import from the host app.** Threading a
callback through `MailjetProviderOptions.onMetric` is cleaner than
reaching for a registry singleton from a leaf package.
- **React Router v7 typegen runs lazily.** New resource routes
(`api.status.ts`, `api.metrics.ts`) work as soon as `react-router typegen`
has run once — committed types in `.react-router/types/` track route
changes automatically when typecheck runs.
## Progress Log (continued)
- 2026-05-18 00:48 UTC — staging-deploy — Cloudflare A record
`grafana.boiler.saas-ctrl.com` → 46.225.5.93 created
(id `2267905b1ee15df4002ca02d76af1fe0`, proxied=false).
Observability stack brought up on `/opt/observability/`: Prometheus
v3.6.0 (`boilerplate-prometheus`) on dedicated
`boilerplate-observability` network, Grafana 11.4.0
(`boilerplate-grafana`) attached to both observability + coolify
networks via on-host `docker-compose.override.yml` (Traefik labels for
TLS via Let's Encrypt). Secrets generated via `openssl rand` on the
host (`METRICS_TOKEN`, `GRAFANA_ADMIN_PASSWORD`,
`GRAFANA_PROMETHEUS_TOKEN` reuses METRICS_TOKEN) stored in
`/opt/observability/.env` (chmod 600) and
`~/.cli-tokens/grafana-staging-env`; the same token is also bind-mounted
into Prometheus at `/etc/prometheus/secrets/metrics_token`. While
deploying, found and fixed a duplicate `jsonData` key in
`infra/grafana/provisioning/datasources/prometheus.yaml` (committed file
was invalid YAML; Grafana refused to start). Smoke green:
`curl -fSs https://grafana.boiler.saas-ctrl.com/api/health` →
`{"database":"ok","version":"11.4.0",...}`. Scrape targets in
`prometheus.yml` point at the public HTTPS hostnames
(`api.boiler.saas-ctrl.com`, etc.) and require the apps to expose
`/metrics` with the same bearer token.
- 2026-05-27 — docs/status reconciliation — A1 verifies
`METRICS_ENABLED=true` + `METRICS_TOKEN` live on web/api/worker and
`/metrics` returns 200. The older "env flips pending" note above is
superseded; any Prometheus target outage after this date should be
treated as a fresh runtime incident, not unfinished #35 implementation.