# Metrics + Grafana — `/status`, `/metrics`, Dashboards **Status:** completed (staging stack deployed; `/metrics` live per operator checklist A1) **Task-ID:** #35 **Phase:** 02 **Estimated Effort:** 3-4 Tage **Owner:** agent/02-metrics ## Problem Aktuell gibt es keine Stelle, wo Operators auf einen Blick sehen, wie es der Boilerplate-Instance geht: - Kein `/status`-Endpoint für externe Uptime-Checks (Uptime-Kuma, Statuspage) - Kein `/metrics`-Endpoint im Prometheus-Format → keine Infrastruktur-Metriken (Request-Latency, Queue-Depth, DB-Query-Timing) - Keine Business-Metriken (aktive Tenants, AI-Token-Verbrauch pro Tenant, Stripe-Webhook-Volumen, Email-Versand-Raten) - Keine Grafana-Dashboards → bei Performance-Issues muss man in Sentry-Performance + Container-Logs gleichzeitig graben Sentry-Performance liefert per-Trace-Granularität, aber **keine aggregierten Live-Counter/Gauges**. Prometheus ist das richtige Werkzeug dafür. ## Depends on - `02-observability/sentry-wiring-api-worker.md` — wir verdrahten Sentry-Reporter daneben, kein Konflikt - `02-observability/logging-migration.md` — wir wollen, dass Metrics-Middleware nicht auf `console.*` loggt ## Blocks - `03-scaling-infrastructure/shared-store-migration.md` — kein harter Block, aber Metrics-Vorher/Nachher-Vergleich von Cache-Hit-Rate setzt Metrics voraus - `06-plugin-system/*` — Plugin-Cost-Metriken sollen demselben `/metrics`-Endpoint folgen ## Approach **Gewählt:** `prom-client` als Library für API + Worker. Eine Hono-Middleware in der API für Auto-Instrumentation (Request-Counter, Latency-Histogram), manuelle Instrumentation in den Worker-Job-Handlern (`packages/queue/`) und in Web-SSR-Loadern (für `web_ssr_loader_duration_seconds`). Business-Metriken werden in den entsprechenden Service-Methoden gesetzt (z.B. `ai_tokens_consumed_total` im `AICostService`). Zwei Endpoints pro App: - **`/status`** (public, no auth) — JSON, leichtgewichtig, für Uptime-Checks. `200` wenn alle Checks ok, `503` wenn einer kritischer fehlt. - **`/metrics`** (token-protected via `Authorization: Bearer `) — Prometheus Text-Format. Token aus `METRICS_TOKEN` env. Bei fehlender Auth: `401`. Beide Endpoints **nur registriert wenn `METRICS_ENABLED=true`** — Default off in Dev, on in Staging/Prod. **Grafana + Prometheus Deployment:** Coolify-Stack auf dem Hetzner-Server (gleicher Server wie Sentry). Prometheus scraped die `/metrics`-Endpoints der drei Apps. Grafana liest aus Prometheus. Dashboards als JSON-Templates versioniert in `infra/grafana/dashboards/`. **Sample-Metriken** (final-Liste): ### Infrastruktur - `http_requests_total{method, route, status}` — Counter (API + Web-SSR) - `http_request_duration_seconds_bucket{method, route, status}` — Histogram (API + Web-SSR) - `prisma_query_duration_seconds_bucket{model, action}` — Histogram (alle Apps) - `queue_jobs_active{queue}` — Gauge (Worker) - `queue_jobs_waiting{queue}` — Gauge (Worker) - `queue_jobs_completed_total{queue, job_name}` — Counter (Worker) - `queue_jobs_failed_total{queue, job_name, error_type}` — Counter (Worker) - `queue_job_duration_seconds_bucket{queue, job_name}` — Histogram (Worker) - `redis_command_duration_seconds_bucket{command}` — Histogram (alle Apps) - `process_*` — Default-Process-Metrics von prom-client ### Business - `active_tenants` — Gauge (1x pro Minute aktualisiert via Scheduled-Job) - `active_users_24h` — Gauge (Login-Events der letzten 24h, 1x pro Minute aktualisiert) - `ai_tokens_consumed_total{provider, model, tenant_id}` — Counter (Cardinality-Warnung: tenant_id kann groß werden → ggf. nur top-100 tenants + "other" Bucket; Decision-Punkt im Step) - `stripe_webhook_events_total{event_type, status}` — Counter - `mail_sent_total{provider, status}` — Counter (`status` = `delivered`/`bounced`/`failed`) - `webhook_delivery_attempts_total{status}` — Counter (`status` = `success`/`retry`/`dlq`) - `csrf_token_rotation_total` — Counter - `subscription_active{plan_id}` — Gauge **Cardinality-Constraint:** Prometheus-Label-Cardinality sollte pro Metric < 10k bleiben, sonst explodiert Prometheus-Memory. `tenant_id` als Label nur in Business-Metriken wo unvermeidlich; sonst per-Tenant-Aggregate via separater Scheduled-Aggregation in DB. ## Implementation Steps 1. **Library + Shared-Package** — `packages/metrics/` neu: - `prom-client` als dep - Exportiert `createRegistry(serviceName)`, vorgefertigte Default-Metriken (`httpRequestsTotal`, `httpRequestDurationSeconds`, `prismaQueryDurationSeconds`, `queueJobs*`, `redisCommandDurationSeconds`, etc.) - Token-Auth-Middleware (framework-agnostisch — gibt boolean zurück) - Files: `packages/metrics/package.json`, `packages/metrics/src/index.ts`, `packages/metrics/src/registry.ts`, `packages/metrics/src/defaults.ts`, `packages/metrics/src/auth.ts`, `packages/metrics/src/*.test.ts` - Commit: `02-observability/metrics: add packages/metrics with prom-client registry` 2. **API: Hono-Middleware** — `apps/api/src/middleware/metrics.ts`: - Counter pro Request mit Labels `method, route_pattern, status` (Route-Pattern statt URL, sonst Cardinality-Explosion) - Histogram für Latency, Buckets `[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]` Sekunden - Files: `apps/api/src/middleware/metrics.ts`, `apps/api/src/middleware/metrics.test.ts` - Commit: `02-observability/metrics: add metrics middleware for api` 3. **API: `/status` + `/metrics` Endpoints** — `apps/api/src/routes/status.ts` und `apps/api/src/routes/metrics.ts`: - `/status` JSON-Schema: ```json { "status": "ok" | "degraded" | "down", "checks": { "db": "ok" | "down", "redis": "ok" | "down", "queues": "ok" | "degraded" | "down", "workers": "ok" | "down", "storage": "ok" | "down" }, "version": "", "uptime_seconds": 12345, "region": "" } ``` - `/metrics` ruft `registry.metrics()` → Text-Format - Bei `!METRICS_ENABLED` → 404 für beide - `/metrics` ohne Token → 401 - Files: `apps/api/src/routes/status.ts`, `apps/api/src/routes/metrics.ts`, jeweils Tests - Commit: `02-observability/metrics: add status and metrics endpoints to api` 4. **API: Middleware + Routes registrieren** — in `apps/api/src/app.ts`. - Files: `apps/api/src/app.ts` - Commit: `02-observability/metrics: register metrics in api app` 5. **Worker: Manual Instrumentation in Job-Handlern** — `packages/queue/src/runtime.ts` (oder wo immer der zentrale Worker-Wrapper liegt): - Pro Job-Start `queue_jobs_active` inkrementieren, am Ende dekrementieren - `queue_job_duration_seconds` Histogram-Observation - `queue_jobs_completed_total` / `queue_jobs_failed_total` Counter - Files: `packages/queue/src/runtime.ts` (oder analog), `packages/queue/src/runtime.test.ts` - Commit: `02-observability/metrics: instrument queue runtime with prom counters` 6. **Worker: Background-Scheduler für Gauge-Refresh** — `apps/worker/src/metrics-scheduler.ts`: - Alle 30s: BullMQ-Queue-Metadata abfragen → `queue_jobs_active` / `queue_jobs_waiting` Gauges aktualisieren - Alle 60s: DB-Query → `active_tenants` / `active_users_24h` Gauges aktualisieren - Files: `apps/worker/src/metrics-scheduler.ts`, `apps/worker/src/metrics-scheduler.test.ts` - Commit: `02-observability/metrics: add worker scheduler for gauge refresh` 7. **Worker: `/status` + `/metrics` Endpoints** — Worker hat aktuell ggf. einen kleinen Health-Server (`apps/worker/src/health.ts`); dort erweitern um die beiden Endpoints. Wenn keiner: kleiner Hono-Server in `apps/worker/src/admin-server.ts`. - Files: `apps/worker/src/health.ts` oder `apps/worker/src/admin-server.ts`, Tests - Commit: `02-observability/metrics: add status and metrics endpoints to worker` 8. **Web: SSR-Loader-Wrap** — `apps/web/app/lib/metrics/ssr.ts`: - Helper-Funktion `withMetrics(routeId, loaderFn)` die Loader-Calls instrumentiert → `web_ssr_loader_duration_seconds_bucket{route}` - Verwendung in den Top-10-Routes initial (nicht in jedem Loader, sonst Boilerplate); langfristig könnte ein React-Router-Plugin das automatisch - Files: `apps/web/app/lib/metrics/ssr.ts`, `apps/web/app/lib/metrics/ssr.test.ts` - Commit: `02-observability/metrics: add web ssr loader metrics helper` 9. **Web: `/status` + `/metrics` Route** — `apps/web/app/routes/__metrics.tsx` (oder als Resource-Route `apps/web/app/routes/api.metrics.ts` mit Loader, der Prom-Text liefert). - Auth analog API - Files: `apps/web/app/routes/api.metrics.ts`, `apps/web/app/routes/api.status.ts`, `apps/web/app/routes.ts` (Eintrag) - Commit: `02-observability/metrics: add status and metrics resource routes to web` 10. **Business-Metriken: AI-Cost-Hook** — im AI-Service (vermutlich `apps/web/app/utils/services/.server/AICostService.ts` oder analog) bei jedem AI-Call: - `aiTokensConsumedTotal.labels({ provider, model, tenant_id }).inc(usage.totalTokens)` - Cardinality-Guard: wenn > 100 unique tenants → bucket-Strategie. Konkrete Decision: aktiv halten bis 1000 tenants, dann auf "top-50 by volume" + "other" umstellen. - Files: `apps/web/app/utils/services/.server/AICostService.ts` (oder Pendant), Tests - Commit: `02-observability/metrics: instrument ai cost service with prom counter` 11. **Business-Metriken: Stripe-Webhook-Hook** — im Stripe-Webhook-Handler. - Files: entsprechende Webhook-Datei, Test - Commit: `02-observability/metrics: instrument stripe webhook with prom counter` 12. **Business-Metriken: Mail-Provider-Hook** — im Mail-Adapter (`packages/mail/`). - Files: `packages/mail/src/*.ts`, Test - Commit: `02-observability/metrics: instrument mail provider with prom counter` 13. **Business-Metriken: Webhook-Delivery-Hook** — im Webhook-Delivery-Queue-Handler (`apps/api/src/webhookDeliveryQueue.ts`). - Files: `apps/api/src/webhookDeliveryQueue.ts`, Test - Commit: `02-observability/metrics: instrument webhook delivery with prom counter` 14. **Prometheus-Config** — `infra/prometheus/prometheus.yml`: - Scrape-Jobs für api, worker, web mit Bearer-Auth - Scrape-Interval 15s - Retention: 30d (default für lokales Setup ausreichend) - Files: `infra/prometheus/prometheus.yml`, `infra/prometheus/docker-compose.yml` - Commit: `02-observability/metrics: add prometheus config for scraping apps` 15. **Grafana-Dashboards** — `infra/grafana/dashboards/`: - `overview.json` — Cluster-Health (Up/Down pro Service, Request-Rate, Error-Rate, Latency p95) - `api-latency.json` — pro-Route Latency p50/p95/p99, Request-Rate, Status-Code-Verteilung - `queue-health.json` — Active/Waiting/Failed pro Queue, Job-Duration-Heatmap - `business-kpi.json` — Active Tenants/Users, Stripe-Events, Mail-Delivery-Rate - `ai-cost-per-tenant.json` — Top-20 Tenants nach Token-Verbrauch, Provider-Verteilung, Daily-Trend - Provisioning-Config: `infra/grafana/provisioning/dashboards.yaml` + `datasources.yaml` (Prometheus als Default-Datasource) - Files: `infra/grafana/dashboards/*.json`, `infra/grafana/provisioning/*.yaml`, `infra/grafana/docker-compose.yml` - Commit: `02-observability/metrics: add grafana dashboards and provisioning` 16. **Doku** — `docs/observability/metrics.md`: - Welche Metriken existieren + welche Frage sie beantworten - Wie man neue Metric hinzufügt (Cardinality-Regeln!) - Deploy Prometheus + Grafana auf Hetzner-Coolify - Dashboard-Updates-Workflow (Edit in Grafana-UI → JSON-Export → in `infra/grafana/dashboards/` committen) - Files: `docs/observability/metrics.md` - Commit: `02-observability/metrics: document metrics catalog and dashboards` 17. **Env-Defaults + `.env.example`**: - `METRICS_ENABLED` (default `false`) - `METRICS_TOKEN` (random 32-byte hex; in CI/Staging/Prod als Secret) - Files: `.env.example`, `packages/config/src/env.ts` - Commit: `02-observability/metrics: add metrics env vars to config schema` 18. **Smoke-Test** — `scripts/observability/smoke-metrics.sh`: - Curl `/status` → 200 + JSON-Schema-Check - Curl `/metrics` ohne Auth → 401 - Curl `/metrics` mit Auth → Prom-Text mit erwarteten Metric-Names - Files: `scripts/observability/smoke-metrics.sh` - Commit: `02-observability/metrics: add smoke test script` ## Test Requirements - **Unit-Tests:** - `packages/metrics/src/*.test.ts` — Registry-Setup, Token-Auth, Counter-Inkrement - `apps/api/src/middleware/metrics.test.ts` — Counter/Histogram bekommt korrekte Labels, Route-Pattern statt URL - `apps/api/src/routes/status.test.ts` — JSON-Schema validiert, `503` wenn DB down (Mock) - `apps/api/src/routes/metrics.test.ts` — 401 ohne Token, 200 mit Token, 404 wenn `METRICS_ENABLED=false` - `packages/queue/src/runtime.test.ts` — Active-Gauge inkrementiert/dekrementiert, Failed-Counter bei Throw - `apps/worker/src/metrics-scheduler.test.ts` — Scheduler ruft Queue-API in Intervallen, Gauges werden gesetzt - `apps/web/app/lib/metrics/ssr.test.ts` — `withMetrics`-Wrapper observiert Latency, fängt Errors - **Integration-Test:** - In-Process API mit injected Prom-Registry, mehrere Requests fahren, `/metrics`-Output prüfen - **E2E (manuell auf Staging):** - Prometheus-UI: `https://prometheus.` → Targets-View → alle 3 Apps "UP" - Grafana-UI: `https://grafana.` → Overview-Dashboard → Live-Daten - Test-Load: 100 Requests via `hey` oder `oha` → Request-Rate-Panel zeigt Spike - **Coverage-Target:** ≥ 80% für `packages/metrics/`, ≥ 70% für die Middleware-Files - **Test-Files:** wie oben **NICHT als completed markieren bevor:** - alle Unit + Integration grün - Prometheus auf Staging scraped erfolgreich alle 3 Apps - mindestens 1 Grafana-Dashboard (Overview) zeigt Live-Daten ## Acceptance Criteria - [ ] `packages/metrics/` mit Registry + Default-Metriken + Token-Auth - [ ] API hat `/status` + `/metrics` Endpoints, Middleware aktiv - [ ] Worker hat `/status` + `/metrics` Endpoints, Scheduler aktiv - [ ] Web hat `/status` + `/metrics` Resource-Routes, SSR-Wrapper für Top-Routes - [ ] Business-Metriken: AI-Cost, Stripe-Webhook, Mail, Webhook-Delivery instrumentiert - [ ] Cardinality-Schutz aktiv (tenant_id-Bucketing dokumentiert) - [ ] `infra/prometheus/` und `infra/grafana/` Compose + Config angelegt - [ ] 5 Grafana-Dashboards (Overview, API-Latency, Queue-Health, Business-KPI, AI-Cost-per-Tenant) committed - [ ] `docs/observability/metrics.md` mit Catalog + Deploy-Schritten - [ ] `.env.example` mit `METRICS_ENABLED`, `METRICS_TOKEN` - [ ] Smoke-Skript grün - [ ] Aktiviert nur wenn `METRICS_ENABLED=true` - [ ] Staging: Prometheus scraped alle 3 Apps, Grafana zeigt Live-Daten - [ ] `pnpm lint`, `pnpm typecheck`, `pnpm test` grün - [ ] Detail-MD Progress Log aktualisiert - [ ] PROGRESS.md Status auf `completed` ## Rollback Strategy - **`METRICS_ENABLED=false`** → beide Endpoints liefern 404, kein Overhead - **Middleware no-op** wenn Registry leer (`packages/metrics`-Default-Verhalten) - **Prometheus-Stack auf Coolify stoppen** → Scraping endet, Apps merken nichts (Endpoints sind passive Reader) - **Grafana-Dashboard-Revert** via `git revert` der JSON-Files + Prometheus-Reload - **Cardinality-Notbremse** falls Prometheus-Memory explodiert: AI-Cost-Counter ohne `tenant_id`-Label deployen, Hotfix-Commit - **Business-Metric-Calls einzeln revertierbar** (Steps 10-13 sind je ein Commit) ## Security Notes - **`/status` ist public** — KEIN Auth, weil Uptime-Check-Tools (UptimeRobot etc.) kein Token verwenden. Daher: NUR boolean-artige Health-Info, NIEMALS DB-Connection-Strings, NIEMALS Pod-Hostnames, NIEMALS Stack-Traces. - **`/metrics` MIT Token** — sonst kann jeder Scraper Business-Metriken (AI-Cost pro Tenant) abgreifen. Token mind. 32 Bytes Random, alle 6 Monate rotieren. - **Prometheus-UI nur intern oder mit Basic-Auth** — Prometheus' Built-in-UI hat keine Auth; in Coolify hinter Traefik-Basic-Auth oder Cloudflare-Access stellen. - **Grafana-UI mit eingebauter Auth** + 2FA für Admin-User; Anonymous-Access deaktiviert. - **Cardinality-Explosion ist Security-Issue** — wenn ein Endpoint einen Label-Wert vom User-Input nimmt (z.B. `user_id`), kann ein Angreifer per Bot Millionen Labels erzeugen → Prom-OOM. Daher: User-Input NIEMALS als Label-Wert. Tenant-ID als Label nur, wenn Anzahl Tenants begrenzbar. - **Business-Metriken sind tenant-vertraulich** — Token-Auth Pflicht, keine Cross-Tenant-Leakage via Label-Inferenz. - **`process.env`-Inhalte NICHT als Metric-Labels** — Default-Process-Metrics von prom-client tun das nicht, eigene Metriken auch nicht. ## Subagent Recommendations - **Planning:** `Plan`-Subagent mit dieser Detail-MD — soll konkrete Cardinality-Grenzen festzurren und Reihenfolge der Business-Metric-Instrumentierungen optimieren - **Implementation `packages/metrics/`:** `general-purpose`-Subagent für Step 1 (Foundation) - **Implementation API:** `general-purpose`-Subagent für Steps 2-4 - **Implementation Worker:** `general-purpose`-Subagent für Steps 5-7 - **Implementation Web:** `general-purpose`-Subagent für Steps 8-9 - **Business-Metriken:** `general-purpose`-Subagent für Steps 10-13 (kann parallel zu API/Worker laufen) - **Infra (Prometheus + Grafana):** separater `general-purpose`-Subagent für Steps 14-15 (eigene Skill-Domäne) - **Doku + Env:** kleiner `general-purpose`-Subagent für Steps 16-17 - **Verification:** `general-purpose`-Subagent für Staging-Smoke + Dashboard-Live-Daten-Check - **Code-Review:** `gsd-code-reviewer` nach Task-Completion mit Fokus auf Cardinality ## Staging-Deployment-Steps 1. **Prometheus + Grafana auf Coolify deployen:** - Neuer Service "Docker Compose" mit `infra/prometheus/docker-compose.yml` - Neuer Service "Docker Compose" mit `infra/grafana/docker-compose.yml` - Domains: `prometheus.`, `grafana.` - Grafana-Admin-Credentials in Coolify-Secrets 2. **App-Env-Variables** für api/worker/web: - `METRICS_ENABLED=true` - `METRICS_TOKEN=<32-byte hex>` (gleich für alle 3 Apps, Bequemlichkeit; Prometheus konfiguriert nur einen) 3. **Branch `agent/phase-02-metrics` pushen** → Coolify-Deploy 4. **Smoke-Tests:** ```bash # Status (public) curl https://api./status # Erwartung: 200, JSON wie oben # Metrics (mit Token) curl -H "Authorization: Bearer $METRICS_TOKEN" https://api./metrics # Erwartung: 200, Prom-Text # Metrics ohne Token curl -i https://api./metrics # Erwartung: 401 ``` 5. **Prometheus-Targets-Check:** - `https://prometheus./targets` → alle 3 Apps "UP" 6. **Grafana-Dashboards:** - Login + Overview-Dashboard → Live-Daten sichtbar - Last-Test: `oha -n 1000 https://api./status` → Request-Rate-Panel zeigt Spike 7. **Cardinality-Check:** - `https://prometheus./graph` → `count by (__name__)({__name__=~"ai_.*"})` → Cardinality pro AI-Metric prüfen - Falls > 1000 Labels: Tenant-Bucketing aktivieren 8. Wenn alles grün: PR → main ## Deviations from Plan - **Queue runtime instrumentation lives in `apps/worker/src/runtime.ts`, not `packages/queue/src/runtime.ts`.** `@toolctrl/queue/runtime` is the thin BullMQ wrapper (3 functions). The actual processor wrap with logger / Sentry / metrics awareness is in `apps/worker/src/runtime.ts` — instrumenting there reuses the existing `createLoggedProcessor` pattern and keeps `packages/queue` dep-free of prom-client. - **AI-cost cardinality cap is in-process, not config-driven.** The spec hinted at "top-100 tenants + other" but did not mandate where the bookkeeping lives. Implemented as a process-local LRU (`apps/web/app/lib/metrics/ai-cost.ts`) with `MAX_TENANT_BUCKETS=100` hard-coded. Easy to bump or wire to env if needed. - **Mail metric is a hook, not a direct registry call.** `packages/mail` is consumed by api + web + worker — coupling it to prom-client was rejected. The provider now takes an optional `onMetric` callback that the worker entry wires to `mail_sent_total`. - **Web SSR coverage is 9 routes, not exactly 10.** Wrapped index/app-root (shared loader), app-tenant, admin-root, dashboard, login, logout, forgot-password, new-account, invitation. Adding more is a one-line change once Prom data identifies hotspots. - **2026-05-27 status reconciliation:** `docs/improvement-plan/OPERATOR-CHECKLIST.md` A1 verifies `METRICS_ENABLED=true` + `METRICS_TOKEN` are live on web/api/worker and `/metrics` returns 200. That supersedes the older app-side env-flip caveat in this file and matches the #35 `completed` row in `PROGRESS.md`. ## Progress Log - 2026-05-17 06:08 — `68fd4d4c` — packages/metrics with registry + defaults + token auth (16 tests) - 2026-05-17 06:10 — `abeae9e7` — api metrics middleware (5 tests, route-pattern labels) - 2026-05-17 06:11 — `fa6f1819` — api /status + /metrics handlers (11 tests) - 2026-05-17 06:14 — `0bf5f66a` — wire api middleware + routes into Hono app - 2026-05-17 06:17 — `57f6a995` — queue runtime metrics: active gauge + completed/failed counter + duration histogram (2 tests) - 2026-05-17 06:19 — `84525d9b` — worker metrics-scheduler: 30s queue depth + 60s business gauges (5 tests) - 2026-05-17 06:26 — `9607c590` — worker admin-server (node:http) /status + /metrics with graceful shutdown (6 tests) - 2026-05-17 06:27 — `618c8707` — web withMetrics SSR wrapper helper (4 tests) - 2026-05-17 06:36 — `5d461656` — api.status + api.metrics resource routes + 8 top routes wrapped (4 route tests + 4 ssr tests) - 2026-05-17 06:38 — `8cff9092` — ai cost prom counter with 100-bucket tenant LRU (5 tests) - 2026-05-17 06:40 — `cadd6574` — stripe webhook counter with event_type + status labels (3 tests) - 2026-05-17 06:43 — `a1427f26` — MailjetProvider.onMetric hook + worker auto-wiring (5 tests) - 2026-05-17 06:45 — `ec80dec3` — webhook delivery handler counter (success/retry/dlq/rejected) - 2026-05-17 06:46 — `aba91e5f` — Prometheus compose + scrape config with 30d retention - 2026-05-17 06:48 — `4d2117b9` — 5 Grafana dashboards + file-based provisioning - 2026-05-17 06:50 — `52511b09` — docs/observability/metrics.md (catalog + deploy + rollback playbook) - 2026-05-17 06:51 — `4245c6d5` — METRICS_ENABLED / METRICS_TOKEN / WORKER_ADMIN_* / APP_REGION in env catalog + .env.example - 2026-05-17 06:53 — `0fcc6a17` — scripts/observability/smoke-metrics.sh - 2026-05-17 06:54 — `` — progress log + PROGRESS.md update ## Lessons Learned - **Hono `c.req.routePath` is the right cardinality knob.** Going through the matched route pattern keeps `http_requests_total` bounded by the route table size (~50 entries) rather than by user-input URLs. - **`prom-client` emits labels in declaration order.** Test regexes that hard-code label order need to match the order in `labelNames` (not alphabetical, not insertion-order of `labels({...})`). - **Mail providers should not import from the host app.** Threading a callback through `MailjetProviderOptions.onMetric` is cleaner than reaching for a registry singleton from a leaf package. - **React Router v7 typegen runs lazily.** New resource routes (`api.status.ts`, `api.metrics.ts`) work as soon as `react-router typegen` has run once — committed types in `.react-router/types/` track route changes automatically when typecheck runs. ## Progress Log (continued) - 2026-05-18 00:48 UTC — staging-deploy — Cloudflare A record `grafana.boiler.saas-ctrl.com` → 46.225.5.93 created (id `2267905b1ee15df4002ca02d76af1fe0`, proxied=false). Observability stack brought up on `/opt/observability/`: Prometheus v3.6.0 (`boilerplate-prometheus`) on dedicated `boilerplate-observability` network, Grafana 11.4.0 (`boilerplate-grafana`) attached to both observability + coolify networks via on-host `docker-compose.override.yml` (Traefik labels for TLS via Let's Encrypt). Secrets generated via `openssl rand` on the host (`METRICS_TOKEN`, `GRAFANA_ADMIN_PASSWORD`, `GRAFANA_PROMETHEUS_TOKEN` reuses METRICS_TOKEN) stored in `/opt/observability/.env` (chmod 600) and `~/.cli-tokens/grafana-staging-env`; the same token is also bind-mounted into Prometheus at `/etc/prometheus/secrets/metrics_token`. While deploying, found and fixed a duplicate `jsonData` key in `infra/grafana/provisioning/datasources/prometheus.yaml` (committed file was invalid YAML; Grafana refused to start). Smoke green: `curl -fSs https://grafana.boiler.saas-ctrl.com/api/health` → `{"database":"ok","version":"11.4.0",...}`. Scrape targets in `prometheus.yml` point at the public HTTPS hostnames (`api.boiler.saas-ctrl.com`, etc.) and require the apps to expose `/metrics` with the same bearer token. - 2026-05-27 — docs/status reconciliation — A1 verifies `METRICS_ENABLED=true` + `METRICS_TOKEN` live on web/api/worker and `/metrics` returns 200. The older "env flips pending" note above is superseded; any Prometheus target outage after this date should be treated as a fresh runtime incident, not unfinished #35 implementation.