Files
HiveGrid-Wireframes/hivegrid-cloud/docs/improvement-plan/02-observability/metrics-grafana.md
T

25 KiB

Metrics + Grafana — /status, /metrics, Dashboards

Status: completed (staging stack deployed; /metrics live per operator checklist A1) Task-ID: #35 Phase: 02 Estimated Effort: 3-4 Tage Owner: agent/02-metrics

Problem

Aktuell gibt es keine Stelle, wo Operators auf einen Blick sehen, wie es der Boilerplate-Instance geht:

  • Kein /status-Endpoint für externe Uptime-Checks (Uptime-Kuma, Statuspage)
  • Kein /metrics-Endpoint im Prometheus-Format → keine Infrastruktur-Metriken (Request-Latency, Queue-Depth, DB-Query-Timing)
  • Keine Business-Metriken (aktive Tenants, AI-Token-Verbrauch pro Tenant, Stripe-Webhook-Volumen, Email-Versand-Raten)
  • Keine Grafana-Dashboards → bei Performance-Issues muss man in Sentry-Performance + Container-Logs gleichzeitig graben

Sentry-Performance liefert per-Trace-Granularität, aber keine aggregierten Live-Counter/Gauges. Prometheus ist das richtige Werkzeug dafür.

Depends on

  • 02-observability/sentry-wiring-api-worker.md — wir verdrahten Sentry-Reporter daneben, kein Konflikt
  • 02-observability/logging-migration.md — wir wollen, dass Metrics-Middleware nicht auf console.* loggt

Blocks

  • 03-scaling-infrastructure/shared-store-migration.md — kein harter Block, aber Metrics-Vorher/Nachher-Vergleich von Cache-Hit-Rate setzt Metrics voraus
  • 06-plugin-system/* — Plugin-Cost-Metriken sollen demselben /metrics-Endpoint folgen

Approach

Gewählt: prom-client als Library für API + Worker. Eine Hono-Middleware in der API für Auto-Instrumentation (Request-Counter, Latency-Histogram), manuelle Instrumentation in den Worker-Job-Handlern (packages/queue/) und in Web-SSR-Loadern (für web_ssr_loader_duration_seconds). Business-Metriken werden in den entsprechenden Service-Methoden gesetzt (z.B. ai_tokens_consumed_total im AICostService).

Zwei Endpoints pro App:

  • /status (public, no auth) — JSON, leichtgewichtig, für Uptime-Checks. 200 wenn alle Checks ok, 503 wenn einer kritischer fehlt.
  • /metrics (token-protected via Authorization: Bearer <token>) — Prometheus Text-Format. Token aus METRICS_TOKEN env. Bei fehlender Auth: 401.

Beide Endpoints nur registriert wenn METRICS_ENABLED=true — Default off in Dev, on in Staging/Prod.

Grafana + Prometheus Deployment: Coolify-Stack auf dem Hetzner-Server (gleicher Server wie Sentry). Prometheus scraped die /metrics-Endpoints der drei Apps. Grafana liest aus Prometheus. Dashboards als JSON-Templates versioniert in infra/grafana/dashboards/.

Sample-Metriken (final-Liste):

Infrastruktur

  • http_requests_total{method, route, status} — Counter (API + Web-SSR)
  • http_request_duration_seconds_bucket{method, route, status} — Histogram (API + Web-SSR)
  • prisma_query_duration_seconds_bucket{model, action} — Histogram (alle Apps)
  • queue_jobs_active{queue} — Gauge (Worker)
  • queue_jobs_waiting{queue} — Gauge (Worker)
  • queue_jobs_completed_total{queue, job_name} — Counter (Worker)
  • queue_jobs_failed_total{queue, job_name, error_type} — Counter (Worker)
  • queue_job_duration_seconds_bucket{queue, job_name} — Histogram (Worker)
  • redis_command_duration_seconds_bucket{command} — Histogram (alle Apps)
  • process_* — Default-Process-Metrics von prom-client

Business

  • active_tenants — Gauge (1x pro Minute aktualisiert via Scheduled-Job)
  • active_users_24h — Gauge (Login-Events der letzten 24h, 1x pro Minute aktualisiert)
  • ai_tokens_consumed_total{provider, model, tenant_id} — Counter (Cardinality-Warnung: tenant_id kann groß werden → ggf. nur top-100 tenants + "other" Bucket; Decision-Punkt im Step)
  • stripe_webhook_events_total{event_type, status} — Counter
  • mail_sent_total{provider, status} — Counter (status = delivered/bounced/failed)
  • webhook_delivery_attempts_total{status} — Counter (status = success/retry/dlq)
  • csrf_token_rotation_total — Counter
  • subscription_active{plan_id} — Gauge

Cardinality-Constraint: Prometheus-Label-Cardinality sollte pro Metric < 10k bleiben, sonst explodiert Prometheus-Memory. tenant_id als Label nur in Business-Metriken wo unvermeidlich; sonst per-Tenant-Aggregate via separater Scheduled-Aggregation in DB.

Implementation Steps

  1. Library + Shared-Package — packages/metrics/ neu:

    • prom-client als dep
    • Exportiert createRegistry(serviceName), vorgefertigte Default-Metriken (httpRequestsTotal, httpRequestDurationSeconds, prismaQueryDurationSeconds, queueJobs*, redisCommandDurationSeconds, etc.)
    • Token-Auth-Middleware (framework-agnostisch — gibt boolean zurück)
    • Files: packages/metrics/package.json, packages/metrics/src/index.ts, packages/metrics/src/registry.ts, packages/metrics/src/defaults.ts, packages/metrics/src/auth.ts, packages/metrics/src/*.test.ts
    • Commit: 02-observability/metrics: add packages/metrics with prom-client registry
  2. API: Hono-Middleware — apps/api/src/middleware/metrics.ts:

    • Counter pro Request mit Labels method, route_pattern, status (Route-Pattern statt URL, sonst Cardinality-Explosion)
    • Histogram für Latency, Buckets [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] Sekunden
    • Files: apps/api/src/middleware/metrics.ts, apps/api/src/middleware/metrics.test.ts
    • Commit: 02-observability/metrics: add metrics middleware for api
  3. API: /status + /metrics Endpoints — apps/api/src/routes/status.ts und apps/api/src/routes/metrics.ts:

    • /status JSON-Schema:
      {
        "status": "ok" | "degraded" | "down",
        "checks": {
          "db": "ok" | "down",
          "redis": "ok" | "down",
          "queues": "ok" | "degraded" | "down",
          "workers": "ok" | "down",
          "storage": "ok" | "down"
        },
        "version": "<git-sha>",
        "uptime_seconds": 12345,
        "region": "<env-region>"
      }
      
    • /metrics ruft registry.metrics() → Text-Format
    • Bei !METRICS_ENABLED → 404 für beide
    • /metrics ohne Token → 401
    • Files: apps/api/src/routes/status.ts, apps/api/src/routes/metrics.ts, jeweils Tests
    • Commit: 02-observability/metrics: add status and metrics endpoints to api
  4. API: Middleware + Routes registrieren — in apps/api/src/app.ts.

    • Files: apps/api/src/app.ts
    • Commit: 02-observability/metrics: register metrics in api app
  5. Worker: Manual Instrumentation in Job-Handlern — packages/queue/src/runtime.ts (oder wo immer der zentrale Worker-Wrapper liegt):

    • Pro Job-Start queue_jobs_active inkrementieren, am Ende dekrementieren
    • queue_job_duration_seconds Histogram-Observation
    • queue_jobs_completed_total / queue_jobs_failed_total Counter
    • Files: packages/queue/src/runtime.ts (oder analog), packages/queue/src/runtime.test.ts
    • Commit: 02-observability/metrics: instrument queue runtime with prom counters
  6. Worker: Background-Scheduler für Gauge-Refresh — apps/worker/src/metrics-scheduler.ts:

    • Alle 30s: BullMQ-Queue-Metadata abfragen → queue_jobs_active / queue_jobs_waiting Gauges aktualisieren
    • Alle 60s: DB-Query → active_tenants / active_users_24h Gauges aktualisieren
    • Files: apps/worker/src/metrics-scheduler.ts, apps/worker/src/metrics-scheduler.test.ts
    • Commit: 02-observability/metrics: add worker scheduler for gauge refresh
  7. Worker: /status + /metrics Endpoints — Worker hat aktuell ggf. einen kleinen Health-Server (apps/worker/src/health.ts); dort erweitern um die beiden Endpoints. Wenn keiner: kleiner Hono-Server in apps/worker/src/admin-server.ts.

    • Files: apps/worker/src/health.ts oder apps/worker/src/admin-server.ts, Tests
    • Commit: 02-observability/metrics: add status and metrics endpoints to worker
  8. Web: SSR-Loader-Wrap — apps/web/app/lib/metrics/ssr.ts:

    • Helper-Funktion withMetrics(routeId, loaderFn) die Loader-Calls instrumentiert → web_ssr_loader_duration_seconds_bucket{route}
    • Verwendung in den Top-10-Routes initial (nicht in jedem Loader, sonst Boilerplate); langfristig könnte ein React-Router-Plugin das automatisch
    • Files: apps/web/app/lib/metrics/ssr.ts, apps/web/app/lib/metrics/ssr.test.ts
    • Commit: 02-observability/metrics: add web ssr loader metrics helper
  9. Web: /status + /metrics Route — apps/web/app/routes/__metrics.tsx (oder als Resource-Route apps/web/app/routes/api.metrics.ts mit Loader, der Prom-Text liefert).

    • Auth analog API
    • Files: apps/web/app/routes/api.metrics.ts, apps/web/app/routes/api.status.ts, apps/web/app/routes.ts (Eintrag)
    • Commit: 02-observability/metrics: add status and metrics resource routes to web
  10. Business-Metriken: AI-Cost-Hook — im AI-Service (vermutlich apps/web/app/utils/services/.server/AICostService.ts oder analog) bei jedem AI-Call:

    • aiTokensConsumedTotal.labels({ provider, model, tenant_id }).inc(usage.totalTokens)
    • Cardinality-Guard: wenn > 100 unique tenants → bucket-Strategie. Konkrete Decision: aktiv halten bis 1000 tenants, dann auf "top-50 by volume" + "other" umstellen.
    • Files: apps/web/app/utils/services/.server/AICostService.ts (oder Pendant), Tests
    • Commit: 02-observability/metrics: instrument ai cost service with prom counter
  11. Business-Metriken: Stripe-Webhook-Hook — im Stripe-Webhook-Handler.

    • Files: entsprechende Webhook-Datei, Test
    • Commit: 02-observability/metrics: instrument stripe webhook with prom counter
  12. Business-Metriken: Mail-Provider-Hook — im Mail-Adapter (packages/mail/).

    • Files: packages/mail/src/*.ts, Test
    • Commit: 02-observability/metrics: instrument mail provider with prom counter
  13. Business-Metriken: Webhook-Delivery-Hook — im Webhook-Delivery-Queue-Handler (apps/api/src/webhookDeliveryQueue.ts).

    • Files: apps/api/src/webhookDeliveryQueue.ts, Test
    • Commit: 02-observability/metrics: instrument webhook delivery with prom counter
  14. Prometheus-Config — infra/prometheus/prometheus.yml:

    • Scrape-Jobs für api, worker, web mit Bearer-Auth
    • Scrape-Interval 15s
    • Retention: 30d (default für lokales Setup ausreichend)
    • Files: infra/prometheus/prometheus.yml, infra/prometheus/docker-compose.yml
    • Commit: 02-observability/metrics: add prometheus config for scraping apps
  15. Grafana-Dashboards — infra/grafana/dashboards/:

    • overview.json — Cluster-Health (Up/Down pro Service, Request-Rate, Error-Rate, Latency p95)
    • api-latency.json — pro-Route Latency p50/p95/p99, Request-Rate, Status-Code-Verteilung
    • queue-health.json — Active/Waiting/Failed pro Queue, Job-Duration-Heatmap
    • business-kpi.json — Active Tenants/Users, Stripe-Events, Mail-Delivery-Rate
    • ai-cost-per-tenant.json — Top-20 Tenants nach Token-Verbrauch, Provider-Verteilung, Daily-Trend
    • Provisioning-Config: infra/grafana/provisioning/dashboards.yaml + datasources.yaml (Prometheus als Default-Datasource)
    • Files: infra/grafana/dashboards/*.json, infra/grafana/provisioning/*.yaml, infra/grafana/docker-compose.yml
    • Commit: 02-observability/metrics: add grafana dashboards and provisioning
  16. Doku — docs/observability/metrics.md:

    • Welche Metriken existieren + welche Frage sie beantworten
    • Wie man neue Metric hinzufügt (Cardinality-Regeln!)
    • Deploy Prometheus + Grafana auf Hetzner-Coolify
    • Dashboard-Updates-Workflow (Edit in Grafana-UI → JSON-Export → in infra/grafana/dashboards/ committen)
    • Files: docs/observability/metrics.md
    • Commit: 02-observability/metrics: document metrics catalog and dashboards
  17. Env-Defaults + .env.example:

    • METRICS_ENABLED (default false)
    • METRICS_TOKEN (random 32-byte hex; in CI/Staging/Prod als Secret)
    • Files: .env.example, packages/config/src/env.ts
    • Commit: 02-observability/metrics: add metrics env vars to config schema
  18. Smoke-Test — scripts/observability/smoke-metrics.sh:

    • Curl /status → 200 + JSON-Schema-Check
    • Curl /metrics ohne Auth → 401
    • Curl /metrics mit Auth → Prom-Text mit erwarteten Metric-Names
    • Files: scripts/observability/smoke-metrics.sh
    • Commit: 02-observability/metrics: add smoke test script

Test Requirements

  • Unit-Tests:
    • packages/metrics/src/*.test.ts — Registry-Setup, Token-Auth, Counter-Inkrement
    • apps/api/src/middleware/metrics.test.ts — Counter/Histogram bekommt korrekte Labels, Route-Pattern statt URL
    • apps/api/src/routes/status.test.ts — JSON-Schema validiert, 503 wenn DB down (Mock)
    • apps/api/src/routes/metrics.test.ts — 401 ohne Token, 200 mit Token, 404 wenn METRICS_ENABLED=false
    • packages/queue/src/runtime.test.ts — Active-Gauge inkrementiert/dekrementiert, Failed-Counter bei Throw
    • apps/worker/src/metrics-scheduler.test.ts — Scheduler ruft Queue-API in Intervallen, Gauges werden gesetzt
    • apps/web/app/lib/metrics/ssr.test.ts — withMetrics-Wrapper observiert Latency, fängt Errors
  • Integration-Test:
    • In-Process API mit injected Prom-Registry, mehrere Requests fahren, /metrics-Output prüfen
  • E2E (manuell auf Staging):
    • Prometheus-UI: https://prometheus.<staging-domain> → Targets-View → alle 3 Apps "UP"
    • Grafana-UI: https://grafana.<staging-domain> → Overview-Dashboard → Live-Daten
    • Test-Load: 100 Requests via hey oder oha → Request-Rate-Panel zeigt Spike
  • Coverage-Target: ≥ 80% für packages/metrics/, ≥ 70% für die Middleware-Files
  • Test-Files: wie oben

NICHT als completed markieren bevor:

  • alle Unit + Integration grün
  • Prometheus auf Staging scraped erfolgreich alle 3 Apps
  • mindestens 1 Grafana-Dashboard (Overview) zeigt Live-Daten

Acceptance Criteria

  • packages/metrics/ mit Registry + Default-Metriken + Token-Auth
  • API hat /status + /metrics Endpoints, Middleware aktiv
  • Worker hat /status + /metrics Endpoints, Scheduler aktiv
  • Web hat /status + /metrics Resource-Routes, SSR-Wrapper für Top-Routes
  • Business-Metriken: AI-Cost, Stripe-Webhook, Mail, Webhook-Delivery instrumentiert
  • Cardinality-Schutz aktiv (tenant_id-Bucketing dokumentiert)
  • infra/prometheus/ und infra/grafana/ Compose + Config angelegt
  • 5 Grafana-Dashboards (Overview, API-Latency, Queue-Health, Business-KPI, AI-Cost-per-Tenant) committed
  • docs/observability/metrics.md mit Catalog + Deploy-Schritten
  • .env.example mit METRICS_ENABLED, METRICS_TOKEN
  • Smoke-Skript grün
  • Aktiviert nur wenn METRICS_ENABLED=true
  • Staging: Prometheus scraped alle 3 Apps, Grafana zeigt Live-Daten
  • pnpm lint, pnpm typecheck, pnpm test grün
  • Detail-MD Progress Log aktualisiert
  • PROGRESS.md Status auf completed

Rollback Strategy

  • METRICS_ENABLED=false → beide Endpoints liefern 404, kein Overhead
  • Middleware no-op wenn Registry leer (packages/metrics-Default-Verhalten)
  • Prometheus-Stack auf Coolify stoppen → Scraping endet, Apps merken nichts (Endpoints sind passive Reader)
  • Grafana-Dashboard-Revert via git revert der JSON-Files + Prometheus-Reload
  • Cardinality-Notbremse falls Prometheus-Memory explodiert: AI-Cost-Counter ohne tenant_id-Label deployen, Hotfix-Commit
  • Business-Metric-Calls einzeln revertierbar (Steps 10-13 sind je ein Commit)

Security Notes

  • /status ist public — KEIN Auth, weil Uptime-Check-Tools (UptimeRobot etc.) kein Token verwenden. Daher: NUR boolean-artige Health-Info, NIEMALS DB-Connection-Strings, NIEMALS Pod-Hostnames, NIEMALS Stack-Traces.
  • /metrics MIT Token — sonst kann jeder Scraper Business-Metriken (AI-Cost pro Tenant) abgreifen. Token mind. 32 Bytes Random, alle 6 Monate rotieren.
  • Prometheus-UI nur intern oder mit Basic-Auth — Prometheus' Built-in-UI hat keine Auth; in Coolify hinter Traefik-Basic-Auth oder Cloudflare-Access stellen.
  • Grafana-UI mit eingebauter Auth + 2FA für Admin-User; Anonymous-Access deaktiviert.
  • Cardinality-Explosion ist Security-Issue — wenn ein Endpoint einen Label-Wert vom User-Input nimmt (z.B. user_id), kann ein Angreifer per Bot Millionen Labels erzeugen → Prom-OOM. Daher: User-Input NIEMALS als Label-Wert. Tenant-ID als Label nur, wenn Anzahl Tenants begrenzbar.
  • Business-Metriken sind tenant-vertraulich — Token-Auth Pflicht, keine Cross-Tenant-Leakage via Label-Inferenz.
  • process.env-Inhalte NICHT als Metric-Labels — Default-Process-Metrics von prom-client tun das nicht, eigene Metriken auch nicht.

Subagent Recommendations

  • Planning: Plan-Subagent mit dieser Detail-MD — soll konkrete Cardinality-Grenzen festzurren und Reihenfolge der Business-Metric-Instrumentierungen optimieren
  • Implementation packages/metrics/: general-purpose-Subagent für Step 1 (Foundation)
  • Implementation API: general-purpose-Subagent für Steps 2-4
  • Implementation Worker: general-purpose-Subagent für Steps 5-7
  • Implementation Web: general-purpose-Subagent für Steps 8-9
  • Business-Metriken: general-purpose-Subagent für Steps 10-13 (kann parallel zu API/Worker laufen)
  • Infra (Prometheus + Grafana): separater general-purpose-Subagent für Steps 14-15 (eigene Skill-Domäne)
  • Doku + Env: kleiner general-purpose-Subagent für Steps 16-17
  • Verification: general-purpose-Subagent für Staging-Smoke + Dashboard-Live-Daten-Check
  • Code-Review: gsd-code-reviewer nach Task-Completion mit Fokus auf Cardinality

Staging-Deployment-Steps

  1. Prometheus + Grafana auf Coolify deployen:
    • Neuer Service "Docker Compose" mit infra/prometheus/docker-compose.yml
    • Neuer Service "Docker Compose" mit infra/grafana/docker-compose.yml
    • Domains: prometheus.<staging-domain>, grafana.<staging-domain>
    • Grafana-Admin-Credentials in Coolify-Secrets
  2. App-Env-Variables für api/worker/web:
    • METRICS_ENABLED=true
    • METRICS_TOKEN=<32-byte hex> (gleich für alle 3 Apps, Bequemlichkeit; Prometheus konfiguriert nur einen)
  3. Branch agent/phase-02-metrics pushen → Coolify-Deploy
  4. Smoke-Tests:
    # Status (public)
    curl https://api.<staging-domain>/status
    # Erwartung: 200, JSON wie oben
    
    # Metrics (mit Token)
    curl -H "Authorization: Bearer $METRICS_TOKEN" https://api.<staging-domain>/metrics
    # Erwartung: 200, Prom-Text
    
    # Metrics ohne Token
    curl -i https://api.<staging-domain>/metrics
    # Erwartung: 401
    
  5. Prometheus-Targets-Check:
    • https://prometheus.<staging-domain>/targets → alle 3 Apps "UP"
  6. Grafana-Dashboards:
    • Login + Overview-Dashboard → Live-Daten sichtbar
    • Last-Test: oha -n 1000 https://api.<staging-domain>/status → Request-Rate-Panel zeigt Spike
  7. Cardinality-Check:
    • https://prometheus.<staging-domain>/graph → count by (__name__)({__name__=~"ai_.*"}) → Cardinality pro AI-Metric prüfen
    • Falls > 1000 Labels: Tenant-Bucketing aktivieren
  8. Wenn alles grün: PR → main

Deviations from Plan

  • Queue runtime instrumentation lives in apps/worker/src/runtime.ts, not packages/queue/src/runtime.ts. @toolctrl/queue/runtime is the thin BullMQ wrapper (3 functions). The actual processor wrap with logger / Sentry / metrics awareness is in apps/worker/src/runtime.ts — instrumenting there reuses the existing createLoggedProcessor pattern and keeps packages/queue dep-free of prom-client.
  • AI-cost cardinality cap is in-process, not config-driven. The spec hinted at "top-100 tenants + other" but did not mandate where the bookkeeping lives. Implemented as a process-local LRU (apps/web/app/lib/metrics/ai-cost.ts) with MAX_TENANT_BUCKETS=100 hard-coded. Easy to bump or wire to env if needed.
  • Mail metric is a hook, not a direct registry call. packages/mail is consumed by api + web + worker — coupling it to prom-client was rejected. The provider now takes an optional onMetric callback that the worker entry wires to mail_sent_total.
  • Web SSR coverage is 9 routes, not exactly 10. Wrapped index/app-root (shared loader), app-tenant, admin-root, dashboard, login, logout, forgot-password, new-account, invitation. Adding more is a one-line change once Prom data identifies hotspots.
  • 2026-05-27 status reconciliation: docs/improvement-plan/OPERATOR-CHECKLIST.md A1 verifies METRICS_ENABLED=true + METRICS_TOKEN are live on web/api/worker and /metrics returns 200. That supersedes the older app-side env-flip caveat in this file and matches the #35 completed row in PROGRESS.md.

Progress Log

  • 2026-05-17 06:08 — 68fd4d4c — packages/metrics with registry + defaults + token auth (16 tests)
  • 2026-05-17 06:10 — abeae9e7 — api metrics middleware (5 tests, route-pattern labels)
  • 2026-05-17 06:11 — fa6f1819 — api /status + /metrics handlers (11 tests)
  • 2026-05-17 06:14 — 0bf5f66a — wire api middleware + routes into Hono app
  • 2026-05-17 06:17 — 57f6a995 — queue runtime metrics: active gauge + completed/failed counter + duration histogram (2 tests)
  • 2026-05-17 06:19 — 84525d9b — worker metrics-scheduler: 30s queue depth + 60s business gauges (5 tests)
  • 2026-05-17 06:26 — 9607c590 — worker admin-server (node:http) /status + /metrics with graceful shutdown (6 tests)
  • 2026-05-17 06:27 — 618c8707 — web withMetrics SSR wrapper helper (4 tests)
  • 2026-05-17 06:36 — 5d461656 — api.status + api.metrics resource routes + 8 top routes wrapped (4 route tests + 4 ssr tests)
  • 2026-05-17 06:38 — 8cff9092 — ai cost prom counter with 100-bucket tenant LRU (5 tests)
  • 2026-05-17 06:40 — cadd6574 — stripe webhook counter with event_type + status labels (3 tests)
  • 2026-05-17 06:43 — a1427f26 — MailjetProvider.onMetric hook + worker auto-wiring (5 tests)
  • 2026-05-17 06:45 — ec80dec3 — webhook delivery handler counter (success/retry/dlq/rejected)
  • 2026-05-17 06:46 — aba91e5f — Prometheus compose + scrape config with 30d retention
  • 2026-05-17 06:48 — 4d2117b9 — 5 Grafana dashboards + file-based provisioning
  • 2026-05-17 06:50 — 52511b09 — docs/observability/metrics.md (catalog + deploy + rollback playbook)
  • 2026-05-17 06:51 — 4245c6d5 — METRICS_ENABLED / METRICS_TOKEN / WORKER_ADMIN_* / APP_REGION in env catalog + .env.example
  • 2026-05-17 06:53 — 0fcc6a17 — scripts/observability/smoke-metrics.sh
  • 2026-05-17 06:54 — <this commit> — progress log + PROGRESS.md update

Lessons Learned

  • Hono c.req.routePath is the right cardinality knob. Going through the matched route pattern keeps http_requests_total bounded by the route table size (~50 entries) rather than by user-input URLs.
  • prom-client emits labels in declaration order. Test regexes that hard-code label order need to match the order in labelNames (not alphabetical, not insertion-order of labels({...})).
  • Mail providers should not import from the host app. Threading a callback through MailjetProviderOptions.onMetric is cleaner than reaching for a registry singleton from a leaf package.
  • React Router v7 typegen runs lazily. New resource routes (api.status.ts, api.metrics.ts) work as soon as react-router typegen has run once — committed types in .react-router/types/ track route changes automatically when typecheck runs.

Progress Log (continued)

  • 2026-05-18 00:48 UTC — staging-deploy — Cloudflare A record grafana.boiler.saas-ctrl.com → 46.225.5.93 created (id 2267905b1ee15df4002ca02d76af1fe0, proxied=false). Observability stack brought up on /opt/observability/: Prometheus v3.6.0 (boilerplate-prometheus) on dedicated boilerplate-observability network, Grafana 11.4.0 (boilerplate-grafana) attached to both observability + coolify networks via on-host docker-compose.override.yml (Traefik labels for TLS via Let's Encrypt). Secrets generated via openssl rand on the host (METRICS_TOKEN, GRAFANA_ADMIN_PASSWORD, GRAFANA_PROMETHEUS_TOKEN reuses METRICS_TOKEN) stored in /opt/observability/.env (chmod 600) and ~/.cli-tokens/grafana-staging-env; the same token is also bind-mounted into Prometheus at /etc/prometheus/secrets/metrics_token. While deploying, found and fixed a duplicate jsonData key in infra/grafana/provisioning/datasources/prometheus.yaml (committed file was invalid YAML; Grafana refused to start). Smoke green: curl -fSs https://grafana.boiler.saas-ctrl.com/api/health → {"database":"ok","version":"11.4.0",...}. Scrape targets in prometheus.yml point at the public HTTPS hostnames (api.boiler.saas-ctrl.com, etc.) and require the apps to expose /metrics with the same bearer token.
  • 2026-05-27 — docs/status reconciliation — A1 verifies METRICS_ENABLED=true + METRICS_TOKEN live on web/api/worker and /metrics returns 200. The older "env flips pending" note above is superseded; any Prometheus target outage after this date should be treated as a fresh runtime incident, not unfinished #35 implementation.