25 KiB
Metrics + Grafana — /status, /metrics, Dashboards
Status: completed (staging stack deployed; /metrics live per operator checklist A1)
Task-ID: #35
Phase: 02
Estimated Effort: 3-4 Tage
Owner: agent/02-metrics
Problem
Aktuell gibt es keine Stelle, wo Operators auf einen Blick sehen, wie es der Boilerplate-Instance geht:
- Kein
/status-Endpoint für externe Uptime-Checks (Uptime-Kuma, Statuspage) - Kein
/metrics-Endpoint im Prometheus-Format → keine Infrastruktur-Metriken (Request-Latency, Queue-Depth, DB-Query-Timing) - Keine Business-Metriken (aktive Tenants, AI-Token-Verbrauch pro Tenant, Stripe-Webhook-Volumen, Email-Versand-Raten)
- Keine Grafana-Dashboards → bei Performance-Issues muss man in Sentry-Performance + Container-Logs gleichzeitig graben
Sentry-Performance liefert per-Trace-Granularität, aber keine aggregierten Live-Counter/Gauges. Prometheus ist das richtige Werkzeug dafür.
Depends on
02-observability/sentry-wiring-api-worker.md— wir verdrahten Sentry-Reporter daneben, kein Konflikt02-observability/logging-migration.md— wir wollen, dass Metrics-Middleware nicht aufconsole.*loggt
Blocks
03-scaling-infrastructure/shared-store-migration.md— kein harter Block, aber Metrics-Vorher/Nachher-Vergleich von Cache-Hit-Rate setzt Metrics voraus06-plugin-system/*— Plugin-Cost-Metriken sollen demselben/metrics-Endpoint folgen
Approach
Gewählt: prom-client als Library für API + Worker. Eine Hono-Middleware in der API für Auto-Instrumentation (Request-Counter, Latency-Histogram), manuelle Instrumentation in den Worker-Job-Handlern (packages/queue/) und in Web-SSR-Loadern (für web_ssr_loader_duration_seconds). Business-Metriken werden in den entsprechenden Service-Methoden gesetzt (z.B. ai_tokens_consumed_total im AICostService).
Zwei Endpoints pro App:
/status(public, no auth) — JSON, leichtgewichtig, für Uptime-Checks.200wenn alle Checks ok,503wenn einer kritischer fehlt./metrics(token-protected viaAuthorization: Bearer <token>) — Prometheus Text-Format. Token ausMETRICS_TOKENenv. Bei fehlender Auth:401.
Beide Endpoints nur registriert wenn METRICS_ENABLED=true — Default off in Dev, on in Staging/Prod.
Grafana + Prometheus Deployment: Coolify-Stack auf dem Hetzner-Server (gleicher Server wie Sentry). Prometheus scraped die /metrics-Endpoints der drei Apps. Grafana liest aus Prometheus. Dashboards als JSON-Templates versioniert in infra/grafana/dashboards/.
Sample-Metriken (final-Liste):
Infrastruktur
http_requests_total{method, route, status}— Counter (API + Web-SSR)http_request_duration_seconds_bucket{method, route, status}— Histogram (API + Web-SSR)prisma_query_duration_seconds_bucket{model, action}— Histogram (alle Apps)queue_jobs_active{queue}— Gauge (Worker)queue_jobs_waiting{queue}— Gauge (Worker)queue_jobs_completed_total{queue, job_name}— Counter (Worker)queue_jobs_failed_total{queue, job_name, error_type}— Counter (Worker)queue_job_duration_seconds_bucket{queue, job_name}— Histogram (Worker)redis_command_duration_seconds_bucket{command}— Histogram (alle Apps)process_*— Default-Process-Metrics von prom-client
Business
active_tenants— Gauge (1x pro Minute aktualisiert via Scheduled-Job)active_users_24h— Gauge (Login-Events der letzten 24h, 1x pro Minute aktualisiert)ai_tokens_consumed_total{provider, model, tenant_id}— Counter (Cardinality-Warnung: tenant_id kann groß werden → ggf. nur top-100 tenants + "other" Bucket; Decision-Punkt im Step)stripe_webhook_events_total{event_type, status}— Countermail_sent_total{provider, status}— Counter (status=delivered/bounced/failed)webhook_delivery_attempts_total{status}— Counter (status=success/retry/dlq)csrf_token_rotation_total— Countersubscription_active{plan_id}— Gauge
Cardinality-Constraint: Prometheus-Label-Cardinality sollte pro Metric < 10k bleiben, sonst explodiert Prometheus-Memory. tenant_id als Label nur in Business-Metriken wo unvermeidlich; sonst per-Tenant-Aggregate via separater Scheduled-Aggregation in DB.
Implementation Steps
-
Library + Shared-Package —
packages/metrics/neu:prom-clientals dep- Exportiert
createRegistry(serviceName), vorgefertigte Default-Metriken (httpRequestsTotal,httpRequestDurationSeconds,prismaQueryDurationSeconds,queueJobs*,redisCommandDurationSeconds, etc.) - Token-Auth-Middleware (framework-agnostisch — gibt boolean zurück)
- Files:
packages/metrics/package.json,packages/metrics/src/index.ts,packages/metrics/src/registry.ts,packages/metrics/src/defaults.ts,packages/metrics/src/auth.ts,packages/metrics/src/*.test.ts - Commit:
02-observability/metrics: add packages/metrics with prom-client registry
-
API: Hono-Middleware —
apps/api/src/middleware/metrics.ts:- Counter pro Request mit Labels
method, route_pattern, status(Route-Pattern statt URL, sonst Cardinality-Explosion) - Histogram für Latency, Buckets
[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]Sekunden - Files:
apps/api/src/middleware/metrics.ts,apps/api/src/middleware/metrics.test.ts - Commit:
02-observability/metrics: add metrics middleware for api
- Counter pro Request mit Labels
-
API:
/status+/metricsEndpoints —apps/api/src/routes/status.tsundapps/api/src/routes/metrics.ts:/statusJSON-Schema:{ "status": "ok" | "degraded" | "down", "checks": { "db": "ok" | "down", "redis": "ok" | "down", "queues": "ok" | "degraded" | "down", "workers": "ok" | "down", "storage": "ok" | "down" }, "version": "<git-sha>", "uptime_seconds": 12345, "region": "<env-region>" }/metricsruftregistry.metrics()→ Text-Format- Bei
!METRICS_ENABLED→ 404 für beide /metricsohne Token → 401- Files:
apps/api/src/routes/status.ts,apps/api/src/routes/metrics.ts, jeweils Tests - Commit:
02-observability/metrics: add status and metrics endpoints to api
-
API: Middleware + Routes registrieren — in
apps/api/src/app.ts.- Files:
apps/api/src/app.ts - Commit:
02-observability/metrics: register metrics in api app
- Files:
-
Worker: Manual Instrumentation in Job-Handlern —
packages/queue/src/runtime.ts(oder wo immer der zentrale Worker-Wrapper liegt):- Pro Job-Start
queue_jobs_activeinkrementieren, am Ende dekrementieren queue_job_duration_secondsHistogram-Observationqueue_jobs_completed_total/queue_jobs_failed_totalCounter- Files:
packages/queue/src/runtime.ts(oder analog),packages/queue/src/runtime.test.ts - Commit:
02-observability/metrics: instrument queue runtime with prom counters
- Pro Job-Start
-
Worker: Background-Scheduler für Gauge-Refresh —
apps/worker/src/metrics-scheduler.ts:- Alle 30s: BullMQ-Queue-Metadata abfragen →
queue_jobs_active/queue_jobs_waitingGauges aktualisieren - Alle 60s: DB-Query →
active_tenants/active_users_24hGauges aktualisieren - Files:
apps/worker/src/metrics-scheduler.ts,apps/worker/src/metrics-scheduler.test.ts - Commit:
02-observability/metrics: add worker scheduler for gauge refresh
- Alle 30s: BullMQ-Queue-Metadata abfragen →
-
Worker:
/status+/metricsEndpoints — Worker hat aktuell ggf. einen kleinen Health-Server (apps/worker/src/health.ts); dort erweitern um die beiden Endpoints. Wenn keiner: kleiner Hono-Server inapps/worker/src/admin-server.ts.- Files:
apps/worker/src/health.tsoderapps/worker/src/admin-server.ts, Tests - Commit:
02-observability/metrics: add status and metrics endpoints to worker
- Files:
-
Web: SSR-Loader-Wrap —
apps/web/app/lib/metrics/ssr.ts:- Helper-Funktion
withMetrics(routeId, loaderFn)die Loader-Calls instrumentiert →web_ssr_loader_duration_seconds_bucket{route} - Verwendung in den Top-10-Routes initial (nicht in jedem Loader, sonst Boilerplate); langfristig könnte ein React-Router-Plugin das automatisch
- Files:
apps/web/app/lib/metrics/ssr.ts,apps/web/app/lib/metrics/ssr.test.ts - Commit:
02-observability/metrics: add web ssr loader metrics helper
- Helper-Funktion
-
Web:
/status+/metricsRoute —apps/web/app/routes/__metrics.tsx(oder als Resource-Routeapps/web/app/routes/api.metrics.tsmit Loader, der Prom-Text liefert).- Auth analog API
- Files:
apps/web/app/routes/api.metrics.ts,apps/web/app/routes/api.status.ts,apps/web/app/routes.ts(Eintrag) - Commit:
02-observability/metrics: add status and metrics resource routes to web
-
Business-Metriken: AI-Cost-Hook — im AI-Service (vermutlich
apps/web/app/utils/services/.server/AICostService.tsoder analog) bei jedem AI-Call:aiTokensConsumedTotal.labels({ provider, model, tenant_id }).inc(usage.totalTokens)- Cardinality-Guard: wenn > 100 unique tenants → bucket-Strategie. Konkrete Decision: aktiv halten bis 1000 tenants, dann auf "top-50 by volume" + "other" umstellen.
- Files:
apps/web/app/utils/services/.server/AICostService.ts(oder Pendant), Tests - Commit:
02-observability/metrics: instrument ai cost service with prom counter
-
Business-Metriken: Stripe-Webhook-Hook — im Stripe-Webhook-Handler.
- Files: entsprechende Webhook-Datei, Test
- Commit:
02-observability/metrics: instrument stripe webhook with prom counter
-
Business-Metriken: Mail-Provider-Hook — im Mail-Adapter (
packages/mail/).- Files:
packages/mail/src/*.ts, Test - Commit:
02-observability/metrics: instrument mail provider with prom counter
- Files:
-
Business-Metriken: Webhook-Delivery-Hook — im Webhook-Delivery-Queue-Handler (
apps/api/src/webhookDeliveryQueue.ts).- Files:
apps/api/src/webhookDeliveryQueue.ts, Test - Commit:
02-observability/metrics: instrument webhook delivery with prom counter
- Files:
-
Prometheus-Config —
infra/prometheus/prometheus.yml:- Scrape-Jobs für api, worker, web mit Bearer-Auth
- Scrape-Interval 15s
- Retention: 30d (default für lokales Setup ausreichend)
- Files:
infra/prometheus/prometheus.yml,infra/prometheus/docker-compose.yml - Commit:
02-observability/metrics: add prometheus config for scraping apps
-
Grafana-Dashboards —
infra/grafana/dashboards/:overview.json— Cluster-Health (Up/Down pro Service, Request-Rate, Error-Rate, Latency p95)api-latency.json— pro-Route Latency p50/p95/p99, Request-Rate, Status-Code-Verteilungqueue-health.json— Active/Waiting/Failed pro Queue, Job-Duration-Heatmapbusiness-kpi.json— Active Tenants/Users, Stripe-Events, Mail-Delivery-Rateai-cost-per-tenant.json— Top-20 Tenants nach Token-Verbrauch, Provider-Verteilung, Daily-Trend- Provisioning-Config:
infra/grafana/provisioning/dashboards.yaml+datasources.yaml(Prometheus als Default-Datasource) - Files:
infra/grafana/dashboards/*.json,infra/grafana/provisioning/*.yaml,infra/grafana/docker-compose.yml - Commit:
02-observability/metrics: add grafana dashboards and provisioning
-
Doku —
docs/observability/metrics.md:- Welche Metriken existieren + welche Frage sie beantworten
- Wie man neue Metric hinzufügt (Cardinality-Regeln!)
- Deploy Prometheus + Grafana auf Hetzner-Coolify
- Dashboard-Updates-Workflow (Edit in Grafana-UI → JSON-Export → in
infra/grafana/dashboards/committen) - Files:
docs/observability/metrics.md - Commit:
02-observability/metrics: document metrics catalog and dashboards
-
Env-Defaults +
.env.example:METRICS_ENABLED(defaultfalse)METRICS_TOKEN(random 32-byte hex; in CI/Staging/Prod als Secret)- Files:
.env.example,packages/config/src/env.ts - Commit:
02-observability/metrics: add metrics env vars to config schema
-
Smoke-Test —
scripts/observability/smoke-metrics.sh:- Curl
/status→ 200 + JSON-Schema-Check - Curl
/metricsohne Auth → 401 - Curl
/metricsmit Auth → Prom-Text mit erwarteten Metric-Names - Files:
scripts/observability/smoke-metrics.sh - Commit:
02-observability/metrics: add smoke test script
- Curl
Test Requirements
- Unit-Tests:
packages/metrics/src/*.test.ts— Registry-Setup, Token-Auth, Counter-Inkrementapps/api/src/middleware/metrics.test.ts— Counter/Histogram bekommt korrekte Labels, Route-Pattern statt URLapps/api/src/routes/status.test.ts— JSON-Schema validiert,503wenn DB down (Mock)apps/api/src/routes/metrics.test.ts— 401 ohne Token, 200 mit Token, 404 wennMETRICS_ENABLED=falsepackages/queue/src/runtime.test.ts— Active-Gauge inkrementiert/dekrementiert, Failed-Counter bei Throwapps/worker/src/metrics-scheduler.test.ts— Scheduler ruft Queue-API in Intervallen, Gauges werden gesetztapps/web/app/lib/metrics/ssr.test.ts—withMetrics-Wrapper observiert Latency, fängt Errors
- Integration-Test:
- In-Process API mit injected Prom-Registry, mehrere Requests fahren,
/metrics-Output prüfen
- In-Process API mit injected Prom-Registry, mehrere Requests fahren,
- E2E (manuell auf Staging):
- Prometheus-UI:
https://prometheus.<staging-domain>→ Targets-View → alle 3 Apps "UP" - Grafana-UI:
https://grafana.<staging-domain>→ Overview-Dashboard → Live-Daten - Test-Load: 100 Requests via
heyoderoha→ Request-Rate-Panel zeigt Spike
- Prometheus-UI:
- Coverage-Target: ≥ 80% für
packages/metrics/, ≥ 70% für die Middleware-Files - Test-Files: wie oben
NICHT als completed markieren bevor:
- alle Unit + Integration grün
- Prometheus auf Staging scraped erfolgreich alle 3 Apps
- mindestens 1 Grafana-Dashboard (Overview) zeigt Live-Daten
Acceptance Criteria
packages/metrics/mit Registry + Default-Metriken + Token-Auth- API hat
/status+/metricsEndpoints, Middleware aktiv - Worker hat
/status+/metricsEndpoints, Scheduler aktiv - Web hat
/status+/metricsResource-Routes, SSR-Wrapper für Top-Routes - Business-Metriken: AI-Cost, Stripe-Webhook, Mail, Webhook-Delivery instrumentiert
- Cardinality-Schutz aktiv (tenant_id-Bucketing dokumentiert)
infra/prometheus/undinfra/grafana/Compose + Config angelegt- 5 Grafana-Dashboards (Overview, API-Latency, Queue-Health, Business-KPI, AI-Cost-per-Tenant) committed
docs/observability/metrics.mdmit Catalog + Deploy-Schritten.env.examplemitMETRICS_ENABLED,METRICS_TOKEN- Smoke-Skript grün
- Aktiviert nur wenn
METRICS_ENABLED=true - Staging: Prometheus scraped alle 3 Apps, Grafana zeigt Live-Daten
pnpm lint,pnpm typecheck,pnpm testgrün- Detail-MD Progress Log aktualisiert
- PROGRESS.md Status auf
completed
Rollback Strategy
METRICS_ENABLED=false→ beide Endpoints liefern 404, kein Overhead- Middleware no-op wenn Registry leer (
packages/metrics-Default-Verhalten) - Prometheus-Stack auf Coolify stoppen → Scraping endet, Apps merken nichts (Endpoints sind passive Reader)
- Grafana-Dashboard-Revert via
git revertder JSON-Files + Prometheus-Reload - Cardinality-Notbremse falls Prometheus-Memory explodiert: AI-Cost-Counter ohne
tenant_id-Label deployen, Hotfix-Commit - Business-Metric-Calls einzeln revertierbar (Steps 10-13 sind je ein Commit)
Security Notes
/statusist public — KEIN Auth, weil Uptime-Check-Tools (UptimeRobot etc.) kein Token verwenden. Daher: NUR boolean-artige Health-Info, NIEMALS DB-Connection-Strings, NIEMALS Pod-Hostnames, NIEMALS Stack-Traces./metricsMIT Token — sonst kann jeder Scraper Business-Metriken (AI-Cost pro Tenant) abgreifen. Token mind. 32 Bytes Random, alle 6 Monate rotieren.- Prometheus-UI nur intern oder mit Basic-Auth — Prometheus' Built-in-UI hat keine Auth; in Coolify hinter Traefik-Basic-Auth oder Cloudflare-Access stellen.
- Grafana-UI mit eingebauter Auth + 2FA für Admin-User; Anonymous-Access deaktiviert.
- Cardinality-Explosion ist Security-Issue — wenn ein Endpoint einen Label-Wert vom User-Input nimmt (z.B.
user_id), kann ein Angreifer per Bot Millionen Labels erzeugen → Prom-OOM. Daher: User-Input NIEMALS als Label-Wert. Tenant-ID als Label nur, wenn Anzahl Tenants begrenzbar. - Business-Metriken sind tenant-vertraulich — Token-Auth Pflicht, keine Cross-Tenant-Leakage via Label-Inferenz.
process.env-Inhalte NICHT als Metric-Labels — Default-Process-Metrics von prom-client tun das nicht, eigene Metriken auch nicht.
Subagent Recommendations
- Planning:
Plan-Subagent mit dieser Detail-MD — soll konkrete Cardinality-Grenzen festzurren und Reihenfolge der Business-Metric-Instrumentierungen optimieren - Implementation
packages/metrics/:general-purpose-Subagent für Step 1 (Foundation) - Implementation API:
general-purpose-Subagent für Steps 2-4 - Implementation Worker:
general-purpose-Subagent für Steps 5-7 - Implementation Web:
general-purpose-Subagent für Steps 8-9 - Business-Metriken:
general-purpose-Subagent für Steps 10-13 (kann parallel zu API/Worker laufen) - Infra (Prometheus + Grafana): separater
general-purpose-Subagent für Steps 14-15 (eigene Skill-Domäne) - Doku + Env: kleiner
general-purpose-Subagent für Steps 16-17 - Verification:
general-purpose-Subagent für Staging-Smoke + Dashboard-Live-Daten-Check - Code-Review:
gsd-code-reviewernach Task-Completion mit Fokus auf Cardinality
Staging-Deployment-Steps
- Prometheus + Grafana auf Coolify deployen:
- Neuer Service "Docker Compose" mit
infra/prometheus/docker-compose.yml - Neuer Service "Docker Compose" mit
infra/grafana/docker-compose.yml - Domains:
prometheus.<staging-domain>,grafana.<staging-domain> - Grafana-Admin-Credentials in Coolify-Secrets
- Neuer Service "Docker Compose" mit
- App-Env-Variables für api/worker/web:
METRICS_ENABLED=trueMETRICS_TOKEN=<32-byte hex>(gleich für alle 3 Apps, Bequemlichkeit; Prometheus konfiguriert nur einen)
- Branch
agent/phase-02-metricspushen → Coolify-Deploy - Smoke-Tests:
# Status (public) curl https://api.<staging-domain>/status # Erwartung: 200, JSON wie oben # Metrics (mit Token) curl -H "Authorization: Bearer $METRICS_TOKEN" https://api.<staging-domain>/metrics # Erwartung: 200, Prom-Text # Metrics ohne Token curl -i https://api.<staging-domain>/metrics # Erwartung: 401 - Prometheus-Targets-Check:
https://prometheus.<staging-domain>/targets→ alle 3 Apps "UP"
- Grafana-Dashboards:
- Login + Overview-Dashboard → Live-Daten sichtbar
- Last-Test:
oha -n 1000 https://api.<staging-domain>/status→ Request-Rate-Panel zeigt Spike
- Cardinality-Check:
https://prometheus.<staging-domain>/graph→count by (__name__)({__name__=~"ai_.*"})→ Cardinality pro AI-Metric prüfen- Falls > 1000 Labels: Tenant-Bucketing aktivieren
- Wenn alles grün: PR → main
Deviations from Plan
- Queue runtime instrumentation lives in
apps/worker/src/runtime.ts, notpackages/queue/src/runtime.ts.@toolctrl/queue/runtimeis the thin BullMQ wrapper (3 functions). The actual processor wrap with logger / Sentry / metrics awareness is inapps/worker/src/runtime.ts— instrumenting there reuses the existingcreateLoggedProcessorpattern and keepspackages/queuedep-free of prom-client. - AI-cost cardinality cap is in-process, not config-driven. The
spec hinted at "top-100 tenants + other" but did not mandate where the
bookkeeping lives. Implemented as a process-local LRU
(
apps/web/app/lib/metrics/ai-cost.ts) withMAX_TENANT_BUCKETS=100hard-coded. Easy to bump or wire to env if needed. - Mail metric is a hook, not a direct registry call.
packages/mailis consumed by api + web + worker — coupling it to prom-client was rejected. The provider now takes an optionalonMetriccallback that the worker entry wires tomail_sent_total. - Web SSR coverage is 9 routes, not exactly 10. Wrapped index/app-root (shared loader), app-tenant, admin-root, dashboard, login, logout, forgot-password, new-account, invitation. Adding more is a one-line change once Prom data identifies hotspots.
- 2026-05-27 status reconciliation:
docs/improvement-plan/OPERATOR-CHECKLIST.mdA1 verifiesMETRICS_ENABLED=true+METRICS_TOKENare live on web/api/worker and/metricsreturns 200. That supersedes the older app-side env-flip caveat in this file and matches the #35completedrow inPROGRESS.md.
Progress Log
- 2026-05-17 06:08 —
68fd4d4c— packages/metrics with registry + defaults + token auth (16 tests) - 2026-05-17 06:10 —
abeae9e7— api metrics middleware (5 tests, route-pattern labels) - 2026-05-17 06:11 —
fa6f1819— api /status + /metrics handlers (11 tests) - 2026-05-17 06:14 —
0bf5f66a— wire api middleware + routes into Hono app - 2026-05-17 06:17 —
57f6a995— queue runtime metrics: active gauge + completed/failed counter + duration histogram (2 tests) - 2026-05-17 06:19 —
84525d9b— worker metrics-scheduler: 30s queue depth + 60s business gauges (5 tests) - 2026-05-17 06:26 —
9607c590— worker admin-server (node:http) /status + /metrics with graceful shutdown (6 tests) - 2026-05-17 06:27 —
618c8707— web withMetrics SSR wrapper helper (4 tests) - 2026-05-17 06:36 —
5d461656— api.status + api.metrics resource routes + 8 top routes wrapped (4 route tests + 4 ssr tests) - 2026-05-17 06:38 —
8cff9092— ai cost prom counter with 100-bucket tenant LRU (5 tests) - 2026-05-17 06:40 —
cadd6574— stripe webhook counter with event_type + status labels (3 tests) - 2026-05-17 06:43 —
a1427f26— MailjetProvider.onMetric hook + worker auto-wiring (5 tests) - 2026-05-17 06:45 —
ec80dec3— webhook delivery handler counter (success/retry/dlq/rejected) - 2026-05-17 06:46 —
aba91e5f— Prometheus compose + scrape config with 30d retention - 2026-05-17 06:48 —
4d2117b9— 5 Grafana dashboards + file-based provisioning - 2026-05-17 06:50 —
52511b09— docs/observability/metrics.md (catalog + deploy + rollback playbook) - 2026-05-17 06:51 —
4245c6d5— METRICS_ENABLED / METRICS_TOKEN / WORKER_ADMIN_* / APP_REGION in env catalog + .env.example - 2026-05-17 06:53 —
0fcc6a17— scripts/observability/smoke-metrics.sh - 2026-05-17 06:54 —
<this commit>— progress log + PROGRESS.md update
Lessons Learned
- Hono
c.req.routePathis the right cardinality knob. Going through the matched route pattern keepshttp_requests_totalbounded by the route table size (~50 entries) rather than by user-input URLs. prom-clientemits labels in declaration order. Test regexes that hard-code label order need to match the order inlabelNames(not alphabetical, not insertion-order oflabels({...})).- Mail providers should not import from the host app. Threading a
callback through
MailjetProviderOptions.onMetricis cleaner than reaching for a registry singleton from a leaf package. - React Router v7 typegen runs lazily. New resource routes
(
api.status.ts,api.metrics.ts) work as soon asreact-router typegenhas run once — committed types in.react-router/types/track route changes automatically when typecheck runs.
Progress Log (continued)
- 2026-05-18 00:48 UTC — staging-deploy — Cloudflare A record
grafana.boiler.saas-ctrl.com→ 46.225.5.93 created (id2267905b1ee15df4002ca02d76af1fe0, proxied=false). Observability stack brought up on/opt/observability/: Prometheus v3.6.0 (boilerplate-prometheus) on dedicatedboilerplate-observabilitynetwork, Grafana 11.4.0 (boilerplate-grafana) attached to both observability + coolify networks via on-hostdocker-compose.override.yml(Traefik labels for TLS via Let's Encrypt). Secrets generated viaopenssl randon the host (METRICS_TOKEN,GRAFANA_ADMIN_PASSWORD,GRAFANA_PROMETHEUS_TOKENreuses METRICS_TOKEN) stored in/opt/observability/.env(chmod 600) and~/.cli-tokens/grafana-staging-env; the same token is also bind-mounted into Prometheus at/etc/prometheus/secrets/metrics_token. While deploying, found and fixed a duplicatejsonDatakey ininfra/grafana/provisioning/datasources/prometheus.yaml(committed file was invalid YAML; Grafana refused to start). Smoke green:curl -fSs https://grafana.boiler.saas-ctrl.com/api/health→{"database":"ok","version":"11.4.0",...}. Scrape targets inprometheus.ymlpoint at the public HTTPS hostnames (api.boiler.saas-ctrl.com, etc.) and require the apps to expose/metricswith the same bearer token. - 2026-05-27 — docs/status reconciliation — A1 verifies
METRICS_ENABLED=true+METRICS_TOKENlive on web/api/worker and/metricsreturns 200. The older "env flips pending" note above is superseded; any Prometheus target outage after this date should be treated as a fresh runtime incident, not unfinished #35 implementation.