Metrics + Grafana

Prometheus + Grafana stack for the boilerplate. Scrapes /metrics from the three application processes and renders five dashboards.

Architecture

+----------+        scrape          +-------------+        query        +---------+
|   api    | <--------------------- |             | <------------------ |         |
|  worker  | <--------------------- | Prometheus  |                     | Grafana |
|   web    | <--------------------- |             |                     |         |
+----------+                        +-------------+                     +---------+
       ^                                   |
       |                                   v
   /status                            tsdb (30d)
   /metrics

All three apps are instrumented via @toolctrl/metrics.
/status is public, JSON, 200 / 503 — suitable for Uptime Kuma.
/metrics is bearer-token protected, returns Prom text-format.
Both endpoints are gated by METRICS_ENABLED=true; default off in dev.

Metric catalog

Infrastructure (all services)

Metric	Type	Labels	Question it answers
`http_requests_total`	counter	method, route, status	Request volume + status distribution.
`http_request_duration_seconds_bucket`	histogram	method, route, status	API latency percentiles.
`prisma_query_duration_seconds_bucket`	histogram	model, action	DB hotspot detection.
`redis_command_duration_seconds_bucket`	histogram	command	Redis cache hit / slow-cmd diagnosis.
`process_*`	counter / gauge	—	Node.js process health (default `prom-client`).

Worker queue

Metric	Type	Labels	Question
`queue_jobs_active`	gauge	queue	Concurrency in-flight.
`queue_jobs_waiting`	gauge	queue	Backlog depth.
`queue_jobs_completed_total`	counter	queue, job_name	Throughput.
`queue_jobs_failed_total`	counter	queue, job_name, error_type	Failure-type breakdown.
`queue_job_duration_seconds_bucket`	histogram	queue, job_name	Job-duration heatmap.

Web SSR

Metric	Type	Labels	Question
`web_ssr_loader_duration_seconds_bucket`	histogram	route, status	Loader-level latency for the top routes.

Status enum: ok | error | redirect. Apply via withMetrics(routeId, loader) from apps/web/app/lib/metrics/ssr.ts. Currently wrapped: index/app-root, app-tenant, admin-root, dashboard, login, logout, forgot-password, new-account, invitation.

Business

Metric	Type	Labels	Question
`active_tenants`	gauge	—	Tenant growth.
`active_users_24h`	gauge	—	DAU.
`ai_tokens_consumed_total`	counter	provider, model, tenant_bucket	AI spend allocation.
`stripe_webhook_events_total`	counter	event_type, status	Stripe traffic + spoof attempts.
`mail_sent_total`	counter	provider, status	Email reliability.
`webhook_delivery_attempts_total`	counter	status	Outbound webhook health (success / retry / dlq / rejected).
`csrf_token_rotation_total`	counter	—	Auth churn.
`subscription_active`	gauge	plan_id	Plan mix.

Cardinality rules

Prometheus dies before you hit a million series. Pre-empt it.

Never use raw user / request input as a label value. Always use a pattern (route pattern, error name, status code) or a bucket.
Cap unknown sets. ai_tokens_consumed_total.tenant_bucket is a process-local LRU of the first 100 tenants seen + "other" overflow (see apps/web/app/lib/metrics/ai-cost.ts). When you add a new metric with a similar shape, apply the same pattern.
No PII labels. Tenant IDs are opaque slugs, never email / name.
No env-var spill. Default prom-client process metrics never leak process.env; verify the same for custom metrics in code review.

Adding a new metric

Declare it in packages/metrics/src/defaults.ts. Pick the smallest useful label set.
Write a unit test in packages/metrics/src/defaults.test.ts that touches the metric and asserts the rendered text contains the name.
Add an instrumentation call at the business-logic site. Wrap the call in try { … } catch { /* never crash */ }.
Add a Grafana panel under infra/grafana/dashboards/ and commit.
Update this catalog.

Deploy on Hetzner staging

The Prometheus and Grafana stacks are intended for the same Hetzner observability box that hosts Sentry (/opt/observability/ directory convention). The compose files in infra/prometheus/ and infra/grafana/ can be deployed as Coolify "Docker Compose" services or as plain docker compose up -d units.

Per-step instructions live in infra/prometheus/README.md.

Coolify vs `/opt/observability/`

Operator decision — both work. Coolify is easier to wire to Traefik for the prometheus.boiler.saas-ctrl.com and grafana.boiler.saas-ctrl.com subdomains. Plain docker compose is lighter but requires a manual reverse-proxy entry. Default recommendation: Coolify-managed Service, same as Sentry.

Dashboard-update workflow

Edit panels in the Grafana UI.
Top-right menu → Share → Export → Save to file (uncheck "Export for sharing externally" so UIDs stay stable).
Drop the JSON into infra/grafana/dashboards/<name>.json.
Commit. Provisioning reloads automatically every 30s.

Smoke check

# /status — should be 200 OK
curl -fsS https://api.boiler.saas-ctrl.com/status | jq

# /metrics without auth — should be 401
curl -i https://api.boiler.saas-ctrl.com/metrics | head -1

# /metrics with auth — should be 200 + prom text
curl -fsS -H "Authorization: Bearer $METRICS_TOKEN" \
  https://api.boiler.saas-ctrl.com/metrics | head -20

scripts/observability/smoke-metrics.sh automates this against any base host (defaults to staging).

Rollback

Disable everything: METRICS_ENABLED=false. All three apps stop exposing /metrics (404) within one restart. Counters keep working in-process; only the surface goes away.
Cardinality emergency (Prometheus OOM): the most likely culprit is ai_tokens_consumed_total. The safety valve is already there (tenant_bucket caps at 101). If a NEW metric explodes, revert the commit that added it and reload Prometheus.
Dashboard rollback: git revert the JSON file and the file provider replaces it within 30 s.

6.4 KiB Raw Blame History