# Metrics + Grafana

Prometheus + Grafana stack for the boilerplate. Scrapes `/metrics` from
the three application processes and renders five dashboards.

## Architecture

```
+----------+        scrape          +-------------+        query        +---------+
|   api    | <--------------------- |             | <------------------ |         |
|  worker  | <--------------------- | Prometheus  |                     | Grafana |
|   web    | <--------------------- |             |                     |         |
+----------+                        +-------------+                     +---------+
       ^                                   |
       |                                   v
   /status                            tsdb (30d)
   /metrics
```

- All three apps are instrumented via [`@toolctrl/metrics`](../../packages/metrics).
- `/status` is public, JSON, 200 / 503 — suitable for Uptime Kuma.
- `/metrics` is bearer-token protected, returns Prom text-format.
- Both endpoints are gated by `METRICS_ENABLED=true`; default off in dev.

## Metric catalog

### Infrastructure (all services)

| Metric | Type | Labels | Question it answers |
|---|---|---|---|
| `http_requests_total` | counter | method, route, status | Request volume + status distribution. |
| `http_request_duration_seconds_bucket` | histogram | method, route, status | API latency percentiles. |
| `prisma_query_duration_seconds_bucket` | histogram | model, action | DB hotspot detection. |
| `redis_command_duration_seconds_bucket` | histogram | command | Redis cache hit / slow-cmd diagnosis. |
| `process_*` | counter / gauge | — | Node.js process health (default `prom-client`). |

### Worker queue

| Metric | Type | Labels | Question |
|---|---|---|---|
| `queue_jobs_active` | gauge | queue | Concurrency in-flight. |
| `queue_jobs_waiting` | gauge | queue | Backlog depth. |
| `queue_jobs_completed_total` | counter | queue, job_name | Throughput. |
| `queue_jobs_failed_total` | counter | queue, job_name, error_type | Failure-type breakdown. |
| `queue_job_duration_seconds_bucket` | histogram | queue, job_name | Job-duration heatmap. |

### Web SSR

| Metric | Type | Labels | Question |
|---|---|---|---|
| `web_ssr_loader_duration_seconds_bucket` | histogram | route, status | Loader-level latency for the top routes. |

Status enum: `ok | error | redirect`. Apply via `withMetrics(routeId, loader)`
from `apps/web/app/lib/metrics/ssr.ts`. Currently wrapped: index/app-root,
app-tenant, admin-root, dashboard, login, logout, forgot-password,
new-account, invitation.

### Business

| Metric | Type | Labels | Question |
|---|---|---|---|
| `active_tenants` | gauge | — | Tenant growth. |
| `active_users_24h` | gauge | — | DAU. |
| `ai_tokens_consumed_total` | counter | provider, model, tenant_bucket | AI spend allocation. |
| `stripe_webhook_events_total` | counter | event_type, status | Stripe traffic + spoof attempts. |
| `mail_sent_total` | counter | provider, status | Email reliability. |
| `webhook_delivery_attempts_total` | counter | status | Outbound webhook health (success / retry / dlq / rejected). |
| `csrf_token_rotation_total` | counter | — | Auth churn. |
| `subscription_active` | gauge | plan_id | Plan mix. |

## Cardinality rules

> Prometheus dies before you hit a million series. Pre-empt it.

1. **Never use raw user / request input as a label value.** Always use a
   pattern (route pattern, error name, status code) or a bucket.
2. **Cap unknown sets.** `ai_tokens_consumed_total.tenant_bucket` is a
   process-local LRU of the first 100 tenants seen + `"other"` overflow
   (see `apps/web/app/lib/metrics/ai-cost.ts`). When you add a new
   metric with a similar shape, apply the same pattern.
3. **No PII labels.** Tenant IDs are opaque slugs, never email / name.
4. **No env-var spill.** Default prom-client process metrics never leak
   `process.env`; verify the same for custom metrics in code review.

## Adding a new metric

1. Declare it in `packages/metrics/src/defaults.ts`. Pick the smallest
   useful label set.
2. Write a unit test in `packages/metrics/src/defaults.test.ts` that
   touches the metric and asserts the rendered text contains the name.
3. Add an instrumentation call at the business-logic site. Wrap the
   call in `try { … } catch { /* never crash */ }`.
4. Add a Grafana panel under `infra/grafana/dashboards/` and commit.
5. Update this catalog.

## Deploy on Hetzner staging

The Prometheus and Grafana stacks are intended for the **same Hetzner
observability box that hosts Sentry** (`/opt/observability/` directory
convention). The compose files in `infra/prometheus/` and
`infra/grafana/` can be deployed as Coolify "Docker Compose" services
or as plain `docker compose up -d` units.

Per-step instructions live in
[`infra/prometheus/README.md`](../../infra/prometheus/README.md).

### Coolify vs `/opt/observability/`

Operator decision — both work. Coolify is easier to wire to Traefik for
the `prometheus.boiler.saas-ctrl.com` and `grafana.boiler.saas-ctrl.com`
subdomains. Plain `docker compose` is lighter but requires a manual
reverse-proxy entry. **Default recommendation**: Coolify-managed
Service, same as Sentry.

## Dashboard-update workflow

1. Edit panels in the Grafana UI.
2. Top-right menu → **Share → Export → Save to file** (uncheck "Export
   for sharing externally" so UIDs stay stable).
3. Drop the JSON into `infra/grafana/dashboards/<name>.json`.
4. Commit. Provisioning reloads automatically every 30s.

## Smoke check

```sh
# /status — should be 200 OK
curl -fsS https://api.boiler.saas-ctrl.com/status | jq

# /metrics without auth — should be 401
curl -i https://api.boiler.saas-ctrl.com/metrics | head -1

# /metrics with auth — should be 200 + prom text
curl -fsS -H "Authorization: Bearer $METRICS_TOKEN" \
  https://api.boiler.saas-ctrl.com/metrics | head -20
```

`scripts/observability/smoke-metrics.sh` automates this against any
base host (defaults to staging).

## Rollback

- **Disable everything**: `METRICS_ENABLED=false`. All three apps stop
  exposing `/metrics` (404) within one restart. Counters keep working
  in-process; only the surface goes away.
- **Cardinality emergency** (Prometheus OOM): the most likely culprit
  is `ai_tokens_consumed_total`. The safety valve is already there
  (tenant_bucket caps at 101). If a NEW metric explodes, revert the
  commit that added it and reload Prometheus.
- **Dashboard rollback**: `git revert` the JSON file and the file
  provider replaces it within 30 s.