151 lines
6.4 KiB
Markdown
151 lines
6.4 KiB
Markdown
# Metrics + Grafana
|
|
|
|
Prometheus + Grafana stack for the boilerplate. Scrapes `/metrics` from
|
|
the three application processes and renders five dashboards.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+----------+ scrape +-------------+ query +---------+
|
|
| api | <--------------------- | | <------------------ | |
|
|
| worker | <--------------------- | Prometheus | | Grafana |
|
|
| web | <--------------------- | | | |
|
|
+----------+ +-------------+ +---------+
|
|
^ |
|
|
| v
|
|
/status tsdb (30d)
|
|
/metrics
|
|
```
|
|
|
|
- All three apps are instrumented via [`@toolctrl/metrics`](../../packages/metrics).
|
|
- `/status` is public, JSON, 200 / 503 — suitable for Uptime Kuma.
|
|
- `/metrics` is bearer-token protected, returns Prom text-format.
|
|
- Both endpoints are gated by `METRICS_ENABLED=true`; default off in dev.
|
|
|
|
## Metric catalog
|
|
|
|
### Infrastructure (all services)
|
|
|
|
| Metric | Type | Labels | Question it answers |
|
|
|---|---|---|---|
|
|
| `http_requests_total` | counter | method, route, status | Request volume + status distribution. |
|
|
| `http_request_duration_seconds_bucket` | histogram | method, route, status | API latency percentiles. |
|
|
| `prisma_query_duration_seconds_bucket` | histogram | model, action | DB hotspot detection. |
|
|
| `redis_command_duration_seconds_bucket` | histogram | command | Redis cache hit / slow-cmd diagnosis. |
|
|
| `process_*` | counter / gauge | — | Node.js process health (default `prom-client`). |
|
|
|
|
### Worker queue
|
|
|
|
| Metric | Type | Labels | Question |
|
|
|---|---|---|---|
|
|
| `queue_jobs_active` | gauge | queue | Concurrency in-flight. |
|
|
| `queue_jobs_waiting` | gauge | queue | Backlog depth. |
|
|
| `queue_jobs_completed_total` | counter | queue, job_name | Throughput. |
|
|
| `queue_jobs_failed_total` | counter | queue, job_name, error_type | Failure-type breakdown. |
|
|
| `queue_job_duration_seconds_bucket` | histogram | queue, job_name | Job-duration heatmap. |
|
|
|
|
### Web SSR
|
|
|
|
| Metric | Type | Labels | Question |
|
|
|---|---|---|---|
|
|
| `web_ssr_loader_duration_seconds_bucket` | histogram | route, status | Loader-level latency for the top routes. |
|
|
|
|
Status enum: `ok | error | redirect`. Apply via `withMetrics(routeId, loader)`
|
|
from `apps/web/app/lib/metrics/ssr.ts`. Currently wrapped: index/app-root,
|
|
app-tenant, admin-root, dashboard, login, logout, forgot-password,
|
|
new-account, invitation.
|
|
|
|
### Business
|
|
|
|
| Metric | Type | Labels | Question |
|
|
|---|---|---|---|
|
|
| `active_tenants` | gauge | — | Tenant growth. |
|
|
| `active_users_24h` | gauge | — | DAU. |
|
|
| `ai_tokens_consumed_total` | counter | provider, model, tenant_bucket | AI spend allocation. |
|
|
| `stripe_webhook_events_total` | counter | event_type, status | Stripe traffic + spoof attempts. |
|
|
| `mail_sent_total` | counter | provider, status | Email reliability. |
|
|
| `webhook_delivery_attempts_total` | counter | status | Outbound webhook health (success / retry / dlq / rejected). |
|
|
| `csrf_token_rotation_total` | counter | — | Auth churn. |
|
|
| `subscription_active` | gauge | plan_id | Plan mix. |
|
|
|
|
## Cardinality rules
|
|
|
|
> Prometheus dies before you hit a million series. Pre-empt it.
|
|
|
|
1. **Never use raw user / request input as a label value.** Always use a
|
|
pattern (route pattern, error name, status code) or a bucket.
|
|
2. **Cap unknown sets.** `ai_tokens_consumed_total.tenant_bucket` is a
|
|
process-local LRU of the first 100 tenants seen + `"other"` overflow
|
|
(see `apps/web/app/lib/metrics/ai-cost.ts`). When you add a new
|
|
metric with a similar shape, apply the same pattern.
|
|
3. **No PII labels.** Tenant IDs are opaque slugs, never email / name.
|
|
4. **No env-var spill.** Default prom-client process metrics never leak
|
|
`process.env`; verify the same for custom metrics in code review.
|
|
|
|
## Adding a new metric
|
|
|
|
1. Declare it in `packages/metrics/src/defaults.ts`. Pick the smallest
|
|
useful label set.
|
|
2. Write a unit test in `packages/metrics/src/defaults.test.ts` that
|
|
touches the metric and asserts the rendered text contains the name.
|
|
3. Add an instrumentation call at the business-logic site. Wrap the
|
|
call in `try { … } catch { /* never crash */ }`.
|
|
4. Add a Grafana panel under `infra/grafana/dashboards/` and commit.
|
|
5. Update this catalog.
|
|
|
|
## Deploy on Hetzner staging
|
|
|
|
The Prometheus and Grafana stacks are intended for the **same Hetzner
|
|
observability box that hosts Sentry** (`/opt/observability/` directory
|
|
convention). The compose files in `infra/prometheus/` and
|
|
`infra/grafana/` can be deployed as Coolify "Docker Compose" services
|
|
or as plain `docker compose up -d` units.
|
|
|
|
Per-step instructions live in
|
|
[`infra/prometheus/README.md`](../../infra/prometheus/README.md).
|
|
|
|
### Coolify vs `/opt/observability/`
|
|
|
|
Operator decision — both work. Coolify is easier to wire to Traefik for
|
|
the `prometheus.boiler.saas-ctrl.com` and `grafana.boiler.saas-ctrl.com`
|
|
subdomains. Plain `docker compose` is lighter but requires a manual
|
|
reverse-proxy entry. **Default recommendation**: Coolify-managed
|
|
Service, same as Sentry.
|
|
|
|
## Dashboard-update workflow
|
|
|
|
1. Edit panels in the Grafana UI.
|
|
2. Top-right menu → **Share → Export → Save to file** (uncheck "Export
|
|
for sharing externally" so UIDs stay stable).
|
|
3. Drop the JSON into `infra/grafana/dashboards/<name>.json`.
|
|
4. Commit. Provisioning reloads automatically every 30s.
|
|
|
|
## Smoke check
|
|
|
|
```sh
|
|
# /status — should be 200 OK
|
|
curl -fsS https://api.boiler.saas-ctrl.com/status | jq
|
|
|
|
# /metrics without auth — should be 401
|
|
curl -i https://api.boiler.saas-ctrl.com/metrics | head -1
|
|
|
|
# /metrics with auth — should be 200 + prom text
|
|
curl -fsS -H "Authorization: Bearer $METRICS_TOKEN" \
|
|
https://api.boiler.saas-ctrl.com/metrics | head -20
|
|
```
|
|
|
|
`scripts/observability/smoke-metrics.sh` automates this against any
|
|
base host (defaults to staging).
|
|
|
|
## Rollback
|
|
|
|
- **Disable everything**: `METRICS_ENABLED=false`. All three apps stop
|
|
exposing `/metrics` (404) within one restart. Counters keep working
|
|
in-process; only the surface goes away.
|
|
- **Cardinality emergency** (Prometheus OOM): the most likely culprit
|
|
is `ai_tokens_consumed_total`. The safety valve is already there
|
|
(tenant_bucket caps at 101). If a NEW metric explodes, revert the
|
|
commit that added it and reload Prometheus.
|
|
- **Dashboard rollback**: `git revert` the JSON file and the file
|
|
provider replaces it within 30 s.
|