Track HiveGrid Cloud planning reference
This commit is contained in:
1 parent
b3d5ea6764
commit
e341c5938a
5784 files changed
+1104929
-3
No files matched your search
@@ -0,0 +1,150 @@
|
||||
# Metrics + Grafana
|
||||
|
||||
Prometheus + Grafana stack for the boilerplate. Scrapes `/metrics` from
|
||||
the three application processes and renders five dashboards.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
+----------+ scrape +-------------+ query +---------+
|
||||
| api | <--------------------- | | <------------------ | |
|
||||
| worker | <--------------------- | Prometheus | | Grafana |
|
||||
| web | <--------------------- | | | |
|
||||
+----------+ +-------------+ +---------+
|
||||
^ |
|
||||
| v
|
||||
/status tsdb (30d)
|
||||
/metrics
|
||||
```
|
||||
|
||||
- All three apps are instrumented via [`@toolctrl/metrics`](../../packages/metrics).
|
||||
- `/status` is public, JSON, 200 / 503 — suitable for Uptime Kuma.
|
||||
- `/metrics` is bearer-token protected, returns Prom text-format.
|
||||
- Both endpoints are gated by `METRICS_ENABLED=true`; default off in dev.
|
||||
|
||||
## Metric catalog
|
||||
|
||||
### Infrastructure (all services)
|
||||
|
||||
| Metric | Type | Labels | Question it answers |
|
||||
|---|---|---|---|
|
||||
| `http_requests_total` | counter | method, route, status | Request volume + status distribution. |
|
||||
| `http_request_duration_seconds_bucket` | histogram | method, route, status | API latency percentiles. |
|
||||
| `prisma_query_duration_seconds_bucket` | histogram | model, action | DB hotspot detection. |
|
||||
| `redis_command_duration_seconds_bucket` | histogram | command | Redis cache hit / slow-cmd diagnosis. |
|
||||
| `process_*` | counter / gauge | — | Node.js process health (default `prom-client`). |
|
||||
|
||||
### Worker queue
|
||||
|
||||
| Metric | Type | Labels | Question |
|
||||
|---|---|---|---|
|
||||
| `queue_jobs_active` | gauge | queue | Concurrency in-flight. |
|
||||
| `queue_jobs_waiting` | gauge | queue | Backlog depth. |
|
||||
| `queue_jobs_completed_total` | counter | queue, job_name | Throughput. |
|
||||
| `queue_jobs_failed_total` | counter | queue, job_name, error_type | Failure-type breakdown. |
|
||||
| `queue_job_duration_seconds_bucket` | histogram | queue, job_name | Job-duration heatmap. |
|
||||
|
||||
### Web SSR
|
||||
|
||||
| Metric | Type | Labels | Question |
|
||||
|---|---|---|---|
|
||||
| `web_ssr_loader_duration_seconds_bucket` | histogram | route, status | Loader-level latency for the top routes. |
|
||||
|
||||
Status enum: `ok | error | redirect`. Apply via `withMetrics(routeId, loader)`
|
||||
from `apps/web/app/lib/metrics/ssr.ts`. Currently wrapped: index/app-root,
|
||||
app-tenant, admin-root, dashboard, login, logout, forgot-password,
|
||||
new-account, invitation.
|
||||
|
||||
### Business
|
||||
|
||||
| Metric | Type | Labels | Question |
|
||||
|---|---|---|---|
|
||||
| `active_tenants` | gauge | — | Tenant growth. |
|
||||
| `active_users_24h` | gauge | — | DAU. |
|
||||
| `ai_tokens_consumed_total` | counter | provider, model, tenant_bucket | AI spend allocation. |
|
||||
| `stripe_webhook_events_total` | counter | event_type, status | Stripe traffic + spoof attempts. |
|
||||
| `mail_sent_total` | counter | provider, status | Email reliability. |
|
||||
| `webhook_delivery_attempts_total` | counter | status | Outbound webhook health (success / retry / dlq / rejected). |
|
||||
| `csrf_token_rotation_total` | counter | — | Auth churn. |
|
||||
| `subscription_active` | gauge | plan_id | Plan mix. |
|
||||
|
||||
## Cardinality rules
|
||||
|
||||
> Prometheus dies before you hit a million series. Pre-empt it.
|
||||
|
||||
1. **Never use raw user / request input as a label value.** Always use a
|
||||
pattern (route pattern, error name, status code) or a bucket.
|
||||
2. **Cap unknown sets.** `ai_tokens_consumed_total.tenant_bucket` is a
|
||||
process-local LRU of the first 100 tenants seen + `"other"` overflow
|
||||
(see `apps/web/app/lib/metrics/ai-cost.ts`). When you add a new
|
||||
metric with a similar shape, apply the same pattern.
|
||||
3. **No PII labels.** Tenant IDs are opaque slugs, never email / name.
|
||||
4. **No env-var spill.** Default prom-client process metrics never leak
|
||||
`process.env`; verify the same for custom metrics in code review.
|
||||
|
||||
## Adding a new metric
|
||||
|
||||
1. Declare it in `packages/metrics/src/defaults.ts`. Pick the smallest
|
||||
useful label set.
|
||||
2. Write a unit test in `packages/metrics/src/defaults.test.ts` that
|
||||
touches the metric and asserts the rendered text contains the name.
|
||||
3. Add an instrumentation call at the business-logic site. Wrap the
|
||||
call in `try { … } catch { /* never crash */ }`.
|
||||
4. Add a Grafana panel under `infra/grafana/dashboards/` and commit.
|
||||
5. Update this catalog.
|
||||
|
||||
## Deploy on Hetzner staging
|
||||
|
||||
The Prometheus and Grafana stacks are intended for the **same Hetzner
|
||||
observability box that hosts Sentry** (`/opt/observability/` directory
|
||||
convention). The compose files in `infra/prometheus/` and
|
||||
`infra/grafana/` can be deployed as Coolify "Docker Compose" services
|
||||
or as plain `docker compose up -d` units.
|
||||
|
||||
Per-step instructions live in
|
||||
[`infra/prometheus/README.md`](../../infra/prometheus/README.md).
|
||||
|
||||
### Coolify vs `/opt/observability/`
|
||||
|
||||
Operator decision — both work. Coolify is easier to wire to Traefik for
|
||||
the `prometheus.boiler.saas-ctrl.com` and `grafana.boiler.saas-ctrl.com`
|
||||
subdomains. Plain `docker compose` is lighter but requires a manual
|
||||
reverse-proxy entry. **Default recommendation**: Coolify-managed
|
||||
Service, same as Sentry.
|
||||
|
||||
## Dashboard-update workflow
|
||||
|
||||
1. Edit panels in the Grafana UI.
|
||||
2. Top-right menu → **Share → Export → Save to file** (uncheck "Export
|
||||
for sharing externally" so UIDs stay stable).
|
||||
3. Drop the JSON into `infra/grafana/dashboards/<name>.json`.
|
||||
4. Commit. Provisioning reloads automatically every 30s.
|
||||
|
||||
## Smoke check
|
||||
|
||||
```sh
|
||||
# /status — should be 200 OK
|
||||
curl -fsS https://api.boiler.saas-ctrl.com/status | jq
|
||||
|
||||
# /metrics without auth — should be 401
|
||||
curl -i https://api.boiler.saas-ctrl.com/metrics | head -1
|
||||
|
||||
# /metrics with auth — should be 200 + prom text
|
||||
curl -fsS -H "Authorization: Bearer $METRICS_TOKEN" \
|
||||
https://api.boiler.saas-ctrl.com/metrics | head -20
|
||||
```
|
||||
|
||||
`scripts/observability/smoke-metrics.sh` automates this against any
|
||||
base host (defaults to staging).
|
||||
|
||||
## Rollback
|
||||
|
||||
- **Disable everything**: `METRICS_ENABLED=false`. All three apps stop
|
||||
exposing `/metrics` (404) within one restart. Counters keep working
|
||||
in-process; only the surface goes away.
|
||||
- **Cardinality emergency** (Prometheus OOM): the most likely culprit
|
||||
is `ai_tokens_consumed_total`. The safety valve is already there
|
||||
(tenant_bucket caps at 101). If a NEW metric explodes, revert the
|
||||
commit that added it and reload Prometheus.
|
||||
- **Dashboard rollback**: `git revert` the JSON file and the file
|
||||
provider replaces it within 30 s.
|
||||
Reference in new issue
Block a user