59 lines
2.1 KiB
Markdown
59 lines
2.1 KiB
Markdown
# Prometheus — HiveGrid
|
|
|
|
Scrapes the HiveGrid runtime metrics every 15s. The web core/admin process and
|
|
the separately deployed portal process each have their own process-local
|
|
registry and therefore their own scrape target.
|
|
|
|
## Files
|
|
|
|
- `prometheus.yml` — scrape jobs (api / worker / web-core / web-portal)
|
|
- `docker-compose.yml` — Coolify-friendly compose stack
|
|
- `secrets/metrics_token` — bearer token (chmod 600, do NOT commit)
|
|
|
|
## First-time deploy on Hetzner staging
|
|
|
|
```sh
|
|
# 1. Create the secret file with the same value as METRICS_TOKEN in Coolify
|
|
mkdir -p secrets
|
|
openssl rand -hex 32 > secrets/metrics_token
|
|
chmod 600 secrets/metrics_token
|
|
|
|
# 2. Create the shared docker network (also used by Grafana)
|
|
docker network create hivegrid-observability
|
|
|
|
# 3. Start the stack
|
|
docker compose up -d
|
|
```
|
|
|
|
Prometheus is deliberately reachable only as `prometheus:9090` on the
|
|
`hivegrid-observability` Docker network. It has no host-port binding and no
|
|
lifecycle HTTP endpoint. Use Grafana for dashboards; temporary operator access
|
|
must use an authenticated tunnel rather than public ingress.
|
|
|
|
## Verifying scrape targets
|
|
|
|
```sh
|
|
curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
|
|
https://api.hivegrid.de/metrics | head -20
|
|
|
|
curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
|
|
https://app.hivegrid.de/api/metrics | head -20
|
|
|
|
curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
|
|
https://portal.hivegrid.de/api/metrics | head -20
|
|
```
|
|
|
|
Through an authenticated operator tunnel, the Prometheus `/targets` view should
|
|
list all four jobs as **UP**. Do not also scrape `admin.hivegrid.de`: admin and
|
|
core share the same process-local registry, so doing so would double-count.
|
|
|
|
## Cardinality budget
|
|
|
|
- `ai_tokens_consumed_total` — capped at ~100 tenant buckets + 1 `other`
|
|
- `subscription_active` — capped by the plan-id table (single digits today)
|
|
- All other metrics have bounded label sets (HTTP routes, queue names)
|
|
|
|
If a panel suddenly shows millions of series, the most likely culprit
|
|
is a new histogram with an unbounded label — search the codebase for
|
|
`labels({` and verify.
|