# Prometheus — HiveGrid Scrapes the HiveGrid runtime metrics every 15s. The web core/admin process and the separately deployed portal process each have their own process-local registry and therefore their own scrape target. ## Files - `prometheus.yml` — scrape jobs (api / worker / web-core / web-portal) - `docker-compose.yml` — Coolify-friendly compose stack - `secrets/metrics_token` — bearer token (chmod 600, do NOT commit) ## First-time deploy on Hetzner staging ```sh # 1. Create the secret file with the same value as METRICS_TOKEN in Coolify mkdir -p secrets openssl rand -hex 32 > secrets/metrics_token chmod 600 secrets/metrics_token # 2. Create the shared docker network (also used by Grafana) docker network create hivegrid-observability # 3. Start the stack docker compose up -d ``` Prometheus is deliberately reachable only as `prometheus:9090` on the `hivegrid-observability` Docker network. It has no host-port binding and no lifecycle HTTP endpoint. Use Grafana for dashboards; temporary operator access must use an authenticated tunnel rather than public ingress. ## Verifying scrape targets ```sh curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \ https://api.hivegrid.de/metrics | head -20 curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \ https://app.hivegrid.de/api/metrics | head -20 curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \ https://portal.hivegrid.de/api/metrics | head -20 ``` Through an authenticated operator tunnel, the Prometheus `/targets` view should list all four jobs as **UP**. Do not also scrape `admin.hivegrid.de`: admin and core share the same process-local registry, so doing so would double-count. ## Cardinality budget - `ai_tokens_consumed_total` — capped at ~100 tenant buckets + 1 `other` - `subscription_active` — capped by the plan-id table (single digits today) - All other metrics have bounded label sets (HTTP routes, queue names) If a panel suddenly shows millions of series, the most likely culprit is a new histogram with an unbounded label — search the codebase for `labels({` and verify.