Scrapes the HiveGrid runtime metrics every 15s. The web core/admin process and the separately deployed portal process each have their own process-local registry and therefore their own scrape target.

Files

prometheus.yml — scrape jobs (api / worker / web-core / web-portal)
docker-compose.yml — Coolify-friendly compose stack
secrets/metrics_token — bearer token (chmod 600, do NOT commit)

First-time deploy on Hetzner staging

# 1. Create the secret file with the same value as METRICS_TOKEN in Coolify
mkdir -p secrets
openssl rand -hex 32 > secrets/metrics_token
chmod 600 secrets/metrics_token

# 2. Create the shared docker network (also used by Grafana)
docker network create hivegrid-observability

# 3. Start the stack
docker compose up -d

Prometheus is deliberately reachable only as prometheus:9090 on the hivegrid-observability Docker network. It has no host-port binding and no lifecycle HTTP endpoint. Use Grafana for dashboards; temporary operator access must use an authenticated tunnel rather than public ingress.

Verifying scrape targets

curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
  https://api.hivegrid.de/metrics | head -20

curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
  https://app.hivegrid.de/api/metrics | head -20

curl -s -H "Authorization: Bearer $(cat secrets/metrics_token)" \
  https://portal.hivegrid.de/api/metrics | head -20

Through an authenticated operator tunnel, the Prometheus /targets view should list all four jobs as UP. Do not also scrape admin.hivegrid.de: admin and core share the same process-local registry, so doing so would double-count.

Cardinality budget

ai_tokens_consumed_total — capped at ~100 tenant buckets + 1 other
subscription_active — capped by the plan-id table (single digits today)
All other metrics have bounded label sets (HTTP routes, queue names)

If a panel suddenly shows millions of series, the most likely culprit is a new histogram with an unbounded label — search the codebase for labels({ and verify.