# Storage traffic monitoring This runbook defines the launch-safe BILL-25 signal for storage downloads. It uses HiveGrid's existing Prometheus and Grafana stack and creates no billing, CDN, proxy, bucket, or server infrastructure. ## What the metrics mean | Metric | Meaning | Never means | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------- | | `storage_authorized_downloads_total{surface,delivery,size_known}` | HiveGrid completed its authorization boundary and issued the delivery response. | The client received or finished the download. | | `storage_authorized_download_object_bytes_total{surface,delivery}` | Sum of trustworthy whole-object sizes associated with those authorizations. | Actual bytes transferred or billed by Hetzner. | Labels are closed application enums. They never contain tenant IDs, user IDs, file IDs, filenames, object keys, bucket names, URLs, request values, or other customer data. Metric writes are fail-open: an observation failure cannot block an otherwise valid download. Current coverage: - `surface="files"`, `delivery="signed_redirect"`: original files report the canonical `File.sizeBytes`; generated previews and thumbnails report `size_known="unknown"` because their sizes are not persisted. - `surface="portal_document"`, `delivery="signed_redirect"`: the authorization count is covered, but size is deliberately unknown. The portal's security-definer resolver does not expose `File.sizeBytes`; BILL-25 does not widen that database contract or add a migration. API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy responses are not part of this initial counter coverage. Do not infer zero traffic for an uncovered surface. ## Why this is not actual egress HiveGrid redirects authorized clients to short-lived presigned URLs in a private Hetzner bucket. After the redirect, the object transfer no longer traverses the HiveGrid process. A URL can remain unused, be retried or reused before expiry, or serve one or more byte ranges. Therefore the authorization estimate can be higher or lower than actual traffic. Hetzner currently does not support S3 server-access logging or S3 request metrics and does not provide an API for account storage/traffic utilization. The Hetzner Console and invoice remain the authoritative account-wide cost source: - - - Hetzner combines storage and traffic across all buckets and projects in the account. Ingress, S3 operations, and internal traffic within `eu-central` are free; externally delivered object data consumes the account-wide egress quota. The included account quota is a platform buffer and is not promised to an individual workspace. Hetzner bills every stored object as at least 64 kB, even when the payload is smaller. HiveGrid's package calculation uses logical GB/TB and therefore does not price a hypothetical worst case made entirely of tiny objects. During the pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte total as a separate risk signal. A growing gap means object-size distribution, unfinished multipart uploads, versions, or non-HiveGrid buckets need review; it is not evidence of customer egress. Do not silently convert that difference into a workspace charge. ## Grafana views Use these queries for the web service. A process restart does not invalidate Prometheus `increase` across a correctly scraped counter reset. ```promql sum by (runtime_surface, surface, delivery, size_known) ( increase(storage_authorized_downloads_total{service="web"}[30d]) ) ``` ```promql sum by (runtime_surface, surface, delivery) ( increase(storage_authorized_download_object_bytes_total{service="web"}[30d]) ) ``` Use a shorter window to spot bursts: ```promql sum by (runtime_surface, surface, delivery) ( rate(storage_authorized_downloads_total{service="web"}[15m]) ) ``` Unknown-size authorizations are a coverage indicator, not zero-byte downloads. Keep them visible next to the known-byte panel. ## Pilot reconciliation and thresholds During the pilot, an operator performs the following comparison daily for the first four weeks and weekly afterwards: 1. Record the redacted, account-wide storage and egress totals from the Hetzner Console. Do not attach screenshots containing account or customer data. 2. Read the 30-day Prometheus authorization totals above. 3. Compare known authorized object bytes and the count of unknown-size authorizations with their own trailing 7- and 30-day baselines. The global counter includes base-plan, extra-seat and add-on storage and therefore must not be divided by sold add-on capacity. 4. Compare logical stored bytes with provider-billed storage so Hetzner's 64-kB object minimum and other account-wide storage overhead become visible. 5. Record the difference and whether uncovered surfaces, unused links, retries, ranges, or cross-project account traffic plausibly explain it. Review thresholds use separately meaningful signals: | Signal | Operator action | | ------ | --------------- | | Authorized bytes and counts remain within `2x` of the trailing comparable period | Normal pilot observation. | | A surface grows to `2x` or more without a matching active-user/storage increase | Warning: inspect burst shape, unknown-size counts and provider total. | | Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget | Elevated review: reconcile all account buckets and identify unexpectedly hot flows. | | Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget | Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture. | These thresholds do not throttle a workspace and never create an overage invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant label or durable tenant traffic ledger in this slice, so workspace attribution remains a support approximation and must not be presented as measured usage. Do not silently enable Grafana email notifications. Dashboard review is the launch default; notification routing, recipients, grouping, and rate limits require a separate operator-approved change so storage telemetry cannot recreate an alert-mail flood. ## Runtime verification Before relying on the panels in staging or production: 1. Confirm `METRICS_ENABLED=true` and both bearer-protected web targets `hivegrid-web-core` and `hivegrid-web-portal` are `UP` in Prometheus. Do not scrape the shared core/admin registry twice. 2. Trigger one authorized original-file download and one preview or thumbnail. 3. Confirm the original increments `size_known="known"` and its object-byte counter; confirm the generated variant increments only the `unknown` count. 4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests do not increment either counter. 5. Confirm no metric label or help text contains a tenant, file, key, bucket, filename, or signed URL. Rollback removes only these observations. Authorization, tenant scope, malware scan gates, and private-bucket signed delivery remain unchanged.