Files
HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/storage-traffic-monitoring.md
T

7.5 KiB

Storage traffic monitoring

This runbook defines the launch-safe BILL-25 signal for storage downloads. It uses HiveGrid's existing Prometheus and Grafana stack and creates no billing, CDN, proxy, bucket, or server infrastructure.

What the metrics mean

Metric Meaning Never means
storage_authorized_downloads_total{surface,delivery,size_known} HiveGrid completed its authorization boundary and issued the delivery response. The client received or finished the download.
storage_authorized_download_object_bytes_total{surface,delivery} Sum of trustworthy whole-object sizes associated with those authorizations. Actual bytes transferred or billed by Hetzner.

Labels are closed application enums. They never contain tenant IDs, user IDs, file IDs, filenames, object keys, bucket names, URLs, request values, or other customer data. Metric writes are fail-open: an observation failure cannot block an otherwise valid download.

Current coverage:

  • surface="files", delivery="signed_redirect": original files report the canonical File.sizeBytes; generated previews and thumbnails report size_known="unknown" because their sizes are not persisted.
  • surface="portal_document", delivery="signed_redirect": the authorization count is covered, but size is deliberately unknown. The portal's security-definer resolver does not expose File.sizeBytes; BILL-25 does not widen that database contract or add a migration.

API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy responses are not part of this initial counter coverage. Do not infer zero traffic for an uncovered surface.

Why this is not actual egress

HiveGrid redirects authorized clients to short-lived presigned URLs in a private Hetzner bucket. After the redirect, the object transfer no longer traverses the HiveGrid process. A URL can remain unused, be retried or reused before expiry, or serve one or more byte ranges. Therefore the authorization estimate can be higher or lower than actual traffic.

Hetzner currently does not support S3 server-access logging or S3 request metrics and does not provide an API for account storage/traffic utilization. The Hetzner Console and invoice remain the authoritative account-wide cost source:

Hetzner combines storage and traffic across all buckets and projects in the account. Ingress, S3 operations, and internal traffic within eu-central are free; externally delivered object data consumes the account-wide egress quota. The included account quota is a platform buffer and is not promised to an individual workspace.

Hetzner bills every stored object as at least 64 kB, even when the payload is smaller. HiveGrid's package calculation uses logical GB/TB and therefore does not price a hypothetical worst case made entirely of tiny objects. During the pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte total as a separate risk signal. A growing gap means object-size distribution, unfinished multipart uploads, versions, or non-HiveGrid buckets need review; it is not evidence of customer egress. Do not silently convert that difference into a workspace charge.

Grafana views

Use these queries for the web service. A process restart does not invalidate Prometheus increase across a correctly scraped counter reset.

sum by (runtime_surface, surface, delivery, size_known) (
  increase(storage_authorized_downloads_total{service="web"}[30d])
)
sum by (runtime_surface, surface, delivery) (
  increase(storage_authorized_download_object_bytes_total{service="web"}[30d])
)

Use a shorter window to spot bursts:

sum by (runtime_surface, surface, delivery) (
  rate(storage_authorized_downloads_total{service="web"}[15m])
)

Unknown-size authorizations are a coverage indicator, not zero-byte downloads. Keep them visible next to the known-byte panel.

Pilot reconciliation and thresholds

During the pilot, an operator performs the following comparison daily for the first four weeks and weekly afterwards:

  1. Record the redacted, account-wide storage and egress totals from the Hetzner Console. Do not attach screenshots containing account or customer data.
  2. Read the 30-day Prometheus authorization totals above.
  3. Compare known authorized object bytes and the count of unknown-size authorizations with their own trailing 7- and 30-day baselines. The global counter includes base-plan, extra-seat and add-on storage and therefore must not be divided by sold add-on capacity.
  4. Compare logical stored bytes with provider-billed storage so Hetzner's 64-kB object minimum and other account-wide storage overhead become visible.
  5. Record the difference and whether uncovered surfaces, unused links, retries, ranges, or cross-project account traffic plausibly explain it.

Review thresholds use separately meaningful signals:

Signal Operator action
Authorized bytes and counts remain within 2x of the trailing comparable period Normal pilot observation.
A surface grows to 2x or more without a matching active-user/storage increase Warning: inspect burst shape, unknown-size counts and provider total.
Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget Elevated review: reconcile all account buckets and identify unexpectedly hot flows.
Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture.

These thresholds do not throttle a workspace and never create an overage invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant label or durable tenant traffic ledger in this slice, so workspace attribution remains a support approximation and must not be presented as measured usage.

Do not silently enable Grafana email notifications. Dashboard review is the launch default; notification routing, recipients, grouping, and rate limits require a separate operator-approved change so storage telemetry cannot recreate an alert-mail flood.

Runtime verification

Before relying on the panels in staging or production:

  1. Confirm METRICS_ENABLED=true and both bearer-protected web targets hivegrid-web-core and hivegrid-web-portal are UP in Prometheus. Do not scrape the shared core/admin registry twice.
  2. Trigger one authorized original-file download and one preview or thumbnail.
  3. Confirm the original increments size_known="known" and its object-byte counter; confirm the generated variant increments only the unknown count.
  4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests do not increment either counter.
  5. Confirm no metric label or help text contains a tenant, file, key, bucket, filename, or signed URL.

Rollback removes only these observations. Authorization, tenant scope, malware scan gates, and private-bucket signed delivery remain unchanged.