Files

146 lines
7.5 KiB
Markdown

# Storage traffic monitoring
This runbook defines the launch-safe BILL-25 signal for storage downloads. It
uses HiveGrid's existing Prometheus and Grafana stack and creates no billing,
CDN, proxy, bucket, or server infrastructure.
## What the metrics mean
| Metric | Meaning | Never means |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------- |
| `storage_authorized_downloads_total{surface,delivery,size_known}` | HiveGrid completed its authorization boundary and issued the delivery response. | The client received or finished the download. |
| `storage_authorized_download_object_bytes_total{surface,delivery}` | Sum of trustworthy whole-object sizes associated with those authorizations. | Actual bytes transferred or billed by Hetzner. |
Labels are closed application enums. They never contain tenant IDs, user IDs,
file IDs, filenames, object keys, bucket names, URLs, request values, or other
customer data. Metric writes are fail-open: an observation failure cannot block
an otherwise valid download.
Current coverage:
- `surface="files"`, `delivery="signed_redirect"`: original files report the
canonical `File.sizeBytes`; generated previews and thumbnails report
`size_known="unknown"` because their sizes are not persisted.
- `surface="portal_document"`, `delivery="signed_redirect"`: the authorization
count is covered, but size is deliberately unknown. The portal's
security-definer resolver does not expose `File.sizeBytes`; BILL-25 does not
widen that database contract or add a migration.
API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy
responses are not part of this initial counter coverage. Do not infer zero
traffic for an uncovered surface.
## Why this is not actual egress
HiveGrid redirects authorized clients to short-lived presigned URLs in a private
Hetzner bucket. After the redirect, the object transfer no longer traverses the
HiveGrid process. A URL can remain unused, be retried or reused before expiry,
or serve one or more byte ranges. Therefore the authorization estimate can be
higher or lower than actual traffic.
Hetzner currently does not support S3 server-access logging or S3 request
metrics and does not provide an API for account storage/traffic utilization.
The Hetzner Console and invoice remain the authoritative account-wide cost
source:
- <https://docs.hetzner.com/storage/object-storage/supported-actions/>
- <https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/>
- <https://docs.hetzner.com/storage/object-storage/overview/>
Hetzner combines storage and traffic across all buckets and projects in the
account. Ingress, S3 operations, and internal traffic within `eu-central` are
free; externally delivered object data consumes the account-wide egress quota.
The included account quota is a platform buffer and is not promised to an
individual workspace.
Hetzner bills every stored object as at least 64 kB, even when the payload is
smaller. HiveGrid's package calculation uses logical GB/TB and therefore does
not price a hypothetical worst case made entirely of tiny objects. During the
pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte
total as a separate risk signal. A growing gap means object-size distribution,
unfinished multipart uploads, versions, or non-HiveGrid buckets need review;
it is not evidence of customer egress. Do not silently convert that difference
into a workspace charge.
## Grafana views
Use these queries for the web service. A process restart does not invalidate
Prometheus `increase` across a correctly scraped counter reset.
```promql
sum by (runtime_surface, surface, delivery, size_known) (
increase(storage_authorized_downloads_total{service="web"}[30d])
)
```
```promql
sum by (runtime_surface, surface, delivery) (
increase(storage_authorized_download_object_bytes_total{service="web"}[30d])
)
```
Use a shorter window to spot bursts:
```promql
sum by (runtime_surface, surface, delivery) (
rate(storage_authorized_downloads_total{service="web"}[15m])
)
```
Unknown-size authorizations are a coverage indicator, not zero-byte downloads.
Keep them visible next to the known-byte panel.
## Pilot reconciliation and thresholds
During the pilot, an operator performs the following comparison daily for the
first four weeks and weekly afterwards:
1. Record the redacted, account-wide storage and egress totals from the Hetzner
Console. Do not attach screenshots containing account or customer data.
2. Read the 30-day Prometheus authorization totals above.
3. Compare known authorized object bytes and the count of unknown-size
authorizations with their own trailing 7- and 30-day baselines. The global
counter includes base-plan, extra-seat and add-on storage and therefore must
not be divided by sold add-on capacity.
4. Compare logical stored bytes with provider-billed storage so Hetzner's
64-kB object minimum and other account-wide storage overhead become visible.
5. Record the difference and whether uncovered surfaces, unused links, retries,
ranges, or cross-project account traffic plausibly explain it.
Review thresholds use separately meaningful signals:
| Signal | Operator action |
| ------ | --------------- |
| Authorized bytes and counts remain within `2x` of the trailing comparable period | Normal pilot observation. |
| A surface grows to `2x` or more without a matching active-user/storage increase | Warning: inspect burst shape, unknown-size counts and provider total. |
| Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget | Elevated review: reconcile all account buckets and identify unexpectedly hot flows. |
| Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget | Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture. |
These thresholds do not throttle a workspace and never create an overage
invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant
label or durable tenant traffic ledger in this slice, so workspace attribution
remains a support approximation and must not be presented as measured usage.
Do not silently enable Grafana email notifications. Dashboard review is the
launch default; notification routing, recipients, grouping, and rate limits
require a separate operator-approved change so storage telemetry cannot recreate
an alert-mail flood.
## Runtime verification
Before relying on the panels in staging or production:
1. Confirm `METRICS_ENABLED=true` and both bearer-protected web targets
`hivegrid-web-core` and `hivegrid-web-portal` are `UP` in Prometheus. Do not
scrape the shared core/admin registry twice.
2. Trigger one authorized original-file download and one preview or thumbnail.
3. Confirm the original increments `size_known="known"` and its object-byte
counter; confirm the generated variant increments only the `unknown` count.
4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests
do not increment either counter.
5. Confirm no metric label or help text contains a tenant, file, key, bucket,
filename, or signed URL.
Rollback removes only these observations. Authorization, tenant scope, malware
scan gates, and private-bucket signed delivery remain unchanged.