146 lines
7.5 KiB
Markdown
146 lines
7.5 KiB
Markdown
# Storage traffic monitoring
|
|
|
|
This runbook defines the launch-safe BILL-25 signal for storage downloads. It
|
|
uses HiveGrid's existing Prometheus and Grafana stack and creates no billing,
|
|
CDN, proxy, bucket, or server infrastructure.
|
|
|
|
## What the metrics mean
|
|
|
|
| Metric | Meaning | Never means |
|
|
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------- |
|
|
| `storage_authorized_downloads_total{surface,delivery,size_known}` | HiveGrid completed its authorization boundary and issued the delivery response. | The client received or finished the download. |
|
|
| `storage_authorized_download_object_bytes_total{surface,delivery}` | Sum of trustworthy whole-object sizes associated with those authorizations. | Actual bytes transferred or billed by Hetzner. |
|
|
|
|
Labels are closed application enums. They never contain tenant IDs, user IDs,
|
|
file IDs, filenames, object keys, bucket names, URLs, request values, or other
|
|
customer data. Metric writes are fail-open: an observation failure cannot block
|
|
an otherwise valid download.
|
|
|
|
Current coverage:
|
|
|
|
- `surface="files"`, `delivery="signed_redirect"`: original files report the
|
|
canonical `File.sizeBytes`; generated previews and thumbnails report
|
|
`size_known="unknown"` because their sizes are not persisted.
|
|
- `surface="portal_document"`, `delivery="signed_redirect"`: the authorization
|
|
count is covered, but size is deliberately unknown. The portal's
|
|
security-definer resolver does not expose `File.sizeBytes`; BILL-25 does not
|
|
widen that database contract or add a migration.
|
|
|
|
API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy
|
|
responses are not part of this initial counter coverage. Do not infer zero
|
|
traffic for an uncovered surface.
|
|
|
|
## Why this is not actual egress
|
|
|
|
HiveGrid redirects authorized clients to short-lived presigned URLs in a private
|
|
Hetzner bucket. After the redirect, the object transfer no longer traverses the
|
|
HiveGrid process. A URL can remain unused, be retried or reused before expiry,
|
|
or serve one or more byte ranges. Therefore the authorization estimate can be
|
|
higher or lower than actual traffic.
|
|
|
|
Hetzner currently does not support S3 server-access logging or S3 request
|
|
metrics and does not provide an API for account storage/traffic utilization.
|
|
The Hetzner Console and invoice remain the authoritative account-wide cost
|
|
source:
|
|
|
|
- <https://docs.hetzner.com/storage/object-storage/supported-actions/>
|
|
- <https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/>
|
|
- <https://docs.hetzner.com/storage/object-storage/overview/>
|
|
|
|
Hetzner combines storage and traffic across all buckets and projects in the
|
|
account. Ingress, S3 operations, and internal traffic within `eu-central` are
|
|
free; externally delivered object data consumes the account-wide egress quota.
|
|
The included account quota is a platform buffer and is not promised to an
|
|
individual workspace.
|
|
|
|
Hetzner bills every stored object as at least 64 kB, even when the payload is
|
|
smaller. HiveGrid's package calculation uses logical GB/TB and therefore does
|
|
not price a hypothetical worst case made entirely of tiny objects. During the
|
|
pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte
|
|
total as a separate risk signal. A growing gap means object-size distribution,
|
|
unfinished multipart uploads, versions, or non-HiveGrid buckets need review;
|
|
it is not evidence of customer egress. Do not silently convert that difference
|
|
into a workspace charge.
|
|
|
|
## Grafana views
|
|
|
|
Use these queries for the web service. A process restart does not invalidate
|
|
Prometheus `increase` across a correctly scraped counter reset.
|
|
|
|
```promql
|
|
sum by (runtime_surface, surface, delivery, size_known) (
|
|
increase(storage_authorized_downloads_total{service="web"}[30d])
|
|
)
|
|
```
|
|
|
|
```promql
|
|
sum by (runtime_surface, surface, delivery) (
|
|
increase(storage_authorized_download_object_bytes_total{service="web"}[30d])
|
|
)
|
|
```
|
|
|
|
Use a shorter window to spot bursts:
|
|
|
|
```promql
|
|
sum by (runtime_surface, surface, delivery) (
|
|
rate(storage_authorized_downloads_total{service="web"}[15m])
|
|
)
|
|
```
|
|
|
|
Unknown-size authorizations are a coverage indicator, not zero-byte downloads.
|
|
Keep them visible next to the known-byte panel.
|
|
|
|
## Pilot reconciliation and thresholds
|
|
|
|
During the pilot, an operator performs the following comparison daily for the
|
|
first four weeks and weekly afterwards:
|
|
|
|
1. Record the redacted, account-wide storage and egress totals from the Hetzner
|
|
Console. Do not attach screenshots containing account or customer data.
|
|
2. Read the 30-day Prometheus authorization totals above.
|
|
3. Compare known authorized object bytes and the count of unknown-size
|
|
authorizations with their own trailing 7- and 30-day baselines. The global
|
|
counter includes base-plan, extra-seat and add-on storage and therefore must
|
|
not be divided by sold add-on capacity.
|
|
4. Compare logical stored bytes with provider-billed storage so Hetzner's
|
|
64-kB object minimum and other account-wide storage overhead become visible.
|
|
5. Record the difference and whether uncovered surfaces, unused links, retries,
|
|
ranges, or cross-project account traffic plausibly explain it.
|
|
|
|
Review thresholds use separately meaningful signals:
|
|
|
|
| Signal | Operator action |
|
|
| ------ | --------------- |
|
|
| Authorized bytes and counts remain within `2x` of the trailing comparable period | Normal pilot observation. |
|
|
| A surface grows to `2x` or more without a matching active-user/storage increase | Warning: inspect burst shape, unknown-size counts and provider total. |
|
|
| Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget | Elevated review: reconcile all account buckets and identify unexpectedly hot flows. |
|
|
| Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget | Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture. |
|
|
|
|
These thresholds do not throttle a workspace and never create an overage
|
|
invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant
|
|
label or durable tenant traffic ledger in this slice, so workspace attribution
|
|
remains a support approximation and must not be presented as measured usage.
|
|
|
|
Do not silently enable Grafana email notifications. Dashboard review is the
|
|
launch default; notification routing, recipients, grouping, and rate limits
|
|
require a separate operator-approved change so storage telemetry cannot recreate
|
|
an alert-mail flood.
|
|
|
|
## Runtime verification
|
|
|
|
Before relying on the panels in staging or production:
|
|
|
|
1. Confirm `METRICS_ENABLED=true` and both bearer-protected web targets
|
|
`hivegrid-web-core` and `hivegrid-web-portal` are `UP` in Prometheus. Do not
|
|
scrape the shared core/admin registry twice.
|
|
2. Trigger one authorized original-file download and one preview or thumbnail.
|
|
3. Confirm the original increments `size_known="known"` and its object-byte
|
|
counter; confirm the generated variant increments only the `unknown` count.
|
|
4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests
|
|
do not increment either counter.
|
|
5. Confirm no metric label or help text contains a tenant, file, key, bucket,
|
|
filename, or signed URL.
|
|
|
|
Rollback removes only these observations. Authorization, tenant scope, malware
|
|
scan gates, and private-bucket signed delivery remain unchanged.
|