Refresh HiveGrid Cloud reference snapshot
This commit is contained in:
1 parent
5e99e14772
commit
7f88026f79
5955 files changed
+1177325
-188362
No files matched your search
@@ -0,0 +1,145 @@
|
||||
# Storage traffic monitoring
|
||||
|
||||
This runbook defines the launch-safe BILL-25 signal for storage downloads. It
|
||||
uses HiveGrid's existing Prometheus and Grafana stack and creates no billing,
|
||||
CDN, proxy, bucket, or server infrastructure.
|
||||
|
||||
## What the metrics mean
|
||||
|
||||
| Metric | Meaning | Never means |
|
||||
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------- |
|
||||
| `storage_authorized_downloads_total{surface,delivery,size_known}` | HiveGrid completed its authorization boundary and issued the delivery response. | The client received or finished the download. |
|
||||
| `storage_authorized_download_object_bytes_total{surface,delivery}` | Sum of trustworthy whole-object sizes associated with those authorizations. | Actual bytes transferred or billed by Hetzner. |
|
||||
|
||||
Labels are closed application enums. They never contain tenant IDs, user IDs,
|
||||
file IDs, filenames, object keys, bucket names, URLs, request values, or other
|
||||
customer data. Metric writes are fail-open: an observation failure cannot block
|
||||
an otherwise valid download.
|
||||
|
||||
Current coverage:
|
||||
|
||||
- `surface="files"`, `delivery="signed_redirect"`: original files report the
|
||||
canonical `File.sizeBytes`; generated previews and thumbnails report
|
||||
`size_known="unknown"` because their sizes are not persisted.
|
||||
- `surface="portal_document"`, `delivery="signed_redirect"`: the authorization
|
||||
count is covered, but size is deliberately unknown. The portal's
|
||||
security-definer resolver does not expose `File.sizeBytes`; BILL-25 does not
|
||||
widen that database contract or add a migration.
|
||||
|
||||
API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy
|
||||
responses are not part of this initial counter coverage. Do not infer zero
|
||||
traffic for an uncovered surface.
|
||||
|
||||
## Why this is not actual egress
|
||||
|
||||
HiveGrid redirects authorized clients to short-lived presigned URLs in a private
|
||||
Hetzner bucket. After the redirect, the object transfer no longer traverses the
|
||||
HiveGrid process. A URL can remain unused, be retried or reused before expiry,
|
||||
or serve one or more byte ranges. Therefore the authorization estimate can be
|
||||
higher or lower than actual traffic.
|
||||
|
||||
Hetzner currently does not support S3 server-access logging or S3 request
|
||||
metrics and does not provide an API for account storage/traffic utilization.
|
||||
The Hetzner Console and invoice remain the authoritative account-wide cost
|
||||
source:
|
||||
|
||||
- <https://docs.hetzner.com/storage/object-storage/supported-actions/>
|
||||
- <https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/>
|
||||
- <https://docs.hetzner.com/storage/object-storage/overview/>
|
||||
|
||||
Hetzner combines storage and traffic across all buckets and projects in the
|
||||
account. Ingress, S3 operations, and internal traffic within `eu-central` are
|
||||
free; externally delivered object data consumes the account-wide egress quota.
|
||||
The included account quota is a platform buffer and is not promised to an
|
||||
individual workspace.
|
||||
|
||||
Hetzner bills every stored object as at least 64 kB, even when the payload is
|
||||
smaller. HiveGrid's package calculation uses logical GB/TB and therefore does
|
||||
not price a hypothetical worst case made entirely of tiny objects. During the
|
||||
pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte
|
||||
total as a separate risk signal. A growing gap means object-size distribution,
|
||||
unfinished multipart uploads, versions, or non-HiveGrid buckets need review;
|
||||
it is not evidence of customer egress. Do not silently convert that difference
|
||||
into a workspace charge.
|
||||
|
||||
## Grafana views
|
||||
|
||||
Use these queries for the web service. A process restart does not invalidate
|
||||
Prometheus `increase` across a correctly scraped counter reset.
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery, size_known) (
|
||||
increase(storage_authorized_downloads_total{service="web"}[30d])
|
||||
)
|
||||
```
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery) (
|
||||
increase(storage_authorized_download_object_bytes_total{service="web"}[30d])
|
||||
)
|
||||
```
|
||||
|
||||
Use a shorter window to spot bursts:
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery) (
|
||||
rate(storage_authorized_downloads_total{service="web"}[15m])
|
||||
)
|
||||
```
|
||||
|
||||
Unknown-size authorizations are a coverage indicator, not zero-byte downloads.
|
||||
Keep them visible next to the known-byte panel.
|
||||
|
||||
## Pilot reconciliation and thresholds
|
||||
|
||||
During the pilot, an operator performs the following comparison daily for the
|
||||
first four weeks and weekly afterwards:
|
||||
|
||||
1. Record the redacted, account-wide storage and egress totals from the Hetzner
|
||||
Console. Do not attach screenshots containing account or customer data.
|
||||
2. Read the 30-day Prometheus authorization totals above.
|
||||
3. Compare known authorized object bytes and the count of unknown-size
|
||||
authorizations with their own trailing 7- and 30-day baselines. The global
|
||||
counter includes base-plan, extra-seat and add-on storage and therefore must
|
||||
not be divided by sold add-on capacity.
|
||||
4. Compare logical stored bytes with provider-billed storage so Hetzner's
|
||||
64-kB object minimum and other account-wide storage overhead become visible.
|
||||
5. Record the difference and whether uncovered surfaces, unused links, retries,
|
||||
ranges, or cross-project account traffic plausibly explain it.
|
||||
|
||||
Review thresholds use separately meaningful signals:
|
||||
|
||||
| Signal | Operator action |
|
||||
| ------ | --------------- |
|
||||
| Authorized bytes and counts remain within `2x` of the trailing comparable period | Normal pilot observation. |
|
||||
| A surface grows to `2x` or more without a matching active-user/storage increase | Warning: inspect burst shape, unknown-size counts and provider total. |
|
||||
| Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget | Elevated review: reconcile all account buckets and identify unexpectedly hot flows. |
|
||||
| Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget | Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture. |
|
||||
|
||||
These thresholds do not throttle a workspace and never create an overage
|
||||
invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant
|
||||
label or durable tenant traffic ledger in this slice, so workspace attribution
|
||||
remains a support approximation and must not be presented as measured usage.
|
||||
|
||||
Do not silently enable Grafana email notifications. Dashboard review is the
|
||||
launch default; notification routing, recipients, grouping, and rate limits
|
||||
require a separate operator-approved change so storage telemetry cannot recreate
|
||||
an alert-mail flood.
|
||||
|
||||
## Runtime verification
|
||||
|
||||
Before relying on the panels in staging or production:
|
||||
|
||||
1. Confirm `METRICS_ENABLED=true` and both bearer-protected web targets
|
||||
`hivegrid-web-core` and `hivegrid-web-portal` are `UP` in Prometheus. Do not
|
||||
scrape the shared core/admin registry twice.
|
||||
2. Trigger one authorized original-file download and one preview or thumbnail.
|
||||
3. Confirm the original increments `size_known="known"` and its object-byte
|
||||
counter; confirm the generated variant increments only the `unknown` count.
|
||||
4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests
|
||||
do not increment either counter.
|
||||
5. Confirm no metric label or help text contains a tenant, file, key, bucket,
|
||||
filename, or signed URL.
|
||||
|
||||
Rollback removes only these observations. Authorization, tenant scope, malware
|
||||
scan gates, and private-bucket signed delivery remain unchanged.
|
||||
Reference in new issue
Block a user