Refresh HiveGrid Cloud reference snapshot

This commit is contained in:
david committed 2026-08-03 15:31:14 +02:00
1 parent 5e99e14772
commit 7f88026f79
5955 files changed
+1177325 -188362

No files matched your search

@@ -1,6 +1,6 @@
# Console-Call Migration Inventory
> Generated by `scripts/observability/scan-console.ts` on 2026-05-29T06:20:36.301Z.
> Generated by `scripts/observability/scan-console.ts` on 2026-07-12T05:34:33.665Z.
> Re-run with: `pnpm exec tsx scripts/observability/scan-console.ts`
**Total `console.*` hits in server-code paths: 0**
@@ -65,6 +65,8 @@ new-account, invitation.
| `billing_provider_webhook_events_total` | counter | event_type, status | Billing webhook traffic + spoof attempts. |
| `mail_sent_total` | counter | provider, status | Email reliability. |
| `webhook_delivery_attempts_total` | counter | status | Outbound webhook health (success / retry / dlq / rejected). |
| `storage_authorized_downloads_total` | counter | surface, delivery, size_known | Successful authorization/delivery decisions; not completed downloads. |
| `storage_authorized_download_object_bytes_total` | counter | surface, delivery | Known whole-object sizes behind successful authorizations; not actual provider egress. |
| `csrf_token_rotation_total` | counter | — | Auth churn. |
| `subscription_active` | gauge | plan_id | Plan mix. |
@@ -106,10 +108,11 @@ Per-step instructions live in
### Coolify vs `/opt/observability/`
Operator decision — both work. Coolify is easier to wire to Traefik for
the `prometheus.boiler.saas-ctrl.com` and `grafana.boiler.saas-ctrl.com`
subdomains. Plain `docker compose` is lighter but requires a manual
reverse-proxy entry. **Default recommendation**: Coolify-managed
Operator decision — both work. Coolify is easier to wire to Traefik for the
Grafana subdomain. Prometheus itself remains internal on the shared
`hivegrid-observability` network and must not receive a public router or host
port. Plain `docker compose` is lighter but still exposes only Grafana through
an authenticated reverse proxy. **Default recommendation**: Coolify-managed
Service, same as Sentry.
## Dashboard-update workflow
@@ -1,82 +0,0 @@
# Performance Baseline — Phase 02 Quickwins (Task #36)
This document tracks the expected and observed effect of the 8 quickwin
optimizations shipped under task #36.
## Status
- **Code-level changes:** shipped to `main` 2026-05-17 (8 atomic commits +
this baseline doc).
- **`/metrics` endpoint status:** code shipped for API and worker
runtimes. The API exposes token-protected `GET /metrics` when
`METRICS_ENABLED=true`; worker admin servers expose token-protected
`/metrics` when enabled and keep health probes on a separate port.
- **Live `/metrics` baseline:** operator-owned. The endpoint code exists,
but the live before/after numbers still require an environment with
`METRICS_ENABLED=true`, `METRICS_TOKEN`, scrape access and a controlled
load run.
## Per-item expected gains (code-level reasoning)
| Item | File | Mechanism | Expected gain |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. N+1 in admin users list | `apps/web/app/routes/admin/accounts/users.tsx` + `apps/web/app/utils/db/logs.db.server.ts` | Replace N `findFirst` calls with 1 `groupBy` + 1 `findMany` | DB queries on the admin users-list loader: `O(N)` → `O(1)` (101→2 for a 100-user page). Largest practical win on this route. |
| 2. Duplicate tenant fetch | `apps/web/app/utils/session.server.ts` + `apps/web/app/routes/admin/accounts/users.tsx` + `apps/web/app/routes/oauth/github/callback.tsx` | `setLoggedUser` now returns `defaultTenant`; callers reuse it instead of fetching again | One fewer `getTenant()` round-trip per sign-in / impersonation start (≈ 1 DB query/login). |
| 3. S3 client + signer caching | `apps/api/src/storage.ts` | Memoize S3 storage adapter, signed-URL adapter and per-day SigV4 signing key by SHA256-of-config | Eliminates per-call adapter construction + HMAC chain (4 HMAC-SHA256 ops) on every signed URL. Per-request CPU win in hot upload/download paths. |
| 4. clearSubscriptionsCache thrash | `apps/web/app/utils/services/.server/subscriptionService.ts` + `apps/web/app/utils/db/subscriptions/tenantSubscriptionProducts.db.server.ts` | New `clearTenantSubscriptionCache(tenantId)` replaces a global wipe in `cancelTenantSubscriptionProduct` | Subscription-change webhooks no longer evict every tenant's cached subscription. Tenant N=1000 → only 1 eviction per event instead of 1000. |
| 5. CSRF header-first check | `apps/web/app/utils/session.server.ts` | Already-resolved in commit `83a57f1c`; regression test added | No additional perf change in #36; documents the historical fix and now-enforced contract: 0 body-clones when the header carries the token. |
| 6. Recovery-code rejection sampling | `apps/web/app/utils/auth/mfa/recovery-codes.server.ts` | Drop `while(set.size < n)` loop; generate batch + max 2 retries | Generates 10 codes in one batch (≥99.999% of calls); throws on systemic failure instead of looping forever. Bounds wall-clock generation cost. |
| 7. UUID collision loop | `apps/api/src/auth/session/session-store.ts` | `while (sessions.has(sessionId))` → single insert; throw on collision | 1 ID generation per `createSession` instead of unbounded retry. UUID-v4 has 122 bits of entropy, so a collision is a systemic error to surface, not mask. |
| 8. CSRF token size | `apps/web/app/utils/session.server.ts` | `randomBytes(100).toString("base64")` → `randomBytes(32).toString("hex")` | Token shrinks ~136 chars → 64 chars. Saves ~70 bytes/cookie and ~70 bytes/response per CSRF-bearing endpoint. Aggregate response-body reduction across the cluster. |
## Test coverage added
- `apps/web/app/utils/db/logs.db.server.test.ts` — `getLastUserLogsByUserIds`
uses exactly 1 `groupBy` + 1 `findMany`.
- `apps/web/app/utils/session.server.test.ts` — `setLoggedUser` returns
`defaultTenant`; `validateCSRFToken` never clones the request body when
the header carries the token; `generateCSRFToken` emits 64 hex chars.
- `apps/api/src/storage.test.ts` — repeated config hits the adapter cache;
bucket change misses; secret rotation misses.
- `apps/web/app/utils/services/.server/subscriptionService.test.ts` —
`clearTenantSubscriptionCache` does not enumerate other tenants;
`clearAllSubscriptionsCache` invalidates all tenant entries.
- `apps/web/app/utils/auth/mfa/recovery-codes.server.test.ts` — generation
completes under a strict wall-clock bound (regression guard against
unbounded rejection sampling).
- `apps/api/src/auth/session/session-store.test.ts` — UUID collision now
throws instead of silently retrying.
## Live baseline (operator-owned)
Populate the table below by running `oha -n 5000 -c 50` against staging
twice for a comparable before/after window and scraping API/worker
`/metrics` snapshots with `Authorization: Bearer <METRICS_TOKEN>`.
The code path is shipped; the live baseline values are operational
evidence and should be recorded only from the target environment.
| Metric | Before | After | Delta |
| ----------------------------------------------- | --------------- | ----- | ----- |
| p50 latency, GET `/admin/users` | TBD | TBD | TBD |
| p95 latency, GET `/admin/users` | TBD | TBD | TBD |
| Prisma query count / request (admin users list) | TBD | TBD | TBD |
| p95 latency, signed-URL endpoint | TBD | TBD | TBD |
| p95 latency, login flow | TBD | TBD | TBD |
| Cache evictions per Legacy-Provider webhook | TBD | TBD | TBD |
| CSRF cookie size (bytes) | ~136 hex equiv. | 64 | -53% |
## Notes
- All eight items shipped as **atomic** commits; any single item can be
reverted in isolation.
- Item 5 was already in tree from `83a57f1c fix: enforce csrf on settings
mutations`. It is recorded here for completeness and now carries an
explicit no-body-read assertion.
- Path corrections were necessary on items 2 and 6.
## Footer
- Date: 2026-05-17
- Owner: Phase-02 quickwins agent
- Live-test hardware: operator-owned (fill with the target environment
used for the controlled load run)
@@ -40,7 +40,6 @@ The following components/routes MUST be annotated by the time the staging-smoke
- `apps/web/app/routes/register.tsx` — password input → `sentry-mask`
- `apps/web/app/routes/forgot-password.tsx` — email input → `sentry-mask`
- `apps/web/app/routes/reset.tsx` — password inputs → `sentry-mask`
- `apps/web/app/routes/auth.totp.tsx` — TOTP code input → `sentry-mask`
- `apps/web/app/routes/auth.magic-link.tsx` — email + token field → `sentry-mask`
- `apps/web/app/modules/users/components/UserProfileSettings.tsx` — email, name, phone → `sentry-mask`
- `apps/web/app/modules/users/components/UserProfileSettings.tsx` — password and account-deletion forms → `sentry-block`
@@ -12,7 +12,7 @@ For the lightweight alternative (GlitchTip, no Performance/Replay/OTLP), see
| Item | Value |
| --- | --- |
| Upstream | `getsentry/self-hosted` tag **`25.12.1`** |
| Upstream | `getsentry/self-hosted` tag **`25.12.1`**, commit **`5c9794c9b1fda551f38515f85d56ec298371d060`** |
| Public hostname | `https://sentry.boiler.saas-ctrl.com` |
| Host | `boilerplate-staging` (`46.225.5.93`, Hetzner Nürnberg) |
| Orchestrator | Coolify-managed Docker Compose, Traefik v3.6 in front |
@@ -262,17 +262,18 @@ a one-time operator pass plus a quarterly review.
host-port bindings inherited from upstream. Verify:
`ss -tlnp | grep -E '5432|9000|9092|6379'` on the host shows
**no** matches.
- [ ] **Backups encrypted.** `backup.sh` uses GPG (recipient-key if
`SENTRY_BACKUP_GPG_RECIPIENT` is set, AES256 symmetric with
`SENTRY_SECRET_KEY` otherwise). Verify: `aws s3 cp` of a recent
- [ ] **Backups encrypted with a separate key.** `backup.sh` requires the
dedicated `SENTRY_BACKUP_GPG_RECIPIENT` and fails closed when it is
absent. Runtime `SENTRY_SECRET_KEY` is never backup key material. Verify:
`aws s3 cp` of a recent
dump returns a file with header bytes `8c 0d` (GPG-armored) or
`85 02` (binary GPG).
- [ ] **Symbolicator: no external sym sources.** `sentry.conf.py` sets
`SYMBOLICATOR_SYMSOURCES = []`. Verify: `docker compose exec
symbolicator cat /etc/symbolicator/config.yml | grep sources`
shows no Microsoft / Mozilla / Apple entries.
- [ ] **SENTRY_SECRET_KEY stored offline.** Lost key = unreadable
backups. Copy from Coolify vault to a hardware-backed password
- [ ] **Backup recipient private key stored offline.** Lost key = unreadable
backups. Store it separately from the Sentry runtime secrets in a hardware-backed password
manager (1Password, BitWarden Premium with TOTP, YubiKey-backed
pass).
- [ ] **Traefik dashboard port 8080 firewalled.** Close at Hetzner Cloud
@@ -286,8 +287,7 @@ per quarter, (c) before opening Sentry to additional users.
## CI Secrets
The CI pipelines in `.github/workflows/_sentry-release.yml` and
`.gitea/workflows/_sentry-release.yml` need three Sentry secrets to tag
The CI pipeline in `.gitea/workflows/_sentry-release.yml` needs three Sentry secrets to tag
releases, upload source-maps, correlate commits, and mark deploys. The
main CI build uploads `apps/web/build/client` as the
`web-client-sourcemaps` workflow artifact; the reusable web release job
@@ -312,30 +312,12 @@ and passes source-maps through the workflow artifact instead.
`member:admin`.
4. Copy the token immediately — Sentry only shows it once.
### 2. Store secrets in GitHub Actions
Navigate to **Settings → Secrets and variables → Actions** in the
GitHub mirror (`github.com/<org>/boilerplate`) and add three repository
secrets:
| Name | Value |
| --- | --- |
| `SENTRY_AUTH_TOKEN` | The token minted in step 1 |
| `SENTRY_ORG` | `sentry` (the auto-generated slug from single-org install) |
| `SENTRY_URL` | `https://sentry.boiler.saas-ctrl.com` |
Once configured, `_sentry-release.yml` is invoked once per project
(api/worker/web) at the end of every CI run via `workflow_call`.
Only the web invocation passes `sourcemaps-artifact:
web-client-sourcemaps`; api and worker releases pass `sourcemaps-path:
none`.
### 3. Store secrets in Gitea Actions
### 2. Store secrets in Gitea Actions
In Gitea (`git.tool-ctrl.de/toolctrl-labs/HiveGrid-Cloud`) open
**Repository Settings → Actions → Secrets** and create the same three
secrets (`SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_URL`) with
identical values. Gitea's secret scoping rules mirror GitHub's, so the
identical values. Gitea's repository secret scoping makes them available to the
reusable workflow at `.gitea/workflows/_sentry-release.yml` (and the
composite-action fallback at `.gitea/actions/sentry-release/action.yml`
for runner versions < 1.21) will pick them up automatically.
@@ -345,22 +327,21 @@ secrets in its **Mark Sentry deploy** step; if the secrets are absent
the step skips silently so existing deploy paths continue to function
during the rollout window.
### 4. Rotation
### 3. Rotation
- **Rotate every 6 months.** Add a recurring calendar entry for the
operator on-call.
- Rotation procedure:
1. Mint a new token (step 1) with the same `project:releases` scope.
2. Update `SENTRY_AUTH_TOKEN` in **both** GitHub and Gitea secret
stores within the same maintenance window.
3. Wait for one CI run on each system to confirm the new token works
2. Update `SENTRY_AUTH_TOKEN` in the Gitea repository secret store.
3. Wait for one CI run to confirm the new token works
(look for a fresh release in the Sentry UI).
4. Revoke the old token via the Sentry UI (Settings → Account →
API → Auth Tokens → trash icon).
- **Next scheduled rotation: 2026-11-17.** Update this date after each
rotation.
### 5. Local development
### 4. Local development
Releases are a CI-only concern. Local developers do **not** need
`SENTRY_AUTH_TOKEN`; the web Vite build only uploads source-maps when