Update hivegrid-cloud reference to latest main

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
davidandClaude Fable 5 committed 2026-07-08 16:15:37 +02:00
1 parent e98d38e716
commit 7b21524c86
3781 files changed
+373484 -225607

No files matched your search

@@ -1,6 +1,6 @@
# Asset CDN Operations Runbook
When A7/B2/#51 is reactivated, front the boilerplate's static assets
When the asset-CDN rollout is reactivated, front HiveGrid's static assets
(`/assets/*`, `/fonts/*`, `/images/*`) with a CDN so the origin no longer pays
the bandwidth + TLS handshake cost for every visitor and so users far from the
Frankfurt DC see sub-second first paints.
@@ -8,13 +8,10 @@ Frankfurt DC see sub-second first paints.
**Default provider:** Bunny.net (EU-based, cheap, simple).
**Documented alternative:** Cloudflare.
Related task: [`docs/improvement-plan/03-scaling-infrastructure/asset-cdn.md`](../improvement-plan/03-scaling-infrastructure/asset-cdn.md).
**Rollout status:** CDN provisioning and `ASSET_CDN_URL` flips remain parked
with A7/B2/#51. The deploy workflow already includes a no-op-safe Bunny purge
step for future activation; keep this runbook as the reactivation guide until
`docs/improvement-plan/OPERATOR-CHECKLIST.md` and
`docs/improvement-plan/PROGRESS.md` move #51 out of backlog.
until a HiveGrid infra ticket explicitly reactivates them. The deploy workflow
already includes a no-op-safe Bunny purge step for future activation; keep this
runbook as the reactivation guide.
## Architecture
@@ -63,9 +60,9 @@ response. Loaders may set their own `Cache-Control` to opt out of `no-store`
Cloudflare and wants a single console — then Cloudflare is the lower-friction
choice.
## Bunny.net setup (default, when A7/B2/#51 is reactivated)
## Bunny.net setup (default, when the rollout is reactivated)
Do not execute these steps while A7/B2/#51 remains parked. They are the
Do not execute these steps while the rollout remains parked. They are the
reactivation checklist for the future CDN rollout. None of this is automated by
the deploy pipeline.
@@ -123,7 +120,7 @@ the deploy pipeline.
7. **Trigger a deploy** and run the smoke tests below.
## Smoke tests (when A7/B2/#51 is reactivated)
## Smoke tests (when the rollout is reactivated)
Do not run these as active acceptance checks while CDN rollout remains parked.
@@ -1,7 +1,7 @@
# Backup CLI Runbook
Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and
is callable as `toolctrl-cli backup <subcommand>` once the workspace is
Operator guide for `@hivegrid/cli backup:*`. The CLI ships in `packages/cli` and
is callable as `hivegrid-cli backup <subcommand>` once the workspace is
installed.
## Quick reference
@@ -21,16 +21,16 @@ bucket so a compromised app credential cannot pivot to the backups.
```bash
# MinIO example (staging)
mc mb staging-minio/toolctrl-backups
mc anonymous set none staging-minio/toolctrl-backups
mc mb staging-minio/hivegrid-backups
mc anonymous set none staging-minio/hivegrid-backups
# AWS S3 example
aws s3api create-bucket --bucket toolctrl-backups \
aws s3api create-bucket --bucket hivegrid-backups \
--create-bucket-configuration LocationConstraint=us-east-1
aws s3api put-bucket-encryption --bucket toolctrl-backups \
aws s3api put-bucket-encryption --bucket hivegrid-backups \
--server-side-encryption-configuration \
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-public-access-block --bucket toolctrl-backups \
aws s3api put-public-access-block --bucket hivegrid-backups \
--public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
```
@@ -49,13 +49,13 @@ The default encryption method is [age](https://age-encryption.org).
```bash
# Generate a key pair on a trusted workstation
age-keygen -o ~/.age-toolctrl-backup.key
age-keygen -o ~/.age-hivegrid-backup.key
# Output:
# Public key: age1example...
# Identity (private key) is at ~/.age-toolctrl-backup.key
# Identity (private key) is at ~/.age-hivegrid-backup.key
# Set the public key in the deployment env
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/hivegrid/backup.env
```
**Store the identity file (private key) somewhere the running app does not have
@@ -75,7 +75,7 @@ If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and
# Backup bucket
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
BACKUP_S3_REGION=us-east-1
BACKUP_S3_BUCKET=toolctrl-backups
BACKUP_S3_BUCKET=hivegrid-backups
BACKUP_S3_ACCESS_KEY_ID=AKIA...
BACKUP_S3_SECRET_ACCESS_KEY=...
BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style
@@ -101,11 +101,11 @@ PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback.
## 4. Daily cron
```cron
# /etc/cron.d/toolctrl-backup
# /etc/cron.d/hivegrid-backup
SHELL=/bin/sh
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=ops@example.com
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1
0 3 * * * deploy /opt/hivegrid/scripts/cron/backup-daily.sh >> /var/log/hivegrid-backup.log 2>&1
```
Example wrapper:
@@ -113,9 +113,9 @@ Example wrapper:
```bash
#!/usr/bin/env bash
set -euo pipefail
cd /opt/toolctrl
node ./packages/cli/bin/toolctrl-cli.mjs backup create
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate
cd /opt/hivegrid
node ./packages/cli/bin/hivegrid-cli.mjs backup create
node ./packages/cli/bin/hivegrid-cli.mjs backup rotate
```
For docker-compose deployments:
@@ -140,15 +140,15 @@ Run **at least monthly**:
```bash
# 1. Pick yesterday's backup
toolctrl-cli backup list
hivegrid-cli backup list
# 2. Verify in-place (creates verify_<uuid>, drops it after)
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
hivegrid-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
# 3. End-to-end DR drill (restore into a scratch host)
DATABASE_URL=postgres://dr-test/restored \
BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
hivegrid-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
```
Document the drill outcome in the incident channel or your internal ops ledger.
@@ -0,0 +1,108 @@
# Billing E2E Sandbox-Flow
Stand: 2026-07-05
Dieses Runbook dokumentiert den reproduzierbaren BILL-07-Proof fuer den v1-Billing-Kern. Der Standardlauf nutzt
lokale Sandbox-Stubs fuer Lago und Mollie, fuehrt aber dieselbe HiveGrid-Sync-Schicht aus, die auch Provider-Webhooks
in den lokalen Billing-State spiegelt.
## Ziel
Der Flow beweist:
- Test-Tenant wird als Lago-Customer angelegt.
- Plan + Seat-Subscription wird in Lago angelegt.
- Mollie-Zahlung wird erzeugt und als bezahlt simuliert.
- Provider-Sync aktualisiert `TenantSubscription` lokal.
- Entitlement-/Modul-Gate und Read-only-State reagieren auf den lokalen Billing-State.
- Invoice- und Payment-Referenzen sind sichtbar.
- Sync-Audit und Activity-Eintraege werden geschrieben.
- Negativfaelle `failed`, `expired` und `frozen` setzen den Tenant read-only.
## Stub-Proof ausfuehren
Vom Repo-Root:
```bash
pnpm exec tsx scripts/billing-e2e-sandbox.ts --run-id=manual-$(date -u +%Y%m%d%H%M%S)
```
Erwartung:
- JSON-Ausgabe mit `"ok": true`.
- `steps[]` enthaelt acht erfolgreiche Schritte.
- `activeState.status` ist `active`.
- `activeState.readOnly` ist `false`.
- `gate.activeAllowed` ist `true`.
- `failedState.status` ist `past_due` und `failedState.readOnly` ist `true`.
- `pastDueState.status` ist `past_due` und `pastDueState.readOnly` ist `true`.
- `frozenState.status` ist `frozen` und `frozenState.readOnly` ist `true`.
- `referencesVisible` ist `true`.
- `audit.auditEvents` und `audit.activities` sind groesser/gleich `5`.
Der Lauf schreibt nicht in die echte Datenbank. Er nutzt deterministische In-Memory-Stubs fuer Provider und DB-Client,
ruft aber `applyLagoSubscriptionSync`, `applyMolliePaymentSync` und `resolveTenantBillingState` aus der echten
HiveGrid-Codebase auf.
## Automatischer Test
```bash
pnpm --dir packages/billing test -- sandbox-flow.test.ts
```
Dieser Test ist CI-tauglich, weil keine externen Lago-/Mollie-Credentials gebraucht werden.
## Testdaten-Konvention fuer echte Sandbox
Falls der Flow spaeter gegen `lago-staging` erweitert wird:
- Customer external ID: `hg-stg-bill-07-<short-id>`
- Plan: `hg-stg-plan-bill-07-<short-id>`
- Subscription external ID: `subscription-<short-id>`
- Payment external ID: `payment-<short-id>`
Zugang und Tunnel fuer `lago-staging` stehen in `docs/ops/lago-staging.md`.
## Live-Sandbox-Erweiterung
Der urspruengliche BILL-07-Abschluss war provider-stubbed. Seit 2026-07-04 ist die echte Staging-Verbindung
vorbereitet:
- `lago-staging` ist initialisiert.
- HiveGrid-Staging erreicht Lago ueber `http://10.42.0.4:3108`.
- HiveGrid-Staging hat `LAGO_*` und `MOLLIE_*` Runtime-Env auf API/Web/Worker.
- Provider-Smoke aus dem laufenden `hivegrid-api`-Container: Lago HTTP 200, Mollie HTTP 200.
- Synthetischer Lago-Customer und Mollie-Testpayment koennen aus Staging erzeugt werden.
- `POST /v1/billing/mollie/webhooks` ist im API-Runtime registriert. Die Route nimmt Mollies `id` entgegen,
refetcht die Zahlung serverseitig bei Mollie und schreibt den lokalen Billing-State ueber
`applyMolliePaymentSync`. Fuer Self-Service-Checkout-Zahlungen mit Plan-Metadaten schreibt sie zusaetzlich den
lokalen Lago-/HiveGrid-Subscription-State ueber `applyLagoSubscriptionSync`.
Erledigt fuer den manuellen Staging-E2E:
- Abschluss eines echten Mollie-Test-Checkout-Flows gegen die echte Staging-Tenant `demo-pro-crm`.
- Mollie-Payment `tr_Nn3WsWmtkTXXYRCenmVTJ` wurde bezahlt, Webhook kam mit HTTP 200 an,
Tenant-Billing-Status steht lokal auf `active`.
- Staging-Admin fuer interne Billing-Pruefung ist unter `https://staging-admin.hivegrid.de/accounts/billing`
erreichbar.
- Lago-Staging hat echte Demo-Rechnungen fuer `demo-starter-crm`, `demo-pro-crm` und `demo-business-crm`.
Der HiveGrid-Kundenbereich `/:tenant/settings/subscription` zeigt die Rechnungen fuer alle drei Demo-Tenants an.
- Lago-Staging speichert neue Invoice-PDFs im Hetzner-Bucket `hivegrid-staging-lago`; der Kunden-Download laeuft
ueber HiveGrid als serverseitiger Proxy nach Permission-/Tenant-Pruefung.
- Die alten lokalen Demo-Invoices `*-001` wurden in Lago voided; aktuelle S3-Demo-Invoices sind `*-002`.
- Billing entity, Logo, German document locale, 14-day payment terms and the neutral Mollie footer are configured in
Lago staging.
- Enriched demo invoice `HIV-6711-002-008` for `demo-pro-crm` verifies the current customer-facing PDF shape through
HiveGrid's download route.
- Die funktionale Self-Service-Basis ist in HiveGrid verdrahtet: `/:tenant/settings/subscription` kann oeffentliche
Plaene/Seats auswaehlen, erstellt Lago-Customer/Subscription, startet Mollie und verarbeitet Return/Webhook
idempotent.
Offen fuer den vollautomatischen Staging-E2E:
- Cleanup-Schritt fuer Lago-/Mollie-Testdaten.
- Wiederholbarer Script-Schritt fuer echte Lago-Plan-/Subscription-/Invoice-Testdaten.
- Browser-Smoke fuer den echten Self-Service-Checkout gegen Staging-Testdaten.
Bis dahin bleibt der Stub-Proof der kanonische, wiederholbare Regressionstest fuer den HiveGrid-Billing-Kern; der
Live-Smoke beweist zusaetzlich Provider-Reachability, Checkout-Link-Erzeugung und die API-WebHook-Grenze aus Staging.
@@ -0,0 +1,64 @@
# CI-Runner auf node1 + CI-Pipeline-Overhaul (2026-06-28)
> Operativer Record der CI-Umstellung. Runner-Host-Themen gehören mittelfristig auch nach
> `infra/LESSONS.md` (Infra-Repo, Akeshan-Domain) — hier dokumentiert, weil die
> `.gitea/workflows/`-Änderungen in diesem Repo liegen.
## Warum: Homeserver OOMte unter paralleler CI
Der geteilte Homeserver (32 GB RAM, neben anderen Diensten ~17 GB belegt) ist bei paralleler
CI wiederholt abgestürzt: parallele Jobs spikten den RAM auf 30+/32 GB, CPU auf 100%, der
Linux-OOM-Killer riss den größten Job → Crash-Recover-Schleife (Runner grabbte sofort den
nächsten Job). Parallelisierung auf der geteilten Box war der falsche Hebel.
## Lösung: node1 als dedizierter CI-Runner
node1 (`176.9.111.179`, eigener Hetzner-Dedi mit self-hosted Sentry, **20 Kerne / 62 GB**)
ist Mesh-Peer und erreicht Gitea über `10.42.0.2`. Gitea-Server bleibt, wo er ist — nur die
Runner-Last zieht um.
- Compose: `/opt/stacks/gitea-ci-runner/docker-compose.yml` (`network_mode: host`,
`--add-host git.tool-ctrl.de:10.42.0.2`, `CONFIG_FILE=/data/config.yaml`).
- Config: `/srv/data/gitea-ci-runner/config.yaml` (capacity, labels, cache.dir).
- Homeserver-Runner steht auf `restart=no` (aus, kein Auto-Restart, keine OOM-Schleife).
### act_runner-Footguns (alle real getroffen)
| Symptom | Ursache | Fix |
|---|---|---|
| `runner.capacity` wird ignoriert | run.sh überschreibt den compose-`command`, läuft ohne `--config` | env `CONFIG_FILE=/data/config.yaml` setzen |
| "no matching runner online with label toolctrl-ci" | leere `runner.labels` → KEIN Fallback auf `.runner` | Labels EXPLIZIT in config.yaml |
| Daemon "token is empty" trotz erfolgreicher Registrierung | `act_runner register` schrieb `.runner` ins Container-CWD → bei `--rm` weg | `docker run -w /data ...` beim Register |
| Register "instance address is empty" | run.sh ignoriert CLI-Args, nutzt leeres `GITEA_INSTANCE_URL` | `--entrypoint act_runner` umgeht run.sh |
| pnpm-Store-Cache greift nie (Cache-Dir 12K) | `cache.dir` default ephemer im Container, Restarts wischen ihn | `cache.dir: /data/actcache` (gemountet) |
## Capacity-Findings (empirisch gemessen)
| Setup | Wallclock | RAM-Peak frei | Notiz |
|---|---|---|---|
| cap 4, docker-smoke auf Tail | ~10m | 35 GB | |
| cap 6 | ~8m | 23-25 GB | |
| cap 6 + docker-smoke parallel | ~7m35s | 23 GB | |
| **cap 8 + docker-smoke parallel** | **~6m53s** | **25 GB** | Sweet-Spot |
**cap 8 ist schneller als cap 6** trotz mehr Contention: Install/Netzwerk/Docker sind
I/O-bound, die 20 Kerne tragen 8 Jobs; die Queueing-Reduktion schlägt die Mehr-Contention.
cap >8 bringt nichts (Kerne saturiert). Reise gesamt: **~32m → ~6m53s**.
## Pipeline-Änderungen (.gitea/workflows/)
- **docker-smoke `needs` entfernt** → läuft parallel zur Quality-Phase (baut Images aus
Source, keine Cross-Job-Artefakt-Abhängigkeit), raus aus dem kritischen Tail.
- **Sentry-Release-Jobs aus ci.yml entfernt** (ci.yml läuft nur auf `pull_request`; ein
PR-Build wird nirgends deployed → die Release/Sourcemaps mappen auf keine Deployment).
Release passiert in `deploy.yml` (push→main / promote).
## Install-Diagnose (warum KEIN Deps-Image)
Gemessen: Install = **~43s Download + ~15s Link** (warmer Store). Die ~2-min-Annahme war zu
hoch. Ein Deps-Image würde nur die 15s Link sparen → verworfen. Der Hebel war der **kalte
Store** (Cache persistierte nicht) → mit `cache.dir` gefixt.
## Offene Punkte
- **Sourcemaps im Deploy:** web nutzt `@sentry/vite-plugin` (lädt Sourcemaps beim Build hoch,
wenn `SENTRY_AUTH_TOKEN` gesetzt). To-do: Token als BuildKit-`--secret` an den
`web-build`-Dockerfile-Stage durchreichen (NICHT build-arg → kein Leak) +
`SENTRY_PROJECT=web` explizit (Default ist alter Slug `hivegrid-web`). Nur auf echtem
Deploy testbar → bis Sentry-Go-Live aufgeschoben.
- **Persistenter Store-Cache:** `cache.dir` gesetzt + befüllt (2,9 GB), aber Hit-Gewinn bei
cap 8 noch ungemessen (Per-Job-Restore-I/O könnte Download-Gewinn auffressen → ggf. auf
geteilten gemounteten Store umstellen statt actions/cache).
@@ -0,0 +1,150 @@
# Lago Production Prep Runbook
This runbook defines the production target for HiveGrid billing after AD-27.
It is a preparation document only. Do not provision or enable production billing
from this document until the external gates in `EXT-16` are complete.
## Scope
`lago-production` will be the dedicated production Lago service for HiveGrid.
It must be separate from `lago-staging` and must never reuse staging database,
Redis, storage, API keys, webhook secrets, customers, plans or invoice records.
Production billing owns:
- Lago customers, plans, seats, subscriptions and invoice records.
- Invoice PDF generation and storage.
- Billing status used by HiveGrid entitlement and read-only gates.
- Mollie payment references for paid, failed, expired, cancelled and refunded
payments.
Out of scope for v1:
- Usage billing.
- Credits.
- Add-on metering.
- Public price promises in code.
- Live activation before legal, tax and Mollie KYC approval.
## Target Topology
| Item | Decision |
|---|---|
| Coolify project | `HiveGrid` |
| Coolify environment | `production` |
| Resource name | `lago-production` |
| Initial host | The current HiveGrid production host, while Hetzner capacity is constrained |
| Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted |
| Public access | None by default |
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
The initial host decision is intentionally pragmatic: production gets its own
Lago resource and data stores, but it may share the physical server with other
HiveGrid production services until infrastructure capacity allows a stricter
split. The isolation boundary is the Coolify resource, secrets, database,
Redis, storage and backup path, not the current machine count.
## Required Production Secrets
Store these only in the runtime secret store. Never put values in docs, Git,
tickets, screenshots or copied logs.
| Secret | Owner | Notes |
|---|---|---|
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
Do not reuse staging/testmode keys in production. Rotate keys when moving from
pilot validation to real customer billing.
## Backup And Restore
Production Lago is stateful. It must not receive production traffic before a
backup and restore path has been verified.
Minimum backup scope:
- Lago Postgres database.
- Lago Redis persistence if configured for durable state.
- Invoice PDF storage volume or bucket.
- Coolify resource settings and image tags.
- Runtime secret names and rotation history, without secret values.
Minimum backup schedule:
- Daily encrypted database dump.
- Backup before every Lago image upgrade.
- Backup before destructive billing test data cleanup.
- Retention aligned with the final accounting and tax retention decision.
Restore drill before go-live:
1. Create a fresh disposable restore target.
2. Restore the latest Lago Postgres dump.
3. Restore or attach invoice PDF storage.
4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
5. Verify `/health`.
6. Read one known customer, subscription and invoice.
7. Verify HiveGrid can resolve the corresponding local billing state without
writing test payments.
The restore drill date, operator and result must be recorded before production
billing is enabled.
## Access And Domain
Default: no public Lago domain.
Allowed access paths:
- Internal service-to-service URL for HiveGrid runtime.
- SSH tunnel for operators.
- Authentik-gated admin URL later, only if operator UX requires browser access.
Do not expose Lago directly to customers. Customer-facing billing screens must
remain HiveGrid-owned and use the server-side Lago/Mollie gateways.
## Mollie Live Gate
Mollie live activation is blocked until:
- Legal entity / firming is final.
- Business address, bank account and KYC documents are accepted by Mollie.
- Rechnungssteller, tax handling, imprint and customer contract data are
approved.
- Live webhook endpoint and return URLs are set for the production host.
- A live low-value payment and refund rehearsal is approved by an operator.
Until then, production must stay on manual billing or disabled self-service
checkout.
## Rollback
Rollback has two layers:
1. **HiveGrid application rollback:** deploy an older immutable HiveGrid image
tag through Coolify. This must not rewrite Lago state.
2. **Lago service rollback:** roll back Lago image tags and restore the pre-
upgrade Lago database/storage backup when the Lago upgrade mutated state.
Never restore Lago production from staging. Never merge staging and production
Lago datasets.
## Go-Live Checklist
- [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
- [ ] Dedicated production DB, Redis and storage are configured.
- [ ] Production secrets are set and staging secrets are absent.
- [ ] Backup job is configured.
- [ ] Restore drill is completed and documented.
- [ ] Mollie live profile is verified.
- [ ] Rechnungssteller, tax and imprint data are approved.
- [ ] Web/API/worker health checks pass with production Lago config.
- [ ] Manual rollback path is tested from image tag and Lago backup.
@@ -0,0 +1,389 @@
# Lago Staging Operations Runbook
This runbook covers the shared `lago-staging` service used for HiveGrid billing
integration tests. It is a staging-only billing engine for the `BILL-*` track,
not a production billing system.
## Scope
`lago-staging` exists to validate:
- Lago customers, plans, seats, subscriptions and invoice records.
- Mollie payment integration against test references.
- HiveGrid billing-state sync and entitlement gates.
It must not contain production tenants, production invoices or real payment
data. Production gets a separate `lago-production` service later.
## Current topology
| Item | Value |
|---|---|
| Coolify project | `node1-estate` |
| Resource | `lago-staging` |
| Resource UUID | `kh7l869ianltiioqgmm45kur` |
| Server | `node1` |
| Operator UI | `https://staging-lago.hivegrid.de` |
| Public protection | Lago login |
| Frontend tunnel | `http://127.0.0.1:8188` |
| API tunnel | `http://127.0.0.1:3108` |
| Mesh API URL for HiveGrid staging | `http://10.42.0.4:3108` |
| Invoice PDF storage | Hetzner Object Storage bucket `hivegrid-staging-lago` |
Services in the stack:
- Lago API
- Lago frontend
- Postgres
- Redis
- Worker
- Billing worker
- PDF service
Invoice PDFs are stored in the dedicated staging object-storage bucket
`hivegrid-staging-lago`. Lago API, worker and billing worker run with
`LAGO_USE_AWS_S3=true` and Hetzner's S3-compatible endpoint. Customer-facing
downloads must still go through HiveGrid's
`/:tenant/settings/subscription/invoices/:invoiceId/download` route; HiveGrid
checks tenant permission/ownership and streams the PDF server-side instead of
redirecting customers to Lago or the bucket origin.
## Access model
Operator browser access is available at:
```text
https://staging-lago.hivegrid.de
```
The public route is protected by the Lago application login.
Lago operator credentials are stored outside Git in
`~/.cli-tokens/lago-staging`.
The Lago API remains internal for HiveGrid staging. Do not expose
`http://10.42.0.4:3108` publicly; HiveGrid reaches it through the WireGuard
mesh.
SSH tunnel access is still a valid fallback:
Example tunnel from an operator workstation:
```bash
ssh -N \
-L 8188:127.0.0.1:8188 \
-L 3108:127.0.0.1:3108 \
root@node1
```
Then verify:
```bash
curl -fsS http://127.0.0.1:3108/health
curl -I http://127.0.0.1:8188
```
Expected API health shape: JSON with `"message":"Success"`.
## HiveGrid staging runtime connection
HiveGrid staging runs on `staging-1` (`10.42.0.3`), while `lago-staging`
runs on `node1` (`10.42.0.4`). The two services do not share a Docker
network. HiveGrid staging therefore reaches Lago through the WireGuard mesh:
```text
HiveGrid staging container -> http://10.42.0.4:3108 -> node1 local Lago API
```
On `node1`, a mesh-only systemd proxy exposes the local Lago API:
```bash
systemctl status lago-staging-mesh-proxy.service
```
The service binds only to `10.42.0.4:3108` and forwards to
`127.0.0.1:3108`, where the Coolify Docker port mapping for Lago API listens.
The public operator domain routes only to the Lago frontend and does not make
the mesh API public.
HiveGrid staging resources currently need these runtime variables:
- `LAGO_API_URL=http://10.42.0.4:3108`
- `LAGO_API_KEY`
- `LAGO_WEBHOOK_SECRET`
- `LAGO_TIMEOUT_MS`
- `MOLLIE_API_KEY`
- `MOLLIE_TIMEOUT_MS`
Configured resources:
- `hivegrid-api`
- `hivegrid-web`
- `worker-general`
- `worker-webhook`
The Lago organization and API key are initialized for staging only. Local
operator credentials are stored outside Git in `~/.cli-tokens/lago-staging`.
The HiveGrid staging webhook secret is stored outside Git in
`~/.cli-tokens/hivegrid-staging-billing`.
## Live staging smoke
Current verified smoke path:
1. From the running `hivegrid-api` staging container, call Lago via
`LAGO_API_URL`.
2. Create a synthetic Lago customer with prefix `hg-stg-manual-`.
3. Call Mollie testmode via `MOLLIE_API_KEY`.
4. Create a synthetic `EUR 1.00` Mollie payment and read the checkout URL.
2026-07-04 proof:
- Lago API from `hivegrid-api`: HTTP 200.
- Mollie payments API from `hivegrid-api`: HTTP 200.
- Synthetic Lago customer created with prefix `hg-stg-manual-`.
- Synthetic Mollie payment created with status `open`.
2026-07-04 live checkout proof:
- Mollie test checkout completed against real staging tenant `demo-pro-crm`.
- Mollie payment `tr_Nn3WsWmtkTXXYRCenmVTJ` reached status `paid`.
- Mollie called HiveGrid staging webhook with HTTP 200.
- HiveGrid updated the tenant billing state to `active`.
- Direct webhook replay returned `replayed: true` and `billingStatus: active`.
2026-07-05 billing-entity and invoice-template proof:
- Default billing entity is configured with HiveGrid staging company data,
German document locale and `14` net payment days.
- Billing entity logo is served through the public Lago ActiveStorage route.
- Organization invoice footer is intentionally payment-neutral because Mollie
may already charge a stored card or mandate:
```text
Zahlung: Automatischer Einzug per Mollie, sofern Mandat oder Karte hinterlegt ist.
Ohne automatischen Einzug bitte innerhalb von 14 Tagen mit Rechnungsnummer zahlen.
Fragen: billing@hivegrid.de · Leistungszeitraum: Juli 2026
```
- Created enriched demo invoice `HIV-6711-002-008` for `demo-pro-crm`.
- HiveGrid customer download route returned `200 application/pdf` for:
```text
https://staging-app.hivegrid.de/demo-pro-crm/settings/subscription/invoices/41c8e519-d871-4ead-9a0f-6b33fc592e71/download
```
- PDF text extraction verified the updated footer, invoice number and customer
data in the generated PDF.
2026-07-04 customer invoice proof:
- Root cause for the empty customer invoice UI was missing Lago data: Lago
staging had no plans, subscriptions or invoices yet.
- Created Lago staging plans `starter`, `pro` and `business`.
- Created Lago customers/subscriptions for the demo tenants
`demo-starter-crm`, `demo-pro-crm` and `demo-business-crm`.
- Lago generated one finalized `EUR 1.00` invoice per demo tenant:
`HIV-6711-003-001`, `HIV-6711-002-001`, `HIV-6711-004-001`.
- Verified the customer UI at `/:tenant/settings/subscription` for all three
demo tenants: invoice number present, empty state absent.
2026-07-04 object-storage cutover proof:
- Created dedicated bucket `hivegrid-staging-lago`.
- Switched Lago staging API/worker/billing-worker from local ActiveStorage
volume to Hetzner S3-compatible storage.
- Switched Lago public URLs from `127.0.0.1:3108` to
`https://staging-lago.hivegrid.de`.
- Created synthetic Lago invoice `HIV-6711-005-001`; a new PDF object appeared
in `hivegrid-staging-lago`.
- HiveGrid download route was changed to proxy/stream PDFs after permission and
tenant ownership checks, because public `/rails/active_storage/...` routes
are not a customer contract.
- The pre-S3 demo invoices `HIV-6711-002-001`, `HIV-6711-003-001` and
`HIV-6711-004-001` were voided because their blobs were created on local
ActiveStorage. Fresh S3-backed demo invoices are `HIV-6711-002-002`,
`HIV-6711-003-002` and `HIV-6711-004-002`.
The internal staging admin surface is available at:
```text
https://staging-admin.hivegrid.de
```
Billing status can be inspected at:
```text
https://staging-admin.hivegrid.de/accounts/billing
```
Admin-host routes do not use the legacy `/admin` prefix.
Webhook endpoint for staging:
```text
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
```
For a direct smoke, post Mollie's payment id as form data. The API will refetch
the payment from Mollie before any local billing write:
```bash
curl -fsS -X POST \
-H 'content-type: application/x-www-form-urlencoded' \
--data-urlencode 'id=<mollie-payment-id>' \
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
```
## Start, stop and restart
Prefer the Coolify UI for routine operations:
1. Open Coolify.
2. Go to project `node1-estate`.
3. Open resource `lago-staging`.
4. Use Start, Stop, Restart or Redeploy from the resource screen.
5. Verify API and frontend through the SSH tunnel.
For API automation, use the Coolify token from the operator token store. Do not
paste tokens into shell history or docs.
## Public operator route
The route `https://staging-lago.hivegrid.de` is configured through
Cloudflare DNS and the Coolify Traefik dynamic config on `node1`:
```text
/data/coolify/proxy/dynamic/lago-staging.yaml
```
The Lago frontend must use the public same-origin API URL:
```text
front.API_URL=https://staging-lago.hivegrid.de
```
Traefik routes `/graphql`, `/api` and `/rails/active_storage` on that host to
the Lago API container. The ActiveStorage route is required for uploaded
billing entity logos and other Lago blob URLs. Do not set the frontend
`API_URL` to `127.0.0.1`; browsers would then post login requests to the
operator workstation instead of the server.
Expected checks:
```bash
curl -sS -o /dev/null -w '%{http_code}\n' https://staging-lago.hivegrid.de
# 200
curl -sS -H 'content-type: application/json' \
--data '{"query":"{ __typename }"}' \
https://staging-lago.hivegrid.de/graphql
# {"data":{"__typename":"Query"}}
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' \
https://staging-lago.hivegrid.de/rails/active_storage/healthcheck
# 404 text/html; charset=UTF-8
# Expected for a fake blob path: this proves the request reaches the Lago API,
# not the Lago frontend SPA.
```
The route also sends `X-Robots-Tag: noindex, nofollow`.
## Backup concept
This is staging, but it is still stateful. Before upgrades or destructive
testing, create a Postgres dump from the Lago Postgres container.
Find the stack containers on `node1`:
```bash
docker ps \
--filter "label=com.docker.compose.project=kh7l869ianltiioqgmm45kur" \
--format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
```
Create a dump:
```bash
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
docker exec -t <lago-postgres-container> \
pg_dump -U <lago-db-user> -d <lago-db-name> \
> "lago-staging-${timestamp}.sql"
```
Store the dump outside the container host after creation. For short-lived
staging tests, an operator-controlled encrypted local archive is acceptable.
For longer-lived staging data, move the dump into the standard HiveGrid backup
bucket flow.
## Restore path
Restore only into `lago-staging` or a disposable local Lago instance. Never
restore staging Lago data into production.
1. Stop Lago API, frontend, worker, billing worker and PDF service in Coolify.
2. Keep Postgres running.
3. Restore the dump:
```bash
cat lago-staging-<timestamp>.sql | docker exec -i <lago-postgres-container> \
psql -U <lago-db-user> -d <lago-db-name>
```
4. Start the full stack again in Coolify.
5. Verify API health and a known test customer/subscription.
If the restore targets an existing DB, drop/recreate the DB first or restore
into a clean disposable instance. Do not merge two Lago staging datasets.
## SMTP decision
Staging outbound mail is disabled by default. Do not configure production
Mailjet credentials on `lago-staging`.
When `BILL-07` needs email verification, use one of these options:
- Mail disabled, assertions based on Lago API state.
- A dedicated Mailjet test configuration with allowlisted recipient addresses.
Any email-capable configuration must be documented in this runbook before use.
## Invoice PDF storage decision
For the current staging pilot, Invoice PDFs may use Lago's local Coolify volume.
This is acceptable because the service has no production data and no public
customer-facing billing UI.
Before production:
- configure a dedicated object storage bucket or production-grade persistent
volume strategy,
- document backup and restore for generated PDFs,
- keep staging and production storage separate.
## Upgrade procedure
1. Read Lago release notes for the target version.
2. Create a Postgres dump.
3. Note current image tags and Coolify resource settings.
4. Update the API/frontend/worker/PDF image tags together.
5. Redeploy through Coolify.
6. Verify:
- API `/health`
- frontend HTTP 200
- worker containers running
- a test customer/subscription still readable
Rollback is image-tag rollback plus DB restore from the dump created before
the upgrade.
## Test data convention
Use deterministic prefixes so the shared staging instance stays clean:
- Customer external IDs: `hg-stg-<ticket>-<short-id>`
- Plans: `hg-stg-plan-<ticket>-<short-id>`
- Coupons or special cases: `hg-stg-case-<ticket>-<short-id>`
Delete test customers and plans after the ticket unless a runbook explicitly
keeps them as regression fixtures.
@@ -5,8 +5,6 @@ of real Postgres backends. The dev compose stack and the staging Coolify
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](../improvement-plan/03-scaling-infrastructure/pgbouncer.md).
## Topology
```
@@ -36,8 +34,8 @@ Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](.
docker compose up postgres pgbouncer
# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \
DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid?pgbouncer=true&connection_limit=10" \
pnpm dev
```
@@ -48,9 +46,8 @@ URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely).
## Coolify (staging) deploy procedure
The pooler must run on the same Docker network as Postgres. The current
operator-owned deployment source is the Coolify service/network config; the
archived [`STAGING-ACCESS.md`](../improvement-plan/archive/STAGING-ACCESS.md)
is historical inventory only and must not override live Coolify state.
operator-owned deployment source is the Coolify service/network config; do not
copy historical staging inventory into a live cutover.
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
@@ -85,7 +85,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
## Coolify (Schritt 3) — erledigt 2026-06-19
- Coolify **v4.1.2** auf `control` installiert (alle Container healthy).
- **Root-Admin headless** angelegt (`raffael@tool-ctrl.de`) via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
- **Root-Admin headless** angelegt via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
- **API aktiviert** + **API-Token** erzeugt (Sanctum, mit `team_id`; Coolifys `createToken`-Override braucht Team-Kontext, daher direkt über `tokens()->create`). `GET /api/v1/servers` → 200.
- Zugang (Secrets in `~/.cli-tokens/coolify-prod`, **nicht** im Repo): UI `http://coolify.hivegrid.de:8000` bzw. `http://157.90.243.228:8000`.
- ⚠ Härtung offen: Coolify-FQDN auf `https://coolify.hivegrid.de` (443) umstellen + Port 8000 wieder schließen.
@@ -101,7 +101,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
- PgBouncer vor Postgres (Compose auf db-VM) → `DATABASE_URL` über Pooler, `DIRECT_URL` direkt.
- DEK-Master-Key + S3-Bucket `hivegrid-prod`.
- App-Deploy (web/api/worker) nach **mitapp-Muster: image-retag / build-once-deploy-via-pull** (nicht rebuild) + `migrate deploy`.
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (ToolCTRL-Instanz, prod-DSNs).
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (HiveGrid-Instanz, prod-DSNs).
## Zentral-Dienste + Storage (Schritt 4) — erledigt 2026-06-19
- **Meilisearch v1.10 + ClamAV 1.4** als Coolify-Compose-Service `hivegrid-prod-search-av` auf worker-VM, privat exposed (`10.0.0.5:7700` / `:3310`). Verifiziert: Meili von api-VM cross-server `{"status":"available"}`, ClamAV healthy. Master-Key in `~/.cli-tokens/hivegrid-prod-services`.
@@ -113,7 +113,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
- **Promote = retag, kein Rebuild:** Prod deployt denselben SHA-Tag, der getestet/gestaged ist (aktuell `9ae3963d…` = live auf Staging → Image existiert in der Gitea-Registry).
- **7 Coolify-Apps** (nicht 3!): `web`, `api`, `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin` — web/api aus `/web`+`/api`-Image, die 5 Worker aus `/worker`-Image mit unterschiedlichem `WORKER_QUEUE_GROUP`.
- Deploy-Trigger: `scripts/coolify-deploy.mjs` (PATCH `docker_registry_image_tag` → deploy), `migrate deploy` als Pre-Deployment-Command.
- **Sentry (ToolCTRL-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
- **Sentry (HiveGrid-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
**Offen (Go-Live-Phase):** Registry-Pull-Credential in Prod-Coolify · 7 Apps anlegen (Docker-Image) · App-Env Prod (aus Staging-Env transformiert: Prod-IPs/Domains, frische Secrets, prod-Sentry-Tag, **kein** WEB_NOINDEX) · erster Promote-Deploy + `migrate deploy` · DNS `app./api./admin.` · Endabnahme · mCaptcha-Sitekey + PgBouncer (Optimierung) als Folgeschritte.
@@ -163,7 +163,7 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
- **mCaptcha:** LE-Cert für `captcha.hivegrid.de` verifizieren, dann proxied; Account+Sitekey anlegen → `MCAPTCHA_*` ins App-Env (bis dahin `CAPTCHA_FAIL_OPEN=true`).
- **Sentry:** Events unter `environment=production` verifizieren (DSN aus Staging übernommen).
- **Härtung:** SSH auf Bastion/Admin-IP einschränken · `admin.hivegrid.de` per Cloudflare-Access/IP schützen · scoped Registry-Bot statt `gitea-admin` · Coolify-FQDN auf https/443 + Port 8000 schließen.
- **Optimierung:** PgBouncer vor Postgres · Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht).
- **Optimierung:** PgBouncer vor Postgres · ~~Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht)~~ → **erledigt 2026-07-03:** Hooks feuerten für Docker-Image-Apps nie; ersetzt durch Migrate-on-Boot im Web-Entrypoint (`start.sh`, `MIGRATE_ON_BOOT`/`SEED_ON_BOOT`), Details in `docs/deployment.md` + `.planning/ticket-docs/AK-P4a.md`.
- **CI-Promote:** `deploy.yml`/Gitea-Variablen auf das Prod-Coolify verdrahten (für künftige `promote`-Deploys ohne manuelle Schritte).
## Härtung (Schritt 7) — teilweise erledigt 2026-06-19
@@ -172,10 +172,10 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
**Offene Härtung/Folgeschritte (priorisiert):**
1. **mCaptcha public-FQDN:** Traefik liefert noch Default-Cert für `captcha.hivegrid.de` (Compose-Service-Domain-Labels nicht generiert; fqdn per tinker reichte nicht). Container laufen intern. Domain am besten über Coolify-**UI** am `mcaptcha`-Service setzen → LE-Cert → proxied. Dann **Account+Sitekey** in der mCaptcha-Admin-UI anlegen → `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` ins App-Env, `CAPTCHA_FAIL_OPEN=false`. Bis dahin: fail-open (kein Captcha-Zwang).
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`toolctrl-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `toolctrl-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`hivegrid-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `hivegrid-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
3. **Coolify-Port 8000** schließen, sobald Coolify hinter `https://coolify.hivegrid.de` (443) läuft (Port aktuell offen, key+token-geschützt; für laufende API-Schritte noch gebraucht).
4. **Registry:** scoped Bot-PAT statt `gitea-admin` für den Image-Pull-Login.
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell; Coolify-Pre-Deploy-Migrate griff nicht → Migration manuell im Container).
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell). Das Migrate-Problem dabei ist seit 2026-07-03 gelöst: Migrationen laufen jetzt im Web-Entrypoint (`MIGRATE_ON_BOOT`), nicht mehr über den nie feuernden Coolify-Hook.
6. **Sentry** Events unter `environment=production` verifizieren · **PgBouncer** (Pooling) ergänzen.
---
@@ -185,7 +185,7 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
**Alles erledigt:**
- ✅ **CI-Promote komplett grün** (Run 8453) — `workflow_dispatch action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`. Drei latente Multi-Server-Promote-Bugs gefixt (migration-status via `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE`, dependent-health via ProxyCommand-SSH, web-health via CF-Cache-Bypass).
- ✅ **SSH-gehärtet** — control = Bastion, web/api/worker/db public-SSH dicht.
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `toolctrl-admin`).
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `hivegrid-admin`).
- ✅ **mCaptcha** scharf + proxied (`captcha.hivegrid.de`).
- ✅ **PgBouncer** vor Postgres, alle 7 Apps über `6432`.
- ✅ **Coolify-Port zu** — nur noch `https://coolify.hivegrid.de` (Traefik+LE), `:8000` firewalled.
@@ -10,7 +10,7 @@ Last observed: 2026-05-28 via `pnpm run build`.
| Warning | Current interpretation | Escalate when |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Source maps are enabled in production` | Expected only when `TOOLCTRL_BUILD_SOURCEMAPS=1` is set for the Sentry release lane. Default web and bundle-budget builds should not emit this warning. Do not put secrets in source code; environment values must stay in `@toolctrl/config` and server-only boundaries. | Source maps expose hard-coded credentials, a default build emits maps unexpectedly, or a deploy target starts publishing server-only artifacts outside the intended build output. |
| `Source maps are enabled in production` | Expected only when `HIVEGRID_BUILD_SOURCEMAPS=1` is set for the Sentry release lane. Default web and bundle-budget builds should not emit this warning. Do not put secrets in source code; environment values must stay in `@hivegrid/config` and server-only boundaries. | Source maps expose hard-coded credentials, a default build emits maps unexpectedly, or a deploy target starts publishing server-only artifacts outside the intended build output. |
| `Generated an empty chunk` | Expected for React Router resource/API routes that have no client runtime. Empty route chunks are checked by `pnpm run verify:bundle-size`; unexpected names should be added only with an explicit rationale. | A new empty chunk is not covered by the bundle-size gate or appears for a user-facing UI route that should ship client code. |
| AWS/Smithy circular chunk warnings | Mitigated in the web Vite build by grouping `@aws-sdk/*` and `@smithy/*` into `vendor-aws-sdk`. The 2026-05-28 build emits `build/server/assets/vendor-aws-sdk-*.js` and no longer prints the circular Smithy/AWS reexport warning. | The circular warning returns, runtime AWS/S3 paths fail after build, the warning points to first-party modules, or the SDK versions/change set introduce a new broken execution order. |
| `dynamically imported ... but also statically imported` | Known Vite server-build warning where a lazy import cannot move a shared server module into a separate chunk because another route imports it statically. Current examples include DB, config, admin SSO audit, subscription entitlements, admin fallback helpers, and password reset helpers. | The warning starts involving a large client-only dependency, moves code into the default client shell, or a new performance slice claims a lazy split that this warning proves ineffective. |
@@ -26,8 +26,8 @@ pnpm run verify:docs-links
To inspect only warning lines from a local build log:
```text
pnpm run build > /tmp/toolctrl-build.log 2>&1
rg -n 'Source maps are enabled|Generated an empty chunk|@smithy|@aws-sdk|vendor-aws-sdk|dynamically imported|dynamic import will not move' /tmp/toolctrl-build.log
pnpm run build > /tmp/hivegrid-build.log 2>&1
rg -n 'Source maps are enabled|Generated an empty chunk|@smithy|@aws-sdk|vendor-aws-sdk|dynamically imported|dynamic import will not move' /tmp/hivegrid-build.log
```
## Triage Notes
@@ -1,8 +1,7 @@
# Streaming Exports — Operator Runbook
This document describes the streaming export pipeline shipped in Phase 03
task #46. It is the operator-facing complement to the detail plan
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).
task #46.
## Why streaming
@@ -1,189 +0,0 @@
# Stripe Meters API — Operator Runbook
**Audience:** operators wiring usage-based billing after the Stripe SDK
13 → 19 migration (B-001, Task #28).
**Why:** Stripe 19 removed the legacy Usage-Records API
(`subscriptionItems.createUsageRecord` /
`subscriptionItems.listUsageRecordSummaries`) and `Recurring.aggregate_usage`
in favor of the **Billing-Meters API**
(`billing.meterEvents.create` / `billing.meters.listEventSummaries`).
The boilerplate worker (`ai-cost-stripe-report`) still drives overage
reporting from `AICostEvent` rows; only the Stripe wire format changed.
Operators must configure a Billing-Meter once and migrate existing
subscriptions to the new meter-backed price.
---
## Step 1 — Create the Billing-Meter
In the Stripe Dashboard (or via the API):
1. **Billing → Meters → Create meter**.
2. Configure:
- **Display name:** `AI overage cents`
- **Event name:** `ai-overage` *(the boilerplate hardcodes this id;
change it only if you also override `aiOverageStripeMeterId` /
`aiOverageMeterEventName` per plan).*
- **Aggregation:** `Sum`
- **Customer mapping:** map the event payload key
`stripe_customer_id` to the Stripe customer id (default — leave
as-is).
- **Value settings:** event payload key `value`, type
`numeric`. The reporter submits the rounded overage block-count
in cents.
3. Copy the resulting **Meter id** (`mtr_…`). You will store this in
`Tenant.aiQuotaOverrideJson.aiOverageStripeMeterId` (per-tenant)
or in the plan default (per-plan, future admin UI).
CLI alternative:
```bash
stripe billing meters create \
--display-name="AI overage cents" \
--event-name="ai-overage" \
--default-aggregation[formula]="sum" \
--customer-mapping[event_payload_key]="stripe_customer_id" \
--customer-mapping[type]="by_id" \
--value-settings[event_payload_key]="value"
```
---
## Step 2 — Create the metered product + price
1. **Products → Create product** → "AI Overage".
2. **Add price**:
- **Pricing model:** `Usage-based`
- **Meter:** select the meter created in Step 1 (`mtr_…`).
- **Per-unit cost:** the per-cent charge (typically `0.01` USD
because the meter aggregates cents).
- **Billing period:** `Monthly` (match the tenant's plan).
3. Copy the resulting **Price id** (`price_…`).
---
## Step 3 — Migrate active subscriptions
For every active subscription that had a metered usage-record line in
SDK 13.x:
1. **Add a new subscription item** referencing the new price from
Step 2. Stripe API:
```bash
stripe subscription_items create \
--subscription="sub_..." \
--price="price_..." \
--quantity=0
```
2. **Remove the old subscription item** that referenced the
pre-migration metered price (or archive its price so no new
subscriptions can attach it).
3. Confirm: `stripe subscriptions retrieve sub_... --expand=items.data.price`
shows the new price and the meter id in `price.recurring.meter`.
The boilerplate's overage configuration uses `aiOverageStripeMeterId`
as the sole identifier (the legacy `aiOverageStripeSubscriptionItemId`
field was retired with the Stripe-19 cleanup). Existing tenant JSON
overrides that still carry the legacy key are silently ignored by the
parser.
---
## Step 4 — Wire the boilerplate
1. **Per-tenant override** (admin route
`/admin/settings/pricing/ai-overage`):
```json
{
"aiOverageStripeMeterId": "mtr_test_abc...",
"aiMarkupMultiplier": 2.0,
"aiOverageBlockCents": 10
}
```
2. **Per-plan default** — edit `packages/entitlements/src/ai-overage.ts`
`standardAIOverageDefaults` and bake the meter id into the plan
blueprint (operator-friendly admin UI is a future task).
3. **Worker bootstrap** — ensure the Stripe-19 client is constructed
with `apiVersion: "2025-10-29.clover"` and the worker injects a
`StripeMeterEventSubmitter` whose `submit()` calls
```ts
stripe.billing.meterEvents.create(
{
event_name: "ai-overage",
payload: {
stripe_customer_id: tenant.stripeCustomerId,
value: String(roundedBlocks),
},
identifier: idempotencyKey,
timestamp,
},
{ idempotencyKey },
);
```
See `apps/worker/src/jobs/ai-cost-stripe-report.ts` for the
pure-by-deps interface (`StripeMeterEventSubmitter`).
---
## Step 5 — Verify with a test event
1. Toggle `AI_OVERAGE_SUBMISSION_ENABLED=true`.
2. Manually emit a test event:
```bash
stripe billing meter_events create \
--event-name="ai-overage" \
--payload[stripe_customer_id]="cus_test123" \
--payload[value]="100" \
--identifier="manual-test-$(date +%s)"
```
3. Confirm aggregation:
```bash
stripe billing meters list_event_summaries mtr_... \
--customer="cus_test123" \
--start-time=$(date -v-1d +%s) \
--end-time=$(date +%s)
```
You should see a non-zero `aggregated_value` for the meter event
you just submitted.
4. End-to-end: run `pnpm --filter @toolctrl/worker test
ai-cost-stripe-report` to confirm the worker keeps passing its 13
unit tests against the new wire.
---
## Rollback
The Meters API is **not trivially reversible** — once events are
submitted, the legacy Usage-Records API cannot ingest them. The safe
rollback path is to leave already-submitted events in place and revert
the SDK to 13.x via Git revert (commit
`4b2350f0c5a098d3c2dd0778f087cac5910ec2f9` and its follow-ups). The
operator-side meter + price can stay in place — Stripe ignores them
when no subscription item references them.
---
## Open items
- Admin UI per-plan meter-id editor (currently only per-tenant via
`aiQuotaOverrideJson`).
- Auto-creation of the meter on first operator-bootstrap (currently
manual via Dashboard / CLI).
- CLI helper `toolctrl-cli ai-cost stripe-init` to scaffold the
meter + product + price idempotently — listed in
`docs/improvement-plan/05-business-features/ai-cost-phase-3.md`
follow-ups.
@@ -4,8 +4,8 @@
operators clone a single tenant's Postgres rows + S3 storage objects into a
PII-anonymized archive, then replay that archive into another environment.
This runbook is the operator-facing companion to the implementation spec at
`docs/improvement-plan/03-scaling-infrastructure/tenant-snapshot.md`.
This runbook is the operator-facing companion to the tenant snapshot
implementation in `packages/cli`.
---
@@ -36,7 +36,7 @@ privacy risk. The full mapping table lives in
| Phone number | `null` | `null` | (untouched) |
| IP address | `127.0.0.<seeded>` | `127.0.0.<seeded>` | (untouched) |
| ZIP code | first two digits kept, rest reseeded | first two digits kept, rest reseeded | (untouched) |
| Stripe customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
| Legacy-Provider customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
| Webhook / SSO / domain URL | `https://anonymized.test/<8 hex>` | same as full | (untouched) |
| Secrets, API keys, refresh tokens, payloads | `[REDACTED]` | `[REDACTED]` | (untouched) |
| Avatar URL | `https://anonymized.test/avatar/<8 hex>.png` | same as full | (untouched) |
@@ -54,9 +54,9 @@ anywhere; only its `sha256` lives in the manifest as a reproducibility marker.
### Create a snapshot (default profile, age-encrypted, upload to S3)
```bash
toolctrl-cli tenant snapshot \
hivegrid-cli tenant snapshot \
--tenant-id ten_abc123 \
--output s3://toolctrl-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
--output s3://hivegrid-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
```
The CLI requires `BACKUP_AGE_PUBLIC_KEY` (recipient public key) plus
@@ -66,9 +66,9 @@ catalog — the snapshot command reuses the same encryption layer.
### Restore into a dev box
```bash
NODE_ENV=development toolctrl-cli tenant restore \
NODE_ENV=development hivegrid-cli tenant restore \
--tenant-id ten_devclone \
--source s3://toolctrl-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
--source s3://hivegrid-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
```
The command refuses to run under `NODE_ENV=production` unless the operator
@@ -78,7 +78,7 @@ with `--yes` for non-interactive pipelines.
### List existing snapshots
```bash
toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
hivegrid-cli tenant list-snapshots --bucket hivegrid-snapshots
```
### Super-admin escape hatch (`--profile=none`)
@@ -86,11 +86,11 @@ toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
Raw, un-anonymized exports require an explicit confirmation flag:
```bash
toolctrl-cli tenant snapshot \
hivegrid-cli tenant snapshot \
--tenant-id ten_abc123 \
--profile none \
--i-am-the-super-admin \
--output s3://toolctrl-snapshots/migration/ten_abc123.tar.gz.age
--output s3://hivegrid-snapshots/migration/ten_abc123.tar.gz.age
```
Without `--i-am-the-super-admin`, the command exits non-zero with a clear
@@ -123,14 +123,14 @@ redirect that into their central audit log; set `auditFilePath` when calling
1. Snapshot a representative tenant from staging:
```bash
toolctrl-cli tenant snapshot --tenant-id <staging-tenant> \
hivegrid-cli tenant snapshot --tenant-id <staging-tenant> \
--profile full-anonymized \
--output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age
```
2. Copy to a dev box and restore into a fresh database:
```bash
NODE_ENV=development DATABASE_URL=postgres://... \
toolctrl-cli tenant restore \
hivegrid-cli tenant restore \
--tenant-id ten_drill_$(date +%s) \
--source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \
--yes
@@ -182,10 +182,10 @@ The restore command accepts one of three strategies (default
```bash
# Default — overlay snapshot rows onto whatever already exists.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
# Sensitive restore: refuse to proceed if any global is already there.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
--globals=error
```
@@ -53,7 +53,7 @@ short app/group/port summary and link back here.
`worker-plugin` is required as a persistent Coolify service so the deploy
topology is stable, but production plugin execution is still gated. With
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
as an idle topology placeholder and must not consume `toolctrl.plugin-invoke`.
as an idle topology placeholder and must not consume `hivegrid.plugin-invoke`.
Only after the external WASM sandbox signoff in
`docs/security/plugin-runtime-signoff.md` may operators set
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
@@ -66,7 +66,7 @@ separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
export, and scan services are wired into the worker bootstrap. In that idle
state `/readyz` may be healthy with an empty queue list, which proves topology
without consuming `toolctrl.imports` or `toolctrl.exports`. When both runtime
without consuming `hivegrid.imports` or `hivegrid.exports`. When both runtime
gates are set, startup fails closed unless the worker has real import, export,
and scan service dependencies.
@@ -74,11 +74,11 @@ and scan service dependencies.
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
| `mail` | `toolctrl.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
| `import` | `toolctrl.imports`, `toolctrl.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
| `plugin` | `toolctrl.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
| `import` | `hivegrid.imports`, `hivegrid.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
Source of truth: `packages/queue/src/groups.ts`. A test
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
@@ -117,11 +117,13 @@ dispatcher or bridge dependency fails startup instead of leaving outbox rows or
`outbox.workflow` silently undrained behind a green process.
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
call-sites stay absent and that only the signed internal trigger bridge may
start `type: "trigger"` executions directly. The `row.created`, `row.updated`,
and `row.deleted` route-backed and workflow-context producers now write the row
mutation, application event, and `workflow.trigger.event` outbox row in the same
transaction when a request or workflow event context is passed. Other internal
requestless mutations remain eventless by default to avoid worker feedback loops.
start `type: "trigger"` executions directly. The generic entity-builder's
`row.created` / `row.updated` / `row.deleted` route-backed producers were
**removed with the entity-builder** (R-P6B-β, AD-2); typed fachentities and
workflow row blocks (via the typed model registry) no longer emit generic
`row.*` outbox events. `workflow.trigger.event` application events still publish
through `EventsService`. Other internal requestless mutations remain eventless by
default to avoid worker feedback loops.
`worker-general` also registers the
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
@@ -146,7 +148,7 @@ outbox dispatcher routes to `outbox.form-submission-route` on
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
marks the submission, and publishes follow-up events. Webhook follow-ups use
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
`worker-mail`, where the bridge enqueues the existing `toolctrl.email`
`worker-mail`, where the bridge enqueues the existing `hivegrid.email`
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
throughput group and leaves the general group responsible only for the
database-backed routing decision.
@@ -290,7 +292,7 @@ Managed Coolify `worker-mail` apps must keep
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
`outbox.mail` cannot subscribe; without that bridge, invite and other
`mail.transactional.*` outbox events can remain pending even if the direct
`toolctrl.email` worker is alive. The Coolify worker-group provisioner writes
`hivegrid.email` worker is alive. The Coolify worker-group provisioner writes
and drift-checks this env value for managed mail workers.
Example: