Update hivegrid-cloud reference to latest main
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e98d38e716
commit
7b21524c86
3781 files changed
+373484
-225607
No files matched your search
@@ -1,6 +1,6 @@
|
||||
# Asset CDN Operations Runbook
|
||||
|
||||
When A7/B2/#51 is reactivated, front the boilerplate's static assets
|
||||
When the asset-CDN rollout is reactivated, front HiveGrid's static assets
|
||||
(`/assets/*`, `/fonts/*`, `/images/*`) with a CDN so the origin no longer pays
|
||||
the bandwidth + TLS handshake cost for every visitor and so users far from the
|
||||
Frankfurt DC see sub-second first paints.
|
||||
@@ -8,13 +8,10 @@ Frankfurt DC see sub-second first paints.
|
||||
**Default provider:** Bunny.net (EU-based, cheap, simple).
|
||||
**Documented alternative:** Cloudflare.
|
||||
|
||||
Related task: [`docs/improvement-plan/03-scaling-infrastructure/asset-cdn.md`](../improvement-plan/03-scaling-infrastructure/asset-cdn.md).
|
||||
|
||||
**Rollout status:** CDN provisioning and `ASSET_CDN_URL` flips remain parked
|
||||
with A7/B2/#51. The deploy workflow already includes a no-op-safe Bunny purge
|
||||
step for future activation; keep this runbook as the reactivation guide until
|
||||
`docs/improvement-plan/OPERATOR-CHECKLIST.md` and
|
||||
`docs/improvement-plan/PROGRESS.md` move #51 out of backlog.
|
||||
until a HiveGrid infra ticket explicitly reactivates them. The deploy workflow
|
||||
already includes a no-op-safe Bunny purge step for future activation; keep this
|
||||
runbook as the reactivation guide.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -63,9 +60,9 @@ response. Loaders may set their own `Cache-Control` to opt out of `no-store`
|
||||
Cloudflare and wants a single console — then Cloudflare is the lower-friction
|
||||
choice.
|
||||
|
||||
## Bunny.net setup (default, when A7/B2/#51 is reactivated)
|
||||
## Bunny.net setup (default, when the rollout is reactivated)
|
||||
|
||||
Do not execute these steps while A7/B2/#51 remains parked. They are the
|
||||
Do not execute these steps while the rollout remains parked. They are the
|
||||
reactivation checklist for the future CDN rollout. None of this is automated by
|
||||
the deploy pipeline.
|
||||
|
||||
@@ -123,7 +120,7 @@ the deploy pipeline.
|
||||
|
||||
7. **Trigger a deploy** and run the smoke tests below.
|
||||
|
||||
## Smoke tests (when A7/B2/#51 is reactivated)
|
||||
## Smoke tests (when the rollout is reactivated)
|
||||
|
||||
Do not run these as active acceptance checks while CDN rollout remains parked.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Backup CLI Runbook
|
||||
|
||||
Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and
|
||||
is callable as `toolctrl-cli backup <subcommand>` once the workspace is
|
||||
Operator guide for `@hivegrid/cli backup:*`. The CLI ships in `packages/cli` and
|
||||
is callable as `hivegrid-cli backup <subcommand>` once the workspace is
|
||||
installed.
|
||||
|
||||
## Quick reference
|
||||
@@ -21,16 +21,16 @@ bucket so a compromised app credential cannot pivot to the backups.
|
||||
|
||||
```bash
|
||||
# MinIO example (staging)
|
||||
mc mb staging-minio/toolctrl-backups
|
||||
mc anonymous set none staging-minio/toolctrl-backups
|
||||
mc mb staging-minio/hivegrid-backups
|
||||
mc anonymous set none staging-minio/hivegrid-backups
|
||||
|
||||
# AWS S3 example
|
||||
aws s3api create-bucket --bucket toolctrl-backups \
|
||||
aws s3api create-bucket --bucket hivegrid-backups \
|
||||
--create-bucket-configuration LocationConstraint=us-east-1
|
||||
aws s3api put-bucket-encryption --bucket toolctrl-backups \
|
||||
aws s3api put-bucket-encryption --bucket hivegrid-backups \
|
||||
--server-side-encryption-configuration \
|
||||
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
|
||||
aws s3api put-public-access-block --bucket toolctrl-backups \
|
||||
aws s3api put-public-access-block --bucket hivegrid-backups \
|
||||
--public-access-block-configuration \
|
||||
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
|
||||
```
|
||||
@@ -49,13 +49,13 @@ The default encryption method is [age](https://age-encryption.org).
|
||||
|
||||
```bash
|
||||
# Generate a key pair on a trusted workstation
|
||||
age-keygen -o ~/.age-toolctrl-backup.key
|
||||
age-keygen -o ~/.age-hivegrid-backup.key
|
||||
# Output:
|
||||
# Public key: age1example...
|
||||
# Identity (private key) is at ~/.age-toolctrl-backup.key
|
||||
# Identity (private key) is at ~/.age-hivegrid-backup.key
|
||||
|
||||
# Set the public key in the deployment env
|
||||
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env
|
||||
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/hivegrid/backup.env
|
||||
```
|
||||
|
||||
**Store the identity file (private key) somewhere the running app does not have
|
||||
@@ -75,7 +75,7 @@ If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and
|
||||
# Backup bucket
|
||||
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
|
||||
BACKUP_S3_REGION=us-east-1
|
||||
BACKUP_S3_BUCKET=toolctrl-backups
|
||||
BACKUP_S3_BUCKET=hivegrid-backups
|
||||
BACKUP_S3_ACCESS_KEY_ID=AKIA...
|
||||
BACKUP_S3_SECRET_ACCESS_KEY=...
|
||||
BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style
|
||||
@@ -101,11 +101,11 @@ PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback.
|
||||
## 4. Daily cron
|
||||
|
||||
```cron
|
||||
# /etc/cron.d/toolctrl-backup
|
||||
# /etc/cron.d/hivegrid-backup
|
||||
SHELL=/bin/sh
|
||||
PATH=/usr/local/bin:/usr/bin:/bin
|
||||
MAILTO=ops@example.com
|
||||
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1
|
||||
0 3 * * * deploy /opt/hivegrid/scripts/cron/backup-daily.sh >> /var/log/hivegrid-backup.log 2>&1
|
||||
```
|
||||
|
||||
Example wrapper:
|
||||
@@ -113,9 +113,9 @@ Example wrapper:
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd /opt/toolctrl
|
||||
node ./packages/cli/bin/toolctrl-cli.mjs backup create
|
||||
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate
|
||||
cd /opt/hivegrid
|
||||
node ./packages/cli/bin/hivegrid-cli.mjs backup create
|
||||
node ./packages/cli/bin/hivegrid-cli.mjs backup rotate
|
||||
```
|
||||
|
||||
For docker-compose deployments:
|
||||
@@ -140,15 +140,15 @@ Run **at least monthly**:
|
||||
|
||||
```bash
|
||||
# 1. Pick yesterday's backup
|
||||
toolctrl-cli backup list
|
||||
hivegrid-cli backup list
|
||||
|
||||
# 2. Verify in-place (creates verify_<uuid>, drops it after)
|
||||
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
|
||||
hivegrid-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
|
||||
|
||||
# 3. End-to-end DR drill (restore into a scratch host)
|
||||
DATABASE_URL=postgres://dr-test/restored \
|
||||
BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
|
||||
toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
|
||||
hivegrid-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
|
||||
```
|
||||
|
||||
Document the drill outcome in the incident channel or your internal ops ledger.
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Billing E2E Sandbox-Flow
|
||||
|
||||
Stand: 2026-07-05
|
||||
|
||||
Dieses Runbook dokumentiert den reproduzierbaren BILL-07-Proof fuer den v1-Billing-Kern. Der Standardlauf nutzt
|
||||
lokale Sandbox-Stubs fuer Lago und Mollie, fuehrt aber dieselbe HiveGrid-Sync-Schicht aus, die auch Provider-Webhooks
|
||||
in den lokalen Billing-State spiegelt.
|
||||
|
||||
## Ziel
|
||||
|
||||
Der Flow beweist:
|
||||
|
||||
- Test-Tenant wird als Lago-Customer angelegt.
|
||||
- Plan + Seat-Subscription wird in Lago angelegt.
|
||||
- Mollie-Zahlung wird erzeugt und als bezahlt simuliert.
|
||||
- Provider-Sync aktualisiert `TenantSubscription` lokal.
|
||||
- Entitlement-/Modul-Gate und Read-only-State reagieren auf den lokalen Billing-State.
|
||||
- Invoice- und Payment-Referenzen sind sichtbar.
|
||||
- Sync-Audit und Activity-Eintraege werden geschrieben.
|
||||
- Negativfaelle `failed`, `expired` und `frozen` setzen den Tenant read-only.
|
||||
|
||||
## Stub-Proof ausfuehren
|
||||
|
||||
Vom Repo-Root:
|
||||
|
||||
```bash
|
||||
pnpm exec tsx scripts/billing-e2e-sandbox.ts --run-id=manual-$(date -u +%Y%m%d%H%M%S)
|
||||
```
|
||||
|
||||
Erwartung:
|
||||
|
||||
- JSON-Ausgabe mit `"ok": true`.
|
||||
- `steps[]` enthaelt acht erfolgreiche Schritte.
|
||||
- `activeState.status` ist `active`.
|
||||
- `activeState.readOnly` ist `false`.
|
||||
- `gate.activeAllowed` ist `true`.
|
||||
- `failedState.status` ist `past_due` und `failedState.readOnly` ist `true`.
|
||||
- `pastDueState.status` ist `past_due` und `pastDueState.readOnly` ist `true`.
|
||||
- `frozenState.status` ist `frozen` und `frozenState.readOnly` ist `true`.
|
||||
- `referencesVisible` ist `true`.
|
||||
- `audit.auditEvents` und `audit.activities` sind groesser/gleich `5`.
|
||||
|
||||
Der Lauf schreibt nicht in die echte Datenbank. Er nutzt deterministische In-Memory-Stubs fuer Provider und DB-Client,
|
||||
ruft aber `applyLagoSubscriptionSync`, `applyMolliePaymentSync` und `resolveTenantBillingState` aus der echten
|
||||
HiveGrid-Codebase auf.
|
||||
|
||||
## Automatischer Test
|
||||
|
||||
```bash
|
||||
pnpm --dir packages/billing test -- sandbox-flow.test.ts
|
||||
```
|
||||
|
||||
Dieser Test ist CI-tauglich, weil keine externen Lago-/Mollie-Credentials gebraucht werden.
|
||||
|
||||
## Testdaten-Konvention fuer echte Sandbox
|
||||
|
||||
Falls der Flow spaeter gegen `lago-staging` erweitert wird:
|
||||
|
||||
- Customer external ID: `hg-stg-bill-07-<short-id>`
|
||||
- Plan: `hg-stg-plan-bill-07-<short-id>`
|
||||
- Subscription external ID: `subscription-<short-id>`
|
||||
- Payment external ID: `payment-<short-id>`
|
||||
|
||||
Zugang und Tunnel fuer `lago-staging` stehen in `docs/ops/lago-staging.md`.
|
||||
|
||||
## Live-Sandbox-Erweiterung
|
||||
|
||||
Der urspruengliche BILL-07-Abschluss war provider-stubbed. Seit 2026-07-04 ist die echte Staging-Verbindung
|
||||
vorbereitet:
|
||||
|
||||
- `lago-staging` ist initialisiert.
|
||||
- HiveGrid-Staging erreicht Lago ueber `http://10.42.0.4:3108`.
|
||||
- HiveGrid-Staging hat `LAGO_*` und `MOLLIE_*` Runtime-Env auf API/Web/Worker.
|
||||
- Provider-Smoke aus dem laufenden `hivegrid-api`-Container: Lago HTTP 200, Mollie HTTP 200.
|
||||
- Synthetischer Lago-Customer und Mollie-Testpayment koennen aus Staging erzeugt werden.
|
||||
- `POST /v1/billing/mollie/webhooks` ist im API-Runtime registriert. Die Route nimmt Mollies `id` entgegen,
|
||||
refetcht die Zahlung serverseitig bei Mollie und schreibt den lokalen Billing-State ueber
|
||||
`applyMolliePaymentSync`. Fuer Self-Service-Checkout-Zahlungen mit Plan-Metadaten schreibt sie zusaetzlich den
|
||||
lokalen Lago-/HiveGrid-Subscription-State ueber `applyLagoSubscriptionSync`.
|
||||
|
||||
Erledigt fuer den manuellen Staging-E2E:
|
||||
|
||||
- Abschluss eines echten Mollie-Test-Checkout-Flows gegen die echte Staging-Tenant `demo-pro-crm`.
|
||||
- Mollie-Payment `tr_Nn3WsWmtkTXXYRCenmVTJ` wurde bezahlt, Webhook kam mit HTTP 200 an,
|
||||
Tenant-Billing-Status steht lokal auf `active`.
|
||||
- Staging-Admin fuer interne Billing-Pruefung ist unter `https://staging-admin.hivegrid.de/accounts/billing`
|
||||
erreichbar.
|
||||
- Lago-Staging hat echte Demo-Rechnungen fuer `demo-starter-crm`, `demo-pro-crm` und `demo-business-crm`.
|
||||
Der HiveGrid-Kundenbereich `/:tenant/settings/subscription` zeigt die Rechnungen fuer alle drei Demo-Tenants an.
|
||||
- Lago-Staging speichert neue Invoice-PDFs im Hetzner-Bucket `hivegrid-staging-lago`; der Kunden-Download laeuft
|
||||
ueber HiveGrid als serverseitiger Proxy nach Permission-/Tenant-Pruefung.
|
||||
- Die alten lokalen Demo-Invoices `*-001` wurden in Lago voided; aktuelle S3-Demo-Invoices sind `*-002`.
|
||||
- Billing entity, Logo, German document locale, 14-day payment terms and the neutral Mollie footer are configured in
|
||||
Lago staging.
|
||||
- Enriched demo invoice `HIV-6711-002-008` for `demo-pro-crm` verifies the current customer-facing PDF shape through
|
||||
HiveGrid's download route.
|
||||
- Die funktionale Self-Service-Basis ist in HiveGrid verdrahtet: `/:tenant/settings/subscription` kann oeffentliche
|
||||
Plaene/Seats auswaehlen, erstellt Lago-Customer/Subscription, startet Mollie und verarbeitet Return/Webhook
|
||||
idempotent.
|
||||
|
||||
Offen fuer den vollautomatischen Staging-E2E:
|
||||
|
||||
- Cleanup-Schritt fuer Lago-/Mollie-Testdaten.
|
||||
- Wiederholbarer Script-Schritt fuer echte Lago-Plan-/Subscription-/Invoice-Testdaten.
|
||||
- Browser-Smoke fuer den echten Self-Service-Checkout gegen Staging-Testdaten.
|
||||
|
||||
Bis dahin bleibt der Stub-Proof der kanonische, wiederholbare Regressionstest fuer den HiveGrid-Billing-Kern; der
|
||||
Live-Smoke beweist zusaetzlich Provider-Reachability, Checkout-Link-Erzeugung und die API-WebHook-Grenze aus Staging.
|
||||
@@ -0,0 +1,64 @@
|
||||
# CI-Runner auf node1 + CI-Pipeline-Overhaul (2026-06-28)
|
||||
|
||||
> Operativer Record der CI-Umstellung. Runner-Host-Themen gehören mittelfristig auch nach
|
||||
> `infra/LESSONS.md` (Infra-Repo, Akeshan-Domain) — hier dokumentiert, weil die
|
||||
> `.gitea/workflows/`-Änderungen in diesem Repo liegen.
|
||||
|
||||
## Warum: Homeserver OOMte unter paralleler CI
|
||||
Der geteilte Homeserver (32 GB RAM, neben anderen Diensten ~17 GB belegt) ist bei paralleler
|
||||
CI wiederholt abgestürzt: parallele Jobs spikten den RAM auf 30+/32 GB, CPU auf 100%, der
|
||||
Linux-OOM-Killer riss den größten Job → Crash-Recover-Schleife (Runner grabbte sofort den
|
||||
nächsten Job). Parallelisierung auf der geteilten Box war der falsche Hebel.
|
||||
|
||||
## Lösung: node1 als dedizierter CI-Runner
|
||||
node1 (`176.9.111.179`, eigener Hetzner-Dedi mit self-hosted Sentry, **20 Kerne / 62 GB**)
|
||||
ist Mesh-Peer und erreicht Gitea über `10.42.0.2`. Gitea-Server bleibt, wo er ist — nur die
|
||||
Runner-Last zieht um.
|
||||
|
||||
- Compose: `/opt/stacks/gitea-ci-runner/docker-compose.yml` (`network_mode: host`,
|
||||
`--add-host git.tool-ctrl.de:10.42.0.2`, `CONFIG_FILE=/data/config.yaml`).
|
||||
- Config: `/srv/data/gitea-ci-runner/config.yaml` (capacity, labels, cache.dir).
|
||||
- Homeserver-Runner steht auf `restart=no` (aus, kein Auto-Restart, keine OOM-Schleife).
|
||||
|
||||
### act_runner-Footguns (alle real getroffen)
|
||||
| Symptom | Ursache | Fix |
|
||||
|---|---|---|
|
||||
| `runner.capacity` wird ignoriert | run.sh überschreibt den compose-`command`, läuft ohne `--config` | env `CONFIG_FILE=/data/config.yaml` setzen |
|
||||
| "no matching runner online with label toolctrl-ci" | leere `runner.labels` → KEIN Fallback auf `.runner` | Labels EXPLIZIT in config.yaml |
|
||||
| Daemon "token is empty" trotz erfolgreicher Registrierung | `act_runner register` schrieb `.runner` ins Container-CWD → bei `--rm` weg | `docker run -w /data ...` beim Register |
|
||||
| Register "instance address is empty" | run.sh ignoriert CLI-Args, nutzt leeres `GITEA_INSTANCE_URL` | `--entrypoint act_runner` umgeht run.sh |
|
||||
| pnpm-Store-Cache greift nie (Cache-Dir 12K) | `cache.dir` default ephemer im Container, Restarts wischen ihn | `cache.dir: /data/actcache` (gemountet) |
|
||||
|
||||
## Capacity-Findings (empirisch gemessen)
|
||||
| Setup | Wallclock | RAM-Peak frei | Notiz |
|
||||
|---|---|---|---|
|
||||
| cap 4, docker-smoke auf Tail | ~10m | 35 GB | |
|
||||
| cap 6 | ~8m | 23-25 GB | |
|
||||
| cap 6 + docker-smoke parallel | ~7m35s | 23 GB | |
|
||||
| **cap 8 + docker-smoke parallel** | **~6m53s** | **25 GB** | Sweet-Spot |
|
||||
|
||||
**cap 8 ist schneller als cap 6** trotz mehr Contention: Install/Netzwerk/Docker sind
|
||||
I/O-bound, die 20 Kerne tragen 8 Jobs; die Queueing-Reduktion schlägt die Mehr-Contention.
|
||||
cap >8 bringt nichts (Kerne saturiert). Reise gesamt: **~32m → ~6m53s**.
|
||||
|
||||
## Pipeline-Änderungen (.gitea/workflows/)
|
||||
- **docker-smoke `needs` entfernt** → läuft parallel zur Quality-Phase (baut Images aus
|
||||
Source, keine Cross-Job-Artefakt-Abhängigkeit), raus aus dem kritischen Tail.
|
||||
- **Sentry-Release-Jobs aus ci.yml entfernt** (ci.yml läuft nur auf `pull_request`; ein
|
||||
PR-Build wird nirgends deployed → die Release/Sourcemaps mappen auf keine Deployment).
|
||||
Release passiert in `deploy.yml` (push→main / promote).
|
||||
|
||||
## Install-Diagnose (warum KEIN Deps-Image)
|
||||
Gemessen: Install = **~43s Download + ~15s Link** (warmer Store). Die ~2-min-Annahme war zu
|
||||
hoch. Ein Deps-Image würde nur die 15s Link sparen → verworfen. Der Hebel war der **kalte
|
||||
Store** (Cache persistierte nicht) → mit `cache.dir` gefixt.
|
||||
|
||||
## Offene Punkte
|
||||
- **Sourcemaps im Deploy:** web nutzt `@sentry/vite-plugin` (lädt Sourcemaps beim Build hoch,
|
||||
wenn `SENTRY_AUTH_TOKEN` gesetzt). To-do: Token als BuildKit-`--secret` an den
|
||||
`web-build`-Dockerfile-Stage durchreichen (NICHT build-arg → kein Leak) +
|
||||
`SENTRY_PROJECT=web` explizit (Default ist alter Slug `hivegrid-web`). Nur auf echtem
|
||||
Deploy testbar → bis Sentry-Go-Live aufgeschoben.
|
||||
- **Persistenter Store-Cache:** `cache.dir` gesetzt + befüllt (2,9 GB), aber Hit-Gewinn bei
|
||||
cap 8 noch ungemessen (Per-Job-Restore-I/O könnte Download-Gewinn auffressen → ggf. auf
|
||||
geteilten gemounteten Store umstellen statt actions/cache).
|
||||
@@ -0,0 +1,150 @@
|
||||
# Lago Production Prep Runbook
|
||||
|
||||
This runbook defines the production target for HiveGrid billing after AD-27.
|
||||
It is a preparation document only. Do not provision or enable production billing
|
||||
from this document until the external gates in `EXT-16` are complete.
|
||||
|
||||
## Scope
|
||||
|
||||
`lago-production` will be the dedicated production Lago service for HiveGrid.
|
||||
It must be separate from `lago-staging` and must never reuse staging database,
|
||||
Redis, storage, API keys, webhook secrets, customers, plans or invoice records.
|
||||
|
||||
Production billing owns:
|
||||
|
||||
- Lago customers, plans, seats, subscriptions and invoice records.
|
||||
- Invoice PDF generation and storage.
|
||||
- Billing status used by HiveGrid entitlement and read-only gates.
|
||||
- Mollie payment references for paid, failed, expired, cancelled and refunded
|
||||
payments.
|
||||
|
||||
Out of scope for v1:
|
||||
|
||||
- Usage billing.
|
||||
- Credits.
|
||||
- Add-on metering.
|
||||
- Public price promises in code.
|
||||
- Live activation before legal, tax and Mollie KYC approval.
|
||||
|
||||
## Target Topology
|
||||
|
||||
| Item | Decision |
|
||||
|---|---|
|
||||
| Coolify project | `HiveGrid` |
|
||||
| Coolify environment | `production` |
|
||||
| Resource name | `lago-production` |
|
||||
| Initial host | The current HiveGrid production host, while Hetzner capacity is constrained |
|
||||
| Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted |
|
||||
| Public access | None by default |
|
||||
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
||||
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
||||
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
||||
|
||||
The initial host decision is intentionally pragmatic: production gets its own
|
||||
Lago resource and data stores, but it may share the physical server with other
|
||||
HiveGrid production services until infrastructure capacity allows a stricter
|
||||
split. The isolation boundary is the Coolify resource, secrets, database,
|
||||
Redis, storage and backup path, not the current machine count.
|
||||
|
||||
## Required Production Secrets
|
||||
|
||||
Store these only in the runtime secret store. Never put values in docs, Git,
|
||||
tickets, screenshots or copied logs.
|
||||
|
||||
| Secret | Owner | Notes |
|
||||
|---|---|---|
|
||||
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
||||
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
||||
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
||||
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
||||
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
||||
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
|
||||
Do not reuse staging/testmode keys in production. Rotate keys when moving from
|
||||
pilot validation to real customer billing.
|
||||
|
||||
## Backup And Restore
|
||||
|
||||
Production Lago is stateful. It must not receive production traffic before a
|
||||
backup and restore path has been verified.
|
||||
|
||||
Minimum backup scope:
|
||||
|
||||
- Lago Postgres database.
|
||||
- Lago Redis persistence if configured for durable state.
|
||||
- Invoice PDF storage volume or bucket.
|
||||
- Coolify resource settings and image tags.
|
||||
- Runtime secret names and rotation history, without secret values.
|
||||
|
||||
Minimum backup schedule:
|
||||
|
||||
- Daily encrypted database dump.
|
||||
- Backup before every Lago image upgrade.
|
||||
- Backup before destructive billing test data cleanup.
|
||||
- Retention aligned with the final accounting and tax retention decision.
|
||||
|
||||
Restore drill before go-live:
|
||||
|
||||
1. Create a fresh disposable restore target.
|
||||
2. Restore the latest Lago Postgres dump.
|
||||
3. Restore or attach invoice PDF storage.
|
||||
4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
|
||||
5. Verify `/health`.
|
||||
6. Read one known customer, subscription and invoice.
|
||||
7. Verify HiveGrid can resolve the corresponding local billing state without
|
||||
writing test payments.
|
||||
|
||||
The restore drill date, operator and result must be recorded before production
|
||||
billing is enabled.
|
||||
|
||||
## Access And Domain
|
||||
|
||||
Default: no public Lago domain.
|
||||
|
||||
Allowed access paths:
|
||||
|
||||
- Internal service-to-service URL for HiveGrid runtime.
|
||||
- SSH tunnel for operators.
|
||||
- Authentik-gated admin URL later, only if operator UX requires browser access.
|
||||
|
||||
Do not expose Lago directly to customers. Customer-facing billing screens must
|
||||
remain HiveGrid-owned and use the server-side Lago/Mollie gateways.
|
||||
|
||||
## Mollie Live Gate
|
||||
|
||||
Mollie live activation is blocked until:
|
||||
|
||||
- Legal entity / firming is final.
|
||||
- Business address, bank account and KYC documents are accepted by Mollie.
|
||||
- Rechnungssteller, tax handling, imprint and customer contract data are
|
||||
approved.
|
||||
- Live webhook endpoint and return URLs are set for the production host.
|
||||
- A live low-value payment and refund rehearsal is approved by an operator.
|
||||
|
||||
Until then, production must stay on manual billing or disabled self-service
|
||||
checkout.
|
||||
|
||||
## Rollback
|
||||
|
||||
Rollback has two layers:
|
||||
|
||||
1. **HiveGrid application rollback:** deploy an older immutable HiveGrid image
|
||||
tag through Coolify. This must not rewrite Lago state.
|
||||
2. **Lago service rollback:** roll back Lago image tags and restore the pre-
|
||||
upgrade Lago database/storage backup when the Lago upgrade mutated state.
|
||||
|
||||
Never restore Lago production from staging. Never merge staging and production
|
||||
Lago datasets.
|
||||
|
||||
## Go-Live Checklist
|
||||
|
||||
- [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
||||
- [ ] Dedicated production DB, Redis and storage are configured.
|
||||
- [ ] Production secrets are set and staging secrets are absent.
|
||||
- [ ] Backup job is configured.
|
||||
- [ ] Restore drill is completed and documented.
|
||||
- [ ] Mollie live profile is verified.
|
||||
- [ ] Rechnungssteller, tax and imprint data are approved.
|
||||
- [ ] Web/API/worker health checks pass with production Lago config.
|
||||
- [ ] Manual rollback path is tested from image tag and Lago backup.
|
||||
@@ -0,0 +1,389 @@
|
||||
# Lago Staging Operations Runbook
|
||||
|
||||
This runbook covers the shared `lago-staging` service used for HiveGrid billing
|
||||
integration tests. It is a staging-only billing engine for the `BILL-*` track,
|
||||
not a production billing system.
|
||||
|
||||
## Scope
|
||||
|
||||
`lago-staging` exists to validate:
|
||||
|
||||
- Lago customers, plans, seats, subscriptions and invoice records.
|
||||
- Mollie payment integration against test references.
|
||||
- HiveGrid billing-state sync and entitlement gates.
|
||||
|
||||
It must not contain production tenants, production invoices or real payment
|
||||
data. Production gets a separate `lago-production` service later.
|
||||
|
||||
## Current topology
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Coolify project | `node1-estate` |
|
||||
| Resource | `lago-staging` |
|
||||
| Resource UUID | `kh7l869ianltiioqgmm45kur` |
|
||||
| Server | `node1` |
|
||||
| Operator UI | `https://staging-lago.hivegrid.de` |
|
||||
| Public protection | Lago login |
|
||||
| Frontend tunnel | `http://127.0.0.1:8188` |
|
||||
| API tunnel | `http://127.0.0.1:3108` |
|
||||
| Mesh API URL for HiveGrid staging | `http://10.42.0.4:3108` |
|
||||
| Invoice PDF storage | Hetzner Object Storage bucket `hivegrid-staging-lago` |
|
||||
|
||||
Services in the stack:
|
||||
|
||||
- Lago API
|
||||
- Lago frontend
|
||||
- Postgres
|
||||
- Redis
|
||||
- Worker
|
||||
- Billing worker
|
||||
- PDF service
|
||||
|
||||
Invoice PDFs are stored in the dedicated staging object-storage bucket
|
||||
`hivegrid-staging-lago`. Lago API, worker and billing worker run with
|
||||
`LAGO_USE_AWS_S3=true` and Hetzner's S3-compatible endpoint. Customer-facing
|
||||
downloads must still go through HiveGrid's
|
||||
`/:tenant/settings/subscription/invoices/:invoiceId/download` route; HiveGrid
|
||||
checks tenant permission/ownership and streams the PDF server-side instead of
|
||||
redirecting customers to Lago or the bucket origin.
|
||||
|
||||
## Access model
|
||||
|
||||
Operator browser access is available at:
|
||||
|
||||
```text
|
||||
https://staging-lago.hivegrid.de
|
||||
```
|
||||
|
||||
The public route is protected by the Lago application login.
|
||||
|
||||
Lago operator credentials are stored outside Git in
|
||||
`~/.cli-tokens/lago-staging`.
|
||||
|
||||
The Lago API remains internal for HiveGrid staging. Do not expose
|
||||
`http://10.42.0.4:3108` publicly; HiveGrid reaches it through the WireGuard
|
||||
mesh.
|
||||
|
||||
SSH tunnel access is still a valid fallback:
|
||||
|
||||
Example tunnel from an operator workstation:
|
||||
|
||||
```bash
|
||||
ssh -N \
|
||||
-L 8188:127.0.0.1:8188 \
|
||||
-L 3108:127.0.0.1:3108 \
|
||||
root@node1
|
||||
```
|
||||
|
||||
Then verify:
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:3108/health
|
||||
curl -I http://127.0.0.1:8188
|
||||
```
|
||||
|
||||
Expected API health shape: JSON with `"message":"Success"`.
|
||||
|
||||
## HiveGrid staging runtime connection
|
||||
|
||||
HiveGrid staging runs on `staging-1` (`10.42.0.3`), while `lago-staging`
|
||||
runs on `node1` (`10.42.0.4`). The two services do not share a Docker
|
||||
network. HiveGrid staging therefore reaches Lago through the WireGuard mesh:
|
||||
|
||||
```text
|
||||
HiveGrid staging container -> http://10.42.0.4:3108 -> node1 local Lago API
|
||||
```
|
||||
|
||||
On `node1`, a mesh-only systemd proxy exposes the local Lago API:
|
||||
|
||||
```bash
|
||||
systemctl status lago-staging-mesh-proxy.service
|
||||
```
|
||||
|
||||
The service binds only to `10.42.0.4:3108` and forwards to
|
||||
`127.0.0.1:3108`, where the Coolify Docker port mapping for Lago API listens.
|
||||
The public operator domain routes only to the Lago frontend and does not make
|
||||
the mesh API public.
|
||||
|
||||
HiveGrid staging resources currently need these runtime variables:
|
||||
|
||||
- `LAGO_API_URL=http://10.42.0.4:3108`
|
||||
- `LAGO_API_KEY`
|
||||
- `LAGO_WEBHOOK_SECRET`
|
||||
- `LAGO_TIMEOUT_MS`
|
||||
- `MOLLIE_API_KEY`
|
||||
- `MOLLIE_TIMEOUT_MS`
|
||||
|
||||
Configured resources:
|
||||
|
||||
- `hivegrid-api`
|
||||
- `hivegrid-web`
|
||||
- `worker-general`
|
||||
- `worker-webhook`
|
||||
|
||||
The Lago organization and API key are initialized for staging only. Local
|
||||
operator credentials are stored outside Git in `~/.cli-tokens/lago-staging`.
|
||||
The HiveGrid staging webhook secret is stored outside Git in
|
||||
`~/.cli-tokens/hivegrid-staging-billing`.
|
||||
|
||||
## Live staging smoke
|
||||
|
||||
Current verified smoke path:
|
||||
|
||||
1. From the running `hivegrid-api` staging container, call Lago via
|
||||
`LAGO_API_URL`.
|
||||
2. Create a synthetic Lago customer with prefix `hg-stg-manual-`.
|
||||
3. Call Mollie testmode via `MOLLIE_API_KEY`.
|
||||
4. Create a synthetic `EUR 1.00` Mollie payment and read the checkout URL.
|
||||
|
||||
2026-07-04 proof:
|
||||
|
||||
- Lago API from `hivegrid-api`: HTTP 200.
|
||||
- Mollie payments API from `hivegrid-api`: HTTP 200.
|
||||
- Synthetic Lago customer created with prefix `hg-stg-manual-`.
|
||||
- Synthetic Mollie payment created with status `open`.
|
||||
|
||||
2026-07-04 live checkout proof:
|
||||
|
||||
- Mollie test checkout completed against real staging tenant `demo-pro-crm`.
|
||||
- Mollie payment `tr_Nn3WsWmtkTXXYRCenmVTJ` reached status `paid`.
|
||||
- Mollie called HiveGrid staging webhook with HTTP 200.
|
||||
- HiveGrid updated the tenant billing state to `active`.
|
||||
- Direct webhook replay returned `replayed: true` and `billingStatus: active`.
|
||||
|
||||
2026-07-05 billing-entity and invoice-template proof:
|
||||
|
||||
- Default billing entity is configured with HiveGrid staging company data,
|
||||
German document locale and `14` net payment days.
|
||||
- Billing entity logo is served through the public Lago ActiveStorage route.
|
||||
- Organization invoice footer is intentionally payment-neutral because Mollie
|
||||
may already charge a stored card or mandate:
|
||||
|
||||
```text
|
||||
Zahlung: Automatischer Einzug per Mollie, sofern Mandat oder Karte hinterlegt ist.
|
||||
Ohne automatischen Einzug bitte innerhalb von 14 Tagen mit Rechnungsnummer zahlen.
|
||||
Fragen: billing@hivegrid.de · Leistungszeitraum: Juli 2026
|
||||
```
|
||||
|
||||
- Created enriched demo invoice `HIV-6711-002-008` for `demo-pro-crm`.
|
||||
- HiveGrid customer download route returned `200 application/pdf` for:
|
||||
|
||||
```text
|
||||
https://staging-app.hivegrid.de/demo-pro-crm/settings/subscription/invoices/41c8e519-d871-4ead-9a0f-6b33fc592e71/download
|
||||
```
|
||||
|
||||
- PDF text extraction verified the updated footer, invoice number and customer
|
||||
data in the generated PDF.
|
||||
|
||||
2026-07-04 customer invoice proof:
|
||||
|
||||
- Root cause for the empty customer invoice UI was missing Lago data: Lago
|
||||
staging had no plans, subscriptions or invoices yet.
|
||||
- Created Lago staging plans `starter`, `pro` and `business`.
|
||||
- Created Lago customers/subscriptions for the demo tenants
|
||||
`demo-starter-crm`, `demo-pro-crm` and `demo-business-crm`.
|
||||
- Lago generated one finalized `EUR 1.00` invoice per demo tenant:
|
||||
`HIV-6711-003-001`, `HIV-6711-002-001`, `HIV-6711-004-001`.
|
||||
- Verified the customer UI at `/:tenant/settings/subscription` for all three
|
||||
demo tenants: invoice number present, empty state absent.
|
||||
|
||||
2026-07-04 object-storage cutover proof:
|
||||
|
||||
- Created dedicated bucket `hivegrid-staging-lago`.
|
||||
- Switched Lago staging API/worker/billing-worker from local ActiveStorage
|
||||
volume to Hetzner S3-compatible storage.
|
||||
- Switched Lago public URLs from `127.0.0.1:3108` to
|
||||
`https://staging-lago.hivegrid.de`.
|
||||
- Created synthetic Lago invoice `HIV-6711-005-001`; a new PDF object appeared
|
||||
in `hivegrid-staging-lago`.
|
||||
- HiveGrid download route was changed to proxy/stream PDFs after permission and
|
||||
tenant ownership checks, because public `/rails/active_storage/...` routes
|
||||
are not a customer contract.
|
||||
- The pre-S3 demo invoices `HIV-6711-002-001`, `HIV-6711-003-001` and
|
||||
`HIV-6711-004-001` were voided because their blobs were created on local
|
||||
ActiveStorage. Fresh S3-backed demo invoices are `HIV-6711-002-002`,
|
||||
`HIV-6711-003-002` and `HIV-6711-004-002`.
|
||||
|
||||
The internal staging admin surface is available at:
|
||||
|
||||
```text
|
||||
https://staging-admin.hivegrid.de
|
||||
```
|
||||
|
||||
Billing status can be inspected at:
|
||||
|
||||
```text
|
||||
https://staging-admin.hivegrid.de/accounts/billing
|
||||
```
|
||||
|
||||
Admin-host routes do not use the legacy `/admin` prefix.
|
||||
|
||||
Webhook endpoint for staging:
|
||||
|
||||
```text
|
||||
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
|
||||
```
|
||||
|
||||
For a direct smoke, post Mollie's payment id as form data. The API will refetch
|
||||
the payment from Mollie before any local billing write:
|
||||
|
||||
```bash
|
||||
curl -fsS -X POST \
|
||||
-H 'content-type: application/x-www-form-urlencoded' \
|
||||
--data-urlencode 'id=<mollie-payment-id>' \
|
||||
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
|
||||
```
|
||||
|
||||
## Start, stop and restart
|
||||
|
||||
Prefer the Coolify UI for routine operations:
|
||||
|
||||
1. Open Coolify.
|
||||
2. Go to project `node1-estate`.
|
||||
3. Open resource `lago-staging`.
|
||||
4. Use Start, Stop, Restart or Redeploy from the resource screen.
|
||||
5. Verify API and frontend through the SSH tunnel.
|
||||
|
||||
For API automation, use the Coolify token from the operator token store. Do not
|
||||
paste tokens into shell history or docs.
|
||||
|
||||
## Public operator route
|
||||
|
||||
The route `https://staging-lago.hivegrid.de` is configured through
|
||||
Cloudflare DNS and the Coolify Traefik dynamic config on `node1`:
|
||||
|
||||
```text
|
||||
/data/coolify/proxy/dynamic/lago-staging.yaml
|
||||
```
|
||||
|
||||
The Lago frontend must use the public same-origin API URL:
|
||||
|
||||
```text
|
||||
front.API_URL=https://staging-lago.hivegrid.de
|
||||
```
|
||||
|
||||
Traefik routes `/graphql`, `/api` and `/rails/active_storage` on that host to
|
||||
the Lago API container. The ActiveStorage route is required for uploaded
|
||||
billing entity logos and other Lago blob URLs. Do not set the frontend
|
||||
`API_URL` to `127.0.0.1`; browsers would then post login requests to the
|
||||
operator workstation instead of the server.
|
||||
|
||||
Expected checks:
|
||||
|
||||
```bash
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' https://staging-lago.hivegrid.de
|
||||
# 200
|
||||
|
||||
curl -sS -H 'content-type: application/json' \
|
||||
--data '{"query":"{ __typename }"}' \
|
||||
https://staging-lago.hivegrid.de/graphql
|
||||
# {"data":{"__typename":"Query"}}
|
||||
|
||||
curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' \
|
||||
https://staging-lago.hivegrid.de/rails/active_storage/healthcheck
|
||||
# 404 text/html; charset=UTF-8
|
||||
# Expected for a fake blob path: this proves the request reaches the Lago API,
|
||||
# not the Lago frontend SPA.
|
||||
```
|
||||
|
||||
The route also sends `X-Robots-Tag: noindex, nofollow`.
|
||||
|
||||
## Backup concept
|
||||
|
||||
This is staging, but it is still stateful. Before upgrades or destructive
|
||||
testing, create a Postgres dump from the Lago Postgres container.
|
||||
|
||||
Find the stack containers on `node1`:
|
||||
|
||||
```bash
|
||||
docker ps \
|
||||
--filter "label=com.docker.compose.project=kh7l869ianltiioqgmm45kur" \
|
||||
--format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
|
||||
```
|
||||
|
||||
Create a dump:
|
||||
|
||||
```bash
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
docker exec -t <lago-postgres-container> \
|
||||
pg_dump -U <lago-db-user> -d <lago-db-name> \
|
||||
> "lago-staging-${timestamp}.sql"
|
||||
```
|
||||
|
||||
Store the dump outside the container host after creation. For short-lived
|
||||
staging tests, an operator-controlled encrypted local archive is acceptable.
|
||||
For longer-lived staging data, move the dump into the standard HiveGrid backup
|
||||
bucket flow.
|
||||
|
||||
## Restore path
|
||||
|
||||
Restore only into `lago-staging` or a disposable local Lago instance. Never
|
||||
restore staging Lago data into production.
|
||||
|
||||
1. Stop Lago API, frontend, worker, billing worker and PDF service in Coolify.
|
||||
2. Keep Postgres running.
|
||||
3. Restore the dump:
|
||||
|
||||
```bash
|
||||
cat lago-staging-<timestamp>.sql | docker exec -i <lago-postgres-container> \
|
||||
psql -U <lago-db-user> -d <lago-db-name>
|
||||
```
|
||||
|
||||
4. Start the full stack again in Coolify.
|
||||
5. Verify API health and a known test customer/subscription.
|
||||
|
||||
If the restore targets an existing DB, drop/recreate the DB first or restore
|
||||
into a clean disposable instance. Do not merge two Lago staging datasets.
|
||||
|
||||
## SMTP decision
|
||||
|
||||
Staging outbound mail is disabled by default. Do not configure production
|
||||
Mailjet credentials on `lago-staging`.
|
||||
|
||||
When `BILL-07` needs email verification, use one of these options:
|
||||
|
||||
- Mail disabled, assertions based on Lago API state.
|
||||
- A dedicated Mailjet test configuration with allowlisted recipient addresses.
|
||||
|
||||
Any email-capable configuration must be documented in this runbook before use.
|
||||
|
||||
## Invoice PDF storage decision
|
||||
|
||||
For the current staging pilot, Invoice PDFs may use Lago's local Coolify volume.
|
||||
This is acceptable because the service has no production data and no public
|
||||
customer-facing billing UI.
|
||||
|
||||
Before production:
|
||||
|
||||
- configure a dedicated object storage bucket or production-grade persistent
|
||||
volume strategy,
|
||||
- document backup and restore for generated PDFs,
|
||||
- keep staging and production storage separate.
|
||||
|
||||
## Upgrade procedure
|
||||
|
||||
1. Read Lago release notes for the target version.
|
||||
2. Create a Postgres dump.
|
||||
3. Note current image tags and Coolify resource settings.
|
||||
4. Update the API/frontend/worker/PDF image tags together.
|
||||
5. Redeploy through Coolify.
|
||||
6. Verify:
|
||||
- API `/health`
|
||||
- frontend HTTP 200
|
||||
- worker containers running
|
||||
- a test customer/subscription still readable
|
||||
|
||||
Rollback is image-tag rollback plus DB restore from the dump created before
|
||||
the upgrade.
|
||||
|
||||
## Test data convention
|
||||
|
||||
Use deterministic prefixes so the shared staging instance stays clean:
|
||||
|
||||
- Customer external IDs: `hg-stg-<ticket>-<short-id>`
|
||||
- Plans: `hg-stg-plan-<ticket>-<short-id>`
|
||||
- Coupons or special cases: `hg-stg-case-<ticket>-<short-id>`
|
||||
|
||||
Delete test customers and plans after the ticket unless a runbook explicitly
|
||||
keeps them as regression fixtures.
|
||||
@@ -5,8 +5,6 @@ of real Postgres backends. The dev compose stack and the staging Coolify
|
||||
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
|
||||
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
|
||||
|
||||
Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](../improvement-plan/03-scaling-infrastructure/pgbouncer.md).
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
@@ -36,8 +34,8 @@ Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](.
|
||||
docker compose up postgres pgbouncer
|
||||
|
||||
# In another shell:
|
||||
DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \
|
||||
DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \
|
||||
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
|
||||
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid?pgbouncer=true&connection_limit=10" \
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
@@ -48,9 +46,8 @@ URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely).
|
||||
## Coolify (staging) deploy procedure
|
||||
|
||||
The pooler must run on the same Docker network as Postgres. The current
|
||||
operator-owned deployment source is the Coolify service/network config; the
|
||||
archived [`STAGING-ACCESS.md`](../improvement-plan/archive/STAGING-ACCESS.md)
|
||||
is historical inventory only and must not override live Coolify state.
|
||||
operator-owned deployment source is the Coolify service/network config; do not
|
||||
copy historical staging inventory into a live cutover.
|
||||
|
||||
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
|
||||
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
|
||||
|
||||
@@ -85,7 +85,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
|
||||
## Coolify (Schritt 3) — erledigt 2026-06-19
|
||||
- Coolify **v4.1.2** auf `control` installiert (alle Container healthy).
|
||||
- **Root-Admin headless** angelegt (`raffael@tool-ctrl.de`) via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
|
||||
- **Root-Admin headless** angelegt via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
|
||||
- **API aktiviert** + **API-Token** erzeugt (Sanctum, mit `team_id`; Coolifys `createToken`-Override braucht Team-Kontext, daher direkt über `tokens()->create`). `GET /api/v1/servers` → 200.
|
||||
- Zugang (Secrets in `~/.cli-tokens/coolify-prod`, **nicht** im Repo): UI `http://coolify.hivegrid.de:8000` bzw. `http://157.90.243.228:8000`.
|
||||
- ⚠ Härtung offen: Coolify-FQDN auf `https://coolify.hivegrid.de` (443) umstellen + Port 8000 wieder schließen.
|
||||
@@ -101,7 +101,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
- PgBouncer vor Postgres (Compose auf db-VM) → `DATABASE_URL` über Pooler, `DIRECT_URL` direkt.
|
||||
- DEK-Master-Key + S3-Bucket `hivegrid-prod`.
|
||||
- App-Deploy (web/api/worker) nach **mitapp-Muster: image-retag / build-once-deploy-via-pull** (nicht rebuild) + `migrate deploy`.
|
||||
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (ToolCTRL-Instanz, prod-DSNs).
|
||||
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (HiveGrid-Instanz, prod-DSNs).
|
||||
|
||||
## Zentral-Dienste + Storage (Schritt 4) — erledigt 2026-06-19
|
||||
- **Meilisearch v1.10 + ClamAV 1.4** als Coolify-Compose-Service `hivegrid-prod-search-av` auf worker-VM, privat exposed (`10.0.0.5:7700` / `:3310`). Verifiziert: Meili von api-VM cross-server `{"status":"available"}`, ClamAV healthy. Master-Key in `~/.cli-tokens/hivegrid-prod-services`.
|
||||
@@ -113,7 +113,7 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
- **Promote = retag, kein Rebuild:** Prod deployt denselben SHA-Tag, der getestet/gestaged ist (aktuell `9ae3963d…` = live auf Staging → Image existiert in der Gitea-Registry).
|
||||
- **7 Coolify-Apps** (nicht 3!): `web`, `api`, `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin` — web/api aus `/web`+`/api`-Image, die 5 Worker aus `/worker`-Image mit unterschiedlichem `WORKER_QUEUE_GROUP`.
|
||||
- Deploy-Trigger: `scripts/coolify-deploy.mjs` (PATCH `docker_registry_image_tag` → deploy), `migrate deploy` als Pre-Deployment-Command.
|
||||
- **Sentry (ToolCTRL-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
|
||||
- **Sentry (HiveGrid-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
|
||||
|
||||
**Offen (Go-Live-Phase):** Registry-Pull-Credential in Prod-Coolify · 7 Apps anlegen (Docker-Image) · App-Env Prod (aus Staging-Env transformiert: Prod-IPs/Domains, frische Secrets, prod-Sentry-Tag, **kein** WEB_NOINDEX) · erster Promote-Deploy + `migrate deploy` · DNS `app./api./admin.` · Endabnahme · mCaptcha-Sitekey + PgBouncer (Optimierung) als Folgeschritte.
|
||||
|
||||
@@ -163,7 +163,7 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
|
||||
- **mCaptcha:** LE-Cert für `captcha.hivegrid.de` verifizieren, dann proxied; Account+Sitekey anlegen → `MCAPTCHA_*` ins App-Env (bis dahin `CAPTCHA_FAIL_OPEN=true`).
|
||||
- **Sentry:** Events unter `environment=production` verifizieren (DSN aus Staging übernommen).
|
||||
- **Härtung:** SSH auf Bastion/Admin-IP einschränken · `admin.hivegrid.de` per Cloudflare-Access/IP schützen · scoped Registry-Bot statt `gitea-admin` · Coolify-FQDN auf https/443 + Port 8000 schließen.
|
||||
- **Optimierung:** PgBouncer vor Postgres · Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht).
|
||||
- **Optimierung:** PgBouncer vor Postgres · ~~Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht)~~ → **erledigt 2026-07-03:** Hooks feuerten für Docker-Image-Apps nie; ersetzt durch Migrate-on-Boot im Web-Entrypoint (`start.sh`, `MIGRATE_ON_BOOT`/`SEED_ON_BOOT`), Details in `docs/deployment.md` + `.planning/ticket-docs/AK-P4a.md`.
|
||||
- **CI-Promote:** `deploy.yml`/Gitea-Variablen auf das Prod-Coolify verdrahten (für künftige `promote`-Deploys ohne manuelle Schritte).
|
||||
|
||||
## Härtung (Schritt 7) — teilweise erledigt 2026-06-19
|
||||
@@ -172,10 +172,10 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
|
||||
|
||||
**Offene Härtung/Folgeschritte (priorisiert):**
|
||||
1. **mCaptcha public-FQDN:** Traefik liefert noch Default-Cert für `captcha.hivegrid.de` (Compose-Service-Domain-Labels nicht generiert; fqdn per tinker reichte nicht). Container laufen intern. Domain am besten über Coolify-**UI** am `mcaptcha`-Service setzen → LE-Cert → proxied. Dann **Account+Sitekey** in der mCaptcha-Admin-UI anlegen → `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` ins App-Env, `CAPTCHA_FAIL_OPEN=false`. Bis dahin: fail-open (kein Captcha-Zwang).
|
||||
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`toolctrl-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `toolctrl-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
|
||||
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`hivegrid-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `hivegrid-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
|
||||
3. **Coolify-Port 8000** schließen, sobald Coolify hinter `https://coolify.hivegrid.de` (443) läuft (Port aktuell offen, key+token-geschützt; für laufende API-Schritte noch gebraucht).
|
||||
4. **Registry:** scoped Bot-PAT statt `gitea-admin` für den Image-Pull-Login.
|
||||
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell; Coolify-Pre-Deploy-Migrate griff nicht → Migration manuell im Container).
|
||||
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell). Das Migrate-Problem dabei ist seit 2026-07-03 gelöst: Migrationen laufen jetzt im Web-Entrypoint (`MIGRATE_ON_BOOT`), nicht mehr über den nie feuernden Coolify-Hook.
|
||||
6. **Sentry** Events unter `environment=production` verifizieren · **PgBouncer** (Pooling) ergänzen.
|
||||
|
||||
---
|
||||
@@ -185,7 +185,7 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
|
||||
**Alles erledigt:**
|
||||
- ✅ **CI-Promote komplett grün** (Run 8453) — `workflow_dispatch action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`. Drei latente Multi-Server-Promote-Bugs gefixt (migration-status via `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE`, dependent-health via ProxyCommand-SSH, web-health via CF-Cache-Bypass).
|
||||
- ✅ **SSH-gehärtet** — control = Bastion, web/api/worker/db public-SSH dicht.
|
||||
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `toolctrl-admin`).
|
||||
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `hivegrid-admin`).
|
||||
- ✅ **mCaptcha** scharf + proxied (`captcha.hivegrid.de`).
|
||||
- ✅ **PgBouncer** vor Postgres, alle 7 Apps über `6432`.
|
||||
- ✅ **Coolify-Port zu** — nur noch `https://coolify.hivegrid.de` (Traefik+LE), `:8000` firewalled.
|
||||
|
||||
@@ -10,7 +10,7 @@ Last observed: 2026-05-28 via `pnpm run build`.
|
||||
|
||||
| Warning | Current interpretation | Escalate when |
|
||||
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Source maps are enabled in production` | Expected only when `TOOLCTRL_BUILD_SOURCEMAPS=1` is set for the Sentry release lane. Default web and bundle-budget builds should not emit this warning. Do not put secrets in source code; environment values must stay in `@toolctrl/config` and server-only boundaries. | Source maps expose hard-coded credentials, a default build emits maps unexpectedly, or a deploy target starts publishing server-only artifacts outside the intended build output. |
|
||||
| `Source maps are enabled in production` | Expected only when `HIVEGRID_BUILD_SOURCEMAPS=1` is set for the Sentry release lane. Default web and bundle-budget builds should not emit this warning. Do not put secrets in source code; environment values must stay in `@hivegrid/config` and server-only boundaries. | Source maps expose hard-coded credentials, a default build emits maps unexpectedly, or a deploy target starts publishing server-only artifacts outside the intended build output. |
|
||||
| `Generated an empty chunk` | Expected for React Router resource/API routes that have no client runtime. Empty route chunks are checked by `pnpm run verify:bundle-size`; unexpected names should be added only with an explicit rationale. | A new empty chunk is not covered by the bundle-size gate or appears for a user-facing UI route that should ship client code. |
|
||||
| AWS/Smithy circular chunk warnings | Mitigated in the web Vite build by grouping `@aws-sdk/*` and `@smithy/*` into `vendor-aws-sdk`. The 2026-05-28 build emits `build/server/assets/vendor-aws-sdk-*.js` and no longer prints the circular Smithy/AWS reexport warning. | The circular warning returns, runtime AWS/S3 paths fail after build, the warning points to first-party modules, or the SDK versions/change set introduce a new broken execution order. |
|
||||
| `dynamically imported ... but also statically imported` | Known Vite server-build warning where a lazy import cannot move a shared server module into a separate chunk because another route imports it statically. Current examples include DB, config, admin SSO audit, subscription entitlements, admin fallback helpers, and password reset helpers. | The warning starts involving a large client-only dependency, moves code into the default client shell, or a new performance slice claims a lazy split that this warning proves ineffective. |
|
||||
@@ -26,8 +26,8 @@ pnpm run verify:docs-links
|
||||
To inspect only warning lines from a local build log:
|
||||
|
||||
```text
|
||||
pnpm run build > /tmp/toolctrl-build.log 2>&1
|
||||
rg -n 'Source maps are enabled|Generated an empty chunk|@smithy|@aws-sdk|vendor-aws-sdk|dynamically imported|dynamic import will not move' /tmp/toolctrl-build.log
|
||||
pnpm run build > /tmp/hivegrid-build.log 2>&1
|
||||
rg -n 'Source maps are enabled|Generated an empty chunk|@smithy|@aws-sdk|vendor-aws-sdk|dynamically imported|dynamic import will not move' /tmp/hivegrid-build.log
|
||||
```
|
||||
|
||||
## Triage Notes
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
# Streaming Exports — Operator Runbook
|
||||
|
||||
This document describes the streaming export pipeline shipped in Phase 03
|
||||
task #46. It is the operator-facing complement to the detail plan
|
||||
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).
|
||||
task #46.
|
||||
|
||||
## Why streaming
|
||||
|
||||
|
||||
@@ -1,189 +0,0 @@
|
||||
# Stripe Meters API — Operator Runbook
|
||||
|
||||
**Audience:** operators wiring usage-based billing after the Stripe SDK
|
||||
13 → 19 migration (B-001, Task #28).
|
||||
|
||||
**Why:** Stripe 19 removed the legacy Usage-Records API
|
||||
(`subscriptionItems.createUsageRecord` /
|
||||
`subscriptionItems.listUsageRecordSummaries`) and `Recurring.aggregate_usage`
|
||||
in favor of the **Billing-Meters API**
|
||||
(`billing.meterEvents.create` / `billing.meters.listEventSummaries`).
|
||||
The boilerplate worker (`ai-cost-stripe-report`) still drives overage
|
||||
reporting from `AICostEvent` rows; only the Stripe wire format changed.
|
||||
Operators must configure a Billing-Meter once and migrate existing
|
||||
subscriptions to the new meter-backed price.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Create the Billing-Meter
|
||||
|
||||
In the Stripe Dashboard (or via the API):
|
||||
|
||||
1. **Billing → Meters → Create meter**.
|
||||
2. Configure:
|
||||
- **Display name:** `AI overage cents`
|
||||
- **Event name:** `ai-overage` *(the boilerplate hardcodes this id;
|
||||
change it only if you also override `aiOverageStripeMeterId` /
|
||||
`aiOverageMeterEventName` per plan).*
|
||||
- **Aggregation:** `Sum`
|
||||
- **Customer mapping:** map the event payload key
|
||||
`stripe_customer_id` to the Stripe customer id (default — leave
|
||||
as-is).
|
||||
- **Value settings:** event payload key `value`, type
|
||||
`numeric`. The reporter submits the rounded overage block-count
|
||||
in cents.
|
||||
3. Copy the resulting **Meter id** (`mtr_…`). You will store this in
|
||||
`Tenant.aiQuotaOverrideJson.aiOverageStripeMeterId` (per-tenant)
|
||||
or in the plan default (per-plan, future admin UI).
|
||||
|
||||
CLI alternative:
|
||||
|
||||
```bash
|
||||
stripe billing meters create \
|
||||
--display-name="AI overage cents" \
|
||||
--event-name="ai-overage" \
|
||||
--default-aggregation[formula]="sum" \
|
||||
--customer-mapping[event_payload_key]="stripe_customer_id" \
|
||||
--customer-mapping[type]="by_id" \
|
||||
--value-settings[event_payload_key]="value"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Create the metered product + price
|
||||
|
||||
1. **Products → Create product** → "AI Overage".
|
||||
2. **Add price**:
|
||||
- **Pricing model:** `Usage-based`
|
||||
- **Meter:** select the meter created in Step 1 (`mtr_…`).
|
||||
- **Per-unit cost:** the per-cent charge (typically `0.01` USD
|
||||
because the meter aggregates cents).
|
||||
- **Billing period:** `Monthly` (match the tenant's plan).
|
||||
3. Copy the resulting **Price id** (`price_…`).
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Migrate active subscriptions
|
||||
|
||||
For every active subscription that had a metered usage-record line in
|
||||
SDK 13.x:
|
||||
|
||||
1. **Add a new subscription item** referencing the new price from
|
||||
Step 2. Stripe API:
|
||||
|
||||
```bash
|
||||
stripe subscription_items create \
|
||||
--subscription="sub_..." \
|
||||
--price="price_..." \
|
||||
--quantity=0
|
||||
```
|
||||
|
||||
2. **Remove the old subscription item** that referenced the
|
||||
pre-migration metered price (or archive its price so no new
|
||||
subscriptions can attach it).
|
||||
|
||||
3. Confirm: `stripe subscriptions retrieve sub_... --expand=items.data.price`
|
||||
shows the new price and the meter id in `price.recurring.meter`.
|
||||
|
||||
The boilerplate's overage configuration uses `aiOverageStripeMeterId`
|
||||
as the sole identifier (the legacy `aiOverageStripeSubscriptionItemId`
|
||||
field was retired with the Stripe-19 cleanup). Existing tenant JSON
|
||||
overrides that still carry the legacy key are silently ignored by the
|
||||
parser.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Wire the boilerplate
|
||||
|
||||
1. **Per-tenant override** (admin route
|
||||
`/admin/settings/pricing/ai-overage`):
|
||||
|
||||
```json
|
||||
{
|
||||
"aiOverageStripeMeterId": "mtr_test_abc...",
|
||||
"aiMarkupMultiplier": 2.0,
|
||||
"aiOverageBlockCents": 10
|
||||
}
|
||||
```
|
||||
|
||||
2. **Per-plan default** — edit `packages/entitlements/src/ai-overage.ts`
|
||||
`standardAIOverageDefaults` and bake the meter id into the plan
|
||||
blueprint (operator-friendly admin UI is a future task).
|
||||
|
||||
3. **Worker bootstrap** — ensure the Stripe-19 client is constructed
|
||||
with `apiVersion: "2025-10-29.clover"` and the worker injects a
|
||||
`StripeMeterEventSubmitter` whose `submit()` calls
|
||||
|
||||
```ts
|
||||
stripe.billing.meterEvents.create(
|
||||
{
|
||||
event_name: "ai-overage",
|
||||
payload: {
|
||||
stripe_customer_id: tenant.stripeCustomerId,
|
||||
value: String(roundedBlocks),
|
||||
},
|
||||
identifier: idempotencyKey,
|
||||
timestamp,
|
||||
},
|
||||
{ idempotencyKey },
|
||||
);
|
||||
```
|
||||
|
||||
See `apps/worker/src/jobs/ai-cost-stripe-report.ts` for the
|
||||
pure-by-deps interface (`StripeMeterEventSubmitter`).
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Verify with a test event
|
||||
|
||||
1. Toggle `AI_OVERAGE_SUBMISSION_ENABLED=true`.
|
||||
2. Manually emit a test event:
|
||||
|
||||
```bash
|
||||
stripe billing meter_events create \
|
||||
--event-name="ai-overage" \
|
||||
--payload[stripe_customer_id]="cus_test123" \
|
||||
--payload[value]="100" \
|
||||
--identifier="manual-test-$(date +%s)"
|
||||
```
|
||||
|
||||
3. Confirm aggregation:
|
||||
|
||||
```bash
|
||||
stripe billing meters list_event_summaries mtr_... \
|
||||
--customer="cus_test123" \
|
||||
--start-time=$(date -v-1d +%s) \
|
||||
--end-time=$(date +%s)
|
||||
```
|
||||
|
||||
You should see a non-zero `aggregated_value` for the meter event
|
||||
you just submitted.
|
||||
|
||||
4. End-to-end: run `pnpm --filter @toolctrl/worker test
|
||||
ai-cost-stripe-report` to confirm the worker keeps passing its 13
|
||||
unit tests against the new wire.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
The Meters API is **not trivially reversible** — once events are
|
||||
submitted, the legacy Usage-Records API cannot ingest them. The safe
|
||||
rollback path is to leave already-submitted events in place and revert
|
||||
the SDK to 13.x via Git revert (commit
|
||||
`4b2350f0c5a098d3c2dd0778f087cac5910ec2f9` and its follow-ups). The
|
||||
operator-side meter + price can stay in place — Stripe ignores them
|
||||
when no subscription item references them.
|
||||
|
||||
---
|
||||
|
||||
## Open items
|
||||
|
||||
- Admin UI per-plan meter-id editor (currently only per-tenant via
|
||||
`aiQuotaOverrideJson`).
|
||||
- Auto-creation of the meter on first operator-bootstrap (currently
|
||||
manual via Dashboard / CLI).
|
||||
- CLI helper `toolctrl-cli ai-cost stripe-init` to scaffold the
|
||||
meter + product + price idempotently — listed in
|
||||
`docs/improvement-plan/05-business-features/ai-cost-phase-3.md`
|
||||
follow-ups.
|
||||
@@ -4,8 +4,8 @@
|
||||
operators clone a single tenant's Postgres rows + S3 storage objects into a
|
||||
PII-anonymized archive, then replay that archive into another environment.
|
||||
|
||||
This runbook is the operator-facing companion to the implementation spec at
|
||||
`docs/improvement-plan/03-scaling-infrastructure/tenant-snapshot.md`.
|
||||
This runbook is the operator-facing companion to the tenant snapshot
|
||||
implementation in `packages/cli`.
|
||||
|
||||
---
|
||||
|
||||
@@ -36,7 +36,7 @@ privacy risk. The full mapping table lives in
|
||||
| Phone number | `null` | `null` | (untouched) |
|
||||
| IP address | `127.0.0.<seeded>` | `127.0.0.<seeded>` | (untouched) |
|
||||
| ZIP code | first two digits kept, rest reseeded | first two digits kept, rest reseeded | (untouched) |
|
||||
| Stripe customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
|
||||
| Legacy-Provider customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
|
||||
| Webhook / SSO / domain URL | `https://anonymized.test/<8 hex>` | same as full | (untouched) |
|
||||
| Secrets, API keys, refresh tokens, payloads | `[REDACTED]` | `[REDACTED]` | (untouched) |
|
||||
| Avatar URL | `https://anonymized.test/avatar/<8 hex>.png` | same as full | (untouched) |
|
||||
@@ -54,9 +54,9 @@ anywhere; only its `sha256` lives in the manifest as a reproducibility marker.
|
||||
### Create a snapshot (default profile, age-encrypted, upload to S3)
|
||||
|
||||
```bash
|
||||
toolctrl-cli tenant snapshot \
|
||||
hivegrid-cli tenant snapshot \
|
||||
--tenant-id ten_abc123 \
|
||||
--output s3://toolctrl-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
|
||||
--output s3://hivegrid-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
|
||||
```
|
||||
|
||||
The CLI requires `BACKUP_AGE_PUBLIC_KEY` (recipient public key) plus
|
||||
@@ -66,9 +66,9 @@ catalog — the snapshot command reuses the same encryption layer.
|
||||
### Restore into a dev box
|
||||
|
||||
```bash
|
||||
NODE_ENV=development toolctrl-cli tenant restore \
|
||||
NODE_ENV=development hivegrid-cli tenant restore \
|
||||
--tenant-id ten_devclone \
|
||||
--source s3://toolctrl-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
|
||||
--source s3://hivegrid-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
|
||||
```
|
||||
|
||||
The command refuses to run under `NODE_ENV=production` unless the operator
|
||||
@@ -78,7 +78,7 @@ with `--yes` for non-interactive pipelines.
|
||||
### List existing snapshots
|
||||
|
||||
```bash
|
||||
toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
|
||||
hivegrid-cli tenant list-snapshots --bucket hivegrid-snapshots
|
||||
```
|
||||
|
||||
### Super-admin escape hatch (`--profile=none`)
|
||||
@@ -86,11 +86,11 @@ toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
|
||||
Raw, un-anonymized exports require an explicit confirmation flag:
|
||||
|
||||
```bash
|
||||
toolctrl-cli tenant snapshot \
|
||||
hivegrid-cli tenant snapshot \
|
||||
--tenant-id ten_abc123 \
|
||||
--profile none \
|
||||
--i-am-the-super-admin \
|
||||
--output s3://toolctrl-snapshots/migration/ten_abc123.tar.gz.age
|
||||
--output s3://hivegrid-snapshots/migration/ten_abc123.tar.gz.age
|
||||
```
|
||||
|
||||
Without `--i-am-the-super-admin`, the command exits non-zero with a clear
|
||||
@@ -123,14 +123,14 @@ redirect that into their central audit log; set `auditFilePath` when calling
|
||||
|
||||
1. Snapshot a representative tenant from staging:
|
||||
```bash
|
||||
toolctrl-cli tenant snapshot --tenant-id <staging-tenant> \
|
||||
hivegrid-cli tenant snapshot --tenant-id <staging-tenant> \
|
||||
--profile full-anonymized \
|
||||
--output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age
|
||||
```
|
||||
2. Copy to a dev box and restore into a fresh database:
|
||||
```bash
|
||||
NODE_ENV=development DATABASE_URL=postgres://... \
|
||||
toolctrl-cli tenant restore \
|
||||
hivegrid-cli tenant restore \
|
||||
--tenant-id ten_drill_$(date +%s) \
|
||||
--source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \
|
||||
--yes
|
||||
@@ -182,10 +182,10 @@ The restore command accepts one of three strategies (default
|
||||
|
||||
```bash
|
||||
# Default — overlay snapshot rows onto whatever already exists.
|
||||
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
|
||||
hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
|
||||
|
||||
# Sensitive restore: refuse to proceed if any global is already there.
|
||||
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
|
||||
hivegrid-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
|
||||
--globals=error
|
||||
```
|
||||
|
||||
|
||||
@@ -53,7 +53,7 @@ short app/group/port summary and link back here.
|
||||
`worker-plugin` is required as a persistent Coolify service so the deploy
|
||||
topology is stable, but production plugin execution is still gated. With
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
|
||||
as an idle topology placeholder and must not consume `toolctrl.plugin-invoke`.
|
||||
as an idle topology placeholder and must not consume `hivegrid.plugin-invoke`.
|
||||
Only after the external WASM sandbox signoff in
|
||||
`docs/security/plugin-runtime-signoff.md` may operators set
|
||||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
|
||||
@@ -66,7 +66,7 @@ separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
|
||||
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
|
||||
export, and scan services are wired into the worker bootstrap. In that idle
|
||||
state `/readyz` may be healthy with an empty queue list, which proves topology
|
||||
without consuming `toolctrl.imports` or `toolctrl.exports`. When both runtime
|
||||
without consuming `hivegrid.imports` or `hivegrid.exports`. When both runtime
|
||||
gates are set, startup fails closed unless the worker has real import, export,
|
||||
and scan service dependencies.
|
||||
|
||||
@@ -74,11 +74,11 @@ and scan service dependencies.
|
||||
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `toolctrl.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | `toolctrl.imports`, `toolctrl.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
||||
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `toolctrl.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | `hivegrid.imports`, `hivegrid.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
||||
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
|
||||
Source of truth: `packages/queue/src/groups.ts`. A test
|
||||
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
||||
@@ -117,11 +117,13 @@ dispatcher or bridge dependency fails startup instead of leaving outbox rows or
|
||||
`outbox.workflow` silently undrained behind a green process.
|
||||
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
|
||||
call-sites stay absent and that only the signed internal trigger bridge may
|
||||
start `type: "trigger"` executions directly. The `row.created`, `row.updated`,
|
||||
and `row.deleted` route-backed and workflow-context producers now write the row
|
||||
mutation, application event, and `workflow.trigger.event` outbox row in the same
|
||||
transaction when a request or workflow event context is passed. Other internal
|
||||
requestless mutations remain eventless by default to avoid worker feedback loops.
|
||||
start `type: "trigger"` executions directly. The generic entity-builder's
|
||||
`row.created` / `row.updated` / `row.deleted` route-backed producers were
|
||||
**removed with the entity-builder** (R-P6B-β, AD-2); typed fachentities and
|
||||
workflow row blocks (via the typed model registry) no longer emit generic
|
||||
`row.*` outbox events. `workflow.trigger.event` application events still publish
|
||||
through `EventsService`. Other internal requestless mutations remain eventless by
|
||||
default to avoid worker feedback loops.
|
||||
|
||||
`worker-general` also registers the
|
||||
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
|
||||
@@ -146,7 +148,7 @@ outbox dispatcher routes to `outbox.form-submission-route` on
|
||||
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
|
||||
marks the submission, and publishes follow-up events. Webhook follow-ups use
|
||||
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
|
||||
`worker-mail`, where the bridge enqueues the existing `toolctrl.email`
|
||||
`worker-mail`, where the bridge enqueues the existing `hivegrid.email`
|
||||
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
|
||||
throughput group and leaves the general group responsible only for the
|
||||
database-backed routing decision.
|
||||
@@ -290,7 +292,7 @@ Managed Coolify `worker-mail` apps must keep
|
||||
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
|
||||
`outbox.mail` cannot subscribe; without that bridge, invite and other
|
||||
`mail.transactional.*` outbox events can remain pending even if the direct
|
||||
`toolctrl.email` worker is alive. The Coolify worker-group provisioner writes
|
||||
`hivegrid.email` worker is alive. The Coolify worker-group provisioner writes
|
||||
and drift-checks this env value for managed mail workers.
|
||||
|
||||
Example:
|
||||
|
||||
Reference in new issue
Block a user