Refresh HiveGrid Cloud reference snapshot
This commit is contained in:
1 parent
5e99e14772
commit
7f88026f79
5955 files changed
+1177325
-188362
No files matched your search
@@ -0,0 +1,663 @@
|
||||
# Customer-Portal Staging-UAT
|
||||
|
||||
Dieses Runbook provisioniert drei voneinander getrennte Operator-Zustaende fuer `VA-P26a`, `VA-P26b`
|
||||
und den Katalogteil von `VA-P26c`:
|
||||
|
||||
1. den kanonischen Customer-Portal-Katalog in Lago und HiveGrid;
|
||||
2. eine wiederholbare Portal-UAT-Fixture im bestehenden Staging-Tenant `demo-pro-crm`.
|
||||
3. drei getrennte Commercial-UAT-Pfade in `demo-starter-crm`, `demo-pro-crm` und
|
||||
`demo-enterprise-crm`, ohne Starter oder Business durch einen Enterprise-Override zu maskieren.
|
||||
|
||||
Die Kommandos laufen standardmaessig im read-only Modus `--plan`. Sie deployen nichts, aendern kein
|
||||
DNS und besitzen keine Coolify-, Traefik-, SSH-, Migrator- oder Produktions-Credentials.
|
||||
|
||||
## Sicherheitsgrenze
|
||||
|
||||
- Ausschliesslich Staging: `SENTRY_ENVIRONMENT=staging` und `RLS_RUNTIME_MODE=required` sind Pflicht.
|
||||
- `NODE_ENV=production` und der kanonische Staging-Wert von `AUDIT_CHAIN_SECRET` sind Pflicht. Damit
|
||||
bleiben Advisory-Lock, persistenter Chain-Head und HMAC auch im einmaligen Operator-Prozess
|
||||
fail-closed aktiv; ein Dev-Fallback darf keine Staging-Auditzeile erzeugen. Vor jedem Plan,
|
||||
Verify oder Apply muss der vorhandene Plattform-`AuditChainHead` mit diesem Secret authentifiziert
|
||||
werden. Ein fehlender Anchor oder ein nur ausreichend langes, aber falsches Secret blockiert den
|
||||
Lauf vor Provider- und Datenbankmutationen; dieses UAT-Tool bootstrappt keine Audit-Kette.
|
||||
- Die Datenbank-URL muss als `hivegrid_app` auf exakt den separat bestaetigten Host, Port und
|
||||
Datenbanknamen zeigen. URL-Queryparameter/-Fragmente und alle gesetzten `PG*`-Variablen werden als
|
||||
potenzielle libpq-Verbindungs-Overrides abgelehnt. `DIRECT_URL`, Boot-Migrationen und alle
|
||||
`SEED_*`-Variablen werden ebenfalls abgelehnt.
|
||||
- Die Datenbank selbst muss unabhaengig von den Operator-Variablen mit
|
||||
`ALTER DATABASE ... SET hivegrid.environment=staging` markiert sein. Der Guard liest den
|
||||
persistenten Datenbank-Eintrag aus `pg_db_role_setting`, nicht den durch `PGOPTIONS` oder
|
||||
Connection-Parameter ueberschreibbaren Session-Wert. Ein fehlender Marker blockiert bereits Plan
|
||||
und Verify fail-closed. `hivegrid_app` muss ausserdem ohne `SUPERUSER`, `BYPASSRLS`, `CREATEROLE`
|
||||
und `CREATEDB` laufen.
|
||||
- `CUSTOMER_PORTAL_UAT_OPERATOR_USER_ID` muss ein aktiver `AdminUser` sein. Der Postgres-Vertrag
|
||||
`hivegrid_is_operator()` bleibt damit auch bei Katalogschreibzugriffen wirksam.
|
||||
- Katalog und Portal-Fixture brauchen fuer jede Mutation zusaetzlich
|
||||
`--apply --confirm=demo-pro-crm`. Der Commercial-Fixture bindet zwei Tenants und akzeptiert nur
|
||||
`--apply --confirm=demo-starter-crm+demo-pro-crm+demo-enterprise-crm`. Die ersten beiden Tenants
|
||||
muessen bereits existieren; den dritten besitzt und erstellt ausschliesslich diese UAT-Fixture mit
|
||||
der explizit gebundenen ID.
|
||||
- Die Fixture verwendet nur die synthetischen Identitaeten
|
||||
`customer-portal-uat@saas-ctrl.com` und
|
||||
`customer-portal-subcontractor-uat@saas-ctrl.com`. Das Passwort wird nie ausgegeben.
|
||||
- Lago-Ressourcen werden erstellt, wenn ihr Code fehlt. Bei einem bestehenden Plan oder Add-on darf
|
||||
der auditierte Operator ausschliesslich einen fehlenden oder abweichenden kanonischen Tax-Code
|
||||
aktualisieren. Preis- oder sonstiger Vertragsdrift stoppt weiterhin fail-closed; das Tool loescht
|
||||
keine Provider-Ressource.
|
||||
- Lago-Aufrufe sind auf die kanonische Mesh-Adresse `http://10.42.0.4:3108` und den dokumentierten
|
||||
lokalen SSH-Tunnel `http://127.0.0.1:3108` begrenzt. Andere Hosts, Pfade oder URL-Credentials
|
||||
werden vor dem Versand des Bearer-Tokens abgelehnt.
|
||||
- Vor jedem Katalog-Read und jeder Provider-Mutation muss Lago selbst den Namen `HiveGrid Staging`
|
||||
und den fest hinterlegten SHA-256-Fingerprint seiner Organisations-ID liefern. Ein abweichendes
|
||||
Ziel wird vor weiteren Katalogaufrufen fail-closed abgelehnt.
|
||||
|
||||
## Einmalige lokale Operator-Umgebung
|
||||
|
||||
Die Werte kommen aus den bestehenden Staging-Ressourcen beziehungsweise dem freigegebenen
|
||||
Operator-Zugang. Keine Werte ins Repository, in Tickets oder in Shell-History kopieren. Fuer das
|
||||
Portal-UAT-Passwort eine lokale Datei mit Modus `600` verwenden.
|
||||
|
||||
```bash
|
||||
export NODE_ENV=production
|
||||
export SENTRY_ENVIRONMENT=staging
|
||||
export RLS_RUNTIME_MODE=required
|
||||
export DATABASE_RUNTIME_ROLE=hivegrid_app
|
||||
export WEB_RUNTIME_SURFACE=core
|
||||
export DATABASE_URL=<restricted-staging-hivegrid-app-url>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_DATABASE_HOST=<exact-host-from-database-url>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_DATABASE_PORT=<exact-port-from-database-url>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_DATABASE=<exact-database-name>
|
||||
export CUSTOMER_PORTAL_UAT_OPERATOR_USER_ID=<active-admin-user-id>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_TENANT_ID=<demo-pro-crm-tenant-id>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_STARTER_TENANT_ID=<demo-starter-crm-tenant-id>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_PRO_TENANT_ID=<demo-pro-crm-tenant-id>
|
||||
export CUSTOMER_PORTAL_UAT_EXPECTED_ENTERPRISE_TENANT_ID=<reserved-demo-enterprise-crm-tenant-id>
|
||||
export CUSTOMER_PORTAL_UAT_PASSWORD="$(<~/.cli-tokens/customer-portal-uat)"
|
||||
export AUDIT_CHAIN_SECRET="$(<~/.cli-tokens/hivegrid-staging-audit-chain)"
|
||||
set -a
|
||||
source ~/.cli-tokens/lago-staging
|
||||
set +a
|
||||
```
|
||||
|
||||
Die lokale Audit-Datei muss Modus `600` besitzen und exakt den Wert der aktiven, nicht als Preview
|
||||
markierten Staging-Web-Ressource enthalten. Den Wert niemals ausgeben oder manuell in die
|
||||
Shell-History schreiben; bei fehlender oder abweichender Datei stoppt jeder UAT-Befehl vor dem
|
||||
ersten Datenbankzugriff.
|
||||
|
||||
Die kanonische Lago-Staging-Topologie und Tunnel-Variante stehen in
|
||||
[`lago-staging.md`](./lago-staging.md). Der eingeschraenkte Runtime-DB-Zugang ist bewusst nicht der
|
||||
Migrator-Zugang.
|
||||
|
||||
## 1. Katalog provisionieren
|
||||
|
||||
Der Plan umfasst:
|
||||
|
||||
- Lago-Plan `customer_portal_monthly`, monatlich, 19,99 EUR netto, im Voraus;
|
||||
- Lago-Add-on `customer_portal_domain_setup`, einmalig 299,99 EUR netto;
|
||||
- die drei HiveGrid-Katalogprodukte fuer Add-on, Domain-Setup und nicht-oeffentlichen
|
||||
Enterprise-Vertrags-Override;
|
||||
- den AD-30-Nachweis: Starter und Pro ohne Basis-Portalrecht, Business inklusive Portalrecht.
|
||||
|
||||
```bash
|
||||
pnpm customer-portal:uat:catalog --plan
|
||||
pnpm customer-portal:uat:catalog --apply --confirm=demo-pro-crm
|
||||
pnpm customer-portal:uat:catalog --verify
|
||||
```
|
||||
|
||||
`--apply` bricht vor der ersten Mutation ab, wenn ein bestehender Provider-Vertrag ausserhalb seines
|
||||
Tax-Codes driftet oder die Basisplan-Abbildung nicht dem AD-30-Vertrag entspricht. Eine isolierte
|
||||
Tax-Code-Abweichung wird auf `de-vat-19` aktualisiert. Bei einem Fehler nach einer Provider-Erstellung
|
||||
oder Tax-Aktualisierung ist ein erneuter Lauf sicher: vorhandene kanonische Provider-Ressourcen werden
|
||||
als `noop` erkannt und nur der fehlende Schritt wird fortgesetzt.
|
||||
|
||||
Mollie besitzt fuer diesen Flow keinen separaten Produktkatalog. Der bestehende generische Checkout
|
||||
verwendet den HiveGrid-Preis, startet die Mollie-Testzahlung und synchronisiert anschliessend die
|
||||
Lago-Subscription beziehungsweise die einmalige Lago-Add-on-Rechnung.
|
||||
|
||||
Bleibt ein Testcheckout nach einem verlorenen Mollie-Response als mehrdeutiger `provider_pending`-
|
||||
Versuch ohne `providerPaymentId` stehen, darf die Fixture keinen zweiten Versuch erzwingen. Der
|
||||
Operator folgt ausschliesslich
|
||||
[`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md); der dortige Plan-Lauf
|
||||
prueft Mollie read-only und `--apply` wird erst nach abgelaufenem Recovery-Fenster sowie belegter
|
||||
Nichtanlage freigegeben.
|
||||
|
||||
## 2. UAT-Fixture provisionieren
|
||||
|
||||
Die Fixture verwendet die deterministischen Projekte `rheinmain-shk-v1-project-0001` und
|
||||
`rheinmain-shk-v1-project-0002` des bestehenden `demo-pro-crm`-Datensatzes. Vor der Mutation werden
|
||||
Tenant-ID, aktives Business-Abo, Projektstatus und UAT-Eigentum aller kollidierenden Zeilen geprueft.
|
||||
|
||||
Sie reconciled idempotent:
|
||||
|
||||
- ein publiziertes Portal und dessen rechtlich vollstaendige Konfiguration;
|
||||
- Customer- und Subunternehmer-PortalUser mit getrennten, expliziten Projektzugriffen;
|
||||
- sichtbare Projektphasen und einen projektgebundenen, publizierten `portal_form`-Reklamationsflow;
|
||||
- ein offenes, sieben Tage gueltiges Service-Inbox-Element fuer den Tenant-Admin-UAT.
|
||||
|
||||
```bash
|
||||
pnpm customer-portal:uat:fixture --plan
|
||||
pnpm customer-portal:uat:fixture --apply --confirm=demo-pro-crm
|
||||
pnpm customer-portal:uat:fixture --verify
|
||||
```
|
||||
|
||||
Die JSON-Ausgabe enthaelt ausschliesslich sichere E2E-Handoff-Werte: Portal-Origin, Tenant-Slug,
|
||||
synthetische E-Mail-Adressen und die erwarteten beziehungsweise verbotenen Projektnamen. Das Passwort
|
||||
wird als `E2E_PORTAL_PASSWORD` und `E2E_PORTAL_SUBCONTRACTOR_PASSWORD` aus derselben lokalen
|
||||
Credential-Datei gesetzt, nicht aus der Kommandoausgabe.
|
||||
|
||||
## 3. Commercial-UAT-Fixture provisionieren
|
||||
|
||||
Der Commercial-Fixture bindet zwei bestehende, explizit bestaetigte Staging-Tenants und besitzt einen
|
||||
dritten, deterministischen Tenant `demo-enterprise-crm`. Er legt die synthetische Identitaet
|
||||
`customer-portal-admin-uat@saas-ctrl.com` als reinen Tenant-Admin in allen drei Tenants an. Sie
|
||||
erhaelt exakt `app.dashboard.view`, `portal.admin`, `portal.read`, `portal.manage_users`,
|
||||
`portal.manage_access`, `portal.activity.read`, `portal.activity.manage`, `crm.read_own` und
|
||||
`projects.read_own` sowie `projects.read_all`, niemals Plattform-Admin-Rechte. `projects.read_own`
|
||||
erfuellt dabei das kanonische Projects-Modul-Gate; `projects.read_all` erlaubt der isolierten
|
||||
UAT-Identitaet den fest gebundenen Projekt-Einstieg.
|
||||
|
||||
Der Starter-Pfad darf ausschliesslich einen leeren, providerlosen Trial-Zustand in einen aktiven
|
||||
Starter-Basisplan ueberfuehren und ein eigenes Demo-Portal anlegen. Er erstellt weder eine
|
||||
`customer_portal_monthly`-Acquisition noch Lago-/Mollie-Providerzustand; der echte Checkout ueber
|
||||
23,79 EUR brutto (19,99 EUR netto plus 19 Prozent Umsatzsteuer) bleibt damit beweisbar. Der Pro-Pfad prueft das vorhandene aktive Business-Abo
|
||||
und die separate `VA-P26`-Portal-Fixture als inkludierten Basisplan-Vertrag.
|
||||
|
||||
Der Enterprise-Pfad ist absichtlich **kein vierter Basisplan**. Die Fixture erstellt atomar einen
|
||||
providerlosen aktiven `pro`-Basisvertrag und gewaehrt darauf genau einmal den nicht oeffentlichen
|
||||
`customer_portal_enterprise_contract` mit `productKind=contract_override`, `scopeKey=tenant` und
|
||||
Menge `1`. Tenant, Basisvertrag, Portal, Rolle/Membership, Grant und Audit-Outbox committen in einer
|
||||
Transaktion. Es entstehen weder SubscriptionPrice noch CheckoutSession, Mollie-Zahlung,
|
||||
Lago-Subscription, Rechnung oder Checkout-URL. Ein stabiler Request-Hash plus CAS-Version `3`
|
||||
beweist den abgeschlossenen idempotenten Grant; abweichende Referenz-, Provider- oder Ledgerdaten
|
||||
blockieren fail-closed.
|
||||
|
||||
```bash
|
||||
pnpm customer-portal:uat:commercial-fixture --plan
|
||||
pnpm customer-portal:uat:commercial-fixture \
|
||||
--apply \
|
||||
--confirm=demo-starter-crm+demo-pro-crm+demo-enterprise-crm
|
||||
pnpm customer-portal:uat:commercial-fixture --verify
|
||||
```
|
||||
|
||||
Vorhandene Portal-Add-on-Ledger, Providerreferenzen, fremde Rollen/Portale oder ID-Kollisionen
|
||||
blockieren fail-closed. Ein Grant auf Starter oder Business bleibt verboten, weil er den jeweils zu
|
||||
pruefenden kommerziellen Pfad verdecken wuerde. Erfolgreiches `--verify` liefert als sicheren
|
||||
Nachweis unter `assessment.liveProof` mindestens:
|
||||
|
||||
- `businessBasePlanReady=true` und damit weiterhin `portalSource=base_plan` fuer `demo-pro-crm`;
|
||||
- `enterpriseContractGrantReady=true`, `enterprisePortalSource=contract_override` und
|
||||
`enterpriseDomainSetupIncludedByContract=true` fuer `demo-enterprise-crm`;
|
||||
- `enterpriseCheckoutRows=0` und eine nicht-leere `enterpriseAcquisitionId`.
|
||||
|
||||
Der Enterprise-Tenant kann danach mit derselben lokalen UAT-Identitaet unter
|
||||
`/app/demo-enterprise-crm/settings/customer-portal/domain` geoeffnet werden. Die Seite muss das Portal
|
||||
ohne Kaufaktion freigeben, eine enthaltene Domain anzeigen und darf weder Portal-Checkout noch
|
||||
Domain-Setup-Checkout anbieten. Ein spaeter beanspruchter erster Host bindet sich an
|
||||
`enterprise-contract:<enterpriseAcquisitionId>`; ein zweiter Host darf denselben Vertragsplatz nicht
|
||||
verwenden.
|
||||
|
||||
## Browser-UAT
|
||||
|
||||
Nach erfolgreichem `--verify` werden die bestehenden Reisen ausschliesslich fail-closed ausgefuehrt.
|
||||
`REQUIRE_E2E_READINESS=1` verwandelt fehlende Inputs, Fixtures, 404-Pfade und Login-Redirects in Fehler;
|
||||
`REQUIRE_E2E_MUTATING_READINESS=1` macht dasselbe fuer fehlende Mutationsfreigaben. Ein Lauf ohne beide
|
||||
Flags ist kein Ticket-Proof, auch wenn Playwright mit Exit-Code 0 endet.
|
||||
Die drei Live-Specs erzwingen `trace: off` und `video: off`, damit das lokale UAT-Passwort nicht in
|
||||
Playwright-Artefakten landet. Nach jedem Lauf wird `test-results/` geprueft und vor der Ablage des
|
||||
redigierten Proofs entfernt.
|
||||
|
||||
- `tests/e2e/va-p26a-customer-portal.smoke.spec.ts` fuer getrennte Portal-Session und Projektzugriff;
|
||||
- `tests/e2e/va-p26b-customer-portal.smoke.spec.ts` fuer Upload, Reklamation, Subunternehmer-Isolation,
|
||||
Service-Inbox und DSGVO-Lifecycle;
|
||||
- `tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts` erst dann, wenn ein eigener Testhostname,
|
||||
DNS-Proof und die Controller-/TLS-Voraussetzungen bereitstehen.
|
||||
|
||||
### P26a/P26b: Standardhost und Portal-Reisen
|
||||
|
||||
Die Tenant-Admin-Credentials muessen vorab aus dem freigegebenen lokalen Staging-Secret geladen sein;
|
||||
das Runbook enthaelt sie absichtlich nicht. Der redigierte Fixture-Verify-Output liefert die beiden
|
||||
Projektbezeichnungen ohne Passwort oder Datenbank-Credential.
|
||||
|
||||
```bash
|
||||
: "${E2E_TENANT_EMAIL:?load the approved demo-pro-crm portal.admin email from the local staging secret}"
|
||||
: "${E2E_TENANT_PASSWORD:?load the approved demo-pro-crm password from the local staging secret}"
|
||||
: "${CUSTOMER_PORTAL_UAT_PASSWORD:?load the local customer-portal UAT password first}"
|
||||
export E2E_TENANT_EMAIL E2E_TENANT_PASSWORD
|
||||
|
||||
export E2E_BASE_URL=https://staging-app.hivegrid.de
|
||||
export E2E_PORTAL_BASE_URL=https://staging-portal.hivegrid.de
|
||||
export E2E_PORTAL_TENANT_SLUG=demo-pro-crm
|
||||
export E2E_PORTAL_EXPECTED_TITLE="Demo Pro CRM Kundenportal"
|
||||
export E2E_PORTAL_EMAIL=customer-portal-uat@saas-ctrl.com
|
||||
export E2E_PORTAL_SUBCONTRACTOR_EMAIL=customer-portal-subcontractor-uat@saas-ctrl.com
|
||||
export E2E_PORTAL_PASSWORD="$CUSTOMER_PORTAL_UAT_PASSWORD"
|
||||
export E2E_PORTAL_SUBCONTRACTOR_PASSWORD="$CUSTOMER_PORTAL_UAT_PASSWORD"
|
||||
export E2E_TENANT_DASHBOARD_PATH=/demo-pro-crm/dashboard
|
||||
|
||||
FIXTURE_VERIFY_JSON="$(mktemp)"
|
||||
pnpm --silent customer-portal:uat:fixture --verify >"$FIXTURE_VERIFY_JSON"
|
||||
E2E_PORTAL_SUBCONTRACTOR_EXPECTED_PROJECT="$(node -e '
|
||||
const fs = require("node:fs");
|
||||
const value = JSON.parse(fs.readFileSync(process.argv[1], "utf8")).assessment.e2e.subcontractorExpectedProject;
|
||||
if (!value) process.exit(1);
|
||||
process.stdout.write(value);
|
||||
' "$FIXTURE_VERIFY_JSON")"
|
||||
E2E_PORTAL_SUBCONTRACTOR_FORBIDDEN_PROJECT="$(node -e '
|
||||
const fs = require("node:fs");
|
||||
const value = JSON.parse(fs.readFileSync(process.argv[1], "utf8")).assessment.e2e.subcontractorForbiddenProject;
|
||||
if (!value) process.exit(1);
|
||||
process.stdout.write(value);
|
||||
' "$FIXTURE_VERIFY_JSON")"
|
||||
rm -f "$FIXTURE_VERIFY_JSON"
|
||||
export E2E_PORTAL_SUBCONTRACTOR_EXPECTED_PROJECT E2E_PORTAL_SUBCONTRACTOR_FORBIDDEN_PROJECT
|
||||
|
||||
export REQUIRE_E2E_READINESS=1
|
||||
export REQUIRE_E2E_MUTATING_READINESS=1
|
||||
export E2E_ALLOW_CUSTOMER_PORTAL_P26B_MUTATION=1
|
||||
|
||||
pnpm exec playwright test \
|
||||
tests/e2e/va-p26a-customer-portal.smoke.spec.ts \
|
||||
tests/e2e/va-p26b-customer-portal.smoke.spec.ts \
|
||||
--grep-invert "stateful invitation lifecycle" \
|
||||
--project=chromium \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Vor einer Wiederholung von P26b muss die Fixture erneut mit `--apply --confirm=demo-pro-crm` und
|
||||
anschliessendem `--verify` auf den offenen Service-Inbox-Zustand reconciled werden. Als Proof zaehlt nur
|
||||
ein Lauf ohne `skipped`-Tests.
|
||||
|
||||
### P26b: verschluesselter DSGVO-Export
|
||||
|
||||
Vor dem Browserlauf muessen Core-Web, Portal-Web und `worker-import` denselben server-only Wert fuer
|
||||
`GDPR_EXPORT_SSE_C_KEY_BASE64` aus dem Staging-Secret-Store erhalten. Er ist kanonisches Base64 von
|
||||
exakt 32 zufaelligen Bytes; weder Wert noch Hash werden in Logs, Tickets oder Proofs ausgegeben.
|
||||
`worker-import /readyz` muss mit `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true`, vollstaendigem HTTPS-
|
||||
`STORAGE_*`-Vertrag und dem Key bereit sein. Beide Web-Ressourcen muessen dieselbe Revision und
|
||||
ebenfalls Readiness liefern.
|
||||
|
||||
Als Live-Proof gilt nur ein **frischer** Portal-Export nach dem Cutover: Der Worker schreibt per
|
||||
SSE-C, ein direkter Object-Store-GET ohne Customer-Key scheitert, und die autorisierte Portalroute
|
||||
liefert den JSON-Download mit Status 200 ohne `Location`-Header. Der Cross-Tenant-Pfad bleibt 404.
|
||||
Vorhandene Klartext-Artefakte erhalten keinen Fallback und werden nach ihrem Ablauf ueber den
|
||||
Retention-Pfad entfernt. Fuer Rotation: Queue pausieren, hoechstens 72 Stunden alte Artefakte
|
||||
ablaufen lassen beziehungsweise gezielt entfernen, denselben neuen Key atomar auf den drei
|
||||
Ressourcen setzen und erst danach wieder freigeben.
|
||||
|
||||
### P26a: Default-on- und Zugriffsnachweis
|
||||
|
||||
Das Kundenportal ist eine Kernfunktion und besitzt kein globales Rollout-Flag mehr. Der Staging-Proof
|
||||
muss zeigen, dass Standardhost, Workspace-Verwaltung, Einladung und Portalprojekt ohne aktive
|
||||
`customer-portal`-Flag-Zeile funktionieren. Negativ geprüft werden weiterhin fehlende Portal-Permission,
|
||||
fehlendes Entitlement, inaktiver beziehungsweise eingefrorener Tenant, unveröffentliches Portal und
|
||||
falsche Portal-Identität. Diese Fälle liefern unverändert 404 beziehungsweise unterdrücken Effekte.
|
||||
`/healthz` bleibt für die Diagnose erreichbar.
|
||||
|
||||
### P26a: vollstaendiger Einladungs-Lifecycle
|
||||
|
||||
Der zustandsveraendernde Einladungs-Proof laeuft getrennt und genau einmal in Chromium. Vor jedem
|
||||
Lauf muss `customer-portal:uat:fixture --verify` gruen sein. Als Ziel wird eine neue synthetische
|
||||
Plus-Adresse des authentifizierten `MAILBOX_EMAIL`-Postfachs verwendet; eine bereits verwendete
|
||||
Adresse ist wegen des bewusst fail-closed Duplicate-E-Mail-Vertrags kein wiederholbarer Testdatensatz.
|
||||
|
||||
Der Mail-Fixture-Adapter ist fuer diesen Staging-Abschluss ein temporaerer, operatorverwalteter
|
||||
Loopback-Dienst vor dem dedizierten SOGo-Testpostfach. Er bindet ausschliesslich an `127.0.0.1`,
|
||||
bleibt nur waehrend der drei Einladungslaeufe aktiv und wird danach beendet. Er muss die
|
||||
Nachricht nach echter Mailjet-Zustellung aus diesem Postfach read-only lesen. Ein
|
||||
Zugriff auf HiveGrid-Datenbank, Outbox, Queue oder Redis ist ausdruecklich kein gueltiger
|
||||
Zustellungsnachweis. Der Adapter:
|
||||
|
||||
- akzeptiert nur `POST /setup-mail`, begrenzt Request- und Mailgroesse und verifiziert IMAP-TLS;
|
||||
- lehnt einen `Authorization`-Header ab; ein Bearer-Token darf den Loopback-Prozess nie erreichen;
|
||||
- akzeptiert ausschliesslich `<MAILBOX_EMAIL-Localpart>+<Korrelationsschluessel>@<MAILBOX_EMAIL-Domain>`;
|
||||
Localpart, Domain und RFC-Laengengrenzen werden fail-closed validiert;
|
||||
- liefert nur die zum Korrelationsschluessel, Empfaenger und Zeitpunkt passende frisch zugestellte
|
||||
Setup-Nachricht genau einmal aus. Der Einstieg wird nicht vom zugestellten Mail-Header getragen,
|
||||
sondern fail-closed ueber die getrennten `p26a-admin-*`, `p26a-contact-*` und `p26a-project-*`
|
||||
Korrelationsschluessel an die zuvor sichtbare Browser-Aktion gebunden;
|
||||
- akzeptiert Setup-Links entweder am Standard-Origin unter `/<tenant>/setup/<token>` oder, nur bei
|
||||
explizit gesetztem `E2E_PORTAL_CUSTOM_DOMAIN_BASE_URL`, an genau diesem HTTPS-Root unter
|
||||
`/setup/<token>`. Derselbe einzelne direkte Link muss im Klartext und im HTML-CTA stehen; der
|
||||
Adapter dekodiert oder verfolgt keine Redirects. Nicht konfigurierte Hosts, Tracking-/Redirect-Links,
|
||||
Query und Fragment bleiben fail-closed abgelehnt;
|
||||
- protokolliert weder Authorization, Empfaenger, Mailtext noch `setupUrl` oder Einmal-Token;
|
||||
- antwortet mit `202`, `204` oder `404`, solange noch keine passende Zustellung vorliegt, und mit dem
|
||||
in `tests/e2e/utils/portal-setup-mail-fixture.ts` beschriebenen `200`-JSON-Vertrag bei Erfolg.
|
||||
|
||||
Die Zugangswerte kommen aus einer Datei mit Modus `600`; nichts davon in Shell-History,
|
||||
Workflow-Inputs, Tickets oder Logs kopieren. Der mitgelieferte Adapter bekommt Credentials nur ueber
|
||||
Environment-Variablen, schreibt keine Access-Logs und wird mit einem `EXIT`-Trap beendet:
|
||||
|
||||
```bash
|
||||
MAILBOX_ENV_FILE="${HOME}/.cli-tokens/hivegrid-sogo-mailbox"
|
||||
test -f "$MAILBOX_ENV_FILE"
|
||||
MAILBOX_ENV_MODE="$(stat -f '%Lp' "$MAILBOX_ENV_FILE" 2>/dev/null || stat -c '%a' "$MAILBOX_ENV_FILE")"
|
||||
test "$MAILBOX_ENV_MODE" = 600
|
||||
unset MAILBOX_ENV_MODE
|
||||
|
||||
: "${E2E_PORTAL_BASE_URL:?load the staging portal origin}"
|
||||
: "${E2E_PORTAL_TENANT_SLUG:?load the staging portal tenant slug}"
|
||||
export E2E_PORTAL_BASE_URL E2E_PORTAL_TENANT_SLUG
|
||||
|
||||
# Nur setzen, wenn die aktive Custom-Domain den Setup-Link erzeugt. Der Wert
|
||||
# muss ein exakter HTTPS-Root ohne Pfad, Query oder Fragment sein.
|
||||
# export E2E_PORTAL_CUSTOM_DOMAIN_BASE_URL=https://portal.customer.example
|
||||
# Fuer den Standardhost bleibt die Variable ungesetzt.
|
||||
|
||||
export PORTAL_SETUP_MAIL_IMAP_HOST=imap.mailbox.org
|
||||
export PORTAL_SETUP_MAIL_IMAP_PORT=993
|
||||
export PORTAL_SETUP_MAIL_ADAPTER_PORT=4010
|
||||
unset E2E_PORTAL_SETUP_MAIL_FIXTURE_TOKEN
|
||||
export E2E_PORTAL_SETUP_MAIL_FIXTURE_URL=http://127.0.0.1:4010/setup-mail
|
||||
|
||||
unset MAILBOX_EMAIL MAILBOX_APP_PASSWORD
|
||||
while IFS='=' read -r key value; do
|
||||
case "$key" in
|
||||
MAILBOX_EMAIL) MAILBOX_EMAIL="$value" ;;
|
||||
MAILBOX_APP_PASSWORD) MAILBOX_APP_PASSWORD="$value" ;;
|
||||
esac
|
||||
done <"$MAILBOX_ENV_FILE"
|
||||
unset key value
|
||||
: "${MAILBOX_EMAIL:?missing MAILBOX_EMAIL in the mode-600 mailbox file}"
|
||||
: "${MAILBOX_APP_PASSWORD:?missing MAILBOX_APP_PASSWORD in the mode-600 mailbox file}"
|
||||
export MAILBOX_EMAIL MAILBOX_APP_PASSWORD
|
||||
MAILBOX_LOCAL_PART="${MAILBOX_EMAIL%@*}"
|
||||
MAILBOX_DOMAIN="${MAILBOX_EMAIL##*@}"
|
||||
|
||||
(
|
||||
exec python3 scripts/customer-portal-uat/portal_setup_mail_adapter.py
|
||||
) >/dev/null 2>&1 &
|
||||
PORTAL_MAIL_ADAPTER_PID=$!
|
||||
unset MAILBOX_APP_PASSWORD
|
||||
cleanup_portal_mail_adapter() {
|
||||
if test -n "${PORTAL_MAIL_ADAPTER_PID:-}"; then
|
||||
kill "$PORTAL_MAIL_ADAPTER_PID" 2>/dev/null || true
|
||||
wait "$PORTAL_MAIL_ADAPTER_PID" 2>/dev/null || true
|
||||
fi
|
||||
if test -n "${PROJECT_INVITATION_FIXTURE_JSON:-}" && test -f "$PROJECT_INVITATION_FIXTURE_JSON"; then
|
||||
/bin/unlink "$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
fi
|
||||
unset PORTAL_MAIL_ADAPTER_PID
|
||||
unset PROJECT_INVITATION_FIXTURE_JSON
|
||||
unset MAILBOX_EMAIL MAILBOX_LOCAL_PART MAILBOX_DOMAIN
|
||||
unset PORTAL_SETUP_MAIL_IMAP_HOST PORTAL_SETUP_MAIL_IMAP_PORT PORTAL_SETUP_MAIL_ADAPTER_PORT
|
||||
}
|
||||
trap 'cleanup_portal_mail_adapter' EXIT
|
||||
trap 'cleanup_portal_mail_adapter; exit 129' HUP
|
||||
trap 'cleanup_portal_mail_adapter; exit 130' INT
|
||||
trap 'cleanup_portal_mail_adapter; exit 143' TERM
|
||||
|
||||
ADAPTER_STATUS=""
|
||||
for _ in $(seq 1 20); do
|
||||
ADAPTER_STATUS="$(curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
--request POST --header 'Content-Type: application/json' --data '{}' \
|
||||
"$E2E_PORTAL_SETUP_MAIL_FIXTURE_URL" || true)"
|
||||
test "$ADAPTER_STATUS" = 400 && break
|
||||
sleep 0.25
|
||||
done
|
||||
test "$ADAPTER_STATUS" = 400
|
||||
unset ADAPTER_STATUS
|
||||
```
|
||||
|
||||
Danach werden die gemeinsamen Browser-Eingaben gesetzt:
|
||||
|
||||
```bash
|
||||
: "${E2E_TENANT_EMAIL:?load the approved demo-pro-crm portal.manage_users account}"
|
||||
: "${E2E_TENANT_PASSWORD:?load the approved tenant-admin password}"
|
||||
: "${E2E_PORTAL_SETUP_MAIL_FIXTURE_URL:?start the loopback delivered-mail adapter first}"
|
||||
: "${E2E_PORTAL_INVITATION_SETUP_PASSWORD:?load a strong disposable setup password}"
|
||||
|
||||
export E2E_TENANT_CUSTOMER_PORTAL_SETTINGS_PATH=/demo-pro-crm/settings/customer-portal
|
||||
export E2E_PORTAL_INVITATION_ENTRYPOINT=admin
|
||||
export E2E_PORTAL_INVITATION_EXPECTED_PROJECT="$E2E_PORTAL_SUBCONTRACTOR_EXPECTED_PROJECT"
|
||||
export E2E_ALLOW_CUSTOMER_PORTAL_P26A_INVITATION_MUTATION=1
|
||||
export REQUIRE_E2E_READINESS=1
|
||||
export REQUIRE_E2E_MUTATING_READINESS=1
|
||||
|
||||
RUN_KEY="p26a-admin-$(date -u +%Y%m%dt%H%M%Sz)-$$"
|
||||
export E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY="$RUN_KEY"
|
||||
export E2E_PORTAL_INVITATION_EMAIL="${MAILBOX_LOCAL_PART}+${RUN_KEY}@${MAILBOX_DOMAIN}"
|
||||
export E2E_PORTAL_INVITATION_FIRST_NAME=Portal
|
||||
export E2E_PORTAL_INVITATION_LAST_NAME="Admin-${RUN_KEY}"
|
||||
|
||||
pnpm run verify:e2e-customer-portal-invitation
|
||||
unset RUN_KEY E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY E2E_PORTAL_INVITATION_EMAIL E2E_PORTAL_INVITATION_FIRST_NAME E2E_PORTAL_INVITATION_LAST_NAME
|
||||
```
|
||||
|
||||
Der Lauf wird danach mit zwei weiteren, zuvor unbenutzten Plus-Adressen wiederholt. Fuer den
|
||||
Kontakt-Einstieg wird zuerst ein eindeutiger CRM-Kontakt mit der frischen Adresse als Primaeradresse
|
||||
angelegt. Der Harness vergleicht die PortalUser-Auswahl vor und nach der Einladung und vergibt den
|
||||
erwarteten Projektzugriff anschliessend ueber die bestehende Portalverwaltung:
|
||||
|
||||
```bash
|
||||
RUN_KEY="p26a-contact-$(date -u +%Y%m%dt%H%M%Sz)-$$"
|
||||
export E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY="$RUN_KEY"
|
||||
export E2E_PORTAL_INVITATION_EMAIL="${MAILBOX_LOCAL_PART}+${RUN_KEY}@${MAILBOX_DOMAIN}"
|
||||
export E2E_PORTAL_INVITATION_ENTRYPOINT=contact
|
||||
CONTACT_FIRST_NAME=Portal
|
||||
CONTACT_LAST_NAME="Contact-${RUN_KEY}"
|
||||
|
||||
# Jetzt einen frischen Kontakt mit exakt CONTACT_FIRST_NAME, CONTACT_LAST_NAME
|
||||
# und E2E_PORTAL_INVITATION_EMAIL als Primaeradresse anlegen.
|
||||
: "${E2E_PORTAL_INVITATION_CONTACT_PATH:?load the fresh contact detail path}"
|
||||
|
||||
pnpm run verify:e2e-customer-portal-invitation
|
||||
unset RUN_KEY CONTACT_FIRST_NAME CONTACT_LAST_NAME E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY E2E_PORTAL_INVITATION_EMAIL E2E_PORTAL_INVITATION_CONTACT_PATH
|
||||
```
|
||||
|
||||
Fuer den Projekt-Einstieg legt der lokale Operator-Fixture einen dritten frischen Kontakt an und
|
||||
ordnet ihn dem erwarteten Projekt privat zu. Er verwendet nur den eingeschraenkten
|
||||
`hivegrid_app`-/RLS-Kontext aus der einmaligen Operator-Umgebung. Plan, Apply und Verify geben weder
|
||||
Empfaenger noch Namen oder Korrelationsschluessel aus; der Verify-Handoff enthaelt nur Pfad und
|
||||
`ProjektKontakt`-Zeilen-ID:
|
||||
|
||||
```bash
|
||||
RUN_KEY="p26a-project-$(date -u +%Y%m%dt%H%M%Sz)-$$"
|
||||
export E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY="$RUN_KEY"
|
||||
export E2E_PORTAL_INVITATION_EMAIL="${MAILBOX_LOCAL_PART}+${RUN_KEY}@${MAILBOX_DOMAIN}"
|
||||
export E2E_PORTAL_INVITATION_ENTRYPOINT=project
|
||||
|
||||
PROJECT_INVITATION_FIXTURE_JSON="$(mktemp)"
|
||||
chmod 600 "$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
pnpm --silent customer-portal:uat:project-invitation-contact --plan >"$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
jq -e '
|
||||
.mode == "plan" and
|
||||
(.complete | type) == "boolean" and
|
||||
(.handoff == null or (
|
||||
(.handoff | keys | sort) == ["projectContactRowId", "projectPath"] and
|
||||
(.handoff.projectPath | type) == "string" and
|
||||
(.handoff.projectContactRowId | type) == "string"
|
||||
))
|
||||
' "$PROJECT_INVITATION_FIXTURE_JSON" >/dev/null
|
||||
if jq -e '.complete == false' "$PROJECT_INVITATION_FIXTURE_JSON" >/dev/null; then
|
||||
pnpm --silent customer-portal:uat:project-invitation-contact \
|
||||
--apply \
|
||||
--confirm=demo-pro-crm >"$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
jq -e '.mode == "apply" and .complete == true' "$PROJECT_INVITATION_FIXTURE_JSON" >/dev/null
|
||||
fi
|
||||
pnpm --silent customer-portal:uat:project-invitation-contact --verify >"$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
jq -e '.mode == "verify" and .complete == true and .handoff != null' "$PROJECT_INVITATION_FIXTURE_JSON" >/dev/null
|
||||
export E2E_PORTAL_INVITATION_PROJECT_PATH="$(jq -er '.handoff.projectPath' "$PROJECT_INVITATION_FIXTURE_JSON")"
|
||||
export E2E_PORTAL_INVITATION_PROJECT_CONTACT_ROW_ID="$(jq -er '.handoff.projectContactRowId' "$PROJECT_INVITATION_FIXTURE_JSON")"
|
||||
/bin/unlink "$PROJECT_INVITATION_FIXTURE_JSON"
|
||||
unset PROJECT_INVITATION_FIXTURE_JSON
|
||||
|
||||
pnpm run verify:e2e-customer-portal-invitation
|
||||
unset RUN_KEY E2E_PORTAL_SETUP_MAIL_FIXTURE_KEY E2E_PORTAL_INVITATION_EMAIL
|
||||
unset E2E_PORTAL_INVITATION_PROJECT_PATH E2E_PORTAL_INVITATION_PROJECT_CONTACT_ROW_ID
|
||||
|
||||
cleanup_portal_mail_adapter
|
||||
trap - EXIT HUP INT TERM
|
||||
```
|
||||
|
||||
Bei einem Loopback-Adapter bleibt `E2E_PORTAL_SETUP_MAIL_FIXTURE_TOKEN` ungesetzt. Ein externer
|
||||
Adapter verlangt weiterhin HTTPS plus Bearer-Token. Empfaenger, Mailinhalt, Setup-URL und Einmal-Token
|
||||
duerfen weder der Adapter noch der Testlauf ausgeben.
|
||||
|
||||
Der manuelle `customer_portal_invitation`-Dispatch im
|
||||
`e2e-readiness.yml`-Workflow fuehrt denselben fokussierten Admin-Test aus. Die normale naechtliche
|
||||
Readiness-Lane bleibt nicht mutierend. Vor dem Dispatch muessen die drei Adapter-/Passwort-Secrets,
|
||||
die beiden Tenant-Admin-Secrets und die Repository-Variable
|
||||
`E2E_PORTAL_INVITATION_EXPECTED_PROJECT` aus dem unmittelbar davor geprueften UAT-Handoff gesetzt
|
||||
sein. Der Workflow erzeugt Empfaenger und Korrelationsschluessel aus der Run-ID nur innerhalb des
|
||||
einzelnen Testschritts und schreibt sie nicht nach `GITHUB_ENV`.
|
||||
|
||||
Der Proof ist nur gueltig, wenn der fokussierte Lauf ohne Skip besteht und nachweist: Einladung,
|
||||
echte Zustellung, Setup-Link-GET ohne Session, ausdrueckliche Datenschutzannahme, erst danach Session
|
||||
und Projektzugriff sowie Replay-Denial in einem frischen Browserkontext.
|
||||
|
||||
### P26c: Custom-Domain schrittweise
|
||||
|
||||
P26c verwendet ausschliesslich den reservierten Staging-Testhost `p26-uat.saas-ctrl.com`. Die drei
|
||||
Reisen werden wegen Checkout, DNS-Propagation und ACME bewusst nacheinander ausgefuehrt; ein kompletter
|
||||
Dateilauf vor Erfuellung dieser Zwischenzustaende ist kein gueltiger Proof.
|
||||
|
||||
Zuerst wird der eigenstaendige Starter-Vertrag bewiesen. Der Commercial-Fixture hat diesen Tenant
|
||||
absichtlich ohne Portal-Add-on und ohne Providerhistorie vorbereitet. Der erste Lauf muss auf den
|
||||
Mollie-Testcheckout mit exakt 23,79 EUR brutto fuehren; ein bereits sichtbares Domain-Eingabefeld ist nur bei
|
||||
einem dokumentierten Wiederholungslauf zulaessig und ersetzt den initialen Checkout-Proof nicht.
|
||||
|
||||
```bash
|
||||
export E2E_STARTER_TENANT_EMAIL=customer-portal-admin-uat@saas-ctrl.com
|
||||
export E2E_STARTER_TENANT_PASSWORD="$CUSTOMER_PORTAL_UAT_PASSWORD"
|
||||
export E2E_STARTER_TENANT_DASHBOARD_PATH=/demo-starter-crm/dashboard
|
||||
export E2E_ALLOW_CUSTOMER_PORTAL_P26C_MUTATION=1
|
||||
export REQUIRE_E2E_READINESS=1
|
||||
export REQUIRE_E2E_MUTATING_READINESS=1
|
||||
E2E_OPERATOR_CHECKPOINTS=1 pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "exact recurring customer-portal add-on checkout" \
|
||||
--project=chromium \
|
||||
--headed \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Am lokalen Operator-Checkpoint die Mollie-Testzahlung als bezahlt abschliessen, zur
|
||||
HiveGrid-Adminroute zurueckkehren und den Playwright-Lauf fortsetzen. Danach muss
|
||||
der wiederkehrende Erwerb aktiv sein und der Domain-Self-Service ohne erneuten Portal-Kauf erscheinen:
|
||||
|
||||
```bash
|
||||
pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "add-on is active after the Mollie return" \
|
||||
--project=chromium \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Erst danach folgt der getrennte Business-Pfad fuer das einmalige Domain-Setup. Die Reservierung startet
|
||||
bewusst noch keinen Checkout: Der Server verlangt vor dem Kauf einen frischen TXT-Eigentumsnachweis.
|
||||
|
||||
```bash
|
||||
export E2E_PORTAL_DOMAIN_CLAIM_HOSTNAME=p26-uat.saas-ctrl.com
|
||||
export E2E_PORTAL_CUSTOM_DOMAIN_BASE_URL=https://p26-uat.saas-ctrl.com
|
||||
export E2E_ALLOW_CUSTOMER_PORTAL_P26C_MUTATION=1
|
||||
export REQUIRE_E2E_READINESS=1
|
||||
export REQUIRE_E2E_MUTATING_READINESS=1
|
||||
|
||||
E2E_OPERATOR_CHECKPOINTS=1 pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "claims a domain" \
|
||||
--project=chromium \
|
||||
--headed \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Am lokalen Operator-Checkpoint den nach der Reservierung einmalig angezeigten TXT-Proof setzen, ohne
|
||||
ihn in Logs, Tickets oder Git zu kopieren, sowie den CNAME. Erst danach den Playwright-Lauf
|
||||
fortsetzen:
|
||||
|
||||
```text
|
||||
p26-uat.saas-ctrl.com CNAME staging-domains.portal.hivegrid.de
|
||||
```
|
||||
|
||||
Nach DNS-Propagation wird zuerst der Eigentumsnachweis bestaetigt und erst dann der claim-gebundene
|
||||
356,99-EUR-Mollie-Testcheckout brutto gestartet:
|
||||
|
||||
```bash
|
||||
E2E_OPERATOR_CHECKPOINTS=1 pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "verifies ownership" \
|
||||
--project=chromium \
|
||||
--headed \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Am lokalen Operator-Checkpoint den gestarteten Mollie-Testcheckout abschliessen, auf die Admin-Route
|
||||
zurueckkehren und den Playwright-Lauf fortsetzen. Danach den reconcilierten Zustand `awaiting_dns`
|
||||
pruefen. Anschliessend wird TXT/CNAME erneut gegen den bezahlten Claim
|
||||
verifiziert, damit die Bereitstellung starten darf:
|
||||
|
||||
```bash
|
||||
pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "verifies DNS" \
|
||||
--project=chromium \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Der letzte DNS-Schritt zaehlt nur, wenn TXT und CNAME bestaetigt sind und der Domain-Zustand in
|
||||
`provisioning`/`active` uebergeht; die UI-Meldung fuer noch unvollstaendige DNS-Propagation ist kein
|
||||
Abschluss-Proof. Nach Controller-Ack, ACME-Zertifikat und aktivem Host folgt der Fallback-Nachweis:
|
||||
|
||||
```bash
|
||||
pnpm exec playwright test \
|
||||
tests/e2e/va-p26c-customer-portal-domain.smoke.spec.ts \
|
||||
--grep "active custom host" \
|
||||
--project=chromium \
|
||||
--reporter=line
|
||||
```
|
||||
|
||||
Die P26b- und P26c-Mutationen bleiben damit explizit hinter
|
||||
`E2E_ALLOW_CUSTOMER_PORTAL_P26B_MUTATION=1` beziehungsweise
|
||||
`E2E_ALLOW_CUSTOMER_PORTAL_P26C_MUTATION=1`. Produktive oder bereits verwendete Hosts sind verboten.
|
||||
`E2E_OPERATOR_CHECKPOINTS=1` wird nur fuer den jeweiligen lokalen `--headed`-Befehl gesetzt und bricht
|
||||
in CI oder headless fail-closed ab. Es bleibt damit nicht fuer einen spaeteren Befehl im Shell-Environment
|
||||
zurueck.
|
||||
|
||||
## Controller-Monitoring und Alerts
|
||||
|
||||
Der Controller bleibt ohne FQDN und bindet seinen einzigen Host-Port ausschliesslich an
|
||||
`10.42.0.3:19120:8080`. Vom zentralen `control`-Host muessen `/healthz` und `/readyz` HTTP 200 liefern;
|
||||
`/metrics` muss ohne Bearer-Token HTTP 401 liefern. Der Token liegt nur als Modus-`0640`-Datei im
|
||||
zentralen Observability-Stack und wird von Prometheus ueber `authorization.credentials_file`
|
||||
gelesen, niemals inline in YAML oder Shell-History.
|
||||
|
||||
Der Scrape-Job traegt die Pflicht-Labels `project=HiveGrid`, `environment=staging`,
|
||||
`server=staging-1` und `service=portal-domain-controller`. Als Live-Proof gelten `up=1`,
|
||||
`hivegrid_portal_domain_controller_ready=1` und ein erfolgreicher Command-Zaehler. Folgende
|
||||
Grafana-managed Alerts sind Pflicht:
|
||||
|
||||
- `up < 1` fuer zwei Minuten, `noDataState=Alerting`;
|
||||
- `hivegrid_portal_domain_controller_ready < 1` fuer fuenf Minuten, `noDataState=Alerting`;
|
||||
- Anstieg von `hivegrid_portal_domain_commands_total{status="failed"}` ueber 15 Minuten,
|
||||
`noDataState=OK`.
|
||||
|
||||
Der Abschluss-Proof umfasst einen kontrollierten Firing- und Resolved-Zyklus samt zugestellter
|
||||
E-Mail ueber den zentralen Estate-Monitoring-Contact-Point. Die SMTP-Zugangsdaten bleiben in der
|
||||
Mode-`0600`-Umgebung des zentralen Grafana und werden weder aus dem HiveGrid-App-Container noch aus
|
||||
diesem Repository bezogen.
|
||||
|
||||
## Evidence und Rollback
|
||||
|
||||
Als Ticket-Proof werden nur Revision, Zeitpunkt, der redigierte `--verify`-Output, Browser-Ergebnis und
|
||||
Request-/Run-IDs erfasst. DB-URLs, API-Keys, Passwoerter, TXT-Tokens und komplette Providerantworten
|
||||
bleiben ausserhalb von Git und Tickets.
|
||||
|
||||
Katalog- und Fixture-Mutationen schreiben einen Audit-Eintrag. Das Tool loescht absichtlich keine
|
||||
Provider- oder Portalzeilen. Falls der Operator abbrechen muss:
|
||||
|
||||
1. Browser-UAT stoppen und keine weitere Mutation freigeben.
|
||||
2. Einen driftenden Lago-Code nicht automatisch korrigieren; Vertrag im Lago-Operator-UI pruefen.
|
||||
3. Synthetische PortalUser und ihre UAT-Zugriffe ueber den Tenant-Admin deaktivieren.
|
||||
4. Falls das Portal vor dem UAT nicht publiziert war, den vorher dokumentierten Publish-Zustand im
|
||||
Tenant-Admin wiederherstellen.
|
||||
5. Die Commercial-UAT-Identitaet bei Bedarf in beiden Demo-Tenants deaktivieren; keine Provider- oder
|
||||
Produktzeile als vermeintlichen Rollback loeschen.
|
||||
6. Erst nach redigiertem Audit-/DB-Proof einen weiteren `--plan`-Lauf starten.
|
||||
|
||||
## Lokale Verifikation
|
||||
|
||||
```bash
|
||||
pnpm verify:customer-portal-uat
|
||||
pnpm exec eslint --max-warnings=0 scripts/customer-portal-uat/*.ts
|
||||
```
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"ticket": "HIVEGRID-73",
|
||||
"captured_at": "2026-07-16T03:37:54Z",
|
||||
"contains_pii": false,
|
||||
"encrypted_backup": {
|
||||
"provider": "Hetzner Object Storage",
|
||||
"bucket": "hivegrid-prod",
|
||||
"prefix": "backups/platform-leads/20260716T033754Z",
|
||||
"encryption": "age",
|
||||
"recipient_fingerprint": "age1pu5es0fx9vrj2c9m6065drvngs8dsaey4rapv77h83xmt4k3epqsu7y2ez"
|
||||
},
|
||||
"sources": [
|
||||
{
|
||||
"source": "lead-inbox-postgres",
|
||||
"rows": 1,
|
||||
"minimum_captured_at": "2026-06-06T18:14:43.014219Z",
|
||||
"maximum_captured_at": "2026-06-06T18:14:43.014219Z",
|
||||
"logical_source_sha256": "d6e96dd79c290f3ed06eaff49b460afc62e7021ae4b2020c673023c7bddaee70",
|
||||
"backup_object": "lead-inbox-postgres.dump.age",
|
||||
"encrypted_sha256": "30f604e9afa29a68af553f9f381ae5f6d64830bb752d05c9c9c99b64e97d1fbc",
|
||||
"restore_proof": {
|
||||
"engine": "postgres:16-alpine",
|
||||
"restored_rows": 1,
|
||||
"result": "passed"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "hivegrid-website-production-sqlite",
|
||||
"rows": 0,
|
||||
"container_has_database_file": false,
|
||||
"persistent_volume_configured": false,
|
||||
"backup_object": null,
|
||||
"restore_proof": {
|
||||
"result": "not_applicable_empty_source"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "hivegrid-website-staging-sqlite",
|
||||
"rows": 1,
|
||||
"minimum_captured_at": "2026-05-30T17:40:01Z",
|
||||
"maximum_captured_at": "2026-05-30T17:40:01Z",
|
||||
"source_sha256": "1b4e98580f23e7910c7081b87e6400f5945ee62f0e243b31cd93efcfb1721bfa",
|
||||
"backup_object": "hivegrid-website-staging-sqlite.db.age",
|
||||
"encrypted_sha256": "0540e246de19e2be57a4c54d5ca260afc103d61bb02c37e01ef8f1471ddf1f13",
|
||||
"restore_proof": {
|
||||
"sqlite_integrity_check": "ok",
|
||||
"restored_rows": 1,
|
||||
"result": "passed"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "toolctrl-website-file-store",
|
||||
"status": "erroneously_inventoried_unrelated_source",
|
||||
"excluded_from_hivegrid_import": true,
|
||||
"canonical_records": 2,
|
||||
"ndjson_lines": 2,
|
||||
"csv_lines_including_header": 3,
|
||||
"ndjson_sha256": "587a310440874f5221da5c1e37bbfbe0803e550b016929d460561556b70a04f4",
|
||||
"csv_sha256": "5981fe23b23a3cf6778fa1d8c76e04fba6cfccebde83ff8747f7f23b34ba3e91",
|
||||
"backup_object": "toolctrl-leads.tar.age",
|
||||
"encrypted_sha256": "483613b87c4b951252cdd144b4a24d6983e0e2b1c6211071e3ad8792b6e82980",
|
||||
"restore_proof": {
|
||||
"hashes_match": true,
|
||||
"result": "passed"
|
||||
},
|
||||
"retention": {
|
||||
"deletion_authorized": false,
|
||||
"decision": "separate_explicit_owner_authorization_required",
|
||||
"action_until_decision": "retain_encrypted_object_without_import"
|
||||
}
|
||||
},
|
||||
{
|
||||
"source": "hivegrid-cloud-production-lead",
|
||||
"rows": 0,
|
||||
"event_rows": 0
|
||||
},
|
||||
{
|
||||
"source": "hivegrid-cloud-staging-lead",
|
||||
"rows": 0,
|
||||
"event_rows": 0
|
||||
}
|
||||
],
|
||||
"cutover_gate": {
|
||||
"backup_created": true,
|
||||
"offsite_copy_created": true,
|
||||
"restore_proved": true,
|
||||
"legacy_service_mutated": false,
|
||||
"legacy_dns_mutated": false,
|
||||
"retention_deletion_authorized": false
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,10 @@
|
||||
# Lago Production Prep Runbook
|
||||
|
||||
This runbook defines the production target for HiveGrid billing after AD-27.
|
||||
It is a preparation document only. Do not provision or enable production billing
|
||||
from this document until the external gates in `EXT-16` are complete.
|
||||
The isolated production foundation was provisioned on 2026-07-27 after explicit
|
||||
operator approval. Provisioning does not authorize live billing: checkout and
|
||||
provider side effects remain disabled until the external gates in `EXT-16` are
|
||||
complete.
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -28,38 +30,54 @@ Out of scope for v1:
|
||||
|
||||
## Target Topology
|
||||
|
||||
| Item | Decision |
|
||||
|---|---|
|
||||
| Coolify project | `HiveGrid` |
|
||||
| Coolify environment | `production` |
|
||||
| Resource name | `lago-production` |
|
||||
| Initial host | The current HiveGrid production host, while Hetzner capacity is constrained |
|
||||
| Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted |
|
||||
| Public access | None by default |
|
||||
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
||||
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
||||
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
||||
| Item | Decision |
|
||||
| ---------------------------- | ----------------------------------------------------------------------------------- |
|
||||
| Coolify project | `HiveGrid` |
|
||||
| Coolify environment | `production` |
|
||||
| Resource name | `lago-production` |
|
||||
| Current host | `hivegrid-prod-worker`; API bound only to the private `10.0.0.0/16` network |
|
||||
| Long-term host | Dedicated billing host if measured capacity or blast-radius requirements justify it |
|
||||
| Public access | None by default |
|
||||
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
||||
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
||||
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
||||
|
||||
The initial host decision is intentionally pragmatic: production gets its own
|
||||
Lago resource and data stores, but it may share the physical server with other
|
||||
HiveGrid production services until infrastructure capacity allows a stricter
|
||||
split. The isolation boundary is the Coolify resource, secrets, database,
|
||||
Redis, storage and backup path, not the current machine count.
|
||||
The current placement is intentionally pragmatic: production has its own Lago
|
||||
Coolify resource, Postgres, Redis, versioned object bucket, secrets and backup
|
||||
path, while the containers share the existing worker VM. The host had 16 GB RAM
|
||||
and approximately 145 GB free disk at provisioning time. No public Lago DNS
|
||||
record or public port exists.
|
||||
|
||||
Pinned runtime images:
|
||||
|
||||
- `getlago/api:v1.50.0`
|
||||
- `getlago/front:v1.50.0`
|
||||
- `postgres:15-alpine`
|
||||
- `redis:7-alpine`
|
||||
- `getlago/lago-gotenberg:8.15`
|
||||
|
||||
API, migration runner, general worker, billing worker and the periodic
|
||||
`api-clock` scheduler use the same pinned Lago API image.
|
||||
|
||||
Invoice/file storage uses the private, versioned Hetzner Object Storage bucket
|
||||
`hivegrid-prod-lago` in `fsn1`. It is separate from the HiveGrid application
|
||||
bucket and from `hivegrid-staging-lago`.
|
||||
|
||||
## Required Production Secrets
|
||||
|
||||
Store these only in the runtime secret store. Never put values in docs, Git,
|
||||
tickets, screenshots or copied logs.
|
||||
|
||||
| Secret | Owner | Notes |
|
||||
|---|---|---|
|
||||
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
||||
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
||||
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
||||
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
||||
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
||||
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| Secret | Owner | Notes |
|
||||
| --------------------- | ----------------------- | ----------------------------------------------------------- |
|
||||
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
||||
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
||||
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
||||
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
||||
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
||||
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `BILLING_MOLLIE_WEBHOOK_URL` | HiveGrid `worker-general` | `https://api.hivegrid.de/v1/billing/mollie/webhooks`; required only once provider side effects are enabled. |
|
||||
|
||||
Do not reuse staging/testmode keys in production. Rotate keys when moving from
|
||||
pilot validation to real customer billing.
|
||||
@@ -84,12 +102,23 @@ Minimum backup schedule:
|
||||
- Backup before destructive billing test data cleanup.
|
||||
- Retention aligned with the final accounting and tax retention decision.
|
||||
|
||||
Implemented production schedule:
|
||||
|
||||
- `hivegrid-lago-backup.timer` runs daily at 03:00 UTC with up to 15 minutes
|
||||
randomized delay.
|
||||
- `workflows/hivegrid-lago-backup` backs up a compressed Postgres dump, Redis
|
||||
persistence, RSA keys and local Lago storage into the encrypted HiveGrid
|
||||
restic repository in `hivegrid-prod-backups`.
|
||||
- Retention is 14 daily, 8 weekly and 12 monthly snapshots.
|
||||
- The versioned `hivegrid-prod-lago` bucket remains the primary file store.
|
||||
|
||||
Restore drill before go-live:
|
||||
|
||||
1. Create a fresh disposable restore target.
|
||||
2. Restore the latest Lago Postgres dump.
|
||||
3. Restore or attach invoice PDF storage.
|
||||
4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
|
||||
4. Start Lago API, frontend, worker, billing worker, clock, PDF service and
|
||||
Redis.
|
||||
5. Verify `/health`.
|
||||
6. Read one known customer, subscription and invoice.
|
||||
7. Verify HiveGrid can resolve the corresponding local billing state without
|
||||
@@ -98,6 +127,133 @@ Restore drill before go-live:
|
||||
The restore drill date, operator and result must be recorded before production
|
||||
billing is enabled.
|
||||
|
||||
### Restore proof
|
||||
|
||||
2026-07-27, Raffael:
|
||||
|
||||
- Versionierter Infra-Proof: Commit `936a32a`.
|
||||
- Encrypted snapshot `3d8a1deb` completed after the production organization and
|
||||
API key existed.
|
||||
- The snapshot was restored into disposable Postgres and Redis containers.
|
||||
- Checksum, schema history, exactly one organization and exactly one API key
|
||||
were verified.
|
||||
- Disposable Lago API, frontend, worker, billing worker and PDF service started;
|
||||
`/health` passed.
|
||||
- A canary object was written to, read from and fully removed from the versioned
|
||||
production bucket with a matching SHA-256.
|
||||
- The disposable containers, volume, network and restore directory were removed.
|
||||
|
||||
2026-07-28, Raffael:
|
||||
|
||||
- Pre-upgrade encrypted snapshot `cc8c5c88` was restored successfully before
|
||||
the first image change.
|
||||
- Bridge snapshot `3c245001` was created and restored on Lago `v1.45.0`
|
||||
immediately before the target deployment.
|
||||
- Post-upgrade encrypted snapshot `176fdf80` was restored with Postgres, Redis,
|
||||
RSA/storage mounts, API, frontend, general worker, billing worker, PDF and
|
||||
the new clock scheduler.
|
||||
- The restored database contained exactly one production organization and one
|
||||
production API key; customer, plan, subscription and invoice tables remained
|
||||
empty.
|
||||
- Every disposable resource and restore directory was removed after the drill.
|
||||
|
||||
## Upgrade record: 2026-07-28
|
||||
|
||||
Lago production was upgraded from `v1.27.1` to `v1.50.0` through the required
|
||||
bridge chain:
|
||||
|
||||
```text
|
||||
v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0
|
||||
-> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0
|
||||
```
|
||||
|
||||
Required migration work completed:
|
||||
|
||||
- `v1.28.1`: invoice billing-entity sequential-ID job, `0` records left.
|
||||
- `v1.30.0`: organization-ID migration, nothing left to migrate.
|
||||
- `v1.32.0`: required payment job, nothing left to migrate.
|
||||
- `v1.42.0`: charge/fixed-charge jobs, nothing left to migrate.
|
||||
- `v1.44.0`: subscription event-date job, nothing left to migrate.
|
||||
- `v1.45.0`: wallet dry-run and write pass, `0` wallets and `0` errors.
|
||||
- `v1.50.0`: no native payment-provider customer or integration exists, so
|
||||
provider-specific payment-method backfills were not applicable.
|
||||
|
||||
The official `api-clock` service was added. It starts `26` scheduled Lago jobs,
|
||||
including invoice finalization, billing, retries and subscription lifecycle
|
||||
work. The production restore drill now restores and verifies the clock
|
||||
container as part of the disposable topology.
|
||||
|
||||
Postgres remains on the patched `postgres:15-alpine` image (`15.18` during the
|
||||
upgrade). The optional Lago Partman image was not adopted because its
|
||||
`15.0-alpine` tag would downgrade the database binaries. Lago's Partman-aware
|
||||
event migrations skip partition setup when the extension is unavailable; this
|
||||
production instance does not run the high-volume events processor.
|
||||
|
||||
Final proof:
|
||||
|
||||
- Private API health: `v1.50.0`.
|
||||
- Database migrations: `0` down.
|
||||
- Production state unchanged: `1` organization, `1` API key, `0` customers,
|
||||
plans, subscriptions, invoices and wallets.
|
||||
- Queues: `0` pending, `0` retries, `0` dead.
|
||||
- PDF service: HTTP `200`.
|
||||
- Object-storage HEAD through Active Storage succeeded.
|
||||
- Running HiveGrid production API to Lago call: authenticated HTTP `200`.
|
||||
- `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains active.
|
||||
- Clock starts all `26` schedules and completes jobs with `error=nil`.
|
||||
- No API, general-worker or billing-worker error was found after deployment.
|
||||
|
||||
## Canonical billing catalog
|
||||
|
||||
Production contains the same controlled `BILL-11` catalog as Staging:
|
||||
|
||||
- Version: `2026-07-29.1`
|
||||
- SHA-256 digest:
|
||||
`a8ba1affd93e1b870d016284457594988c7fc5f58fdf70c3d57be80b3c915b2c`
|
||||
- Public packages: Core / Swarm / Colony with stable technical codes
|
||||
`starter` / `pro` / `business`
|
||||
- Two monthly-billed variants per package, including the 12-month SEPA
|
||||
commitment
|
||||
- Four fixed recurring storage packages
|
||||
- German VAT tax code `de-vat-19` at `19 %`, exclusive
|
||||
- No usage metering
|
||||
|
||||
The Production apply and follow-up verify completed on 2026-07-29. The verify
|
||||
returned no changed actions and the digest exactly matches Staging. Redacted
|
||||
pre-change snapshot:
|
||||
|
||||
```text
|
||||
tmp/billing-catalog-snapshots/production-2026-07-29T06-52-23-908Z.json
|
||||
```
|
||||
|
||||
Operator sequence:
|
||||
|
||||
```bash
|
||||
NODE_ENV=production pnpm billing:catalog --target=production --plan
|
||||
NODE_ENV=production pnpm billing:catalog \
|
||||
--target=production \
|
||||
--apply \
|
||||
--confirm=production:2026-07-29.1
|
||||
NODE_ENV=production pnpm billing:catalog --target=production --verify
|
||||
```
|
||||
|
||||
The catalog is active, but real Production payment side effects are not.
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains mandatory
|
||||
until all Mollie/Legal gates below are closed. Production must not receive a
|
||||
test-mode Mollie key.
|
||||
|
||||
2026-07-29 `BILL-11` closeout proof:
|
||||
|
||||
- The read-only Production verify completed with catalog version
|
||||
`2026-07-29.1`, the same Staging digest and only `noop` actions.
|
||||
- Gitea promotion run `11477` promoted the live Staging revision
|
||||
`549f708d9809e9ba839c15366943918679a034d3` without rebuilding it.
|
||||
- Core, Admin, Portal and API health/readiness plus MCP, Realtime and all five
|
||||
worker groups are healthy on that revision.
|
||||
- `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remained active
|
||||
after promotion and no Mollie API key is configured in Production. No
|
||||
Production payment or customer invoice was created by this proof.
|
||||
|
||||
## Access And Domain
|
||||
|
||||
Default: no public Lago domain.
|
||||
@@ -121,10 +277,25 @@ Mollie live activation is blocked until:
|
||||
approved.
|
||||
- Live webhook endpoint and return URLs are set for the production host.
|
||||
- A live low-value payment and refund rehearsal is approved by an operator.
|
||||
- Mollie has explicitly raised the SEPA Direct Debit transaction limit above
|
||||
the highest gross monthly invoice. The default documented limit is
|
||||
`EUR 1,000`; Colony is already above it before optional seats or storage.
|
||||
|
||||
Until then, production must stay on manual billing or disabled self-service
|
||||
checkout.
|
||||
|
||||
The first SEPA authorization uses a customer-present payment method and
|
||||
`sequenceType=first`; it does not attempt a direct-debit payment before a
|
||||
mandate exists. Later Lago invoices use `sequenceType=recurring` with the
|
||||
stored Mollie customer and mandate. See Mollie's
|
||||
[recurring-payments guide](https://docs.mollie.com/gu/docs/recurring-payments)
|
||||
and [SEPA Direct Debit limits](https://docs.mollie.com/docs/sepa-direct-debit).
|
||||
|
||||
The current production gate is
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on
|
||||
`worker-general`. This disables the durable provider subscriber even under
|
||||
required RLS. Do not replace it with a test-mode Mollie key.
|
||||
|
||||
## Rollback
|
||||
|
||||
Rollback has two layers:
|
||||
@@ -139,12 +310,12 @@ Lago datasets.
|
||||
|
||||
## Go-Live Checklist
|
||||
|
||||
- [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
||||
- [ ] Dedicated production DB, Redis and storage are configured.
|
||||
- [ ] Production secrets are set and staging secrets are absent.
|
||||
- [ ] Backup job is configured.
|
||||
- [ ] Restore drill is completed and documented.
|
||||
- [x] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
||||
- [x] Dedicated production DB, Redis and storage are configured.
|
||||
- [x] Production secrets are set and staging secrets are absent.
|
||||
- [x] Backup job is configured.
|
||||
- [x] Restore drill is completed and documented.
|
||||
- [ ] Mollie live profile is verified.
|
||||
- [ ] Rechnungssteller, tax and imprint data are approved.
|
||||
- [ ] Web/API/worker health checks pass with production Lago config.
|
||||
- [x] Web/API/worker health checks pass with production Lago config.
|
||||
- [ ] Manual rollback path is tested from image tag and Lago backup.
|
||||
@@ -17,18 +17,19 @@ data. Production gets a separate `lago-production` service later.
|
||||
|
||||
## Current topology
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Coolify project | `node1-estate` |
|
||||
| Resource | `lago-staging` |
|
||||
| Resource UUID | `kh7l869ianltiioqgmm45kur` |
|
||||
| Server | `node1` |
|
||||
| Operator UI | `https://staging-lago.hivegrid.de` |
|
||||
| Public protection | Lago login |
|
||||
| Frontend tunnel | `http://127.0.0.1:8188` |
|
||||
| API tunnel | `http://127.0.0.1:3108` |
|
||||
| Mesh API URL for HiveGrid staging | `http://10.42.0.4:3108` |
|
||||
| Invoice PDF storage | Hetzner Object Storage bucket `hivegrid-staging-lago` |
|
||||
| Item | Value |
|
||||
| --------------------------------- | ----------------------------------------------------- |
|
||||
| Coolify project | `node1-estate` |
|
||||
| Resource | `lago-staging` |
|
||||
| Resource UUID | `kh7l869ianltiioqgmm45kur` |
|
||||
| Server | `node1` |
|
||||
| Lago version | `v1.50.0` |
|
||||
| Operator UI | `https://staging-lago.hivegrid.de` |
|
||||
| Public protection | Lago login |
|
||||
| Frontend tunnel | `http://127.0.0.1:8188` |
|
||||
| API tunnel | `http://127.0.0.1:3108` |
|
||||
| Mesh API URL for HiveGrid staging | `http://10.42.0.4:3108` |
|
||||
| Invoice PDF storage | Hetzner Object Storage bucket `hivegrid-staging-lago` |
|
||||
|
||||
Services in the stack:
|
||||
|
||||
@@ -38,6 +39,7 @@ Services in the stack:
|
||||
- Redis
|
||||
- Worker
|
||||
- Billing worker
|
||||
- Clock scheduler
|
||||
- PDF service
|
||||
|
||||
Invoice PDFs are stored in the dedicated staging object-storage bucket
|
||||
@@ -114,6 +116,7 @@ HiveGrid staging resources currently need these runtime variables:
|
||||
- `LAGO_TIMEOUT_MS`
|
||||
- `MOLLIE_API_KEY`
|
||||
- `MOLLIE_TIMEOUT_MS`
|
||||
- `BILLING_MOLLIE_WEBHOOK_URL=https://staging-api.hivegrid.de/v1/billing/mollie/webhooks`
|
||||
|
||||
Configured resources:
|
||||
|
||||
@@ -235,6 +238,50 @@ curl -fsS -X POST \
|
||||
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
|
||||
```
|
||||
|
||||
Lago subscription lifecycle webhook for staging:
|
||||
|
||||
```text
|
||||
https://staging-api.hivegrid.de/v1/billing/lago/webhooks
|
||||
```
|
||||
|
||||
Lago v1.50 signs the exact JSON body with the organization HMAC key and sends
|
||||
the result as Base64 in `X-Lago-Signature`. HiveGrid additionally requires
|
||||
`X-Lago-Signature-Algorithm: hmac` and the provider replay key in
|
||||
`X-Lago-Unique-Key`.
|
||||
|
||||
Roll out in this order:
|
||||
|
||||
1. Deploy an API revision containing the Lago webhook route.
|
||||
2. Verify an unsigned request returns `401`, not `404`.
|
||||
3. Set the Lago organization HMAC key as `LAGO_WEBHOOK_SECRET` on
|
||||
`hivegrid-api`; do not copy it into Git, command output or this runbook.
|
||||
4. Redeploy `hivegrid-api`.
|
||||
5. Create one Lago webhook endpoint with URL
|
||||
`https://staging-api.hivegrid.de/v1/billing/lago/webhooks` and
|
||||
`signature_algo=hmac`.
|
||||
6. Trigger a base-plan subscription test, then verify a `200` delivery,
|
||||
exactly one local `billing:provider-sync` idempotency row and no second
|
||||
Audit/Activity write on replay.
|
||||
|
||||
Do not create the Lago endpoint while the route still returns `404`: Lago
|
||||
retries non-2xx responses and would only create avoidable failed deliveries.
|
||||
|
||||
2026-07-29 live proof:
|
||||
|
||||
- The unsigned public probe returned HTTP `401`, proving the route is live and
|
||||
fails closed before payload processing.
|
||||
- Exactly one Lago webhook endpoint exists and uses HMAC.
|
||||
- The organization HMAC matches the local operator record and the running
|
||||
`hivegrid-api` environment.
|
||||
- A signed lifecycle event for the canonical active `business` subscription of
|
||||
`demo-pro-crm` returned `synced`; the identical replay returned `replayed`.
|
||||
- The database contained exactly one matching idempotency row, audit event and
|
||||
activity entry after both requests; the tenant remained `business|active`.
|
||||
- Six obsolete synthetic active subscriptions left by earlier checkout tests
|
||||
were terminated without a closing invoice or credit note. Exactly one active
|
||||
`business` subscription remains, and the invoice count stayed at `27`.
|
||||
- The cleanup is recoverable from the encrypted post-upgrade v1.50 snapshot.
|
||||
|
||||
## Start, stop and restart
|
||||
|
||||
Prefer the Coolify UI for routine operations:
|
||||
@@ -302,13 +349,14 @@ docker ps \
|
||||
--format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
|
||||
```
|
||||
|
||||
Create a dump:
|
||||
Create a compressed custom-format dump and encrypt it immediately:
|
||||
|
||||
```bash
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
docker exec -t <lago-postgres-container> \
|
||||
pg_dump -U <lago-db-user> -d <lago-db-name> \
|
||||
> "lago-staging-${timestamp}.sql"
|
||||
docker exec <lago-postgres-container> \
|
||||
pg_dump -U <lago-db-user> -d <lago-db-name> -Fc \
|
||||
| age -r <operator-age-recipient> \
|
||||
> "lago-staging-${timestamp}.dump.age"
|
||||
```
|
||||
|
||||
Store the dump outside the container host after creation. For short-lived
|
||||
@@ -316,18 +364,35 @@ staging tests, an operator-controlled encrypted local archive is acceptable.
|
||||
For longer-lived staging data, move the dump into the standard HiveGrid backup
|
||||
bucket flow.
|
||||
|
||||
Verify every archive before continuing:
|
||||
|
||||
```bash
|
||||
age -d -i ~/.config/sops/age/keys.txt lago-staging-<timestamp>.dump.age \
|
||||
| docker exec -i <lago-postgres-container> pg_restore --list \
|
||||
>/dev/null
|
||||
```
|
||||
|
||||
The 2026-07-28 upgrade used an encrypted pre-upgrade archive and restored it
|
||||
into a disposable database before the first image change. The restore drill
|
||||
reproduced `6` customers, `16` subscriptions and `27` invoices. A second
|
||||
encrypted archive was created and validated on the `v1.45.0` bridge before the
|
||||
final target deployment.
|
||||
|
||||
## Restore path
|
||||
|
||||
Restore only into `lago-staging` or a disposable local Lago instance. Never
|
||||
restore staging Lago data into production.
|
||||
|
||||
1. Stop Lago API, frontend, worker, billing worker and PDF service in Coolify.
|
||||
1. Stop Lago API, frontend, worker, billing worker, clock and PDF service in
|
||||
Coolify.
|
||||
2. Keep Postgres running.
|
||||
3. Restore the dump:
|
||||
3. Decrypt and restore the custom-format dump:
|
||||
|
||||
```bash
|
||||
cat lago-staging-<timestamp>.sql | docker exec -i <lago-postgres-container> \
|
||||
psql -U <lago-db-user> -d <lago-db-name>
|
||||
age -d -i ~/.config/sops/age/keys.txt lago-staging-<timestamp>.dump.age \
|
||||
| docker exec -i <lago-postgres-container> \
|
||||
pg_restore -U <lago-db-user> -d <lago-db-name> \
|
||||
--clean --if-exists --no-owner
|
||||
```
|
||||
|
||||
4. Start the full stack again in Coolify.
|
||||
@@ -350,33 +415,91 @@ Any email-capable configuration must be documented in this runbook before use.
|
||||
|
||||
## Invoice PDF storage decision
|
||||
|
||||
For the current staging pilot, Invoice PDFs may use Lago's local Coolify volume.
|
||||
This is acceptable because the service has no production data and no public
|
||||
customer-facing billing UI.
|
||||
Staging Invoice PDFs use the dedicated Hetzner Object Storage bucket
|
||||
`hivegrid-staging-lago`. API, worker and billing worker use the same
|
||||
S3-compatible endpoint and credentials. The local Coolify storage volume is
|
||||
not the authoritative PDF store.
|
||||
|
||||
Before production:
|
||||
|
||||
- configure a dedicated object storage bucket or production-grade persistent
|
||||
volume strategy,
|
||||
- configure a separate production object-storage bucket,
|
||||
- document backup and restore for generated PDFs,
|
||||
- keep staging and production storage separate.
|
||||
|
||||
## Upgrade procedure
|
||||
|
||||
1. Read Lago release notes for the target version.
|
||||
2. Create a Postgres dump.
|
||||
3. Note current image tags and Coolify resource settings.
|
||||
4. Update the API/frontend/worker/PDF image tags together.
|
||||
5. Redeploy through Coolify.
|
||||
6. Verify:
|
||||
1. Read Lago release notes and every migration guide between the current and
|
||||
target versions. Do not skip required bridge releases or backfill jobs.
|
||||
2. Create an encrypted Postgres dump and prove it restores into a disposable
|
||||
database.
|
||||
3. Export and encrypt the current Coolify resource configuration.
|
||||
4. Update API, frontend, RSA-key initializer, migration container, worker,
|
||||
billing worker and clock image tags together. Keep Postgres, Redis and PDF
|
||||
tags unchanged unless the release notes require a compatible change.
|
||||
5. Redeploy through Coolify with the deploy action, not restart.
|
||||
6. After each bridge release, wait for API health and run its required jobs
|
||||
before continuing.
|
||||
7. Verify:
|
||||
- API `/health`
|
||||
- frontend HTTP 200
|
||||
- real operator login
|
||||
- worker containers running
|
||||
- a test customer/subscription still readable
|
||||
- zero down migrations and empty failed/retry queues
|
||||
- a test customer/subscription still readable through HiveGrid staging
|
||||
- an existing invoice PDF downloads from object storage
|
||||
|
||||
Rollback is image-tag rollback plus DB restore from the dump created before
|
||||
the upgrade.
|
||||
|
||||
## Upgrade record: 2026-07-28
|
||||
|
||||
Lago staging was upgraded from `v1.27.1` to `v1.50.0` through the required
|
||||
bridge chain:
|
||||
|
||||
```text
|
||||
v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0
|
||||
-> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0
|
||||
```
|
||||
|
||||
Required migration work completed:
|
||||
|
||||
- `v1.28.1`: invoice billing-entity sequential-ID job, `0` records left.
|
||||
- `v1.30.0`: organization-ID migration, including `4` credit-note items.
|
||||
- `v1.32.0`: required payment job, nothing left to migrate.
|
||||
- `v1.42.0`: charge/fixed-charge jobs, nothing left to migrate.
|
||||
- `v1.44.0`: `last_received_event_on` backfill for the staging organization.
|
||||
- `v1.45.0`: wallet dry-run and write pass, `0` wallets and `0` errors.
|
||||
- `v1.50.0`: no native payment-provider customer or integration exists, so
|
||||
provider-specific payment-method backfills were not applicable.
|
||||
|
||||
The official `api-clock` service was added. It starts `26` scheduled Lago jobs,
|
||||
including invoice finalization, billing, retries and subscription lifecycle
|
||||
work. The general and billing workers consume the resulting queues.
|
||||
|
||||
Postgres remains on the patched `postgres:15-alpine` image (`15.18` during the
|
||||
upgrade). The optional Lago Partman image was not adopted because its
|
||||
`15.0-alpine` tag would downgrade the database binaries. Lago's Partman-aware
|
||||
event migrations skip partition setup when the extension is unavailable; this
|
||||
staging instance does not run the high-volume events processor.
|
||||
|
||||
Final proof:
|
||||
|
||||
- API health: `v1.50.0`, local and mesh.
|
||||
- Database migrations: `0` down.
|
||||
- Data unchanged: `1` organization, `6` customers, `4` plans,
|
||||
`16` subscriptions and `27` invoices.
|
||||
- Queues: `0` pending, `0` retries, `0` dead.
|
||||
- Public frontend and GraphQL: HTTP `200`.
|
||||
- Real Lago operator login: token returned.
|
||||
- REST API: HTTP `200`, all `6` customers visible.
|
||||
- Fixed-charge field is available on plan responses.
|
||||
- Existing invoice PDF: HTTP `200`, `application/pdf`, valid `%PDF` header.
|
||||
- Running HiveGrid staging API to Lago mesh call: HTTP `200`, all `6`
|
||||
customers visible.
|
||||
- A fresh encrypted `v1.50.0` post-upgrade dump was restored into a disposable
|
||||
database; the restore again produced `6` customers, `16` subscriptions and
|
||||
`27` invoices.
|
||||
|
||||
## Test data convention
|
||||
|
||||
Use deterministic prefixes so the shared staging instance stays clean:
|
||||
@@ -387,3 +510,116 @@ Use deterministic prefixes so the shared staging instance stays clean:
|
||||
|
||||
Delete test customers and plans after the ticket unless a runbook explicitly
|
||||
keeps them as regression fixtures.
|
||||
|
||||
## Canonical billing catalog
|
||||
|
||||
`BILL-11` replaces manually maintained plan prices with the versioned catalog
|
||||
in `packages/entitlements/src/commercial-catalog.ts`.
|
||||
|
||||
Current contract:
|
||||
|
||||
- Version: `2026-08-03.1`
|
||||
- SHA-256 digest:
|
||||
`e400ac119050602009c7a27bd51e4d17aa92e003de3fcd4d9852b09a8cd934c9`
|
||||
- Public packages: `Core` (`starter`), `Swarm` (`pro`) and `Colony`
|
||||
(`business`)
|
||||
- Two monthly-billed variants per package: monthly cancellation and a
|
||||
12-month commitment with a reduced monthly SEPA rate
|
||||
- Newly purchasable fixed storage packages: `storage_500gb`, `storage_1tb`,
|
||||
`storage_5tb`; legacy `storage_100gb` remains provider-resolvable for
|
||||
existing contracts but is not offered for new purchases
|
||||
- German VAT tax code: `de-vat-19`, `19 %`, exclusive
|
||||
- No usage-based storage meter
|
||||
|
||||
The controlled operator is always run in this order:
|
||||
|
||||
```bash
|
||||
NODE_ENV=production pnpm billing:catalog --target=staging --plan
|
||||
NODE_ENV=production pnpm billing:catalog \
|
||||
--target=staging \
|
||||
--apply \
|
||||
--confirm=staging:2026-08-03.1
|
||||
NODE_ENV=production pnpm billing:catalog --target=staging --verify
|
||||
```
|
||||
|
||||
`LAGO_API_URL` and `LAGO_API_KEY` must refer to the staging organization.
|
||||
The operator pins the expected organization name and a hash of its ID, writes
|
||||
a redacted pre-change snapshot with mode `0600`, and fails if the target or
|
||||
post-apply catalog differs.
|
||||
|
||||
2026-07-31 provider proof:
|
||||
|
||||
- Apply converged successfully.
|
||||
- A second verify returned all 16 resources as no-op.
|
||||
- Redacted pre-change snapshot:
|
||||
`tmp/billing-catalog-snapshots/staging-2026-07-30T22-37-23-338Z.json`
|
||||
- The production-mode verify returned the exact same version and digest.
|
||||
|
||||
2026-08-02 `BILL-23` provider proof:
|
||||
|
||||
- The storage recut kept all technical plan and add-on codes stable while
|
||||
changing the four fixed storage-package prices to the maximum-50-percent
|
||||
gross-margin contract.
|
||||
- The controlled staging apply converged on catalog version `2026-08-02.2`
|
||||
and digest
|
||||
`492992bcefd923d35c94f84b564372b7cfd89c08a9ecb9439f25d9b6becc5977`.
|
||||
- The immediate follow-up verify returned all 16 tax and plan resources as
|
||||
no-op.
|
||||
- Redacted pre-change snapshot:
|
||||
`tmp/billing-catalog-snapshots/staging-2026-08-02T21-20-18-902Z.json`
|
||||
- The production provider was not read through a write-capable flow and was
|
||||
not changed.
|
||||
|
||||
2026-08-03 `BILL-24` provider proof:
|
||||
|
||||
- PR #367 merged normally as `5fba54e72c8b351a37c158ce5be27aba65db1f4d`;
|
||||
all 13 PR checks and all four main/deploy checks passed.
|
||||
- The controlled staging apply converged on catalog version `2026-08-03.1`
|
||||
and digest
|
||||
`e400ac119050602009c7a27bd51e4d17aa92e003de3fcd4d9852b09a8cd934c9`.
|
||||
- The immediate follow-up verify returned all 16 tax and plan resources as
|
||||
no-op.
|
||||
- Newly purchasable storage is 500 GB for EUR 4.99, 1 TB for EUR 9.99 and 5
|
||||
TB for EUR 49.99 net per month. `storage_100gb` remains provider-resolvable
|
||||
with non-purchasable metadata for legacy contracts.
|
||||
- Redacted pre-change snapshot:
|
||||
`tmp/billing-catalog-snapshots/staging-2026-08-02T23-35-27-833Z.json`.
|
||||
- Staging web and API health and readiness returned HTTP 200 on the exact merge
|
||||
revision. The public registration journey recommended Swarm for 10 office
|
||||
and 20 field seats and rendered the corrected 100 GB, 500 GB and 2 TB base
|
||||
storage values without creating an account or payment.
|
||||
- The production Lago provider was not accessed through the catalog operator
|
||||
and was not changed.
|
||||
|
||||
The Staging runtime has the Lago and Mollie test credentials required for the
|
||||
real checkout/mandate/recurring-payment rehearsal. The canonical Mollie
|
||||
webhook URL on `worker-general` is:
|
||||
|
||||
```text
|
||||
https://staging-api.hivegrid.de/v1/billing/mollie/webhooks
|
||||
```
|
||||
|
||||
The final `BILL-11` rehearsal must cover a monthly checkout, 12-month
|
||||
commitment, both seat roles, one storage package, the first customer-present
|
||||
payment and one simulated subsequent Lago invoice. Provider fixtures must use
|
||||
the `hg-stg-bill-11-*` prefix and be removed after the proof.
|
||||
|
||||
2026-07-29 `BILL-11` closeout proof on revision
|
||||
`549f708d9809e9ba839c15366943918679a034d3`:
|
||||
|
||||
- A monthly Core preview with 18 office seats, 33 field seats and the
|
||||
`storage_100gb` package returned the expected net amount, German VAT and
|
||||
gross amount.
|
||||
- The 12-month Core checkout used 4 office seats, 8 field seats and
|
||||
`storage_500gb`. The customer-present Mollie test payment used iDEAL with
|
||||
`sequenceType=first`, completed successfully and created a valid direct
|
||||
debit mandate.
|
||||
- A simulated following Lago subscription invoice created exactly one Mollie
|
||||
payment for EUR 454.22 with `method=directdebit` and
|
||||
`sequenceType=recurring`. The paid webhook updated the HiveGrid ledger and
|
||||
the Lago invoice to `succeeded`.
|
||||
- Replaying the signed Lago invoice event returned `replayed` and left exactly
|
||||
one recurring payment for that invoice.
|
||||
- The deterministic staging tenant, Lago customer, generated managed price
|
||||
plan and Mollie customer were removed after verification. The superseded
|
||||
open Mollie test payment was cancelled; no open fixture payment remains.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Lokaler Demo-Seed (HiveGrid SHK)
|
||||
|
||||
Befüllt einen reichhaltigen, realistischen SHK/Handwerk-Demo-Tenant zum lokalen
|
||||
Erkunden der App. Nach dem Seed ist der Workspace voll bestückt (Kontakte,
|
||||
Pipelines, Projekte, Leads, Forms, Bookings, Kalender, Automationen, Tasks,
|
||||
Rechnungen, Activities, Benachrichtigungen, API-Key, eingehender Webhook) statt
|
||||
leer.
|
||||
|
||||
Quelle: `scripts/seed-hivegrid-demo.ts` (Muster: `scripts/seed-e2e-readiness.ts`).
|
||||
|
||||
## Ausführen
|
||||
|
||||
```bash
|
||||
pnpm seed:demo
|
||||
```
|
||||
|
||||
Das Skript setzt das Guard-Flag `HIVEGRID_DEMO_SEED=1` selbst. Es läuft **nie in
|
||||
Produktion** (`NODE_ENV=production` → Abbruch) und braucht ein erreichbares
|
||||
`DATABASE_URL`. Der Seed ist **idempotent**: mehrfaches Ausführen aktualisiert die
|
||||
Daten in-place, es entstehen keine Duplikate (deterministische `demo-*`-IDs bzw.
|
||||
`@@unique`-Upserts). Alle Fachdaten tragen `isDemoData = true`, sofern die jeweilige Tabelle dieses Feld unterstützt.
|
||||
|
||||
## Login-Daten
|
||||
|
||||
Alle Nutzer haben das Passwort `password`.
|
||||
|
||||
| E-Mail | Rolle im Betrieb | Tenant-Rolle | App-Rolle(n) |
|
||||
|---|---|---|---|
|
||||
| `demo@hivegrid.de` | Geschäftsführer | Owner | SuperUser (voll berechtigt) |
|
||||
| `buero@hivegrid.de` | Büro/Verwaltung | Admin | Admin, Billing Admin |
|
||||
| `monteur@hivegrid.de` | Monteur | Member | User |
|
||||
| `vertrieb@hivegrid.de` | Vertrieb | Member | User |
|
||||
|
||||
Login-URL: `http://localhost:3000/hivegrid-demo/dashboard`
|
||||
|
||||
Tenant: Slug `hivegrid-demo`, Name `HiveGrid Demo SHK`, Blueprint `crm-saas`,
|
||||
aktive Business-Subscription, die wichtigsten v1-Feature-Gates freigeschaltet
|
||||
(`dashboard`, CRM, Leads, Projekte, Tasks, Files, HR, Integrationen, Bookings,
|
||||
Forms, Automationen, E-Mail-Inbox, API, Suche).
|
||||
|
||||
## Was wird geseedet
|
||||
|
||||
- **14 Kontakte:** 6 Firmen (mit Adresse) + 8 Privatpersonen (mit E-Mail + Telefon)
|
||||
- **3 Pipelines** mit Stages (Vertriebs- und Auftrags-Pipeline)
|
||||
- **8 Projekte** über mehrere Stages verteilt (inkl. gewonnen/verloren)
|
||||
- **6 Leads** (verschiedene Quellen + Status) mit 3 Lead-Sources
|
||||
- **5 Forms** (u.a. `SHK Angebotsanfrage`, `SHK Abnahmeprotokoll`) mit
|
||||
**9 Submissions** und Action-Logs
|
||||
- **2 Booking-Pages** mit **3 Terminen**, Provider-Zuordnung, Verfügbarkeits-Regeln
|
||||
(Mo–Fr für 3 Nutzer) plus **4 Datums-Ausnahmen** (Urlaub + Custom-Window, für den
|
||||
`/availability`-Editor); die Seite `Beratung vor Ort` ist mit dem Intake-Form
|
||||
`SHK Angebotsanfrage` verknüpft (AK-P16b/AK-P16c)
|
||||
- **6 Kalender-Events** inkl. Booking-Spiegelung, externem Privat-Termin und einer
|
||||
zweitägigen ganztägigen Messe (Quell-Marker fürs Kalender-Grid) + 1 Konflikt-Log-Eintrag
|
||||
- **2 Automationen** mit **4 Executions** inkl. Erfolg, Dry-Run und Fehlerfall
|
||||
- **6 Tasks** (an Projekte/Kontakte verknüpft)
|
||||
- **2 Rechnungen**, **4 Benachrichtigungen**, **5 Activity-Einträge**
|
||||
- **1 API-Key** (`Website Lead Intake`) + **1 eingehender Webhook**
|
||||
(`website-kontaktformular`)
|
||||
|
||||
## Sichtbarkeit von Demo-Daten
|
||||
|
||||
Einige Listen filtern `isDemoData = true` standardmäßig aus, damit echte
|
||||
Tenant-Daten nicht mit Demo-Füllstand vermischt werden. Für Review-Screens ggf.
|
||||
`?demo=show` anhängen oder den UI-Toggle „Demo-Daten anzeigen“ nutzen:
|
||||
|
||||
- `http://localhost:3000/hivegrid-demo/contacts?demo=show`
|
||||
- `http://localhost:3000/hivegrid-demo/projects?demo=show`
|
||||
- `http://localhost:3000/hivegrid-demo/tasks?demo=show`
|
||||
|
||||
## Hinweis: lokale DB-Erreichbarkeit
|
||||
|
||||
Das Runtime-`DATABASE_URL` zeigt auf PgBouncer (`localhost:6432`). Läuft PgBouncer
|
||||
lokal nicht, ist der direkte Postgres-Backend-Port erreichbar (Container
|
||||
`hivegrid-local-pg`). Zum einmaligen Seeden gegen den direkten Port ohne `.env`
|
||||
zu ändern:
|
||||
|
||||
```bash
|
||||
DATABASE_URL="postgres://<user>:<pass>@localhost:5434/boilerplate" pnpm seed:demo
|
||||
```
|
||||
|
||||
Es ist dieselbe Datenbank, die die App über PgBouncer liest — Seeding über den
|
||||
direkten Port schreibt in denselben `boilerplate`-Datenbestand.
|
||||
@@ -2,29 +2,34 @@
|
||||
|
||||
PgBouncer multiplexes Prisma's per-process connection pools onto a small set
|
||||
of real Postgres backends. The dev compose stack and the staging Coolify
|
||||
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
|
||||
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
|
||||
stack both terminate application traffic at `pgbouncer:6432`; only the
|
||||
short-lived deployment migrator receives a separate `DIRECT_URL` path through
|
||||
`postgres:5432`.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
```text
|
||||
+--------------+ +-------------+ +----------+
|
||||
| apps (web, | 6432 | pgbouncer | 5432 | postgres |
|
||||
| api, worker) +--------->+ transaction +-------->+ 16-alp. |
|
||||
| app runtimes | 6432 | pgbouncer | 5432 | postgres |
|
||||
| app / worker +--------->+ transaction +-------->+ 16-alp. |
|
||||
| Prisma client| | pool, 25/db | | |
|
||||
+------+-------+ +-------------+ +-----^----+
|
||||
| |
|
||||
| DIRECT_URL (5432) — only for prisma migrate ---+
|
||||
v
|
||||
prisma migrate / db push / db pull
|
||||
|
|
||||
one-shot migrator -- DIRECT_URL (5432) ---------------+
|
||||
```
|
||||
|
||||
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>?pgbouncer=true&connection_limit=10`
|
||||
- `pgbouncer=true` tells Prisma to skip prepared-statement caching.
|
||||
- `connection_limit=10` caps Prisma's internal pool — PgBouncer pools again
|
||||
on top, doubling the budget without this would defeat the purpose.
|
||||
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — read by
|
||||
`prisma.config.ts` for `migrate`, `db push`, `db pull`, `db seed`.
|
||||
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>`
|
||||
- PrismaPg uses unnamed statements by default, which is compatible with
|
||||
transaction-mode pooling.
|
||||
- `DATABASE_POOL_MAX=10` caps the explicit node-postgres pool. A URL
|
||||
`connection_limit` parameter does not configure PrismaPg/node-postgres.
|
||||
- `APP_DATABASE_URL`, `PORTAL_DATABASE_URL`, `WORKER_DATABASE_URL`, and the callback-only
|
||||
`AUTHENTIK_ADMIN_JIT_DATABASE_URL` contain distinct RLS-enforced roles and
|
||||
point through PgBouncer. Coolify exposes the JIT URL only to the core-web
|
||||
resource; the portal and other services receive only their normal `DATABASE_URL`.
|
||||
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — supplied only to
|
||||
`start.sh migrate` in a short-lived deployment step. It must never exist on
|
||||
web, API, MCP, Hocuspocus, or worker resources.
|
||||
|
||||
## Local development
|
||||
|
||||
@@ -35,7 +40,8 @@ docker compose up postgres pgbouncer
|
||||
|
||||
# In another shell:
|
||||
DIRECT_URL="postgres://user:pass@localhost:5432/hivegrid" \
|
||||
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid?pgbouncer=true&connection_limit=10" \
|
||||
DATABASE_URL="postgres://user:pass@localhost:6432/hivegrid" \
|
||||
DATABASE_POOL_MAX=10 \
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
@@ -49,23 +55,31 @@ The pooler must run on the same Docker network as Postgres. The current
|
||||
operator-owned deployment source is the Coolify service/network config; do not
|
||||
copy historical staging inventory into a live cutover.
|
||||
|
||||
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
|
||||
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
|
||||
`pg_authid.rolpassword`. From the Postgres container:
|
||||
1. **Provision the reviewed roles and generate `userlist.txt`.** Run the
|
||||
idempotent role provisioner from [`docs/database.md`](../database.md) first.
|
||||
Postgres uses `password_encryption=scram-sha-256`, so the role-specific
|
||||
SCRAM hashes then live in `pg_authid.rolpassword`. From the Postgres
|
||||
container:
|
||||
|
||||
```bash
|
||||
psql -U postgres -tAc \
|
||||
"SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
|
||||
FROM pg_authid
|
||||
WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt
|
||||
WHERE rolname IN (
|
||||
'hivegrid_app','hivegrid_portal','hivegrid_worker','hivegrid_authentik_jit','pgb_admin'
|
||||
);" > userlist.txt
|
||||
chmod 600 userlist.txt
|
||||
```
|
||||
|
||||
Drop the file into a Coolify volume mount that the pgbouncer container
|
||||
will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
|
||||
The query must return all five rows before rollout. `hivegrid_app`,
|
||||
`hivegrid_portal`, and `hivegrid_worker` handle their isolated runtime traffic;
|
||||
`hivegrid_authentik_jit` is the single-function callback capability and
|
||||
`pgb_admin` is reserved for PgBouncer administration and health checks. Drop the file
|
||||
into a Coolify volume mount that the pgbouncer container will see at
|
||||
`/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
|
||||
|
||||
2. **Create the Coolify service.**
|
||||
- Image: `edoburu/pgbouncer:latest`
|
||||
- Image: `edoburu/pgbouncer:latest@sha256:…` (reviewed manifest digest)
|
||||
- Network: `coolify` (must match Postgres)
|
||||
- Mounts:
|
||||
- `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro`
|
||||
@@ -80,9 +94,23 @@ copy historical staging inventory into a live cutover.
|
||||
docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
|
||||
```
|
||||
|
||||
4. **Switch app env vars.** In Coolify, on `web`, `api`, `worker`:
|
||||
- `DATABASE_URL=postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>?pgbouncer=true&connection_limit=10`
|
||||
- `DIRECT_URL=postgres://<database-user>:${PG_PASS}@<postgres-service>:5432/<database>`
|
||||
4. **Switch runtime env vars.** In Coolify:
|
||||
- core web/API/MCP/Hocuspocus: `DATABASE_URL` uses the `hivegrid_app` role.
|
||||
- customer portal: `DATABASE_URL` uses the `hivegrid_portal` role.
|
||||
- every worker group: `DATABASE_URL` uses the `hivegrid_worker` role.
|
||||
- only core web: `AUTHENTIK_ADMIN_JIT_DATABASE_URL` uses the
|
||||
`hivegrid_authentik_jit` role. Remove every Admin-SSO key from API, MCP,
|
||||
Hocuspocus and worker resources, including inherited groups.
|
||||
- every runtime sets `RLS_RUNTIME_MODE=required` and the matching
|
||||
`DATABASE_RUNTIME_ROLE`.
|
||||
- `DATABASE_POOL_MAX` ist auf jeder Runtime explizit gesetzt. Für den ersten
|
||||
Produktionsrollout gilt je Prozess/Instanz: Core-Web `10`, Customer-Portal
|
||||
`5`; API/MCP/Hocuspocus und Worker folgen ihrem separat freigegebenen
|
||||
Kapazitätsplan.
|
||||
- `DATABASE_POOL_IDLE_TIMEOUT_MS=10000`
|
||||
- `DATABASE_POOL_CONNECTION_TIMEOUT_MS=5000`
|
||||
- remove `DIRECT_URL`, `MIGRATOR_*`, `MIGRATE_ON_BOOT`, `SEED_ON_BOOT`, and
|
||||
all `SEED_*` keys from every runtime resource, including inherited env groups.
|
||||
|
||||
Use service and database names from the current Coolify project state; do
|
||||
not copy archived staging inventory into a live cutover.
|
||||
@@ -95,19 +123,22 @@ copy historical staging inventory into a live cutover.
|
||||
1. Provision PgBouncer in Coolify but leave app env vars unchanged.
|
||||
2. Verify `SHOW STATS` and `SHOW POOLS` respond.
|
||||
3. Smoke-test from a one-off container:
|
||||
|
||||
```bash
|
||||
docker run --rm --network coolify \
|
||||
postgres:16-alpine psql \
|
||||
"postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
|
||||
-c "SELECT 1;"
|
||||
```
|
||||
4. Roll out `DIRECT_URL` to all apps first (no behaviour change because
|
||||
`prisma.config.ts` falls back to `DATABASE_URL`).
|
||||
5. Roll out the new `DATABASE_URL` (with `pgbouncer=true&connection_limit=10`).
|
||||
|
||||
4. Configure the migrator credential as a CI secret reachable from the runner;
|
||||
do not copy it to Coolify applications.
|
||||
5. Run the immutable web image once with `DIRECT_URL` and argument `migrate`.
|
||||
A missing/unreachable credential or failed migration stops the rollout.
|
||||
6. Roll out the role-specific `DATABASE_URL` and explicit `DATABASE_POOL_*` limits.
|
||||
Coolify will recreate containers — they must reconnect to the pooler.
|
||||
6. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
|
||||
7. Confirm `prisma migrate status` from a CI container still passes (uses
|
||||
`DIRECT_URL`).
|
||||
7. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
|
||||
8. Confirm `prisma migrate status` from the same step-scoped CI credential.
|
||||
|
||||
## Tuning knobs
|
||||
|
||||
@@ -131,10 +162,10 @@ Panels of interest: pool wait time p95, server connection count vs.
|
||||
### `FATAL: prepared statement "s0" already exists`
|
||||
|
||||
PgBouncer in transaction pool mode reused a server-side connection that
|
||||
still has the prepared statement cached. Confirm `DATABASE_URL` has
|
||||
`?pgbouncer=true` and Prisma 5+ is in use. Without the flag, Prisma keeps
|
||||
local prepared statements and the next transaction lands on a different
|
||||
backend.
|
||||
still has a prepared statement cached. HiveGrid's PrismaPg adapter uses
|
||||
unnamed statements by default; a URL `pgbouncer=true` flag is ignored. Check
|
||||
for a custom `statementNameGenerator`, an older non-adapter Prisma client, or
|
||||
another client sharing the same PgBouncer user.
|
||||
|
||||
### `error: too many clients already`
|
||||
|
||||
@@ -142,9 +173,8 @@ Postgres ran out of connections. Either:
|
||||
|
||||
- `default_pool_size` × number-of-app-pools > `max_connections` — lower
|
||||
pool size or raise `max_connections` on Postgres.
|
||||
- Prisma's `connection_limit` is unset, so each app process opens its own
|
||||
default pool (CPU + 1) against PgBouncer. Set `connection_limit=10` (or
|
||||
lower) in the query string of `DATABASE_URL`.
|
||||
- `DATABASE_POOL_MAX` is too high or unset for the deployment budget. Set it
|
||||
explicitly (10 by default, or lower) for each application process.
|
||||
|
||||
### `cl_waiting > 0` persistently in `SHOW POOLS`
|
||||
|
||||
@@ -156,13 +186,13 @@ Application sustained pool exhaustion. Options:
|
||||
|
||||
### Migrations fail with "prepared statement" or DDL errors
|
||||
|
||||
`DIRECT_URL` is wrong or missing. `prisma migrate` must hit Postgres
|
||||
directly. Verify `prisma.config.ts` reads `DIRECT_URL`:
|
||||
The step-scoped migrator credential is wrong, missing, or cannot reach
|
||||
Postgres directly. Verify it only inside a one-shot container:
|
||||
|
||||
```bash
|
||||
DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \
|
||||
DIRECT_URL=postgres://...@postgres:5432/db \
|
||||
pnpm exec prisma migrate status
|
||||
docker run --rm \
|
||||
-e DIRECT_URL=postgres://...@postgres:5432/db \
|
||||
<immutable-web-image> migrate
|
||||
```
|
||||
|
||||
### Healthcheck flapping
|
||||
@@ -176,8 +206,11 @@ unhealthy.
|
||||
To bypass PgBouncer without code changes:
|
||||
|
||||
1. In Coolify, change `DATABASE_URL` to point at Postgres directly
|
||||
(port 5432, drop `?pgbouncer=true&connection_limit=10`).
|
||||
2. Restart `web`, `api`, `worker`.
|
||||
(port 5432; pool sizing remains in `DATABASE_POOL_*`).
|
||||
2. Preserve the distinct `hivegrid_app`, `hivegrid_portal`, `hivegrid_worker`, and callback-only
|
||||
`hivegrid_authentik_jit` credentials and restart core web, customer portal,
|
||||
API, MCP, Hocuspocus, and workers. The JIT URL remains core-web-only.
|
||||
3. Stop the `pgbouncer` Coolify service.
|
||||
|
||||
`DIRECT_URL` stays valid — it never depended on the pooler.
|
||||
The migrator secret remains step-scoped and unchanged; never add it to the
|
||||
runtime resources during rollback.
|
||||
@@ -0,0 +1,188 @@
|
||||
# Plattform-Leads: Backup, Backfill und Cutover
|
||||
|
||||
Dieses Runbook ist der operative Vertrag fuer `R-LEADS` (`HIVEGRID-73`). Es
|
||||
ersetzt den separaten `lead-inbox`-Schreibpfad kontrolliert durch
|
||||
`POST /v1/platform/leads`, ohne die letzte restorable Kopie der Altdaten zu
|
||||
loeschen.
|
||||
|
||||
## Sicherheitsregeln
|
||||
|
||||
- Kein Producer-Deploy ohne unmittelbar davor wiederholte Inventur und Sicherung
|
||||
seines tatsaechlich beschriebenen Fallback-Stores.
|
||||
- Backups enthalten PII und werden deshalb nie unverschluesselt im Repo, in
|
||||
Tickets, Logs oder Import-Manifests abgelegt.
|
||||
- Der verschluesselte Offsite-Stand liegt im privaten Hetzner-Bucket
|
||||
`hivegrid-prod`; Objektpraefixe und Hashes duerfen als PII-freier Proof
|
||||
dokumentiert werden.
|
||||
- Import-Manifests enthalten nur Quelle, Zaehler, Zeitbereich, Hashes und
|
||||
technische Run-IDs.
|
||||
- Ein Import erzeugt weder Benachrichtigungen noch Marketing-Automationen.
|
||||
- Das letzte verschluesselte Backup bleibt erhalten, bis ein eigener
|
||||
Retention-/Loeschentscheid bestaetigt wurde.
|
||||
|
||||
## Erster Safety-Snapshot
|
||||
|
||||
Der Snapshot vom 2026-07-16 liegt unter
|
||||
`backups/platform-leads/20260716T033754Z`. Sein PII-freies Inventar und die
|
||||
Restore-Ergebnisse stehen in
|
||||
`docs/ops/evidence/r-leads-safety-inventory-20260716.json`.
|
||||
|
||||
Ergebnis:
|
||||
|
||||
- `lead-inbox` Postgres: 1 Datensatz, Custom-Dump in frisches Postgres 16
|
||||
restauriert.
|
||||
- HiveGrid-Website Produktion: kein SQLite-File im laufenden Container und kein
|
||||
persistentes Volume; 0 Datensaetze.
|
||||
- HiveGrid-Website Staging: 1 Datensatz, konsistentes SQLite-Backup,
|
||||
`integrity_check=ok`.
|
||||
- HiveGrid-Cloud `Lead`/`LeadEvent`: Produktion 0/0, Staging 0/0.
|
||||
|
||||
Zusaetzlich wurde irrtuemlich ein nicht zu HiveGrid gehoerender Website-Dateistore
|
||||
inventarisiert und verschluesselt gesichert. Das Objekt bleibt zur Vermeidung einer
|
||||
unautorisierten Datenvernichtung erhalten, ist im Evidence-Inventar jedoch mit
|
||||
`excluded_from_hivegrid_import: true` markiert und darf weder als Quelle
|
||||
konfiguriert noch importiert werden. Aufbewahrung oder Loeschung erfordern einen
|
||||
separaten, ausdruecklich autorisierten Retention-/Loeschentscheid.
|
||||
|
||||
## Operator-Tooling
|
||||
|
||||
Vor dem Source-Dry-Run muss der Canary `/leads` mit einer aktiven AdminUser-
|
||||
Identität und der vorgesehenen Sales-Rolle erreichen. Der API-Schalter
|
||||
`PLATFORM_LEADS_INGRESS_ENABLED` schützt unabhängig davon nur den externen Ingress.
|
||||
|
||||
Der Routing-Operator erhaelt vor dem Bootstrap eine Plattformrolle mit mindestens
|
||||
`admin.leads.read_own` und `admin.leads.write`. Der ausfuehrende Operator braucht
|
||||
`admin.leads.admin`. Das Bootstrap-Tool prueft aktiven AdminUser und diese
|
||||
Rechte fail-closed, bevor es einen Plan erzeugt oder Quellen mutiert. Fuer den
|
||||
Haris-Canary wird zusaetzlich verifiziert: Login auf `admin.hivegrid.de`,
|
||||
`/leads` erreichbar, zugewiesener synthetischer Lead sichtbar und eine
|
||||
Status-/Notizmutation erfolgreich.
|
||||
|
||||
Die nativen Quellen werden idempotent ueber den AdminLead-Domain-Service
|
||||
konfiguriert. Haris' Admin-User-ID wird zur Laufzeit uebergeben und ist keine
|
||||
Code-Konstante. Der Dry-Run erzeugt einen PII-freien Plan-Hash; Apply akzeptiert
|
||||
nur genau diesen Stand:
|
||||
|
||||
```bash
|
||||
pnpm platform-leads:source-bootstrap -- \
|
||||
--mode dry-run \
|
||||
--actor-admin-user-id "$PLATFORM_ADMIN_USER_ID" \
|
||||
--routing-admin-user-id "$HARIS_ADMIN_USER_ID" \
|
||||
--manifest "$EVIDENCE_DIR/source-bootstrap.dry-run.json"
|
||||
|
||||
pnpm platform-leads:source-bootstrap -- \
|
||||
--mode apply \
|
||||
--actor-admin-user-id "$PLATFORM_ADMIN_USER_ID" \
|
||||
--routing-admin-user-id "$HARIS_ADMIN_USER_ID" \
|
||||
--expected-plan-sha256 "$PLAN_SHA256" \
|
||||
--manifest "$EVIDENCE_DIR/source-bootstrap.apply.json"
|
||||
```
|
||||
|
||||
Die aktuelle Website-Quelle wird aktiviert, Haris wird als Default-Assignee
|
||||
und mindestens als Notification-Empfaenger konfiguriert. Bereits vorhandene
|
||||
weitere Empfaenger bleiben erhalten. Die beiden reinen Importquellen bleiben
|
||||
deaktiviert. Das Bootstrap-Tool stellt bewusst keine Credentials aus; deren
|
||||
einmalig angezeigter Klartext wird ueber `/leads/sources` erzeugt und direkt in
|
||||
den jeweiligen Server-Runtime-Secrets hinterlegt.
|
||||
|
||||
Der Backfill liest UTF-8-JSONL oder ein JSON-Array. Die feste Zuordnung ist:
|
||||
|
||||
| Legacy-Store | Native Source |
|
||||
|---|---|
|
||||
| `lead-inbox-postgres` | `lead-inbox-legacy` |
|
||||
| `hivegrid-website-production-sqlite` | `hivegrid-website` |
|
||||
| `hivegrid-website-staging-sqlite` | `hivegrid-website` |
|
||||
| `hivegrid-cloud-legacy` | `hivegrid-legacy-cloud` |
|
||||
|
||||
Pro Store wird zuerst ein Dry-Run mit einer neuen Manifest-Datei ausgefuehrt.
|
||||
Apply und Replay verlangen sowohl dessen exakten Input-Hash als auch die
|
||||
akzeptierte Zeilenzahl und schreiben bei Drift oder Rejections keine Zeile:
|
||||
|
||||
```bash
|
||||
pnpm platform-leads:backfill -- \
|
||||
--mode dry-run \
|
||||
--store "$LEGACY_STORE" \
|
||||
--input "$RESTORED_EXPORT" \
|
||||
--manifest "$EVIDENCE_DIR/$LEGACY_STORE.dry-run.json"
|
||||
|
||||
pnpm platform-leads:backfill -- \
|
||||
--mode apply \
|
||||
--store "$LEGACY_STORE" \
|
||||
--input "$RESTORED_EXPORT" \
|
||||
--actor-admin-user-id "$PLATFORM_ADMIN_USER_ID" \
|
||||
--expected-input-sha256 "$INPUT_SHA256" \
|
||||
--expected-accepted "$ACCEPTED_COUNT" \
|
||||
--manifest "$EVIDENCE_DIR/$LEGACY_STORE.apply.json"
|
||||
|
||||
pnpm platform-leads:backfill -- \
|
||||
--mode replay \
|
||||
--store "$LEGACY_STORE" \
|
||||
--input "$RESTORED_EXPORT" \
|
||||
--actor-admin-user-id "$PLATFORM_ADMIN_USER_ID" \
|
||||
--expected-input-sha256 "$INPUT_SHA256" \
|
||||
--expected-accepted "$ACCEPTED_COUNT" \
|
||||
--manifest "$EVIDENCE_DIR/$LEGACY_STORE.replay.json"
|
||||
```
|
||||
|
||||
Ein erfolgreicher Replay hat `created=0`, `failed=0` und keine Gate-Failures.
|
||||
Die Manifeste enthalten nur Hashes, Zaehler, technische IDs und Zeitbereiche;
|
||||
Input-Pfade, Lead-Inhalte und Klartext-Legacy-IDs werden nicht aufgenommen. Jede
|
||||
Manifest-Datei wird exklusiv mit Modus `0600` angelegt und nicht ueberschrieben.
|
||||
Importe laufen ausschliesslich ueber `importLead`, bewahren Originalzeitpunkte,
|
||||
native Legacy-Status und unterstuetzte Activities und loesen weder M011-
|
||||
Benachrichtigungen noch Automationen aus.
|
||||
|
||||
Cloud-Legacy-Status werden verlustarm auf den nativen M040a-Vertrag abgebildet:
|
||||
`trial` und `customer` werden `qualified`, `churned` wird `disqualified`; der
|
||||
Originalwert bleibt als `metadata.legacyStatus` erhalten. Historische
|
||||
`score-changed`- und `converted`-Events werden als `legacy_event` mit
|
||||
`legacyType` im begrenzten Activity-Payload uebernommen. Unbekannte Status- oder
|
||||
Eventtypen bleiben Rejections und stoppen Apply. Damit wird keine M040b-
|
||||
Deal-Semantik vorgezogen, aber der historische Verlauf geht nicht verloren.
|
||||
|
||||
## Cutover-Reihenfolge
|
||||
|
||||
1. Native Tabellen, RLS/Pre-Tenant-Capability, Domain-Service, API, Worker und
|
||||
Admin-Inbox nach Staging deployen; Operatorrolle wie oben anlegen und
|
||||
Berechtigungs- sowie Direkt-Routen gegen 404 pruefen.
|
||||
2. Source-Credentials einmalig erzeugen und ausschliesslich in den jeweiligen
|
||||
Server-Runtime-Secrets speichern.
|
||||
3. Staging-Canary je Source ausfuehren: Create, identischer Replay, Hash-Konflikt,
|
||||
Auth-Fehler, Rate-Limit, Delivery/Retry und Admin-Bearbeitung.
|
||||
4. Alle Altquellen dry-run importieren. Zaehler und Hashes gegen das Safety-
|
||||
Inventar pruefen.
|
||||
5. Apply und anschliessenden Replay ausfuehren. Der Replay muss `created=0`
|
||||
liefern.
|
||||
6. Direkt vor jedem Website-Deploy den jeweiligen aktuellen Store erneut sichern
|
||||
und nur das Finaldelta importieren.
|
||||
7. Producer einzeln umstellen; nach jeder Quelle einen synthetisch markierten
|
||||
Produktions-Canary und den Admin-/Delivery-Nachweis abwarten.
|
||||
8. Legacy-Endpoint read-only/drain-faehig belassen und mindestens sieben volle
|
||||
Tage auf eingehenden Traffic, Fehler, Queue-Failures und Source-Lag beobachten.
|
||||
9. Nur bei Nulltraffic: App stoppen, Secrets revoken, DNS entfernen und Repo
|
||||
archivieren. Die Datenbank nicht irreversibel loeschen.
|
||||
|
||||
## Rollback
|
||||
|
||||
Innerhalb des Drain-Fensters werden Producer-Secrets/URLs auf den weiterhin
|
||||
laufenden Legacy-Endpoint zurueckgestellt. Nach einem bereits erfolgten Stop wird
|
||||
das letzte bekannte Image gestartet und Postgres aus dem geprueften Custom-Dump
|
||||
restauriert. Native Importe sind idempotent und loeschen keine Quelle; sie muessen
|
||||
fuer einen Rollback nicht rueckgaengig gemacht werden.
|
||||
|
||||
## Decommission-Gates
|
||||
|
||||
Der externe Dienst darf erst gestoppt werden, wenn alle folgenden Nachweise gruen
|
||||
sind:
|
||||
|
||||
- der HiveGrid-Website-Producer produktiv auf dem nativen Endpoint;
|
||||
- Finaldelta importiert und Replay `created=0`;
|
||||
- sieben volle Tage kein Legacy-Write-Traffic;
|
||||
- Native-API-, Admin- und Notification-Monitoring ohne offene kritische Fehler;
|
||||
- letzter Dump offsite vorhanden und Restore erneut erfolgreich;
|
||||
- DNS-/Secret-/Mailjet-Abhaengigkeiten inventarisiert und Rollback-Werte sicher
|
||||
hinterlegt.
|
||||
|
||||
DNS-Loeschung, Secret-Revoke und App-Stop werden mit Zeitstempel und externer
|
||||
Run-ID in der Ticket-Doc dokumentiert. Eine endgueltige Backup- oder DB-Loeschung
|
||||
ist nicht Teil dieses Tickets.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Prisma-Baseline-Cutover
|
||||
|
||||
Seit dem Wartungsschnitt vom 2026-07-22 besteht die aktive Prisma-Historie aus
|
||||
`00000000000000_hivegrid_baseline`. Die Baseline ersetzt 195 Migrationen bis
|
||||
`20260721160000_ak_p21_detach_guard_null_safety`, ohne das Fachschema zu ändern.
|
||||
Die ersetzte Historie bleibt am Git-Commit
|
||||
`090e2558a6965e1c5804eb24d3eab5da8c8611a8` rekonstruierbar. Zusätzlich liegt
|
||||
sie für flache CI-Checkouts als checksum-geprüftes Audit-Archiv unter
|
||||
`prisma/baseline-legacy-migrations.tar.gz`; Prisma behandelt dieses Archiv
|
||||
nicht als aktive Migrationshistorie.
|
||||
|
||||
## Laufzeitvertrag
|
||||
|
||||
- Frische Datenbanken haben noch keine Prisma-Historie. Der Adopter führt keine
|
||||
Schreibaktion aus; `prisma migrate deploy` wendet die Baseline normal an.
|
||||
- Bestehende Datenbanken werden nur übernommen, wenn ihre erfolgreichen
|
||||
Legacy-Einträge entweder alle 195 Migrationen oder ein lückenloses Präfix
|
||||
dieser Kette bilden. Zulässig sind die kanonische SHA-256 oder eine der zehn
|
||||
im Manifest mit ihrem Git-Quellcommit belegten historischen
|
||||
SHA-256-Varianten. Ein exaktes Präfix wird ausschließlich über
|
||||
`prisma migrate deploy` aus dem checksum-geprüften Audit-Archiv auf die
|
||||
vollständige 195er-Kette vorwärts migriert. Lücken, fremde Namen, doppelte
|
||||
erfolgreiche Einträge und unbekannte Checksums bleiben harte Abbrüche. Bei
|
||||
einer historischen Variante werden vor der Baseline-Adoption zusätzlich
|
||||
Nullability/FK-Semantik, der tenant-gebundene Booking-Constraint sowie Owner
|
||||
und ACLs der betroffenen Security-Definer-Funktionen direkt im
|
||||
PostgreSQL-Katalog geprüft. Erst dann markiert
|
||||
`prisma migrate resolve --applied` die Baseline als angewendet. Die
|
||||
Baseline-DDL läuft dabei nicht. Nach dem regulären `prisma migrate deploy` muss ein
|
||||
`prisma migrate diff` die reale Datenbank gegen das aktuelle Prisma-Schema
|
||||
driftfrei bestätigen. Diese Reihenfolge erlaubt Forward-Migrationen nach der
|
||||
Baseline, ohne den noch nicht migrierten Legacy-Stand fälschlich gegen das
|
||||
bereits weiterentwickelte Schema zu vergleichen.
|
||||
- Bereits baselinierte Datenbanken sind für die Adoption ein No-op.
|
||||
Fehlgeschlagene, lückenhafte, unbekannt checksum-abweichende,
|
||||
katalog-abweichende oder fremde Legacy-Historien brechen vor der Adoption ab.
|
||||
Scheitert eine Forward-Migration des exakten Präfixes, bleibt Primas eigener
|
||||
fehlgeschlagener Ledger-Eintrag sichtbar und jeder weitere Lauf stoppt
|
||||
fail-closed.
|
||||
Eine Abweichung vom aktuellen Prisma-Schema bricht nach den atomaren
|
||||
Forward-Migrationen und vor Permission-Sync beziehungsweise Rollout ab.
|
||||
- Historie und Varianten-Katalog werden nach dem unveränderten
|
||||
Migrator-Rollen-Probe und vor `prisma migrate deploy` geprüft; die
|
||||
Prisma-Schema-Parität wird direkt danach im selben kurzlebigen
|
||||
`start.sh migrate`-Container geprüft. Nur dieser Prozess erhält `DIRECT_URL`;
|
||||
Runtime-Ressourcen erhalten sie nie.
|
||||
- `_prisma_migrations` wird weder gelöscht noch umgeschrieben. Der zusätzliche
|
||||
Baseline-Eintrag ist der auditierbare Übergang zwischen alter und neuer
|
||||
Historie.
|
||||
|
||||
## Einmaliger Merge- und Rollout-Ablauf
|
||||
|
||||
1. Während der PR offen ist, gilt Schema-Freeze. Parallele Schema-Branches
|
||||
werden nach dem Merge auf `main` rebased und erzeugen ihre Merge-Migration
|
||||
anschließend neu gegen die Baseline.
|
||||
2. Vor dem Merge müssen Baseline-Äquivalenz, Migrations-Parity, der strikte
|
||||
RLS-Live-Test und die normalen Commit-/Push-Gates grün sein.
|
||||
3. Der reguläre Main-Deploy baut zuerst das unveränderliche Image. Der
|
||||
Staging-One-shot-Migrator prüft die Rolle, vervollständigt bei Bedarf ein
|
||||
checksum-geprüftes Legacy-Präfix aus dem Audit-Archiv, übernimmt die
|
||||
vollständige Legacy-Historie, führt `migrate deploy` aus, prüft danach die
|
||||
aktuelle Schema-Parität und synchronisiert erst dann den Permission-Katalog.
|
||||
4. Erst nach grünem Staging-Deploy und geprüftem Restore-Punkt wird derselbe
|
||||
Image-SHA über den bestehenden Promote-Ablauf nach Produktion gebracht.
|
||||
Der input-freie Production-Promote darf erst nach bestätigter Backup- oder
|
||||
Restore-Point-Bereitschaft gestartet werden; der manuelle Run ist die
|
||||
auditierte Operator-Bestätigung.
|
||||
5. Der Produktions-One-shot-Migrator führt dieselbe fail-closed Adoption aus.
|
||||
Erst danach werden die Runtime-Ressourcen ausgerollt.
|
||||
|
||||
Keine manuelle `migrate resolve`-Ausführung ist Teil dieses Ablaufs. Wenn der
|
||||
Adopter stoppt, wird zuerst die gemeldete Historienabweichung untersucht; die
|
||||
Prisma-Tabelle wird nicht per SQL „repariert".
|
||||
|
||||
## Verifikation
|
||||
|
||||
```bash
|
||||
pnpm run verify:prisma-baseline
|
||||
pnpm run verify:prisma-baseline-equivalence
|
||||
pnpm exec prisma validate --config prisma.config.ts
|
||||
pnpm run verify:migrations-parity
|
||||
pnpm run verify:tenant-policy-contract
|
||||
pnpm run verify:rls-release-readiness
|
||||
HIVEGRID_RLS_REQUIRE_MIGRATION_CHAIN_PARITY=1 pnpm run verify:rls-live
|
||||
```
|
||||
|
||||
Der Äquivalenztest entpackt die 195 Legacy-Dateien aus dem Audit-Archiv, prüft
|
||||
Archiv- und Einzeldatei-Checksums, spielt Legacy und Baseline in getrennte PostgreSQL-16-
|
||||
Datenbanken ein, simuliert alle acht Migrationen mit den zehn belegten
|
||||
historischen Ledger-Checksums, probt Adoption und wiederholten No-op und
|
||||
vergleicht Schema, RLS, Funktionen, ACLs, Extensions, Constraints, Indizes sowie
|
||||
die von Migrationen angelegten Referenzdaten.
|
||||
|
||||
## Rollback
|
||||
|
||||
- Vor Merge: PR verwerfen. Es gab keine externe Datenbankmutation.
|
||||
- Nach Merge, vor Adoption: Merge revertieren und die Legacy-Verzeichnisse aus
|
||||
Git wiederherstellen.
|
||||
- Nach Adoption: Die Adoption hat keine DDL ausgeführt. Für einen Code-Rollback
|
||||
werden die Legacy-Dateien wiederhergestellt; ein altes Image darf jedoch
|
||||
nicht erneut `migrate deploy` ausführen, solange der zusätzliche Baseline-
|
||||
Eintrag nicht mit dem getesteten Rollback-Ablauf vereinbar ist.
|
||||
- Bei einer frischen, nur aus der Baseline aufgebauten Wegwerf-Datenbank wird
|
||||
die Datenbank verworfen und neu erstellt. In Produktion wird niemals ein
|
||||
`DROP SCHEMA` als Rollback ausgeführt; dort gilt Restore-Punkt/Backup plus
|
||||
forward-only Korrekturmigration.
|
||||
@@ -3,8 +3,25 @@
|
||||
> Umsetzung von **AD-24** (Prod unter eigenem Hetzner-Account, self-contained). Stand 2026-06-19.
|
||||
> **Go-Gates:** Jeder Schritt mit 🟥 ist kostenwirksam / schwer reversibel und wird **vor** dem `create`
|
||||
> einzeln mit dem Owner (Raffael) bestätigt. 🟩 = read-only / vorbereitend.
|
||||
>
|
||||
> **Sicherheits-Nachzug 2026-07-12:** CAPTCHA-Verifikation ist in deployed
|
||||
> Runtimes jetzt code-seitig immer fail-closed. Historische Einträge mit
|
||||
> `CAPTCHA_FAIL_OPEN=true` dokumentieren den damaligen Rollout-Zustand, können
|
||||
> Production aber nicht mehr aufweichen. Die Coolify-Variable ist auf `false`
|
||||
> zu setzen, damit Konfiguration und wirksames Verhalten übereinstimmen.
|
||||
>
|
||||
> **Deploy-Vertrag 2026-07-12:** Die folgenden nummerierten Schritte beschreiben
|
||||
> den heute gültigen Ablauf. Spätere Abschnitte mit Datumsüberschriften sind
|
||||
> historische Rollout-Protokolle und keine aktuelle Bedienungsanleitung.
|
||||
>
|
||||
> **Offener Security-Nachzug 2026-07-22:** Die historischen Rollout-Protokolle
|
||||
> belegen noch einen Registry-Pull mit `gitea-admin`; außerdem ist Cosign noch
|
||||
> nicht fail-closed. Die bewusst verschobenen Risiken und Abnahmekriterien sind
|
||||
> zentral in [`SECURITY-AUDIT-2026-07-22`](../security/SECURITY-AUDIT-2026-07-22.md)
|
||||
> dokumentiert.
|
||||
|
||||
## 0. Voraussetzungen (🟩 vorhanden)
|
||||
|
||||
- Hetzner-API-Token (Prod): `~/.cli-tokens/hcloud-hivegrid-prod` — verifiziert (Account leer).
|
||||
- S3 (Prod): `~/.cli-tokens/prod-s3-bucket-hetzner-credentials` — Auth verifiziert (Projekt p15040939, fsn1), Bucket noch nicht angelegt.
|
||||
- Cloudflare (Zone `hivegrid.de`): `~/.cli-tokens/cloudflare-token` (Global Key + Email → `X-Auth-Email`/`X-Auth-Key`).
|
||||
@@ -12,6 +29,7 @@
|
||||
- Entscheidungen: Ingress = **Cloudflare-Proxy + Coolify/Traefik** (kein Hetzner-LB). Region **fsn1**, x86 **cx**-Linie.
|
||||
|
||||
## 1. Netzwerk-Fundament (🟥 Provisionierung, aber günstig/reversibel)
|
||||
|
||||
1. **SSH-Key** im Prod-Account hinterlegen (aktuell 0) — Public-Key des Deploy-/Admin-Keys.
|
||||
2. **Private Network** (Hetzner Cloud Network), z.B. `10.0.0.0/16`, Subnetz fsn1.
|
||||
3. **Firewall**:
|
||||
@@ -20,36 +38,55 @@
|
||||
- `db` + `services`: **kein** eingehender Public-Traffic, nur über das Private Network.
|
||||
|
||||
## 2. Server provisionieren (🟥 kostenwirksam — Go pro Batch)
|
||||
|
||||
Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
|
||||
| VM | Typ | Specs | Zweck |
|
||||
|---|---|---|---|
|
||||
| `hivegrid-control` | cx33 | 4/8/80 | Coolify + Orchestrierung/Bastion |
|
||||
| `hivegrid-web` | cx33 | 4/8/80 | Web (SSR) |
|
||||
| `hivegrid-api` | cx33 | 4/8/80 | API |
|
||||
| `hivegrid-worker` | cx43 | 8/16/160 | 5 Worker-Queue-Gruppen + Meili + ClamAV + mCaptcha |
|
||||
| `hivegrid-db` | cx33 | 4/8/80 | Postgres + PgBouncer + Redis |
|
||||
| VM | Typ | Specs | Zweck |
|
||||
| ------------------ | ---- | -------- | -------------------------------------------------- |
|
||||
| `hivegrid-control` | cx33 | 4/8/80 | Coolify + Orchestrierung/Bastion |
|
||||
| `hivegrid-web` | cx33 | 4/8/80 | Web (SSR) |
|
||||
| `hivegrid-api` | cx33 | 4/8/80 | API |
|
||||
| `hivegrid-worker` | cx43 | 8/16/160 | 5 Worker-Queue-Gruppen + Meili + ClamAV + mCaptcha |
|
||||
| `hivegrid-db` | cx33 | 4/8/80 | Postgres + PgBouncer + Redis |
|
||||
|
||||
4. **Volume** (~50 GB) an `hivegrid-db` für Postgres-Daten (nicht auf Boot-Disk).
|
||||
5. (optional) Server-Backups für `hivegrid-db` aktivieren.
|
||||
|
||||
## 3. Coolify auf control (🟥)
|
||||
|
||||
6. Coolify auf `hivegrid-control` installieren (eigenständige Prod-Instanz).
|
||||
7. `web/api/worker/db` als **remote Server** in Coolify einhängen (über Private Network).
|
||||
8. Container-Registry-Zugang (Gitea-Registry) + Cosign-Keys in Coolify hinterlegen.
|
||||
8. Container-Registry-Zugang mit einem dedizierten read-only
|
||||
`hivegrid-registry-pull`-Servicekonto hinterlegen. Der private Cosign-Key
|
||||
gehört ausschließlich in geschützte Gitea-CI-Secrets; Coolify benötigt für
|
||||
die spätere Verifikation nur den öffentlichen Schlüssel. Die externe
|
||||
Credential-Rotation ist unter `SECURITY-FOLLOWUP-REGISTRY-PULL` noch offen.
|
||||
|
||||
## 4. Daten- + Zentral-Dienste (🟥)
|
||||
|
||||
9. **db-VM:** Postgres (+ DB `hivegrid`, DB `mcaptcha`), **PgBouncer** (Transaction-Pooling, Port 6432), Redis. `DATABASE_URL` → PgBouncer, `DIRECT_URL` → 5432.
|
||||
10. **worker+services-VM:** Meilisearch (Master-Key), ClamAV (clamd), mCaptcha (zeigt auf db-Postgres `mcaptcha` + Redis).
|
||||
11. **DEK-Master-Key** (AD-8) generieren + sicher hinterlegen (nicht ins Repo).
|
||||
12. **S3-Bucket** `hivegrid-prod` (fsn1) anlegen + Versioning; `prod-s3-bucket-hetzner-credentials` um `bucket=hivegrid-prod` ergänzen.
|
||||
|
||||
## 5. App-Env + Deploy (🟥)
|
||||
|
||||
13. Prod-Env je App setzen (analog Staging-Keys, aber **Prod-Werte**): `DATABASE_URL/DIRECT_URL`, `REDIS/BULLMQ_REDIS_URL`, `MEILI_URL/MEILI_API_KEY`, `MCAPTCHA_*`, `CLAMAV_HOST/PORT`, `SESSION/CRYPTO/JWT`-Secrets, `APP_BASE_DOMAIN`/Host-Vars (Prod-Domains), Sentry `SENTRY_ENVIRONMENT=production`. **`WEB_NOINDEX` NICHT setzen** (Prod soll indexierbar sein).
|
||||
14. Migrationen: `prisma migrate deploy` als Pre-Deployment-Command (wie Staging).
|
||||
15. Promote-Deploy über die Pipeline (oder Coolify) → Images ausrollen.
|
||||
14. Auf der Web-Ressource `MIGRATE_ON_BOOT=true` und `SEED_ON_BOOT=true` setzen sowie
|
||||
`pre_deployment_command` und `post_deployment_command` leer halten. Das Web-Image
|
||||
startet über `start.sh`: zuerst `prisma migrate deploy`, danach der idempotente
|
||||
Production-Seed, anschließend der Web-Prozess. Ein Fehler beendet den Container
|
||||
und lässt das Health-Gate fehlschlagen. Coolify-Hooks sind für Docker-Image-Apps
|
||||
kein gültiger Migrationspfad.
|
||||
15. Promote-Deploy über `.gitea/workflows/deploy.yml` auslösen. Nach dem Patch auf den
|
||||
unveränderlichen SHA-Tag ist der kanonische Coolify-Trigger
|
||||
`POST /api/v1/deploy?uuid=<resource-uuid>&force=true`; `scripts/coolify-deploy.mjs`
|
||||
kapselt diesen Aufruf, übernimmt die exakt zurückgegebene Deployment-UUID und
|
||||
wartet auf deren Ergebnis. Nicht `/applications/<uuid>/start` und nicht
|
||||
`/restart` verwenden. Web wird vor API, MCP und den fünf Worker-Gruppen ausgerollt.
|
||||
|
||||
## 6. Ingress / DNS (🟥 outward-facing)
|
||||
|
||||
16. **Cloudflare-Records** (Zone `hivegrid.de`):
|
||||
- `app.` , `api.` , `captcha.` → Prod-IPs, **proxied** (orange).
|
||||
- `admin.` → Prod-IP, **dns-only** + Firewall/Cloudflare-Access-Restriktion.
|
||||
@@ -57,33 +94,52 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
17. Coolify/Traefik: Domains je App + Let's-Encrypt am Origin; Cloudflare SSL-Modus **Full (strict)**.
|
||||
|
||||
## 7. Endabnahme (🟩 verifizieren)
|
||||
|
||||
- `app./api.` `/healthz` + `/readyz` = 200 (DB erreichbar, Migration angewandt).
|
||||
- Login inkl. 2FA-Gate; mCaptcha greift; Suche (Meili) antwortet; File-Upload-Scan (ClamAV) ok.
|
||||
- **Prod ist indexierbar:** `/login` trägt **kein** `X-Robots-Tag: noindex`; `robots.txt` allow-all.
|
||||
- Sentry-Events landen unter `environment=production`.
|
||||
|
||||
## 8. Rollback / Notfall
|
||||
- Deploy-Rollback: vorheriges Image-Tag in Coolify re-deployen.
|
||||
|
||||
- Moderner Deploy-Rollback: vorherige volle 40-stellige SHA verwenden und
|
||||
Image-Tag plus revisiongebundenen Health-Command als eine atomare
|
||||
Konfiguration setzen. Vor den Helpern die exakten HiveGrid-Images über
|
||||
`WORKER_IMAGE`/`MCP_IMAGE` oder das vollständige
|
||||
`REGISTRY`+`IMAGE_NAMESPACE`+`IMAGE_PREFIX`-Tupel vorgeben; reine
|
||||
Suffix-Erkennung ist absichtlich verboten. Worker werden dabei mit
|
||||
`coolify-ensure-worker-groups.mjs production` als komplette Fünfer-Flotte
|
||||
abgeglichen, anschließend alle fünf Ressourcen force-deployt und über ihre
|
||||
exakten Deployment-IDs verifiziert. Nur ein Worker-Redeploy nach einem
|
||||
flottenweiten Reconcile ist nicht sicher.
|
||||
- MCP-Rollback: `configure-mcp-healthcheck` bindet altes Image und Health-Command
|
||||
an dieselbe SHA, bevor der kanonische Force-Deploy gestartet wird.
|
||||
- Legacy-Images ohne revisiongebundenen Health-Entrypoint ausschließlich über
|
||||
`configure-legacy-rollback-healthcheck` vorbereiten; der Helper setzt Tag,
|
||||
deaktivierten Image-Healthcheck und leeren Command gemeinsam und liest die
|
||||
Persistenz fail-closed zurück.
|
||||
- DB: Volume-Snapshot / Backup einspielen (forward-only Migrationen → Inverse-SQL als Kommentar je Migration).
|
||||
- DNS: TTL niedrig halten während des Cutovers.
|
||||
|
||||
## Inventar (provisioniert 2026-06-19, fsn1)
|
||||
| Ressource | Name | ID | Public IP | Private IP |
|
||||
|---|---|---|---|---|
|
||||
| Server control | `hivegrid-prod-control` (cx33) | 143044798 | 157.90.243.228 | 10.0.0.2 |
|
||||
| Server web | `hivegrid-prod-web` (cx33) | 143044800 | 167.233.109.142 | 10.0.0.3 |
|
||||
| Server api | `hivegrid-prod-api` (cx33) | 143044801 | 167.233.110.100 | 10.0.0.4 |
|
||||
| Server worker+services | `hivegrid-prod-worker` (cx43) | 143044803 | 49.13.24.190 | 10.0.0.5 |
|
||||
| Server db | `hivegrid-prod-db` (cx33) | 143044805 | 178.105.208.72 | 10.0.0.6 |
|
||||
| Volume (db-Daten) | `hivegrid-prod-db-data` 50 GB | 106081942 | — | `/dev/disk/by-id/scsi-0HC_Volume_106081942` |
|
||||
| Network | `hivegrid-prod-net` 10.0.0.0/16 | 12347309 | — | — |
|
||||
| Firewall app (22/80/443) | `hivegrid-prod-fw-app` | 11162316 | — | — |
|
||||
| Firewall db (22) | `hivegrid-prod-fw-db` | 11162317 | — | — |
|
||||
| SSH-Key | `hivegrid-prod-node1` (`~/.ssh/node1`) | 113985925 | — | — |
|
||||
|
||||
| Ressource | Name | ID | Public IP | Private IP |
|
||||
| ------------------------ | -------------------------------------- | --------- | --------------- | ------------------------------------------- |
|
||||
| Server control | `hivegrid-prod-control` (cx33) | 143044798 | 157.90.243.228 | 10.0.0.2 |
|
||||
| Server web | `hivegrid-prod-web` (cx33) | 143044800 | 167.233.109.142 | 10.0.0.3 |
|
||||
| Server api | `hivegrid-prod-api` (cx33) | 143044801 | 167.233.110.100 | 10.0.0.4 |
|
||||
| Server worker+services | `hivegrid-prod-worker` (cx43) | 143044803 | 49.13.24.190 | 10.0.0.5 |
|
||||
| Server db | `hivegrid-prod-db` (cx33) | 143044805 | 178.105.208.72 | 10.0.0.6 |
|
||||
| Volume (db-Daten) | `hivegrid-prod-db-data` 50 GB | 106081942 | — | `/dev/disk/by-id/scsi-0HC_Volume_106081942` |
|
||||
| Network | `hivegrid-prod-net` 10.0.0.0/16 | 12347309 | — | — |
|
||||
| Firewall app (22/80/443) | `hivegrid-prod-fw-app` | 11162316 | — | — |
|
||||
| Firewall db (22) | `hivegrid-prod-fw-db` | 11162317 | — | — |
|
||||
| SSH-Key | `hivegrid-prod-node1` (`~/.ssh/node1`) | 113985925 | — | — |
|
||||
|
||||
**Offen / Härtung:** SSH derzeit von `0.0.0.0/0` (key-only) erreichbar → später auf Admin-/Bastion-IP bzw. nur über `control` einschränken. Öffentliche 80/443 später auf Cloudflare-IP-Ranges begrenzen.
|
||||
|
||||
## Coolify (Schritt 3) — erledigt 2026-06-19
|
||||
|
||||
- Coolify **v4.1.2** auf `control` installiert (alle Container healthy).
|
||||
- **Root-Admin headless** angelegt via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
|
||||
- **API aktiviert** + **API-Token** erzeugt (Sanctum, mit `team_id`; Coolifys `createToken`-Override braucht Team-Kontext, daher direkt über `tokens()->create`). `GET /api/v1/servers` → 200.
|
||||
@@ -91,33 +147,42 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
- ⚠ Härtung offen: Coolify-FQDN auf `https://coolify.hivegrid.de` (443) umstellen + Port 8000 wieder schließen.
|
||||
|
||||
## Coolify-Remote-Server + DB (Schritt 3–4) — erledigt 2026-06-19
|
||||
|
||||
- **Docker 29.6.0** auf web/api/worker/db installiert (get.docker.com), in Coolify als Remote-Server eingehängt → alle **`usable=true`**. node1-Key in Coolify (`privkey_uuid`). UUIDs in `~/.cli-tokens/coolify-prod`.
|
||||
- Coolify-Projekt **HiveGrid** (`m74l9z6an57cw0316pir9rrb`) / Environment **production** (`h3woz1vm70yf6ajpzj0ch2fz`).
|
||||
- **Postgres 16** (managed) + **Redis 7** (managed) auf db-VM. Creds + URLs in `~/.cli-tokens/hivegrid-prod-services`.
|
||||
- **Cross-Server-Konnektivität (Designentscheidung):** Coolify-managed DBs laufen im Docker-Netz der db-VM; die App-Server (web/api/worker) erreichen sie über das **private Hetzner-Netz** (`10.0.0.6:5432/6379`), nicht über Container-DNS. DBs sind via `public_port` auf dem Host gebunden (`0.0.0.0`), aber **public durch fw-db (nur 22) dicht** — verifiziert: web→db privat OPEN, public DICHT.
|
||||
|
||||
**Offen (nächste Schritte):**
|
||||
**Historischer damaliger Reststand (2026-06-19, nicht als aktueller Ablauf verwenden):**
|
||||
|
||||
- Meilisearch + ClamAV + mCaptcha als Coolify-**Applications/Services** auf worker-VM (kein managed-DB-Typ), exposed aufs private Netz. mCaptcha nutzt Postgres-DB `mcaptcha` + Redis.
|
||||
- PgBouncer vor Postgres (Compose auf db-VM) → `DATABASE_URL` über Pooler, `DIRECT_URL` direkt.
|
||||
- DEK-Master-Key + S3-Bucket `hivegrid-prod`.
|
||||
- App-Deploy (web/api/worker) nach **mitapp-Muster: image-retag / build-once-deploy-via-pull** (nicht rebuild) + `migrate deploy`.
|
||||
- App-Deploy (web/api/worker) nach **mitapp-Muster: image-retag / build-once-deploy-via-pull** (nicht rebuild). Die damals noch offene Migrationsmechanik wurde inzwischen durch den Web-Entrypoint mit `MIGRATE_ON_BOOT`/`SEED_ON_BOOT` ersetzt.
|
||||
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (HiveGrid-Instanz, prod-DSNs).
|
||||
|
||||
## Zentral-Dienste + Storage (Schritt 4) — erledigt 2026-06-19
|
||||
|
||||
- **Meilisearch v1.10 + ClamAV 1.4** als Coolify-Compose-Service `hivegrid-prod-search-av` auf worker-VM, privat exposed (`10.0.0.5:7700` / `:3310`). Verifiziert: Meili von api-VM cross-server `{"status":"available"}`, ClamAV healthy. Master-Key in `~/.cli-tokens/hivegrid-prod-services`.
|
||||
- **mCaptcha-Stack** (mcaptcha + eigene Postgres + mcaptcha/cache) als Coolify-Service `hivegrid-prod-mcaptcha` auf worker-VM, FQDN `captcha.hivegrid.de` (DNS dns-only → worker, LE-Cert via Traefik im Hochlauf). Secrets gespeichert. **Folgeschritt:** Account+Sitekey in der mCaptcha-Instanz anlegen → neue `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` in App-Env (Code-/Env-Anpassung).
|
||||
- **S3-Bucket** `hivegrid-prod` (fsn1) angelegt + Versioning; `prod-s3-bucket-hetzner-credentials` ergänzt.
|
||||
|
||||
## App-Deploy — Muster verstanden (mitapp / build-once-retag)
|
||||
## App-Deploy — historischer Planungsstand 2026-06-19 (mitapp / build-once-retag)
|
||||
|
||||
> Historisches Rollout-Protokoll. Der aktuelle operative Vertrag steht in §5;
|
||||
> insbesondere gelten heute der Entrypoint-basierte Migrationslauf und der
|
||||
> kanonische `POST /api/v1/deploy?uuid=<resource-uuid>&force=true`-Trigger.
|
||||
|
||||
- Image-Schema: `git.tool-ctrl.de/toolctrl-labs/hivegrid/{web,api,worker}:<commit-sha>`, build_pack `dockerimage`, Port **8080**.
|
||||
- **Promote = retag, kein Rebuild:** Prod deployt denselben SHA-Tag, der getestet/gestaged ist (aktuell `9ae3963d…` = live auf Staging → Image existiert in der Gitea-Registry).
|
||||
- **7 Coolify-Apps** (nicht 3!): `web`, `api`, `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin` — web/api aus `/web`+`/api`-Image, die 5 Worker aus `/worker`-Image mit unterschiedlichem `WORKER_QUEUE_GROUP`.
|
||||
- Deploy-Trigger: `scripts/coolify-deploy.mjs` (PATCH `docker_registry_image_tag` → deploy), `migrate deploy` als Pre-Deployment-Command.
|
||||
- Damalige Annahme: `scripts/coolify-deploy.mjs` patcht `docker_registry_image_tag` und ein Coolify-Pre-Deployment-Command führt `migrate deploy` aus. Der Hook-Teil dieser Annahme ist überholt: Heute setzt die Pipeline die Boot-Flags, leert beide Hook-Slots und triggert nach dem Image-Patch mit `POST /api/v1/deploy?uuid=<resource-uuid>&force=true`; Migration und Seed laufen im neuen Web-Container über `start.sh`.
|
||||
- **Sentry (HiveGrid-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
|
||||
|
||||
**Offen (Go-Live-Phase):** Registry-Pull-Credential in Prod-Coolify · 7 Apps anlegen (Docker-Image) · App-Env Prod (aus Staging-Env transformiert: Prod-IPs/Domains, frische Secrets, prod-Sentry-Tag, **kein** WEB_NOINDEX) · erster Promote-Deploy + `migrate deploy` · DNS `app./api./admin.` · Endabnahme · mCaptcha-Sitekey + PgBouncer (Optimierung) als Folgeschritte.
|
||||
**Historisch offen in dieser Go-Live-Phase:** Registry-Pull-Credential in Prod-Coolify · 7 Apps anlegen (Docker-Image) · App-Env Prod (aus Staging-Env transformiert: Prod-IPs/Domains, frische Secrets, prod-Sentry-Tag, **kein** WEB_NOINDEX) · erster Promote-Deploy; die damals separat genannte Migration läuft heute über `MIGRATE_ON_BOOT=true` und `SEED_ON_BOOT=true` im Web-Entrypoint · DNS `app./api./admin.` · Endabnahme · mCaptcha-Sitekey + PgBouncer (Optimierung) als Folgeschritte.
|
||||
|
||||
## App-Deploy + Go-Live (Schritt 5–6) — LIVE 2026-06-19
|
||||
|
||||
- **Registry-Pull:** `docker login git.tool-ctrl.de` auf web/api/worker (User `gitea-admin`; Härtung: scoped Bot-PAT). Image `hivegrid/{web,api,worker}:9ae3963d…` (= Staging-Stand) gepullt.
|
||||
- **7 Coolify-Apps** (Docker-Image): `hivegrid-web`, `-api`, `-worker-{general,mail,import,webhook,plugin}` (Worker je `WORKER_QUEUE_GROUP`). UUIDs in `~/.cli-tokens/coolify-prod`.
|
||||
- **Prod-Env** aus Staging transformiert (Bug gefixt: bei doppelten build/runtime-Keys nicht-leeren Wert bevorzugen — sonst Mailjet u.a. leer → Worker-Crash). Frische, app-konsistente Secrets in `~/.cli-tokens/hivegrid-prod-app-secrets`. Demo-Seed AUS, `WEB_NOINDEX` raus (Prod indexierbar).
|
||||
@@ -126,63 +191,79 @@ Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
|
||||
- **Endabnahme grün:** `app.hivegrid.de` + `api.hivegrid.de` `/healthz` + `/readyz` = 200 (DB-backed), `/login` rendert „Anmelden | HiveGrid", **kein** X-Robots-Tag (indexierbar). Alle 7 Container healthy.
|
||||
|
||||
## Fix: Admin-SSO-Login-Loop (Cookie-Domain bei Multi-Host) — 2026-06-19
|
||||
|
||||
**Symptom:** SSO-Login loopte (Audit zeigte wiederholt `admin.login.sso.success`, aber Folge-Request nicht eingeloggt). **Ursache:** `resolveCookieDomain(APP_BASE_DOMAIN)` band das Session-Cookie an `app.hivegrid.de` → galt **nicht** für `admin.hivegrid.de` (getrennte Hosts). Auf Staging unsichtbar, weil dort app+admin denselben Host teilen. **Fix:** `APP_BASE_DOMAIN=hivegrid.de` (Parent-Domain) auf allen Apps → Cookie-Domain `hivegrid.de` gilt für app./admin./api.; abgeleitete Hosts bleiben korrekt (`getDerivedHosts` → app./admin./api.hivegrid.de). Verifiziert: `Set-Cookie: Domain=hivegrid.de`. **Lesson:** Multi-Host-Prod braucht `APP_BASE_DOMAIN` = Registrable-Domain, nicht den app-Host.
|
||||
|
||||
> **Sicherheits-Nachzug 2026-07-12:** Der obige Eintrag beschreibt nur den damaligen historischen Fix. Auth-Cookies werden inzwischen bewusst **nicht mehr** über die Parent-Domain geteilt. App und Admin nutzen getrennte host-only `__Host-`-Cookies ohne `Domain`; Admin→App-Impersonation läuft über einen kurzlebigen, einmalig konsumierbaren Handoff. `APP_BASE_DOMAIN` bleibt für das Host-Routing die Registrable-Domain, darf aber nicht mehr als Auth-Cookie-Domain interpretiert werden.
|
||||
|
||||
## Härtung/Abschluss-Runde — 2026-06-19 (nach SSO-Bestätigung)
|
||||
|
||||
- ✅ **Admin-Fallback aus:** `ADMIN_FALLBACK_LOGIN_ENABLED=false` → SSO (Authentik) ist einziger Admin-Pfad.
|
||||
- ✅ **Sentry aktiviert:** `SENTRY_DSN_WEB/API/WORKER` aus zentraler Instanz gesetzt (`environment=production`), alle 7 Apps redeployed + healthy. (Waren vorher leer → Prod lief ohne Error-Tracking.)
|
||||
- ✅ **mCaptcha komplett scharf:** Root-Cause des fehlenden LE-Certs war `service ... port is missing` (Traefik kannte Ziel-Port nicht — Compose-Service ohne Port-Deklaration). Fix: Compose mit korrekter Coolify-Magic `- SERVICE_FQDN_MCAPTCHA_7000=captcha.hivegrid.de` + `expose: 7000` → Host+Port-Label generiert, **LE-Cert geholt** (`captcha.hivegrid.de` HTTP 302, gültiges Cert). Account `hivegrid` + Captcha-Widget via mCaptcha-API angelegt → `MCAPTCHA_SITEKEY_LOGIN` + `MCAPTCHA_ACCOUNT_SECRET` + `MCAPTCHA_INSTANCE_URL` auf web gesetzt + redeployed. Creds in `~/.cli-tokens/hivegrid-prod-services`. `CAPTCHA_FAIL_OPEN=true` bewusst belassen (captcha aktiv, aber mCaptcha-Ausfall sperrt nicht den Login).
|
||||
|
||||
## CI-Promote verdrahtet + getestet — 2026-06-19
|
||||
|
||||
- **Gitea-Repo-Variablen (Production-Slot):** `COOLIFY_API_URL`, `COOLIFY_PROJECT_UUID`, `COOLIFY_ENVIRONMENT_NAME/UUID`, `COOLIFY_RESOURCE_UUIDS` (7 Apps), `COOLIFY_PROVISION_WORKER_GROUPS=false` + Secret `COOLIFY_TOKEN`. (Multi-Server ok: Resource-Auflösung ist server-agnostisch per Image-Name+Queue-Group; Auto-Provisioning aus, da Apps existieren.)
|
||||
- **App-Tags auf `main` umgestellt** + `:main`-Alias in Registry erstellt (= aktueller SHA). Promote aliased Staging-Image → `:main` → deployt.
|
||||
- **Worker-Apps korrekt konfiguriert** (vom CI-Drift-Check erzwungen, macht Prod robuster): Health-Ports 9101–9105 + `/readyz`-Health-Checks; Worker-ENV (WORKER_ENABLED/HEALTH_PORT/ADMIN_PORT/CONCURRENCY je Gruppe; general: OUTBOX_DISPATCHER_*/WORKFLOW_TRIGGER_OUTBOX_*; mail: TRANSACTIONAL_MAIL_OUTBOX_REQUIRED/MARKETING_SEQUENCES_ENABLED).
|
||||
- **Worker-Apps korrekt konfiguriert** (vom CI-Drift-Check erzwungen, macht Prod robuster): Health-Ports 9101–9105 + `/readyz`-Health-Checks; Worker-ENV (WORKER*ENABLED/HEALTH_PORT/ADMIN_PORT/CONCURRENCY je Gruppe; general: OUTBOX_DISPATCHER*_/WORKFLOW*TRIGGER_OUTBOX*_; mail: TRANSACTIONAL_MAIL_OUTBOX_REQUIRED/MARKETING_SEQUENCES_ENABLED).
|
||||
- **Test-Promote (runs 8436–8438):** iterativ alle Drifts gefixt; **8438 deployt Prod erfolgreich** (production worker groups + web-health + dependent deploys grün; Prod app/api `/readyz` 200, alle 7 Container healthy, `migrate status` = up to date).
|
||||
- ⚠ **Ein Step bleibt rot:** „Verify production migration status after web deploy" läuft auf dem **CI-Runner** und braucht DB-Zugang — die Prod-DB liegt aber im **privaten Netz des separaten Prod-Accounts** (`10.0.0.6`), für den Runner strukturell unerreichbar. **Kein Prod-Problem** (Migration lief als Pre-Deploy + `migrate status` im Container = up to date). **Kein Prod-Problem** (Migration lief als Pre-Deploy + `migrate status` im Container = up to date).
|
||||
- ⚠ **Historischer Befund:** „Verify production migration status after web deploy" lief auf dem **CI-Runner** und brauchte DB-Zugang — die Prod-DB liegt aber im **privaten Netz des separaten Prod-Accounts** (`10.0.0.6`), für den Runner strukturell unerreichbar. Die damalige Formulierung „Migration lief als Pre-Deploy" bezeichnete den zu diesem Zeitpunkt angenommenen Ablauf und ist kein aktueller Vertrag. Heute führt `start.sh` Migration und Seed im neuen Web-Container aus; ein Fehler verhindert dessen erfolgreichen Start.
|
||||
- **Container-Verify-Versuch (revertiert `b833c82e`):** Step auf SSH-Jump (Runner→control→web-Container) umgeschrieben → scheiterte am **Preflight-Wiring-Guard** `scripts/verify-gitea-deploy-workflow.mjs` (erzwingt exakte CI-Runner-Step-Struktur + verbietet secret-backed env in `echo`/SSH). Der Guard läuft im **Preflight bei jedem Push** → die Änderung machte **die geteilte Pipeline rot** (auch Staging). **Sofort revertiert**, Preflight wieder grün. **Konsequenz:** Jede Änderung am Step erfordert, **zwei bewusste CI-Guards (Struktur + Secret-Leak) + Tests** umzuschreiben → invasiv, **mit Owner abzustimmen**. Optionen: (A) Guard so anpassen, dass der Step bei privater DB nicht erzwungen wird; (B) Container-SSH-Verify voll umsetzen **mit Host-Key-Pinning** (Security-Review: kein `StrictHostKeyChecking=no` in committetem CI-Code) + Guards/Tests; (C) so lassen — Promote deployt erfolgreich, nur der redundante Schluss-Check ist bei Prod rot.
|
||||
|
||||
## CI-Promote KOMPLETT GRÜN — 2026-06-20 (Run 8453)
|
||||
|
||||
Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Production-Pfad** der geteilten `deploy.yml` auf (Prod gab es vorher nicht). Alle gefixt, lokal/manuell verifiziert, mit committeten Zwischenständen:
|
||||
|
||||
1. **migration-status-verify (war rot):** CI-Runner erreicht die private Prod-DB nicht (separater Account, AD-24). Fix: `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE=true` (eingebauter Script-Mechanismus, erkennt P1001 → skip) + `DATABASE_URL`-Secret. Struktur unverändert → Wiring-Guard + Tests grün.
|
||||
2. **production-dependent-health worker-check (war rot):** Step nutzte `remote_ssh`, **definierte es aber nie** (Copy-Paste-Lücke ggü. Staging) → „worker container not found". Fix: SSH-Setup + `remote_ssh` mit **ProxyCommand** (nicht ProxyJump — Key-Passthrough) über control-Bastion zur worker-VM (Multi-Server), gepinnte Host-Keys, Secret via Zwischen-var (umgeht Secret-Guard). Lokal verifiziert: Container gefunden + health-probe `ready`.
|
||||
3. **web-health-revision-mismatch (war rot):** `/healthz` über `app.hivegrid.de` lieferte eine **40 min alte Cloudflare-gecachte** revision (`cf-cache-status: HIT`) statt der frisch deployten. Fix: **CF-Cache-Rule** `/healthz`+`/readyz` → bypass (Health-Endpoints nie cachen) + Purge. Kein deploy.yml-Eingriff.
|
||||
|
||||
- **Gitea-Config:** Prod-Coolify-Variablen + Secrets (`COOLIFY_TOKEN`, `DATABASE_URL`, `COOLIFY_PROD_SSH_KEY/KNOWN_HOSTS`, Host/Worker-IP-vars). Apps auf Tag `main`, `:main`-Alias in Registry.
|
||||
- **Ergebnis:** Promote 8453 **komplett grün** (web-health + migration + dependent-health alle success). Prod läuft auf `94959b31`, app/api `/readyz` = ready. **Künftige Prod-Deploys:** `deploy.yml` workflow_dispatch `action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`.
|
||||
- **Ergebnis:** Promote 8453 **komplett grün** (web-health + migration + dependent-health alle success). Prod läuft auf `94959b31`, app/api `/readyz` = ready. **Künftige Prod-Deploys:** `deploy.yml` manuell von `main` starten; der input-freie Run promotet automatisch den aktuell live geprüften Staging-SHA und darf erst nach bestätigter Backup-/Restore-Point-Bereitschaft ausgelöst werden.
|
||||
|
||||
## Härtungs-/Service-Runde 2 — 2026-06-20
|
||||
|
||||
- ✅ **mCaptcha scharf + proxied:** `captcha.hivegrid.de` jetzt Cloudflare-proxied; Sitekey + Account-Secret bereits gesetzt (`CAPTCHA_FAIL_OPEN=true` als Safety belassen).
|
||||
- ✅ **PgBouncer (alle 7 Apps):** Coolify-Compose-Service `hivegrid-prod-pgbouncer` auf db-VM (`edoburu/pgbouncer:v1.23.1-p3`, transaction-pool, scram, `6432→5432`). Apps: `DATABASE_URL`→`10.0.0.6:6432?pgbouncer=true&connection_limit=10`, `DIRECT_URL`→`10.0.0.6:5432`. Kanarienvogel web `/readyz`=ready verifiziert, dann api+5 Worker nachgezogen — alle healthy. Creds in `~/.cli-tokens/hivegrid-prod-services`.
|
||||
- ✅ **Coolify-Port 8000 zu (ohne Tunnel):** Coolify hinter den **schon laufenden Traefik** (coolify-proxy) gehängt via **Dynamic-Config-Datei** `/data/coolify/proxy/dynamic/coolify-ui.yaml` (`Host(coolify.hivegrid.de)`→`http://coolify:8080`, certresolver letsencrypt) — file-watch, **kein coolify-Restart, kein Selbst-Aussperr-Risiko**. LE-Cert geholt, `https://coolify.hivegrid.de/api/v1`=200. Firewall fw-app: **8000 entfernt** (jetzt 22/80/443). CI `COOLIFY_API_URL`→`https://coolify.hivegrid.de/api/v1`. Verifiziert: :8000 dicht, https-API 200, SSH+app/api weiter ok. (CF-Tunnel wurde erwogen, vom Owner abgelehnt → Traefik-Weg.)
|
||||
- ✅ **Sentry gelöst (DSN-Host war falsch):** Die alte DSN nutzte `sentry.boiler.saas-ctrl.com` (→Staging-VM, 503) — toter Host, **richtiger Key/Projekt**. Über node1-SSH zum Dedi (`176.9.111.179`, self-hosted Sentry) die Projekte/Keys aus der Sentry-Postgres geholt: org `sentry`, Projekte `web=4`/`api=2`/`worker=3`. Korrekte DSNs (`https://<key>@sentry.saas-ctrl.com/<id>`) gesetzt auf web/api/worker + `SENTRY_ENVIRONMENT=production`. **Verifiziert:** Test-Envelope von der web-VM → HTTP 200; Container hat `…@sentry.saas-ctrl.com/4`; alle 7 healthy. Creds in `~/.cli-tokens/hivegrid-prod-services`. **Hinweis:** nutzt die generischen boilerplate-Projekte (web/api/worker, getrennt per `environment`-Tag) — dedizierte `hivegrid-*`-Sentry-Projekte wären Estate-Standard-Verfeinerung (Folgeschritt).
|
||||
|
||||
**Bewusst NICHT blind ausgeführt (echte Aufgaben, kein Einzeiler — Risiko/Begründung):**
|
||||
|
||||
- **Registry-Pull-Bot:** scoped Bot-PAT statt `gitea-admin` (Least-Privilege; aktuell funktional).
|
||||
- **CF-Tunnel-Rest:** der bei Coolify-Port erwogene CF-Tunnel wurde verworfen; ein leerer Tunnel-Eintrag `hivegrid-prod-coolify` ohne Connector kann in Cloudflare gelöscht werden (API-Delete gab 400, harmlos da ohne cloudflared inaktiv).
|
||||
|
||||
## Offene Folgeschritte (nicht-blockierend)
|
||||
- **mCaptcha:** LE-Cert für `captcha.hivegrid.de` verifizieren, dann proxied; Account+Sitekey anlegen → `MCAPTCHA_*` ins App-Env (bis dahin `CAPTCHA_FAIL_OPEN=true`).
|
||||
|
||||
- **mCaptcha:** Verfügbarkeit und Sitekeys regelmäßig prüfen; `CAPTCHA_FAIL_OPEN=false` in deployed Environments beibehalten.
|
||||
- **Sentry:** Events unter `environment=production` verifizieren (DSN aus Staging übernommen).
|
||||
- **Härtung:** SSH auf Bastion/Admin-IP einschränken · `admin.hivegrid.de` per Cloudflare-Access/IP schützen · scoped Registry-Bot statt `gitea-admin` · Coolify-FQDN auf https/443 + Port 8000 schließen.
|
||||
- **Optimierung:** PgBouncer vor Postgres · ~~Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht)~~ → **erledigt 2026-07-03:** Hooks feuerten für Docker-Image-Apps nie; ersetzt durch Migrate-on-Boot im Web-Entrypoint (`start.sh`, `MIGRATE_ON_BOOT`/`SEED_ON_BOOT`), Details in `docs/deployment.md` + `.planning/ticket-docs/AK-P4a.md`.
|
||||
- **CI-Promote:** `deploy.yml`/Gitea-Variablen auf das Prod-Coolify verdrahten (für künftige `promote`-Deploys ohne manuelle Schritte).
|
||||
|
||||
## Härtung (Schritt 7) — teilweise erledigt 2026-06-19
|
||||
|
||||
- ✅ **SSH-Angriffsfläche reduziert:** neue FW `hivegrid-prod-fw-web` (nur 80/443) auf web/api/worker; **public SSH dort dicht**, Zugang nur über `control` (Bastion) + privates Netz. `control` behält 22/80/443/8000. `db`: **kein** public Port (fw-db rules geleert), nur privat. Verifiziert: control-SSH ok, web-SSH public dicht, web via control ok, app/api öffentlich ok, Coolify-Deploys (privates Netz) laufen weiter.
|
||||
- ✅ Demo-Seed AUS, `WEB_NOINDEX` raus (prod indexierbar).
|
||||
|
||||
**Offene Härtung/Folgeschritte (priorisiert):**
|
||||
1. **mCaptcha public-FQDN:** Traefik liefert noch Default-Cert für `captcha.hivegrid.de` (Compose-Service-Domain-Labels nicht generiert; fqdn per tinker reichte nicht). Container laufen intern. Domain am besten über Coolify-**UI** am `mcaptcha`-Service setzen → LE-Cert → proxied. Dann **Account+Sitekey** in der mCaptcha-Admin-UI anlegen → `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` ins App-Env, `CAPTCHA_FAIL_OPEN=false`. Bis dahin: fail-open (kein Captcha-Zwang).
|
||||
|
||||
1. **mCaptcha public-FQDN:** Traefik liefert noch Default-Cert für `captcha.hivegrid.de` (Compose-Service-Domain-Labels nicht generiert; fqdn per tinker reichte nicht). Container laufen intern. Domain am besten über Coolify-**UI** am `mcaptcha`-Service setzen → LE-Cert → proxied. Dann **Account+Sitekey** in der mCaptcha-Admin-UI anlegen → `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` ins App-Env, `CAPTCHA_FAIL_OPEN=false`. Bis dahin bleiben CAPTCHA-geschützte Aktionen fail-closed.
|
||||
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`hivegrid-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `hivegrid-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
|
||||
3. **Coolify-Port 8000** schließen, sobald Coolify hinter `https://coolify.hivegrid.de` (443) läuft (Port aktuell offen, key+token-geschützt; für laufende API-Schritte noch gebraucht).
|
||||
4. **Registry:** scoped Bot-PAT statt `gitea-admin` für den Image-Pull-Login.
|
||||
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell). Das Migrate-Problem dabei ist seit 2026-07-03 gelöst: Migrationen laufen jetzt im Web-Entrypoint (`MIGRATE_ON_BOOT`), nicht mehr über den nie feuernden Coolify-Hook.
|
||||
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell). Das Migrate-Problem dabei ist seit 2026-07-03 gelöst: Migration und Production-Seed laufen jetzt im Web-Entrypoint (`MIGRATE_ON_BOOT=true`, `SEED_ON_BOOT=true`), nicht mehr über den nie feuernden Coolify-Hook. Der Rollout wird kanonisch mit `POST /api/v1/deploy?uuid=<resource-uuid>&force=true` ausgelöst.
|
||||
6. **Sentry** Events unter `environment=production` verifizieren · **PgBouncer** (Pooling) ergänzen.
|
||||
|
||||
---
|
||||
## Gesamt-Status (Stand 2026-06-20) — ✅ VOLLSTÄNDIG LIVE + GEHÄRTET
|
||||
|
||||
## Gesamt-Status (Stand 2026-06-20) — LIVE · Credential-Härtung offen
|
||||
|
||||
`app.`/`api.`/`admin.hivegrid.de` öffentlich (HTTPS), DB-backed (Postgres 16/pgvector + **PgBouncer** + Redis), P2-Set + 5 Worker-Gruppen, Revision `94959b31`.
|
||||
|
||||
**Alles erledigt:**
|
||||
|
||||
- ✅ **CI-Promote komplett grün** (Run 8453) — `workflow_dispatch action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`. Drei latente Multi-Server-Promote-Bugs gefixt (migration-status via `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE`, dependent-health via ProxyCommand-SSH, web-health via CF-Cache-Bypass).
|
||||
- ✅ **SSH-gehärtet** — control = Bastion, web/api/worker/db public-SSH dicht.
|
||||
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `hivegrid-admin`).
|
||||
@@ -194,3 +275,18 @@ Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Produc
|
||||
**Estate-Doku nachgezogen:** `infra/` README (Host-Inventar + DNS) + **ADR-016** (dedizierter Account + eigenes Coolify) + **LESSONS #13–#17** + onboard-pro-saas-Runbook.
|
||||
|
||||
**Rest-Folgeschritte (Least-Privilege, kein Druck):** scoped Registry-Bot statt `gitea-admin` · dedizierte `hivegrid-*`-Sentry-Projekte (statt geteilter boilerplate-Projekte) · Sentry-DSN-Host-Fix auch auf **Staging** nachziehen (dort steckt dieselbe tote `boiler.`-DSN) · leeren CF-Tunnel `hivegrid-prod-coolify` löschen.
|
||||
|
||||
## BILL-11-Promote und Admin-Erreichbarkeit — 2026-07-29
|
||||
|
||||
- Promote Run `11477` hat die live auf Staging geprüfte Revision
|
||||
`549f708d9809e9ba839c15366943918679a034d3` ohne Rebuild nach Production
|
||||
übernommen. Core, Admin, Portal, API, MCP, Realtime und alle fünf
|
||||
Worker-Gruppen sind gesund.
|
||||
- `admin.hivegrid.de` fehlte noch im öffentlichen DNS. Der Record zeigt jetzt
|
||||
Cloudflare-proxied auf den dedizierten Admin-Host; dessen Hetzner-Firewall
|
||||
erlaubt ausschließlich die öffentlichen HTTP-/HTTPS-Ports.
|
||||
- Der alte konkurrierende `admin.hivegrid.de`-Domain-Eintrag am Core-Web wurde
|
||||
entfernt. Damit routet die Domain eindeutig auf `hivegrid-admin`.
|
||||
- `/healthz` und `/readyz` liefern HTTP 200 mit `service=hivegrid-admin`,
|
||||
`surface=admin` und der promoteten Revision. Der Admin-Login leitet per
|
||||
OIDC/PKCE zu Authentik weiter.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Ambiguous Product-Acquisition sicher aufloesen
|
||||
|
||||
Dieses Runbook gilt fuer eine `TenantProductAcquisition`, deren Mollie-`POST /payments`
|
||||
moeglicherweise erfolgreich war, deren Antwort HiveGrid aber nicht erhalten hat. Der Ledger bleibt dann
|
||||
absichtlich als `provider_pending` ohne `providerPaymentId` stehen. Dieser Zustand darf weder per
|
||||
manuellem Datenbank-Update noch durch einen neuen Checkout freigegeben werden: Ein noch existierendes
|
||||
Mollie-Payment koennte sonst parallel zum neuen Versuch bezahlt werden.
|
||||
|
||||
Der einzige freigegebene Operatorpfad ist:
|
||||
|
||||
```bash
|
||||
pnpm exec tsx packages/cli/bin/hivegrid-cli.mjs ops product-acquisition-recovery \
|
||||
--tenant <tenant-id> \
|
||||
--acquisition-id <acquisition-id> \
|
||||
--json
|
||||
```
|
||||
|
||||
Das Kommando ist standardmaessig read-only. Es setzt den Tenant im RLS-Kontext, verlangt einen aktiven
|
||||
Plattform-`AdminUser` aus `HIVEGRID_CLI_OPERATOR_USER_ID` und benoetigt die vorhandenen
|
||||
`DATABASE_URL`-, `MOLLIE_API_KEY`-, `LAGO_API_URL`- und `LAGO_API_KEY`-Werte des Ziel-Environments.
|
||||
Provider-Secrets oder Provider-Payloads duerfen nicht in Shell-History, Tickets oder Logs kopiert werden.
|
||||
|
||||
## Sicherheitsvertrag
|
||||
|
||||
Das Kommando gibt einen neuen Versuch nur frei, wenn alle Bedingungen gleichzeitig erfuellt sind:
|
||||
|
||||
1. Tenant und Acquisition stimmen exakt mit dem RLS-gebundenen Datensatz ueberein.
|
||||
2. Der Zustand ist ein explizit mehrdeutiger `provider_pending`-Versuch ohne `providerPaymentId` oder
|
||||
ein bereits begonnenes, weiterhin blockiertes `cleanup.operator_provider_non_creation_confirmed`.
|
||||
3. Das unveraenderliche Mollie-Maximalfenster plus Reconciliation-Grace ist abgelaufen. Vorher bleibt
|
||||
der automatische Reconciler alleiniger Owner.
|
||||
4. Die Mollie-Customer-ID und das exakte Attempt-Reference sind vorhanden.
|
||||
5. Eine live ausgefuehrte, vollstaendig paginierte Mollie-Suche hat auf keiner Seite ein Payment mit
|
||||
diesem Attempt-Reference gefunden. Der Apply-Lauf wiederholt diese Suche nach dem dauerhaften
|
||||
Retry-Block ein zweites Mal und schliesst damit auch die Provider-/DB-Race. Fremde, zyklische,
|
||||
ungueltige oder unbegrenzt fortgesetzte Pagination-Links brechen fail-closed ab.
|
||||
6. Der Operator bestaetigt bei `--apply` exakt das vom Plan-Lauf ausgegebene Attempt-Reference und gibt
|
||||
eine Audit-Begruendung an.
|
||||
7. Der Ledger wird zuerst in einen `cleanup.*`-Zustand versetzt. Damit sind Reconciler und User-Retry
|
||||
blockiert, bevor ein externer Cleanup beginnt.
|
||||
8. Erst nach erfolgreichem Cleanup der exakt gespeicherten Lago-Subscription beziehungsweise
|
||||
Lago-Invoice wird `operator.provider_non_creation_confirmed` gespeichert. Erst dieser Zustand darf
|
||||
durch den normalen Checkout in einen neuen, hochgezaehlten Provider-Attempt ueberfuehrt werden.
|
||||
|
||||
Ein `provider_payment_found` ist ausdruecklich **kein** Retry-Proof. In diesem Fall nichts freigeben:
|
||||
solange der Ledger noch `provider_pending` ist, den `worker-general`-Reconciler beobachten. Steht er
|
||||
bereits in `cleanup.*`, bleibt er sicher blockiert und der gefundene Providerzustand muss als
|
||||
Billing-Incident untersucht werden.
|
||||
|
||||
## Anwenden
|
||||
|
||||
1. Den read-only Lauf ausfuehren und nur bei `status=absence_verified` fortfahren. Das ausgegebene
|
||||
`attemptReference` und `observedAt` sind der redigierte Proof; keine Mollie-Antwort anhaengen.
|
||||
2. Das gleiche Kommando erneut ausfuehren. Der Apply-Lauf wiederholt die exhaustive Live-Suche, er
|
||||
vertraut nicht auf das Ergebnis des ersten Prozesses:
|
||||
|
||||
```bash
|
||||
pnpm exec tsx packages/cli/bin/hivegrid-cli.mjs ops product-acquisition-recovery \
|
||||
--tenant <tenant-id> \
|
||||
--acquisition-id <acquisition-id> \
|
||||
--apply \
|
||||
--confirm-attempt '<attempt-reference-aus-dem-plan>' \
|
||||
--reason 'Mollie-Suche und Incident-Pruefung bestaetigen: kein Payment angelegt.' \
|
||||
--json
|
||||
```
|
||||
|
||||
3. Nur `status=released_for_retry` erlaubt einen neuen User-Checkout. Der naechste Checkout verwendet
|
||||
einen hoeheren `providerAttempt`, ein neues externes Attempt-Reference und einen dazu gebundenen
|
||||
Idempotency-Key.
|
||||
4. Im Audit muessen die beiden Aktionen
|
||||
`billing.product_acquisition.operator_recovery_started` und
|
||||
`billing.product_acquisition.operator_recovery_released` fuer dieselbe Acquisition vorhanden sein.
|
||||
|
||||
## Fehler und Wiederaufnahme
|
||||
|
||||
| Ergebnis | Bedeutung | Aktion |
|
||||
|---|---|---|
|
||||
| `absence_verified` | Read-only Mollie-Negativproof erfolgreich | Mit exaktem Attempt und Begruendung erneut als `--apply` ausfuehren |
|
||||
| `provider_payment_found` | Payment existiert oder ist inzwischen sichtbar | Kein Retry; Reconciler beziehungsweise Billing-Incident verfolgen |
|
||||
| `refused` | Status, Zeitfenster, Customer-ID oder Provider-Target erfuellt den Vertrag nicht | Ursache korrigieren, nichts manuell freigeben |
|
||||
| `raced` | Ledger hat sich waehrend der CAS-Operation geaendert | Von vorne read-only inspizieren |
|
||||
| Prozessfehler nach `operator_recovery_started` | Lago-Cleanup oder Abschluss-Persistenz fehlgeschlagen | Dasselbe Kommando erneut ausfuehren; `cleanup.*` bleibt retry-blocking und der Cleanup ist idempotent |
|
||||
| `released_for_retry` | Mollie-Nichtanlage und Lago-Cleanup sind belegt | Neuer Checkout darf normal durch den User gestartet werden |
|
||||
|
||||
Direktes BullMQ-Retry, `hivegrid-cli outbox requeue`, ein Status-Update per SQL oder das Loeschen des
|
||||
Acquisition-Datensatzes ersetzen diesen Ablauf nicht. Sie koennen weder die Mollie-Nichtanlage noch den
|
||||
exakten Lago-Cleanup beweisen.
|
||||
|
||||
## Nachweis ohne Geheimnisse
|
||||
|
||||
Fuer Ticket oder Incident reichen: Environment, Acquisition-ID, Status, Attempt-Reference,
|
||||
`observedAt`, Audit-Aktionsnamen und der spaetere Reconciler-/Checkout-Status. Nicht aufnehmen:
|
||||
Provider-API-Antworten, Provider-Customer-/Payment-IDs, Credentials, Datenbank-URLs, Cookies oder
|
||||
personenbezogene Checkoutdaten.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Storage traffic monitoring
|
||||
|
||||
This runbook defines the launch-safe BILL-25 signal for storage downloads. It
|
||||
uses HiveGrid's existing Prometheus and Grafana stack and creates no billing,
|
||||
CDN, proxy, bucket, or server infrastructure.
|
||||
|
||||
## What the metrics mean
|
||||
|
||||
| Metric | Meaning | Never means |
|
||||
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ---------------------------------------------- |
|
||||
| `storage_authorized_downloads_total{surface,delivery,size_known}` | HiveGrid completed its authorization boundary and issued the delivery response. | The client received or finished the download. |
|
||||
| `storage_authorized_download_object_bytes_total{surface,delivery}` | Sum of trustworthy whole-object sizes associated with those authorizations. | Actual bytes transferred or billed by Hetzner. |
|
||||
|
||||
Labels are closed application enums. They never contain tenant IDs, user IDs,
|
||||
file IDs, filenames, object keys, bucket names, URLs, request values, or other
|
||||
customer data. Metric writes are fail-open: an observation failure cannot block
|
||||
an otherwise valid download.
|
||||
|
||||
Current coverage:
|
||||
|
||||
- `surface="files"`, `delivery="signed_redirect"`: original files report the
|
||||
canonical `File.sizeBytes`; generated previews and thumbnails report
|
||||
`size_known="unknown"` because their sizes are not persisted.
|
||||
- `surface="portal_document"`, `delivery="signed_redirect"`: the authorization
|
||||
count is covered, but size is deliberately unknown. The portal's
|
||||
security-definer resolver does not expose `File.sizeBytes`; BILL-25 does not
|
||||
widen that database contract or add a migration.
|
||||
|
||||
API exports, avatars, branding, admin-only artifacts, bulk ZIPs, and GDPR proxy
|
||||
responses are not part of this initial counter coverage. Do not infer zero
|
||||
traffic for an uncovered surface.
|
||||
|
||||
## Why this is not actual egress
|
||||
|
||||
HiveGrid redirects authorized clients to short-lived presigned URLs in a private
|
||||
Hetzner bucket. After the redirect, the object transfer no longer traverses the
|
||||
HiveGrid process. A URL can remain unused, be retried or reused before expiry,
|
||||
or serve one or more byte ranges. Therefore the authorization estimate can be
|
||||
higher or lower than actual traffic.
|
||||
|
||||
Hetzner currently does not support S3 server-access logging or S3 request
|
||||
metrics and does not provide an API for account storage/traffic utilization.
|
||||
The Hetzner Console and invoice remain the authoritative account-wide cost
|
||||
source:
|
||||
|
||||
- <https://docs.hetzner.com/storage/object-storage/supported-actions/>
|
||||
- <https://docs.hetzner.com/storage/object-storage/faq/buckets-objects/>
|
||||
- <https://docs.hetzner.com/storage/object-storage/overview/>
|
||||
|
||||
Hetzner combines storage and traffic across all buckets and projects in the
|
||||
account. Ingress, S3 operations, and internal traffic within `eu-central` are
|
||||
free; externally delivered object data consumes the account-wide egress quota.
|
||||
The included account quota is a platform buffer and is not promised to an
|
||||
individual workspace.
|
||||
|
||||
Hetzner bills every stored object as at least 64 kB, even when the payload is
|
||||
smaller. HiveGrid's package calculation uses logical GB/TB and therefore does
|
||||
not price a hypothetical worst case made entirely of tiny objects. During the
|
||||
pilot, reconcile provider-billed storage against HiveGrid's logical stored-byte
|
||||
total as a separate risk signal. A growing gap means object-size distribution,
|
||||
unfinished multipart uploads, versions, or non-HiveGrid buckets need review;
|
||||
it is not evidence of customer egress. Do not silently convert that difference
|
||||
into a workspace charge.
|
||||
|
||||
## Grafana views
|
||||
|
||||
Use these queries for the web service. A process restart does not invalidate
|
||||
Prometheus `increase` across a correctly scraped counter reset.
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery, size_known) (
|
||||
increase(storage_authorized_downloads_total{service="web"}[30d])
|
||||
)
|
||||
```
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery) (
|
||||
increase(storage_authorized_download_object_bytes_total{service="web"}[30d])
|
||||
)
|
||||
```
|
||||
|
||||
Use a shorter window to spot bursts:
|
||||
|
||||
```promql
|
||||
sum by (runtime_surface, surface, delivery) (
|
||||
rate(storage_authorized_downloads_total{service="web"}[15m])
|
||||
)
|
||||
```
|
||||
|
||||
Unknown-size authorizations are a coverage indicator, not zero-byte downloads.
|
||||
Keep them visible next to the known-byte panel.
|
||||
|
||||
## Pilot reconciliation and thresholds
|
||||
|
||||
During the pilot, an operator performs the following comparison daily for the
|
||||
first four weeks and weekly afterwards:
|
||||
|
||||
1. Record the redacted, account-wide storage and egress totals from the Hetzner
|
||||
Console. Do not attach screenshots containing account or customer data.
|
||||
2. Read the 30-day Prometheus authorization totals above.
|
||||
3. Compare known authorized object bytes and the count of unknown-size
|
||||
authorizations with their own trailing 7- and 30-day baselines. The global
|
||||
counter includes base-plan, extra-seat and add-on storage and therefore must
|
||||
not be divided by sold add-on capacity.
|
||||
4. Compare logical stored bytes with provider-billed storage so Hetzner's
|
||||
64-kB object minimum and other account-wide storage overhead become visible.
|
||||
5. Record the difference and whether uncovered surfaces, unused links, retries,
|
||||
ranges, or cross-project account traffic plausibly explain it.
|
||||
|
||||
Review thresholds use separately meaningful signals:
|
||||
|
||||
| Signal | Operator action |
|
||||
| ------ | --------------- |
|
||||
| Authorized bytes and counts remain within `2x` of the trailing comparable period | Normal pilot observation. |
|
||||
| A surface grows to `2x` or more without a matching active-user/storage increase | Warning: inspect burst shape, unknown-size counts and provider total. |
|
||||
| Hetzner's projected monthly external egress cost reaches 75% of the approved operations budget | Elevated review: reconcile all account buckets and identify unexpectedly hot flows. |
|
||||
| Hetzner's projected monthly external egress cost reaches 100% of the approved operations budget | Critical commercial threshold: involve product/operations before changing prices, limits or delivery architecture. |
|
||||
|
||||
These thresholds do not throttle a workspace and never create an overage
|
||||
invoice. BILL-25 and AD-27 prohibit traffic-based billing. There is no tenant
|
||||
label or durable tenant traffic ledger in this slice, so workspace attribution
|
||||
remains a support approximation and must not be presented as measured usage.
|
||||
|
||||
Do not silently enable Grafana email notifications. Dashboard review is the
|
||||
launch default; notification routing, recipients, grouping, and rate limits
|
||||
require a separate operator-approved change so storage telemetry cannot recreate
|
||||
an alert-mail flood.
|
||||
|
||||
## Runtime verification
|
||||
|
||||
Before relying on the panels in staging or production:
|
||||
|
||||
1. Confirm `METRICS_ENABLED=true` and both bearer-protected web targets
|
||||
`hivegrid-web-core` and `hivegrid-web-portal` are `UP` in Prometheus. Do not
|
||||
scrape the shared core/admin registry twice.
|
||||
2. Trigger one authorized original-file download and one preview or thumbnail.
|
||||
3. Confirm the original increments `size_known="known"` and its object-byte
|
||||
counter; confirm the generated variant increments only the `unknown` count.
|
||||
4. Confirm rejected, foreign-tenant, scan-pending, and signing-failure requests
|
||||
do not increment either counter.
|
||||
5. Confirm no metric label or help text contains a tenant, file, key, bucket,
|
||||
filename, or signed URL.
|
||||
|
||||
Rollback removes only these observations. Authorization, tenant scope, malware
|
||||
scan gates, and private-bucket signed delivery remain unchanged.
|
||||
@@ -37,13 +37,13 @@ transactional mail, mail outbox, form-submission webhook delivery, and
|
||||
email-sequence worker contracts are intentionally isolated from general outbox
|
||||
fanout.
|
||||
|
||||
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
||||
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
||||
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
|
||||
| ---------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
|
||||
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
|
||||
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Dedicated KWP provider queue plus optional generic import/export or portal/privacy GDPR export processing | `9103` | `9203` | KWP dependencies are always required; GDPR modes additionally require HTTPS `STORAGE_*`, canonical `GDPR_EXPORT_SSE_C_KEY_BASE64`, and both GDPR services; full mode also needs both `IMPORT_QUEUE_*` gates plus generic import/export/scan services |
|
||||
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
|
||||
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
|
||||
|
||||
The detailed queue mapping below is the source of truth for package queues,
|
||||
subscriber queues, concurrency, and workload profile. Update this page first
|
||||
@@ -61,24 +61,66 @@ accept plugin invoke consumption as production-ready. The final live proof is
|
||||
operator-run, inside both `worker-general` and `worker-plugin`:
|
||||
`pnpm run verify:plugin-runtime-live`.
|
||||
|
||||
`worker-import` follows the same placeholder-first principle. Provision the
|
||||
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
|
||||
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
|
||||
export, and scan services are wired into the worker bootstrap. In that idle
|
||||
state `/readyz` may be healthy with an empty queue list, which proves topology
|
||||
without consuming `hivegrid.imports` or `hivegrid.exports`. When both runtime
|
||||
gates are set, startup fails closed unless the worker has real import, export,
|
||||
and scan service dependencies.
|
||||
`worker-import` always consumes `hivegrid.kwp-imports`. KWP provider work is
|
||||
kept off the generic file-import queue so automatic project catalog and
|
||||
binding synchronization remain active while tenant file imports are gated.
|
||||
Keep `IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until
|
||||
generic import, export, scan, portal/privacy GDPR, HTTPS storage, and SSE-C
|
||||
services are wired into the worker bootstrap. In that KWP-only state `/readyz`
|
||||
is healthy with exactly the dedicated KWP queue. When both generic runtime
|
||||
gates are set, startup fails closed unless the worker also has real import,
|
||||
generic export, scan, portal/privacy GDPR service dependencies and the
|
||||
complete encrypted storage contract described below.
|
||||
|
||||
The internal portal/privacy GDPR closeout uses a narrower alternative on the
|
||||
same `worker-import` resource. With both `IMPORT_QUEUE_*` gates unset or `false`,
|
||||
`PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` adds `hivegrid.exports` beside the
|
||||
always-active `hivegrid.kwp-imports` queue and wires only the portal and privacy
|
||||
GDPR export services. Complete
|
||||
`STORAGE_ENDPOINT`, `STORAGE_REGION`, `STORAGE_BUCKET`,
|
||||
`STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, and a canonical
|
||||
`GDPR_EXPORT_SSE_C_KEY_BASE64` for exactly 32 random bytes is mandatory. The
|
||||
same key must be present on core web and portal web. Both writers use SSE-C;
|
||||
both authorized download routes fetch, fully validate and proxy the object server-side, so
|
||||
the key never enters a redirect, browser header, URL, access log, or ticket.
|
||||
The dedicated and full modes are mutually exclusive; mixed gates,
|
||||
another worker group, missing storage, or missing GDPR service wiring fail
|
||||
closed. Full mode uses the same encrypted GDPR services in addition to generic
|
||||
imports/exports/scans. Dedicated mode does not activate `hivegrid.imports` or
|
||||
generic exports; neither mode can disable the isolated KWP queue.
|
||||
|
||||
`worker-general` owns tenant hard-delete. In production, required-RLS, and
|
||||
staging/production Sentry modes it therefore requires both `MEILI_URL` and the
|
||||
HiveGrid-scoped `MEILI_API_KEY` in addition to its deletion-only storage
|
||||
credential. A missing search client aborts startup before queue ownership;
|
||||
provider failures leave the request retryable and logs contain only the error
|
||||
class, never the provider message or credentials. Local development/test may
|
||||
skip external search cleanup only while none of those required-runtime markers
|
||||
is active.
|
||||
|
||||
The privacy and portal writers on `worker-import` and tenant hard-delete on
|
||||
`worker-general` share one distributed tenant-export barrier. Their effective
|
||||
SharedStore target is the first non-empty value of `REDIS_URL`, then
|
||||
`BULLMQ_REDIS_URL`; both resources must resolve to the same Redis protocol,
|
||||
host, port, and database. Credentials may differ. The deploy preflight rejects
|
||||
missing, invalid, or drifting targets without printing either URL. The general
|
||||
worker is delete-only and must never receive `GDPR_EXPORT_SSE_C_KEY_BASE64`.
|
||||
|
||||
Key rotation is deliberately fail-closed: pause the export queue, wait until
|
||||
all 72-hour privacy artifacts have expired (the portal window is 24 hours) or
|
||||
delete those exact expired artifacts through the retention workflow, update
|
||||
the same key atomically on both web resources and `worker-import`, then resume
|
||||
and request fresh exports. There is no plaintext or old-key fallback.
|
||||
|
||||
## Group → queue mapping
|
||||
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | `hivegrid.imports`, `hivegrid.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
|
||||
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
|
||||
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `general` | `hivegrid.notifications`, `hivegrid.maintenance`, `hivegrid.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
|
||||
| `mail` | `hivegrid.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
|
||||
| `import` | Always: `hivegrid.kwp-imports`. Full adds `hivegrid.imports`, `hivegrid.exports` after both `IMPORT_QUEUE_*` gates. Dedicated portal/privacy mode adds only `hivegrid.exports` after `PORTAL_GDPR_EXPORT_QUEUE_ENABLED=true` | none | 1 | 1 | Isolated provider and long-running import/export work. KWP remains active in every valid gate mode; generic imports and GDPR exports remain independently fail-closed |
|
||||
| `webhook` | `hivegrid.ai-chatbot`, `hivegrid.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
|
||||
| `plugin` | `hivegrid.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
|
||||
|
||||
Source of truth: `packages/queue/src/groups.ts`. A test
|
||||
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
|
||||
@@ -181,22 +223,43 @@ Coolify worker apps before deploy: every managed worker group must have
|
||||
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
|
||||
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
|
||||
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
|
||||
The same general-worker preflight requires the non-empty Lago/Mollie provider
|
||||
contract (`LAGO_API_URL`, `LAGO_API_KEY`, `LAGO_TIMEOUT_MS`, `MOLLIE_API_KEY`,
|
||||
`MOLLIE_PROFILE_ID`, `MOLLIE_TIMEOUT_MS`, `BILLING_MOLLIE_WEBHOOK_URL`) because payment reconciliation and
|
||||
durable provider side effects run only in that group. Only `LAGO_API_URL`,
|
||||
`LAGO_API_KEY`, `MOLLIE_API_KEY`, and the environment-specific HTTPS
|
||||
`BILLING_MOLLIE_WEBHOOK_URL` are mandatory; `MOLLIE_PROFILE_ID` and both timeout
|
||||
overrides are optional.
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
||||
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
||||
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
||||
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
||||
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
||||
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
||||
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
||||
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
||||
Production may hold this subscriber before the external payment gates are
|
||||
complete by setting
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`.
|
||||
That explicit value disables the subscriber even under required RLS and waives
|
||||
only the Lago/Mollie provider-key preflight. Unset keeps the required-RLS default,
|
||||
`true` forces the provider contract, and any other value fails closed. The gate
|
||||
must not be copied to another worker group.
|
||||
|
||||
Ein mehrdeutiger Mollie-Payment-Start ohne persistierte `providerPaymentId` wird vom
|
||||
Product-Acquisition-Reconciler absichtlich nicht fuer einen neuen Checkout freigegeben. Der sichere,
|
||||
auditierte Operatorpfad inklusive vollstaendiger Provider-Negativpruefung und exaktem Lago-Cleanup ist
|
||||
in [`docs/ops/product-acquisition-recovery.md`](product-acquisition-recovery.md) beschrieben. Direktes
|
||||
BullMQ-/Outbox-Retry oder ein manuelles Ledger-Update ist fuer diesen Fall verboten.
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
|
||||
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
|
||||
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
|
||||
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
|
||||
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
|
||||
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
|
||||
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
|
||||
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
|
||||
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
|
||||
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
|
||||
|
||||
## Coolify resources
|
||||
|
||||
@@ -231,7 +294,14 @@ webhook, `9205` plugin) so the admin server does not collide with the health ser
|
||||
|
||||
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
|
||||
applications during the deploy workflow when provisioning is explicitly
|
||||
enabled. Staging uses:
|
||||
enabled. It also reconciles every managed worker app to an enabled Coolify
|
||||
`cmd` healthcheck with
|
||||
`node apps/worker/dist/deploy-health.js <40-character lowercase Git SHA> <worker-group>`
|
||||
for the exact immutable image revision and worker resource before rollout. That
|
||||
command verifies the image-baked revision, `APP_REVISION`, enablement, and group,
|
||||
then probes the running process over loopback `/readyz`. A mutable alias, wrong
|
||||
queue group, disabled worker, or healthy stale container cannot satisfy the
|
||||
deploy gate. Staging uses:
|
||||
|
||||
```bash
|
||||
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
|
||||
@@ -261,6 +331,7 @@ Optional overrides:
|
||||
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
|
||||
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
|
||||
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
|
||||
| `PORTAL_GDPR_EXPORT_QUEUE_ENABLED` | false | Internal exports-only mode for `WORKER_QUEUE_GROUP=import`; both `IMPORT_QUEUE_*` gates must remain unset/false and complete `STORAGE_*`, `GDPR_EXPORT_SSE_C_KEY_BASE64`, plus GDPR service wiring is mandatory |
|
||||
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
|
||||
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
|
||||
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
|
||||
@@ -278,8 +349,10 @@ Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
|
||||
for required/expected runtime bridges such as `outbox.mail` on
|
||||
`WORKER_QUEUE_GROUP=mail`.
|
||||
|
||||
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
|
||||
as the liveness probe.
|
||||
Use `/readyz` as the Kubernetes and manual HTTP readiness probe and `/healthz`
|
||||
as the liveness probe. Managed Coolify worker apps instead use the revision-bound
|
||||
`cmd` healthcheck documented above; this avoids relying on HTTP-probe utilities
|
||||
inside the minimal worker image and binds health to the exact deployed SHA.
|
||||
|
||||
Worker health is primarily a queue-runtime check. For managed `worker-mail`
|
||||
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
|
||||
@@ -316,13 +389,13 @@ CLIENT LIST
|
||||
|
||||
## Scaling cookbook
|
||||
|
||||
| Symptom | Action |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
||||
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
||||
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
||||
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
||||
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
||||
| Symptom | Action |
|
||||
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
|
||||
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
|
||||
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
|
||||
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
|
||||
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
|
||||
|
||||
## Operator commands
|
||||
|
||||
@@ -378,6 +451,56 @@ supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
|
||||
standalone default is `general`, so removing the marker during rollback can
|
||||
silently turn a non-general app into another general worker, while an unknown
|
||||
group fails startup by design. For an incident rollback, redeploy the previous
|
||||
known-good image/tag or compose/deploy definition, or keep the five group apps
|
||||
and roll the affected group back independently. In-flight BullMQ job state
|
||||
lives in Redis, so jobs are not lost while workers are rolled.
|
||||
known-good full 40-character SHA or compose/deploy definition. The current
|
||||
reconciler is fleet-wide: it updates all five managed group applications in one
|
||||
validated pass. A modern rollback must therefore redeploy and verify all five
|
||||
groups; it is not safe to reconcile the old SHA and trigger only one group,
|
||||
because a later restart of another group would consume the already-rolled-back
|
||||
configuration. For images that contain the revision-bound deployment-health
|
||||
entrypoint, a Coolify rollback
|
||||
must never patch only `docker_registry_image_tag`: the image tag and
|
||||
`health_check_command` are one atomic configuration unit. The command must be
|
||||
`node apps/worker/dist/deploy-health.js <old-full-sha> <existing-worker-group>`,
|
||||
where the final argument remains the resource's exact existing `general`,
|
||||
`mail`, `import`, `webhook`, or `plugin` group.
|
||||
|
||||
Before the canonical force-deploy is triggered, set `IMAGE_TAG="$OLD_FULL_SHA"`
|
||||
and provide the exact HiveGrid runtime image identity through `WORKER_IMAGE` and
|
||||
`MCP_IMAGE`, or through the complete `REGISTRY`, `IMAGE_NAMESPACE`, and
|
||||
`IMAGE_PREFIX` tuple. The rollback helpers reject suffix-only image matches so
|
||||
a valid CUID from another product cannot be mutated accidentally. Then run
|
||||
`node scripts/coolify-ensure-worker-groups.mjs <staging-or-production>` with the
|
||||
already configured environment-specific Coolify variables. The reconciler
|
||||
atomically sets and rereads both fields for every managed worker resource and
|
||||
rejects group, revision, or persistence drift. Only after it succeeds may the
|
||||
operator run `node scripts/coolify-deploy.mjs trigger "$WORKER_RESOURCE_UUID"`
|
||||
for each of the five worker resources and wait on every exact returned
|
||||
deployment UUID. A scoped single-group rollback requires a future dedicated
|
||||
helper that leaves the other four persisted configurations untouched.
|
||||
For an MCP rollback, the equivalent prerequisite is
|
||||
`node scripts/coolify-deploy.mjs configure-mcp-healthcheck "$MCP_RESOURCE_UUID" "$OLD_FULL_SHA"`;
|
||||
it atomically binds the MCP image tag and command healthcheck to the same old
|
||||
SHA before the same canonical force-deploy path is used. All identifiers above
|
||||
are placeholders; never record real resource UUIDs, endpoints, tokens, or
|
||||
credentials.
|
||||
|
||||
Images built before `apps/worker/dist/deploy-health.js` and
|
||||
`/myapp/.image-revision` were introduced need the bootstrap-compatible legacy
|
||||
path. Do not run the current reconciler against such an old SHA. Instead run
|
||||
`node scripts/coolify-deploy.mjs configure-legacy-rollback-healthcheck "$WORKER_RESOURCE_UUID" "$OLD_FULL_SHA" <existing-worker-group>`.
|
||||
The helper rejects mutable tags, Coolify `APP_REVISION` overrides, and group
|
||||
drift, then atomically sets the old tag, clears `health_check_command`, disables
|
||||
command health, and rereads the result. Force-deploy only after it succeeds and
|
||||
wait on the exact returned deployment UUID.
|
||||
|
||||
This is a degraded, emergency-only bootstrap rollback: disabling command health
|
||||
removes it from Coolify's candidate gate, so brief downtime is possible. Accept
|
||||
the worker only after Coolify reports `running:healthy`, an approved read-only
|
||||
host check resolves the live container digest/OCI revision to the old SHA image,
|
||||
and `node apps/worker/dist/health.js` inside that live container reports the
|
||||
image-baked old revision, enabled processing, the exact existing group, and
|
||||
ready dependencies. `APP_REVISION` must be absent only as a Coolify runtime
|
||||
override; the old image's baked environment value must equal the old SHA.
|
||||
Re-enable and verify the revision-bound `cmd` healthcheck before the next
|
||||
roll-forward to a current image. In-flight BullMQ job state lives in Redis, so
|
||||
jobs are not lost while workers are rolled.
|
||||
Reference in new issue
Block a user