Files
HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/lago-production-prep.md
T
2026-07-08 16:15:37 +02:00

5.8 KiB

Lago Production Prep Runbook

This runbook defines the production target for HiveGrid billing after AD-27. It is a preparation document only. Do not provision or enable production billing from this document until the external gates in EXT-16 are complete.

Scope

lago-production will be the dedicated production Lago service for HiveGrid. It must be separate from lago-staging and must never reuse staging database, Redis, storage, API keys, webhook secrets, customers, plans or invoice records.

Production billing owns:

  • Lago customers, plans, seats, subscriptions and invoice records.
  • Invoice PDF generation and storage.
  • Billing status used by HiveGrid entitlement and read-only gates.
  • Mollie payment references for paid, failed, expired, cancelled and refunded payments.

Out of scope for v1:

  • Usage billing.
  • Credits.
  • Add-on metering.
  • Public price promises in code.
  • Live activation before legal, tax and Mollie KYC approval.

Target Topology

Item Decision
Coolify project HiveGrid
Coolify environment production
Resource name lago-production
Initial host The current HiveGrid production host, while Hetzner capacity is constrained
Long-term host Dedicated production host or isolated capacity once the Hetzner limit is lifted
Public access None by default
Operator access Internal network or SSH tunnel; optional Authentik-gated admin access later
HiveGrid app access Server-to-server from web/API/worker to Lago API
Production payment processor Mollie live profile, only after KYC and legal/tax approval

The initial host decision is intentionally pragmatic: production gets its own Lago resource and data stores, but it may share the physical server with other HiveGrid production services until infrastructure capacity allows a stricter split. The isolation boundary is the Coolify resource, secrets, database, Redis, storage and backup path, not the current machine count.

Required Production Secrets

Store these only in the runtime secret store. Never put values in docs, Git, tickets, screenshots or copied logs.

Secret Owner Notes
LAGO_API_URL HiveGrid runtime Internal production Lago API URL; do not include /api/v1.
LAGO_API_KEY HiveGrid runtime Production Lago API bearer token.
LAGO_WEBHOOK_SECRET HiveGrid runtime / Lago Dedicated production webhook secret.
LAGO_TIMEOUT_MS HiveGrid runtime Optional; default stays 10000.
MOLLIE_API_KEY HiveGrid runtime Mollie live API key after KYC approval.
MOLLIE_PROFILE_ID HiveGrid runtime Production website profile when required.
MOLLIE_TIMEOUT_MS HiveGrid runtime Optional; default stays 10000.

Do not reuse staging/testmode keys in production. Rotate keys when moving from pilot validation to real customer billing.

Backup And Restore

Production Lago is stateful. It must not receive production traffic before a backup and restore path has been verified.

Minimum backup scope:

  • Lago Postgres database.
  • Lago Redis persistence if configured for durable state.
  • Invoice PDF storage volume or bucket.
  • Coolify resource settings and image tags.
  • Runtime secret names and rotation history, without secret values.

Minimum backup schedule:

  • Daily encrypted database dump.
  • Backup before every Lago image upgrade.
  • Backup before destructive billing test data cleanup.
  • Retention aligned with the final accounting and tax retention decision.

Restore drill before go-live:

  1. Create a fresh disposable restore target.
  2. Restore the latest Lago Postgres dump.
  3. Restore or attach invoice PDF storage.
  4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
  5. Verify /health.
  6. Read one known customer, subscription and invoice.
  7. Verify HiveGrid can resolve the corresponding local billing state without writing test payments.

The restore drill date, operator and result must be recorded before production billing is enabled.

Access And Domain

Default: no public Lago domain.

Allowed access paths:

  • Internal service-to-service URL for HiveGrid runtime.
  • SSH tunnel for operators.
  • Authentik-gated admin URL later, only if operator UX requires browser access.

Do not expose Lago directly to customers. Customer-facing billing screens must remain HiveGrid-owned and use the server-side Lago/Mollie gateways.

Mollie Live Gate

Mollie live activation is blocked until:

  • Legal entity / firming is final.
  • Business address, bank account and KYC documents are accepted by Mollie.
  • Rechnungssteller, tax handling, imprint and customer contract data are approved.
  • Live webhook endpoint and return URLs are set for the production host.
  • A live low-value payment and refund rehearsal is approved by an operator.

Until then, production must stay on manual billing or disabled self-service checkout.

Rollback

Rollback has two layers:

  1. HiveGrid application rollback: deploy an older immutable HiveGrid image tag through Coolify. This must not rewrite Lago state.
  2. Lago service rollback: roll back Lago image tags and restore the pre- upgrade Lago database/storage backup when the Lago upgrade mutated state.

Never restore Lago production from staging. Never merge staging and production Lago datasets.

Go-Live Checklist

  • lago-production Coolify resource exists in HiveGrid / production.
  • Dedicated production DB, Redis and storage are configured.
  • Production secrets are set and staging secrets are absent.
  • Backup job is configured.
  • Restore drill is completed and documented.
  • Mollie live profile is verified.
  • Rechnungssteller, tax and imprint data are approved.
  • Web/API/worker health checks pass with production Lago config.
  • Manual rollback path is tested from image tag and Lago backup.