# Lago Production Prep Runbook This runbook defines the production target for HiveGrid billing after AD-27. It is a preparation document only. Do not provision or enable production billing from this document until the external gates in `EXT-16` are complete. ## Scope `lago-production` will be the dedicated production Lago service for HiveGrid. It must be separate from `lago-staging` and must never reuse staging database, Redis, storage, API keys, webhook secrets, customers, plans or invoice records. Production billing owns: - Lago customers, plans, seats, subscriptions and invoice records. - Invoice PDF generation and storage. - Billing status used by HiveGrid entitlement and read-only gates. - Mollie payment references for paid, failed, expired, cancelled and refunded payments. Out of scope for v1: - Usage billing. - Credits. - Add-on metering. - Public price promises in code. - Live activation before legal, tax and Mollie KYC approval. ## Target Topology | Item | Decision | |---|---| | Coolify project | `HiveGrid` | | Coolify environment | `production` | | Resource name | `lago-production` | | Initial host | The current HiveGrid production host, while Hetzner capacity is constrained | | Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted | | Public access | None by default | | Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later | | HiveGrid app access | Server-to-server from web/API/worker to Lago API | | Production payment processor | Mollie live profile, only after KYC and legal/tax approval | The initial host decision is intentionally pragmatic: production gets its own Lago resource and data stores, but it may share the physical server with other HiveGrid production services until infrastructure capacity allows a stricter split. The isolation boundary is the Coolify resource, secrets, database, Redis, storage and backup path, not the current machine count. ## Required Production Secrets Store these only in the runtime secret store. Never put values in docs, Git, tickets, screenshots or copied logs. | Secret | Owner | Notes | |---|---|---| | `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. | | `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. | | `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. | | `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. | | `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. | | `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. | | `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. | Do not reuse staging/testmode keys in production. Rotate keys when moving from pilot validation to real customer billing. ## Backup And Restore Production Lago is stateful. It must not receive production traffic before a backup and restore path has been verified. Minimum backup scope: - Lago Postgres database. - Lago Redis persistence if configured for durable state. - Invoice PDF storage volume or bucket. - Coolify resource settings and image tags. - Runtime secret names and rotation history, without secret values. Minimum backup schedule: - Daily encrypted database dump. - Backup before every Lago image upgrade. - Backup before destructive billing test data cleanup. - Retention aligned with the final accounting and tax retention decision. Restore drill before go-live: 1. Create a fresh disposable restore target. 2. Restore the latest Lago Postgres dump. 3. Restore or attach invoice PDF storage. 4. Start Lago API, frontend, worker, billing worker, PDF service and Redis. 5. Verify `/health`. 6. Read one known customer, subscription and invoice. 7. Verify HiveGrid can resolve the corresponding local billing state without writing test payments. The restore drill date, operator and result must be recorded before production billing is enabled. ## Access And Domain Default: no public Lago domain. Allowed access paths: - Internal service-to-service URL for HiveGrid runtime. - SSH tunnel for operators. - Authentik-gated admin URL later, only if operator UX requires browser access. Do not expose Lago directly to customers. Customer-facing billing screens must remain HiveGrid-owned and use the server-side Lago/Mollie gateways. ## Mollie Live Gate Mollie live activation is blocked until: - Legal entity / firming is final. - Business address, bank account and KYC documents are accepted by Mollie. - Rechnungssteller, tax handling, imprint and customer contract data are approved. - Live webhook endpoint and return URLs are set for the production host. - A live low-value payment and refund rehearsal is approved by an operator. Until then, production must stay on manual billing or disabled self-service checkout. ## Rollback Rollback has two layers: 1. **HiveGrid application rollback:** deploy an older immutable HiveGrid image tag through Coolify. This must not rewrite Lago state. 2. **Lago service rollback:** roll back Lago image tags and restore the pre- upgrade Lago database/storage backup when the Lago upgrade mutated state. Never restore Lago production from staging. Never merge staging and production Lago datasets. ## Go-Live Checklist - [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`. - [ ] Dedicated production DB, Redis and storage are configured. - [ ] Production secrets are set and staging secrets are absent. - [ ] Backup job is configured. - [ ] Restore drill is completed and documented. - [ ] Mollie live profile is verified. - [ ] Rechnungssteller, tax and imprint data are approved. - [ ] Web/API/worker health checks pass with production Lago config. - [ ] Manual rollback path is tested from image tag and Lago backup.