# Lago Production Prep Runbook This runbook defines the production target for HiveGrid billing after AD-27. The isolated production foundation was provisioned on 2026-07-27 after explicit operator approval. Provisioning does not authorize live billing: checkout and provider side effects remain disabled until the external gates in `EXT-16` are complete. ## Scope `lago-production` will be the dedicated production Lago service for HiveGrid. It must be separate from `lago-staging` and must never reuse staging database, Redis, storage, API keys, webhook secrets, customers, plans or invoice records. Production billing owns: - Lago customers, plans, seats, subscriptions and invoice records. - Invoice PDF generation and storage. - Billing status used by HiveGrid entitlement and read-only gates. - Mollie payment references for paid, failed, expired, cancelled and refunded payments. Out of scope for v1: - Usage billing. - Credits. - Add-on metering. - Public price promises in code. - Live activation before legal, tax and Mollie KYC approval. ## Target Topology | Item | Decision | | ---------------------------- | ----------------------------------------------------------------------------------- | | Coolify project | `HiveGrid` | | Coolify environment | `production` | | Resource name | `lago-production` | | Current host | `hivegrid-prod-worker`; API bound only to the private `10.0.0.0/16` network | | Long-term host | Dedicated billing host if measured capacity or blast-radius requirements justify it | | Public access | None by default | | Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later | | HiveGrid app access | Server-to-server from web/API/worker to Lago API | | Production payment processor | Mollie live profile, only after KYC and legal/tax approval | The current placement is intentionally pragmatic: production has its own Lago Coolify resource, Postgres, Redis, versioned object bucket, secrets and backup path, while the containers share the existing worker VM. The host had 16 GB RAM and approximately 145 GB free disk at provisioning time. No public Lago DNS record or public port exists. Pinned runtime images: - `getlago/api:v1.50.0` - `getlago/front:v1.50.0` - `postgres:15-alpine` - `redis:7-alpine` - `getlago/lago-gotenberg:8.15` API, migration runner, general worker, billing worker and the periodic `api-clock` scheduler use the same pinned Lago API image. Invoice/file storage uses the private, versioned Hetzner Object Storage bucket `hivegrid-prod-lago` in `fsn1`. It is separate from the HiveGrid application bucket and from `hivegrid-staging-lago`. ## Required Production Secrets Store these only in the runtime secret store. Never put values in docs, Git, tickets, screenshots or copied logs. | Secret | Owner | Notes | | --------------------- | ----------------------- | ----------------------------------------------------------- | | `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. | | `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. | | `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. | | `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. | | `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. | | `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. | | `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. | | `BILLING_MOLLIE_WEBHOOK_URL` | HiveGrid `worker-general` | `https://api.hivegrid.de/v1/billing/mollie/webhooks`; required only once provider side effects are enabled. | Do not reuse staging/testmode keys in production. Rotate keys when moving from pilot validation to real customer billing. ## Backup And Restore Production Lago is stateful. It must not receive production traffic before a backup and restore path has been verified. Minimum backup scope: - Lago Postgres database. - Lago Redis persistence if configured for durable state. - Invoice PDF storage volume or bucket. - Coolify resource settings and image tags. - Runtime secret names and rotation history, without secret values. Minimum backup schedule: - Daily encrypted database dump. - Backup before every Lago image upgrade. - Backup before destructive billing test data cleanup. - Retention aligned with the final accounting and tax retention decision. Implemented production schedule: - `hivegrid-lago-backup.timer` runs daily at 03:00 UTC with up to 15 minutes randomized delay. - `workflows/hivegrid-lago-backup` backs up a compressed Postgres dump, Redis persistence, RSA keys and local Lago storage into the encrypted HiveGrid restic repository in `hivegrid-prod-backups`. - Retention is 14 daily, 8 weekly and 12 monthly snapshots. - The versioned `hivegrid-prod-lago` bucket remains the primary file store. Restore drill before go-live: 1. Create a fresh disposable restore target. 2. Restore the latest Lago Postgres dump. 3. Restore or attach invoice PDF storage. 4. Start Lago API, frontend, worker, billing worker, clock, PDF service and Redis. 5. Verify `/health`. 6. Read one known customer, subscription and invoice. 7. Verify HiveGrid can resolve the corresponding local billing state without writing test payments. The restore drill date, operator and result must be recorded before production billing is enabled. ### Restore proof 2026-07-27, Raffael: - Versionierter Infra-Proof: Commit `936a32a`. - Encrypted snapshot `3d8a1deb` completed after the production organization and API key existed. - The snapshot was restored into disposable Postgres and Redis containers. - Checksum, schema history, exactly one organization and exactly one API key were verified. - Disposable Lago API, frontend, worker, billing worker and PDF service started; `/health` passed. - A canary object was written to, read from and fully removed from the versioned production bucket with a matching SHA-256. - The disposable containers, volume, network and restore directory were removed. 2026-07-28, Raffael: - Pre-upgrade encrypted snapshot `cc8c5c88` was restored successfully before the first image change. - Bridge snapshot `3c245001` was created and restored on Lago `v1.45.0` immediately before the target deployment. - Post-upgrade encrypted snapshot `176fdf80` was restored with Postgres, Redis, RSA/storage mounts, API, frontend, general worker, billing worker, PDF and the new clock scheduler. - The restored database contained exactly one production organization and one production API key; customer, plan, subscription and invoice tables remained empty. - Every disposable resource and restore directory was removed after the drill. ## Upgrade record: 2026-07-28 Lago production was upgraded from `v1.27.1` to `v1.50.0` through the required bridge chain: ```text v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0 -> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0 ``` Required migration work completed: - `v1.28.1`: invoice billing-entity sequential-ID job, `0` records left. - `v1.30.0`: organization-ID migration, nothing left to migrate. - `v1.32.0`: required payment job, nothing left to migrate. - `v1.42.0`: charge/fixed-charge jobs, nothing left to migrate. - `v1.44.0`: subscription event-date job, nothing left to migrate. - `v1.45.0`: wallet dry-run and write pass, `0` wallets and `0` errors. - `v1.50.0`: no native payment-provider customer or integration exists, so provider-specific payment-method backfills were not applicable. The official `api-clock` service was added. It starts `26` scheduled Lago jobs, including invoice finalization, billing, retries and subscription lifecycle work. The production restore drill now restores and verifies the clock container as part of the disposable topology. Postgres remains on the patched `postgres:15-alpine` image (`15.18` during the upgrade). The optional Lago Partman image was not adopted because its `15.0-alpine` tag would downgrade the database binaries. Lago's Partman-aware event migrations skip partition setup when the extension is unavailable; this production instance does not run the high-volume events processor. Final proof: - Private API health: `v1.50.0`. - Database migrations: `0` down. - Production state unchanged: `1` organization, `1` API key, `0` customers, plans, subscriptions, invoices and wallets. - Queues: `0` pending, `0` retries, `0` dead. - PDF service: HTTP `200`. - Object-storage HEAD through Active Storage succeeded. - Running HiveGrid production API to Lago call: authenticated HTTP `200`. - `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains active. - Clock starts all `26` schedules and completes jobs with `error=nil`. - No API, general-worker or billing-worker error was found after deployment. ## Canonical billing catalog Production contains the same controlled `BILL-11` catalog as Staging: - Version: `2026-07-29.1` - SHA-256 digest: `a8ba1affd93e1b870d016284457594988c7fc5f58fdf70c3d57be80b3c915b2c` - Public packages: Core / Swarm / Colony with stable technical codes `starter` / `pro` / `business` - Two monthly-billed variants per package, including the 12-month SEPA commitment - Four fixed recurring storage packages - German VAT tax code `de-vat-19` at `19 %`, exclusive - No usage metering The Production apply and follow-up verify completed on 2026-07-29. The verify returned no changed actions and the digest exactly matches Staging. Redacted pre-change snapshot: ```text tmp/billing-catalog-snapshots/production-2026-07-29T06-52-23-908Z.json ``` Operator sequence: ```bash NODE_ENV=production pnpm billing:catalog --target=production --plan NODE_ENV=production pnpm billing:catalog \ --target=production \ --apply \ --confirm=production:2026-07-29.1 NODE_ENV=production pnpm billing:catalog --target=production --verify ``` The catalog is active, but real Production payment side effects are not. `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains mandatory until all Mollie/Legal gates below are closed. Production must not receive a test-mode Mollie key. 2026-07-29 `BILL-11` closeout proof: - The read-only Production verify completed with catalog version `2026-07-29.1`, the same Staging digest and only `noop` actions. - Gitea promotion run `11477` promoted the live Staging revision `549f708d9809e9ba839c15366943918679a034d3` without rebuilding it. - Core, Admin, Portal and API health/readiness plus MCP, Realtime and all five worker groups are healthy on that revision. - `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remained active after promotion and no Mollie API key is configured in Production. No Production payment or customer invoice was created by this proof. ## Access And Domain Default: no public Lago domain. Allowed access paths: - Internal service-to-service URL for HiveGrid runtime. - SSH tunnel for operators. - Authentik-gated admin URL later, only if operator UX requires browser access. Do not expose Lago directly to customers. Customer-facing billing screens must remain HiveGrid-owned and use the server-side Lago/Mollie gateways. ## Mollie Live Gate Mollie live activation is blocked until: - Legal entity / firming is final. - Business address, bank account and KYC documents are accepted by Mollie. - Rechnungssteller, tax handling, imprint and customer contract data are approved. - Live webhook endpoint and return URLs are set for the production host. - A live low-value payment and refund rehearsal is approved by an operator. - Mollie has explicitly raised the SEPA Direct Debit transaction limit above the highest gross monthly invoice. The default documented limit is `EUR 1,000`; Colony is already above it before optional seats or storage. Until then, production must stay on manual billing or disabled self-service checkout. The first SEPA authorization uses a customer-present payment method and `sequenceType=first`; it does not attempt a direct-debit payment before a mandate exists. Later Lago invoices use `sequenceType=recurring` with the stored Mollie customer and mandate. See Mollie's [recurring-payments guide](https://docs.mollie.com/gu/docs/recurring-payments) and [SEPA Direct Debit limits](https://docs.mollie.com/docs/sepa-direct-debit). The current production gate is `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on `worker-general`. This disables the durable provider subscriber even under required RLS. Do not replace it with a test-mode Mollie key. ## Rollback Rollback has two layers: 1. **HiveGrid application rollback:** deploy an older immutable HiveGrid image tag through Coolify. This must not rewrite Lago state. 2. **Lago service rollback:** roll back Lago image tags and restore the pre- upgrade Lago database/storage backup when the Lago upgrade mutated state. Never restore Lago production from staging. Never merge staging and production Lago datasets. ## Go-Live Checklist - [x] `lago-production` Coolify resource exists in `HiveGrid` / `production`. - [x] Dedicated production DB, Redis and storage are configured. - [x] Production secrets are set and staging secrets are absent. - [x] Backup job is configured. - [x] Restore drill is completed and documented. - [ ] Mollie live profile is verified. - [ ] Rechnungssteller, tax and imprint data are approved. - [x] Web/API/worker health checks pass with production Lago config. - [ ] Manual rollback path is tested from image tag and Lago backup.