Files
HiveGrid-Wireframes/03 Quellen und Referenzen/hivegrid-cloud/docs/ops/lago-production-prep.md
T

14 KiB

Lago Production Prep Runbook

This runbook defines the production target for HiveGrid billing after AD-27. The isolated production foundation was provisioned on 2026-07-27 after explicit operator approval. Provisioning does not authorize live billing: checkout and provider side effects remain disabled until the external gates in EXT-16 are complete.

Scope

lago-production will be the dedicated production Lago service for HiveGrid. It must be separate from lago-staging and must never reuse staging database, Redis, storage, API keys, webhook secrets, customers, plans or invoice records.

Production billing owns:

  • Lago customers, plans, seats, subscriptions and invoice records.
  • Invoice PDF generation and storage.
  • Billing status used by HiveGrid entitlement and read-only gates.
  • Mollie payment references for paid, failed, expired, cancelled and refunded payments.

Out of scope for v1:

  • Usage billing.
  • Credits.
  • Add-on metering.
  • Public price promises in code.
  • Live activation before legal, tax and Mollie KYC approval.

Target Topology

Item Decision
Coolify project HiveGrid
Coolify environment production
Resource name lago-production
Current host hivegrid-prod-worker; API bound only to the private 10.0.0.0/16 network
Long-term host Dedicated billing host if measured capacity or blast-radius requirements justify it
Public access None by default
Operator access Internal network or SSH tunnel; optional Authentik-gated admin access later
HiveGrid app access Server-to-server from web/API/worker to Lago API
Production payment processor Mollie live profile, only after KYC and legal/tax approval

The current placement is intentionally pragmatic: production has its own Lago Coolify resource, Postgres, Redis, versioned object bucket, secrets and backup path, while the containers share the existing worker VM. The host had 16 GB RAM and approximately 145 GB free disk at provisioning time. No public Lago DNS record or public port exists.

Pinned runtime images:

  • getlago/api:v1.50.0
  • getlago/front:v1.50.0
  • postgres:15-alpine
  • redis:7-alpine
  • getlago/lago-gotenberg:8.15

API, migration runner, general worker, billing worker and the periodic api-clock scheduler use the same pinned Lago API image.

Invoice/file storage uses the private, versioned Hetzner Object Storage bucket hivegrid-prod-lago in fsn1. It is separate from the HiveGrid application bucket and from hivegrid-staging-lago.

Required Production Secrets

Store these only in the runtime secret store. Never put values in docs, Git, tickets, screenshots or copied logs.

Secret Owner Notes
LAGO_API_URL HiveGrid runtime Internal production Lago API URL; do not include /api/v1.
LAGO_API_KEY HiveGrid runtime Production Lago API bearer token.
LAGO_WEBHOOK_SECRET HiveGrid runtime / Lago Dedicated production webhook secret.
LAGO_TIMEOUT_MS HiveGrid runtime Optional; default stays 10000.
MOLLIE_API_KEY HiveGrid runtime Mollie live API key after KYC approval.
MOLLIE_PROFILE_ID HiveGrid runtime Production website profile when required.
MOLLIE_TIMEOUT_MS HiveGrid runtime Optional; default stays 10000.
BILLING_MOLLIE_WEBHOOK_URL HiveGrid worker-general https://api.hivegrid.de/v1/billing/mollie/webhooks; required only once provider side effects are enabled.

Do not reuse staging/testmode keys in production. Rotate keys when moving from pilot validation to real customer billing.

Backup And Restore

Production Lago is stateful. It must not receive production traffic before a backup and restore path has been verified.

Minimum backup scope:

  • Lago Postgres database.
  • Lago Redis persistence if configured for durable state.
  • Invoice PDF storage volume or bucket.
  • Coolify resource settings and image tags.
  • Runtime secret names and rotation history, without secret values.

Minimum backup schedule:

  • Daily encrypted database dump.
  • Backup before every Lago image upgrade.
  • Backup before destructive billing test data cleanup.
  • Retention aligned with the final accounting and tax retention decision.

Implemented production schedule:

  • hivegrid-lago-backup.timer runs daily at 03:00 UTC with up to 15 minutes randomized delay.
  • workflows/hivegrid-lago-backup backs up a compressed Postgres dump, Redis persistence, RSA keys and local Lago storage into the encrypted HiveGrid restic repository in hivegrid-prod-backups.
  • Retention is 14 daily, 8 weekly and 12 monthly snapshots.
  • The versioned hivegrid-prod-lago bucket remains the primary file store.

Restore drill before go-live:

  1. Create a fresh disposable restore target.
  2. Restore the latest Lago Postgres dump.
  3. Restore or attach invoice PDF storage.
  4. Start Lago API, frontend, worker, billing worker, clock, PDF service and Redis.
  5. Verify /health.
  6. Read one known customer, subscription and invoice.
  7. Verify HiveGrid can resolve the corresponding local billing state without writing test payments.

The restore drill date, operator and result must be recorded before production billing is enabled.

Restore proof

2026-07-27, Raffael:

  • Versionierter Infra-Proof: Commit 936a32a.
  • Encrypted snapshot 3d8a1deb completed after the production organization and API key existed.
  • The snapshot was restored into disposable Postgres and Redis containers.
  • Checksum, schema history, exactly one organization and exactly one API key were verified.
  • Disposable Lago API, frontend, worker, billing worker and PDF service started; /health passed.
  • A canary object was written to, read from and fully removed from the versioned production bucket with a matching SHA-256.
  • The disposable containers, volume, network and restore directory were removed.

2026-07-28, Raffael:

  • Pre-upgrade encrypted snapshot cc8c5c88 was restored successfully before the first image change.
  • Bridge snapshot 3c245001 was created and restored on Lago v1.45.0 immediately before the target deployment.
  • Post-upgrade encrypted snapshot 176fdf80 was restored with Postgres, Redis, RSA/storage mounts, API, frontend, general worker, billing worker, PDF and the new clock scheduler.
  • The restored database contained exactly one production organization and one production API key; customer, plan, subscription and invoice tables remained empty.
  • Every disposable resource and restore directory was removed after the drill.

Upgrade record: 2026-07-28

Lago production was upgraded from v1.27.1 to v1.50.0 through the required bridge chain:

v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0
          -> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0

Required migration work completed:

  • v1.28.1: invoice billing-entity sequential-ID job, 0 records left.
  • v1.30.0: organization-ID migration, nothing left to migrate.
  • v1.32.0: required payment job, nothing left to migrate.
  • v1.42.0: charge/fixed-charge jobs, nothing left to migrate.
  • v1.44.0: subscription event-date job, nothing left to migrate.
  • v1.45.0: wallet dry-run and write pass, 0 wallets and 0 errors.
  • v1.50.0: no native payment-provider customer or integration exists, so provider-specific payment-method backfills were not applicable.

The official api-clock service was added. It starts 26 scheduled Lago jobs, including invoice finalization, billing, retries and subscription lifecycle work. The production restore drill now restores and verifies the clock container as part of the disposable topology.

Postgres remains on the patched postgres:15-alpine image (15.18 during the upgrade). The optional Lago Partman image was not adopted because its 15.0-alpine tag would downgrade the database binaries. Lago's Partman-aware event migrations skip partition setup when the extension is unavailable; this production instance does not run the high-volume events processor.

Final proof:

  • Private API health: v1.50.0.
  • Database migrations: 0 down.
  • Production state unchanged: 1 organization, 1 API key, 0 customers, plans, subscriptions, invoices and wallets.
  • Queues: 0 pending, 0 retries, 0 dead.
  • PDF service: HTTP 200.
  • Object-storage HEAD through Active Storage succeeded.
  • Running HiveGrid production API to Lago call: authenticated HTTP 200.
  • PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false remains active.
  • Clock starts all 26 schedules and completes jobs with error=nil.
  • No API, general-worker or billing-worker error was found after deployment.

Canonical billing catalog

Production contains the same controlled BILL-11 catalog as Staging:

  • Version: 2026-07-29.1
  • SHA-256 digest: a8ba1affd93e1b870d016284457594988c7fc5f58fdf70c3d57be80b3c915b2c
  • Public packages: Core / Swarm / Colony with stable technical codes starter / pro / business
  • Two monthly-billed variants per package, including the 12-month SEPA commitment
  • Four fixed recurring storage packages
  • German VAT tax code de-vat-19 at 19 %, exclusive
  • No usage metering

The Production apply and follow-up verify completed on 2026-07-29. The verify returned no changed actions and the digest exactly matches Staging. Redacted pre-change snapshot:

tmp/billing-catalog-snapshots/production-2026-07-29T06-52-23-908Z.json

Operator sequence:

NODE_ENV=production pnpm billing:catalog --target=production --plan
NODE_ENV=production pnpm billing:catalog \
  --target=production \
  --apply \
  --confirm=production:2026-07-29.1
NODE_ENV=production pnpm billing:catalog --target=production --verify

The catalog is active, but real Production payment side effects are not. PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false remains mandatory until all Mollie/Legal gates below are closed. Production must not receive a test-mode Mollie key.

2026-07-29 BILL-11 closeout proof:

  • The read-only Production verify completed with catalog version 2026-07-29.1, the same Staging digest and only noop actions.
  • Gitea promotion run 11477 promoted the live Staging revision 549f708d9809e9ba839c15366943918679a034d3 without rebuilding it.
  • Core, Admin, Portal and API health/readiness plus MCP, Realtime and all five worker groups are healthy on that revision.
  • PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false remained active after promotion and no Mollie API key is configured in Production. No Production payment or customer invoice was created by this proof.

Access And Domain

Default: no public Lago domain.

Allowed access paths:

  • Internal service-to-service URL for HiveGrid runtime.
  • SSH tunnel for operators.
  • Authentik-gated admin URL later, only if operator UX requires browser access.

Do not expose Lago directly to customers. Customer-facing billing screens must remain HiveGrid-owned and use the server-side Lago/Mollie gateways.

Mollie Live Gate

Mollie live activation is blocked until:

  • Legal entity / firming is final.
  • Business address, bank account and KYC documents are accepted by Mollie.
  • Rechnungssteller, tax handling, imprint and customer contract data are approved.
  • Live webhook endpoint and return URLs are set for the production host.
  • A live low-value payment and refund rehearsal is approved by an operator.
  • Mollie has explicitly raised the SEPA Direct Debit transaction limit above the highest gross monthly invoice. The default documented limit is EUR 1,000; Colony is already above it before optional seats or storage.

Until then, production must stay on manual billing or disabled self-service checkout.

The first SEPA authorization uses a customer-present payment method and sequenceType=first; it does not attempt a direct-debit payment before a mandate exists. Later Lago invoices use sequenceType=recurring with the stored Mollie customer and mandate. See Mollie's recurring-payments guide and SEPA Direct Debit limits.

The current production gate is PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false on worker-general. This disables the durable provider subscriber even under required RLS. Do not replace it with a test-mode Mollie key.

Rollback

Rollback has two layers:

  1. HiveGrid application rollback: deploy an older immutable HiveGrid image tag through Coolify. This must not rewrite Lago state.
  2. Lago service rollback: roll back Lago image tags and restore the pre- upgrade Lago database/storage backup when the Lago upgrade mutated state.

Never restore Lago production from staging. Never merge staging and production Lago datasets.

Go-Live Checklist

  • lago-production Coolify resource exists in HiveGrid / production.
  • Dedicated production DB, Redis and storage are configured.
  • Production secrets are set and staging secrets are absent.
  • Backup job is configured.
  • Restore drill is completed and documented.
  • Mollie live profile is verified.
  • Rechnungssteller, tax and imprint data are approved.
  • Web/API/worker health checks pass with production Lago config.
  • Manual rollback path is tested from image tag and Lago backup.