14 KiB
Lago Production Prep Runbook
This runbook defines the production target for HiveGrid billing after AD-27.
The isolated production foundation was provisioned on 2026-07-27 after explicit
operator approval. Provisioning does not authorize live billing: checkout and
provider side effects remain disabled until the external gates in EXT-16 are
complete.
Scope
lago-production will be the dedicated production Lago service for HiveGrid.
It must be separate from lago-staging and must never reuse staging database,
Redis, storage, API keys, webhook secrets, customers, plans or invoice records.
Production billing owns:
- Lago customers, plans, seats, subscriptions and invoice records.
- Invoice PDF generation and storage.
- Billing status used by HiveGrid entitlement and read-only gates.
- Mollie payment references for paid, failed, expired, cancelled and refunded payments.
Out of scope for v1:
- Usage billing.
- Credits.
- Add-on metering.
- Public price promises in code.
- Live activation before legal, tax and Mollie KYC approval.
Target Topology
| Item | Decision |
|---|---|
| Coolify project | HiveGrid |
| Coolify environment | production |
| Resource name | lago-production |
| Current host | hivegrid-prod-worker; API bound only to the private 10.0.0.0/16 network |
| Long-term host | Dedicated billing host if measured capacity or blast-radius requirements justify it |
| Public access | None by default |
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
The current placement is intentionally pragmatic: production has its own Lago Coolify resource, Postgres, Redis, versioned object bucket, secrets and backup path, while the containers share the existing worker VM. The host had 16 GB RAM and approximately 145 GB free disk at provisioning time. No public Lago DNS record or public port exists.
Pinned runtime images:
getlago/api:v1.50.0getlago/front:v1.50.0postgres:15-alpineredis:7-alpinegetlago/lago-gotenberg:8.15
API, migration runner, general worker, billing worker and the periodic
api-clock scheduler use the same pinned Lago API image.
Invoice/file storage uses the private, versioned Hetzner Object Storage bucket
hivegrid-prod-lago in fsn1. It is separate from the HiveGrid application
bucket and from hivegrid-staging-lago.
Required Production Secrets
Store these only in the runtime secret store. Never put values in docs, Git, tickets, screenshots or copied logs.
| Secret | Owner | Notes |
|---|---|---|
LAGO_API_URL |
HiveGrid runtime | Internal production Lago API URL; do not include /api/v1. |
LAGO_API_KEY |
HiveGrid runtime | Production Lago API bearer token. |
LAGO_WEBHOOK_SECRET |
HiveGrid runtime / Lago | Dedicated production webhook secret. |
LAGO_TIMEOUT_MS |
HiveGrid runtime | Optional; default stays 10000. |
MOLLIE_API_KEY |
HiveGrid runtime | Mollie live API key after KYC approval. |
MOLLIE_PROFILE_ID |
HiveGrid runtime | Production website profile when required. |
MOLLIE_TIMEOUT_MS |
HiveGrid runtime | Optional; default stays 10000. |
BILLING_MOLLIE_WEBHOOK_URL |
HiveGrid worker-general |
https://api.hivegrid.de/v1/billing/mollie/webhooks; required only once provider side effects are enabled. |
Do not reuse staging/testmode keys in production. Rotate keys when moving from pilot validation to real customer billing.
Backup And Restore
Production Lago is stateful. It must not receive production traffic before a backup and restore path has been verified.
Minimum backup scope:
- Lago Postgres database.
- Lago Redis persistence if configured for durable state.
- Invoice PDF storage volume or bucket.
- Coolify resource settings and image tags.
- Runtime secret names and rotation history, without secret values.
Minimum backup schedule:
- Daily encrypted database dump.
- Backup before every Lago image upgrade.
- Backup before destructive billing test data cleanup.
- Retention aligned with the final accounting and tax retention decision.
Implemented production schedule:
hivegrid-lago-backup.timerruns daily at 03:00 UTC with up to 15 minutes randomized delay.workflows/hivegrid-lago-backupbacks up a compressed Postgres dump, Redis persistence, RSA keys and local Lago storage into the encrypted HiveGrid restic repository inhivegrid-prod-backups.- Retention is 14 daily, 8 weekly and 12 monthly snapshots.
- The versioned
hivegrid-prod-lagobucket remains the primary file store.
Restore drill before go-live:
- Create a fresh disposable restore target.
- Restore the latest Lago Postgres dump.
- Restore or attach invoice PDF storage.
- Start Lago API, frontend, worker, billing worker, clock, PDF service and Redis.
- Verify
/health. - Read one known customer, subscription and invoice.
- Verify HiveGrid can resolve the corresponding local billing state without writing test payments.
The restore drill date, operator and result must be recorded before production billing is enabled.
Restore proof
2026-07-27, Raffael:
- Versionierter Infra-Proof: Commit
936a32a. - Encrypted snapshot
3d8a1debcompleted after the production organization and API key existed. - The snapshot was restored into disposable Postgres and Redis containers.
- Checksum, schema history, exactly one organization and exactly one API key were verified.
- Disposable Lago API, frontend, worker, billing worker and PDF service started;
/healthpassed. - A canary object was written to, read from and fully removed from the versioned production bucket with a matching SHA-256.
- The disposable containers, volume, network and restore directory were removed.
2026-07-28, Raffael:
- Pre-upgrade encrypted snapshot
cc8c5c88was restored successfully before the first image change. - Bridge snapshot
3c245001was created and restored on Lagov1.45.0immediately before the target deployment. - Post-upgrade encrypted snapshot
176fdf80was restored with Postgres, Redis, RSA/storage mounts, API, frontend, general worker, billing worker, PDF and the new clock scheduler. - The restored database contained exactly one production organization and one production API key; customer, plan, subscription and invoice tables remained empty.
- Every disposable resource and restore directory was removed after the drill.
Upgrade record: 2026-07-28
Lago production was upgraded from v1.27.1 to v1.50.0 through the required
bridge chain:
v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0
-> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0
Required migration work completed:
v1.28.1: invoice billing-entity sequential-ID job,0records left.v1.30.0: organization-ID migration, nothing left to migrate.v1.32.0: required payment job, nothing left to migrate.v1.42.0: charge/fixed-charge jobs, nothing left to migrate.v1.44.0: subscription event-date job, nothing left to migrate.v1.45.0: wallet dry-run and write pass,0wallets and0errors.v1.50.0: no native payment-provider customer or integration exists, so provider-specific payment-method backfills were not applicable.
The official api-clock service was added. It starts 26 scheduled Lago jobs,
including invoice finalization, billing, retries and subscription lifecycle
work. The production restore drill now restores and verifies the clock
container as part of the disposable topology.
Postgres remains on the patched postgres:15-alpine image (15.18 during the
upgrade). The optional Lago Partman image was not adopted because its
15.0-alpine tag would downgrade the database binaries. Lago's Partman-aware
event migrations skip partition setup when the extension is unavailable; this
production instance does not run the high-volume events processor.
Final proof:
- Private API health:
v1.50.0. - Database migrations:
0down. - Production state unchanged:
1organization,1API key,0customers, plans, subscriptions, invoices and wallets. - Queues:
0pending,0retries,0dead. - PDF service: HTTP
200. - Object-storage HEAD through Active Storage succeeded.
- Running HiveGrid production API to Lago call: authenticated HTTP
200. PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=falseremains active.- Clock starts all
26schedules and completes jobs witherror=nil. - No API, general-worker or billing-worker error was found after deployment.
Canonical billing catalog
Production contains the same controlled BILL-11 catalog as Staging:
- Version:
2026-07-29.1 - SHA-256 digest:
a8ba1affd93e1b870d016284457594988c7fc5f58fdf70c3d57be80b3c915b2c - Public packages: Core / Swarm / Colony with stable technical codes
starter/pro/business - Two monthly-billed variants per package, including the 12-month SEPA commitment
- Four fixed recurring storage packages
- German VAT tax code
de-vat-19at19 %, exclusive - No usage metering
The Production apply and follow-up verify completed on 2026-07-29. The verify returned no changed actions and the digest exactly matches Staging. Redacted pre-change snapshot:
tmp/billing-catalog-snapshots/production-2026-07-29T06-52-23-908Z.json
Operator sequence:
NODE_ENV=production pnpm billing:catalog --target=production --plan
NODE_ENV=production pnpm billing:catalog \
--target=production \
--apply \
--confirm=production:2026-07-29.1
NODE_ENV=production pnpm billing:catalog --target=production --verify
The catalog is active, but real Production payment side effects are not.
PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false remains mandatory
until all Mollie/Legal gates below are closed. Production must not receive a
test-mode Mollie key.
2026-07-29 BILL-11 closeout proof:
- The read-only Production verify completed with catalog version
2026-07-29.1, the same Staging digest and onlynoopactions. - Gitea promotion run
11477promoted the live Staging revision549f708d9809e9ba839c15366943918679a034d3without rebuilding it. - Core, Admin, Portal and API health/readiness plus MCP, Realtime and all five worker groups are healthy on that revision.
PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=falseremained active after promotion and no Mollie API key is configured in Production. No Production payment or customer invoice was created by this proof.
Access And Domain
Default: no public Lago domain.
Allowed access paths:
- Internal service-to-service URL for HiveGrid runtime.
- SSH tunnel for operators.
- Authentik-gated admin URL later, only if operator UX requires browser access.
Do not expose Lago directly to customers. Customer-facing billing screens must remain HiveGrid-owned and use the server-side Lago/Mollie gateways.
Mollie Live Gate
Mollie live activation is blocked until:
- Legal entity / firming is final.
- Business address, bank account and KYC documents are accepted by Mollie.
- Rechnungssteller, tax handling, imprint and customer contract data are approved.
- Live webhook endpoint and return URLs are set for the production host.
- A live low-value payment and refund rehearsal is approved by an operator.
- Mollie has explicitly raised the SEPA Direct Debit transaction limit above
the highest gross monthly invoice. The default documented limit is
EUR 1,000; Colony is already above it before optional seats or storage.
Until then, production must stay on manual billing or disabled self-service checkout.
The first SEPA authorization uses a customer-present payment method and
sequenceType=first; it does not attempt a direct-debit payment before a
mandate exists. Later Lago invoices use sequenceType=recurring with the
stored Mollie customer and mandate. See Mollie's
recurring-payments guide
and SEPA Direct Debit limits.
The current production gate is
PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false on
worker-general. This disables the durable provider subscriber even under
required RLS. Do not replace it with a test-mode Mollie key.
Rollback
Rollback has two layers:
- HiveGrid application rollback: deploy an older immutable HiveGrid image tag through Coolify. This must not rewrite Lago state.
- Lago service rollback: roll back Lago image tags and restore the pre- upgrade Lago database/storage backup when the Lago upgrade mutated state.
Never restore Lago production from staging. Never merge staging and production Lago datasets.
Go-Live Checklist
lago-productionCoolify resource exists inHiveGrid/production.- Dedicated production DB, Redis and storage are configured.
- Production secrets are set and staging secrets are absent.
- Backup job is configured.
- Restore drill is completed and documented.
- Mollie live profile is verified.
- Rechnungssteller, tax and imprint data are approved.
- Web/API/worker health checks pass with production Lago config.
- Manual rollback path is tested from image tag and Lago backup.