151 lines
5.8 KiB
Markdown
151 lines
5.8 KiB
Markdown
# Lago Production Prep Runbook
|
|
|
|
This runbook defines the production target for HiveGrid billing after AD-27.
|
|
It is a preparation document only. Do not provision or enable production billing
|
|
from this document until the external gates in `EXT-16` are complete.
|
|
|
|
## Scope
|
|
|
|
`lago-production` will be the dedicated production Lago service for HiveGrid.
|
|
It must be separate from `lago-staging` and must never reuse staging database,
|
|
Redis, storage, API keys, webhook secrets, customers, plans or invoice records.
|
|
|
|
Production billing owns:
|
|
|
|
- Lago customers, plans, seats, subscriptions and invoice records.
|
|
- Invoice PDF generation and storage.
|
|
- Billing status used by HiveGrid entitlement and read-only gates.
|
|
- Mollie payment references for paid, failed, expired, cancelled and refunded
|
|
payments.
|
|
|
|
Out of scope for v1:
|
|
|
|
- Usage billing.
|
|
- Credits.
|
|
- Add-on metering.
|
|
- Public price promises in code.
|
|
- Live activation before legal, tax and Mollie KYC approval.
|
|
|
|
## Target Topology
|
|
|
|
| Item | Decision |
|
|
|---|---|
|
|
| Coolify project | `HiveGrid` |
|
|
| Coolify environment | `production` |
|
|
| Resource name | `lago-production` |
|
|
| Initial host | The current HiveGrid production host, while Hetzner capacity is constrained |
|
|
| Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted |
|
|
| Public access | None by default |
|
|
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
|
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
|
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
|
|
|
The initial host decision is intentionally pragmatic: production gets its own
|
|
Lago resource and data stores, but it may share the physical server with other
|
|
HiveGrid production services until infrastructure capacity allows a stricter
|
|
split. The isolation boundary is the Coolify resource, secrets, database,
|
|
Redis, storage and backup path, not the current machine count.
|
|
|
|
## Required Production Secrets
|
|
|
|
Store these only in the runtime secret store. Never put values in docs, Git,
|
|
tickets, screenshots or copied logs.
|
|
|
|
| Secret | Owner | Notes |
|
|
|---|---|---|
|
|
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
|
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
|
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
|
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
|
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
|
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
|
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
|
|
|
Do not reuse staging/testmode keys in production. Rotate keys when moving from
|
|
pilot validation to real customer billing.
|
|
|
|
## Backup And Restore
|
|
|
|
Production Lago is stateful. It must not receive production traffic before a
|
|
backup and restore path has been verified.
|
|
|
|
Minimum backup scope:
|
|
|
|
- Lago Postgres database.
|
|
- Lago Redis persistence if configured for durable state.
|
|
- Invoice PDF storage volume or bucket.
|
|
- Coolify resource settings and image tags.
|
|
- Runtime secret names and rotation history, without secret values.
|
|
|
|
Minimum backup schedule:
|
|
|
|
- Daily encrypted database dump.
|
|
- Backup before every Lago image upgrade.
|
|
- Backup before destructive billing test data cleanup.
|
|
- Retention aligned with the final accounting and tax retention decision.
|
|
|
|
Restore drill before go-live:
|
|
|
|
1. Create a fresh disposable restore target.
|
|
2. Restore the latest Lago Postgres dump.
|
|
3. Restore or attach invoice PDF storage.
|
|
4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
|
|
5. Verify `/health`.
|
|
6. Read one known customer, subscription and invoice.
|
|
7. Verify HiveGrid can resolve the corresponding local billing state without
|
|
writing test payments.
|
|
|
|
The restore drill date, operator and result must be recorded before production
|
|
billing is enabled.
|
|
|
|
## Access And Domain
|
|
|
|
Default: no public Lago domain.
|
|
|
|
Allowed access paths:
|
|
|
|
- Internal service-to-service URL for HiveGrid runtime.
|
|
- SSH tunnel for operators.
|
|
- Authentik-gated admin URL later, only if operator UX requires browser access.
|
|
|
|
Do not expose Lago directly to customers. Customer-facing billing screens must
|
|
remain HiveGrid-owned and use the server-side Lago/Mollie gateways.
|
|
|
|
## Mollie Live Gate
|
|
|
|
Mollie live activation is blocked until:
|
|
|
|
- Legal entity / firming is final.
|
|
- Business address, bank account and KYC documents are accepted by Mollie.
|
|
- Rechnungssteller, tax handling, imprint and customer contract data are
|
|
approved.
|
|
- Live webhook endpoint and return URLs are set for the production host.
|
|
- A live low-value payment and refund rehearsal is approved by an operator.
|
|
|
|
Until then, production must stay on manual billing or disabled self-service
|
|
checkout.
|
|
|
|
## Rollback
|
|
|
|
Rollback has two layers:
|
|
|
|
1. **HiveGrid application rollback:** deploy an older immutable HiveGrid image
|
|
tag through Coolify. This must not rewrite Lago state.
|
|
2. **Lago service rollback:** roll back Lago image tags and restore the pre-
|
|
upgrade Lago database/storage backup when the Lago upgrade mutated state.
|
|
|
|
Never restore Lago production from staging. Never merge staging and production
|
|
Lago datasets.
|
|
|
|
## Go-Live Checklist
|
|
|
|
- [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
|
- [ ] Dedicated production DB, Redis and storage are configured.
|
|
- [ ] Production secrets are set and staging secrets are absent.
|
|
- [ ] Backup job is configured.
|
|
- [ ] Restore drill is completed and documented.
|
|
- [ ] Mollie live profile is verified.
|
|
- [ ] Rechnungssteller, tax and imprint data are approved.
|
|
- [ ] Web/API/worker health checks pass with production Lago config.
|
|
- [ ] Manual rollback path is tested from image tag and Lago backup.
|