Refresh HiveGrid Cloud reference snapshot
This commit is contained in:
1 parent
5e99e14772
commit
7f88026f79
5955 files changed
+1177325
-188362
No files matched your search
@@ -1,8 +1,10 @@
|
||||
# Lago Production Prep Runbook
|
||||
|
||||
This runbook defines the production target for HiveGrid billing after AD-27.
|
||||
It is a preparation document only. Do not provision or enable production billing
|
||||
from this document until the external gates in `EXT-16` are complete.
|
||||
The isolated production foundation was provisioned on 2026-07-27 after explicit
|
||||
operator approval. Provisioning does not authorize live billing: checkout and
|
||||
provider side effects remain disabled until the external gates in `EXT-16` are
|
||||
complete.
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -28,38 +30,54 @@ Out of scope for v1:
|
||||
|
||||
## Target Topology
|
||||
|
||||
| Item | Decision |
|
||||
|---|---|
|
||||
| Coolify project | `HiveGrid` |
|
||||
| Coolify environment | `production` |
|
||||
| Resource name | `lago-production` |
|
||||
| Initial host | The current HiveGrid production host, while Hetzner capacity is constrained |
|
||||
| Long-term host | Dedicated production host or isolated capacity once the Hetzner limit is lifted |
|
||||
| Public access | None by default |
|
||||
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
||||
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
||||
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
||||
| Item | Decision |
|
||||
| ---------------------------- | ----------------------------------------------------------------------------------- |
|
||||
| Coolify project | `HiveGrid` |
|
||||
| Coolify environment | `production` |
|
||||
| Resource name | `lago-production` |
|
||||
| Current host | `hivegrid-prod-worker`; API bound only to the private `10.0.0.0/16` network |
|
||||
| Long-term host | Dedicated billing host if measured capacity or blast-radius requirements justify it |
|
||||
| Public access | None by default |
|
||||
| Operator access | Internal network or SSH tunnel; optional Authentik-gated admin access later |
|
||||
| HiveGrid app access | Server-to-server from web/API/worker to Lago API |
|
||||
| Production payment processor | Mollie live profile, only after KYC and legal/tax approval |
|
||||
|
||||
The initial host decision is intentionally pragmatic: production gets its own
|
||||
Lago resource and data stores, but it may share the physical server with other
|
||||
HiveGrid production services until infrastructure capacity allows a stricter
|
||||
split. The isolation boundary is the Coolify resource, secrets, database,
|
||||
Redis, storage and backup path, not the current machine count.
|
||||
The current placement is intentionally pragmatic: production has its own Lago
|
||||
Coolify resource, Postgres, Redis, versioned object bucket, secrets and backup
|
||||
path, while the containers share the existing worker VM. The host had 16 GB RAM
|
||||
and approximately 145 GB free disk at provisioning time. No public Lago DNS
|
||||
record or public port exists.
|
||||
|
||||
Pinned runtime images:
|
||||
|
||||
- `getlago/api:v1.50.0`
|
||||
- `getlago/front:v1.50.0`
|
||||
- `postgres:15-alpine`
|
||||
- `redis:7-alpine`
|
||||
- `getlago/lago-gotenberg:8.15`
|
||||
|
||||
API, migration runner, general worker, billing worker and the periodic
|
||||
`api-clock` scheduler use the same pinned Lago API image.
|
||||
|
||||
Invoice/file storage uses the private, versioned Hetzner Object Storage bucket
|
||||
`hivegrid-prod-lago` in `fsn1`. It is separate from the HiveGrid application
|
||||
bucket and from `hivegrid-staging-lago`.
|
||||
|
||||
## Required Production Secrets
|
||||
|
||||
Store these only in the runtime secret store. Never put values in docs, Git,
|
||||
tickets, screenshots or copied logs.
|
||||
|
||||
| Secret | Owner | Notes |
|
||||
|---|---|---|
|
||||
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
||||
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
||||
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
||||
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
||||
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
||||
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| Secret | Owner | Notes |
|
||||
| --------------------- | ----------------------- | ----------------------------------------------------------- |
|
||||
| `LAGO_API_URL` | HiveGrid runtime | Internal production Lago API URL; do not include `/api/v1`. |
|
||||
| `LAGO_API_KEY` | HiveGrid runtime | Production Lago API bearer token. |
|
||||
| `LAGO_WEBHOOK_SECRET` | HiveGrid runtime / Lago | Dedicated production webhook secret. |
|
||||
| `LAGO_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `MOLLIE_API_KEY` | HiveGrid runtime | Mollie live API key after KYC approval. |
|
||||
| `MOLLIE_PROFILE_ID` | HiveGrid runtime | Production website profile when required. |
|
||||
| `MOLLIE_TIMEOUT_MS` | HiveGrid runtime | Optional; default stays `10000`. |
|
||||
| `BILLING_MOLLIE_WEBHOOK_URL` | HiveGrid `worker-general` | `https://api.hivegrid.de/v1/billing/mollie/webhooks`; required only once provider side effects are enabled. |
|
||||
|
||||
Do not reuse staging/testmode keys in production. Rotate keys when moving from
|
||||
pilot validation to real customer billing.
|
||||
@@ -84,12 +102,23 @@ Minimum backup schedule:
|
||||
- Backup before destructive billing test data cleanup.
|
||||
- Retention aligned with the final accounting and tax retention decision.
|
||||
|
||||
Implemented production schedule:
|
||||
|
||||
- `hivegrid-lago-backup.timer` runs daily at 03:00 UTC with up to 15 minutes
|
||||
randomized delay.
|
||||
- `workflows/hivegrid-lago-backup` backs up a compressed Postgres dump, Redis
|
||||
persistence, RSA keys and local Lago storage into the encrypted HiveGrid
|
||||
restic repository in `hivegrid-prod-backups`.
|
||||
- Retention is 14 daily, 8 weekly and 12 monthly snapshots.
|
||||
- The versioned `hivegrid-prod-lago` bucket remains the primary file store.
|
||||
|
||||
Restore drill before go-live:
|
||||
|
||||
1. Create a fresh disposable restore target.
|
||||
2. Restore the latest Lago Postgres dump.
|
||||
3. Restore or attach invoice PDF storage.
|
||||
4. Start Lago API, frontend, worker, billing worker, PDF service and Redis.
|
||||
4. Start Lago API, frontend, worker, billing worker, clock, PDF service and
|
||||
Redis.
|
||||
5. Verify `/health`.
|
||||
6. Read one known customer, subscription and invoice.
|
||||
7. Verify HiveGrid can resolve the corresponding local billing state without
|
||||
@@ -98,6 +127,133 @@ Restore drill before go-live:
|
||||
The restore drill date, operator and result must be recorded before production
|
||||
billing is enabled.
|
||||
|
||||
### Restore proof
|
||||
|
||||
2026-07-27, Raffael:
|
||||
|
||||
- Versionierter Infra-Proof: Commit `936a32a`.
|
||||
- Encrypted snapshot `3d8a1deb` completed after the production organization and
|
||||
API key existed.
|
||||
- The snapshot was restored into disposable Postgres and Redis containers.
|
||||
- Checksum, schema history, exactly one organization and exactly one API key
|
||||
were verified.
|
||||
- Disposable Lago API, frontend, worker, billing worker and PDF service started;
|
||||
`/health` passed.
|
||||
- A canary object was written to, read from and fully removed from the versioned
|
||||
production bucket with a matching SHA-256.
|
||||
- The disposable containers, volume, network and restore directory were removed.
|
||||
|
||||
2026-07-28, Raffael:
|
||||
|
||||
- Pre-upgrade encrypted snapshot `cc8c5c88` was restored successfully before
|
||||
the first image change.
|
||||
- Bridge snapshot `3c245001` was created and restored on Lago `v1.45.0`
|
||||
immediately before the target deployment.
|
||||
- Post-upgrade encrypted snapshot `176fdf80` was restored with Postgres, Redis,
|
||||
RSA/storage mounts, API, frontend, general worker, billing worker, PDF and
|
||||
the new clock scheduler.
|
||||
- The restored database contained exactly one production organization and one
|
||||
production API key; customer, plan, subscription and invoice tables remained
|
||||
empty.
|
||||
- Every disposable resource and restore directory was removed after the drill.
|
||||
|
||||
## Upgrade record: 2026-07-28
|
||||
|
||||
Lago production was upgraded from `v1.27.1` to `v1.50.0` through the required
|
||||
bridge chain:
|
||||
|
||||
```text
|
||||
v1.27.1 -> v1.28.1 -> v1.30.0 -> v1.31.0 -> v1.32.0
|
||||
-> v1.42.0 -> v1.43.0 -> v1.44.0 -> v1.45.0 -> v1.50.0
|
||||
```
|
||||
|
||||
Required migration work completed:
|
||||
|
||||
- `v1.28.1`: invoice billing-entity sequential-ID job, `0` records left.
|
||||
- `v1.30.0`: organization-ID migration, nothing left to migrate.
|
||||
- `v1.32.0`: required payment job, nothing left to migrate.
|
||||
- `v1.42.0`: charge/fixed-charge jobs, nothing left to migrate.
|
||||
- `v1.44.0`: subscription event-date job, nothing left to migrate.
|
||||
- `v1.45.0`: wallet dry-run and write pass, `0` wallets and `0` errors.
|
||||
- `v1.50.0`: no native payment-provider customer or integration exists, so
|
||||
provider-specific payment-method backfills were not applicable.
|
||||
|
||||
The official `api-clock` service was added. It starts `26` scheduled Lago jobs,
|
||||
including invoice finalization, billing, retries and subscription lifecycle
|
||||
work. The production restore drill now restores and verifies the clock
|
||||
container as part of the disposable topology.
|
||||
|
||||
Postgres remains on the patched `postgres:15-alpine` image (`15.18` during the
|
||||
upgrade). The optional Lago Partman image was not adopted because its
|
||||
`15.0-alpine` tag would downgrade the database binaries. Lago's Partman-aware
|
||||
event migrations skip partition setup when the extension is unavailable; this
|
||||
production instance does not run the high-volume events processor.
|
||||
|
||||
Final proof:
|
||||
|
||||
- Private API health: `v1.50.0`.
|
||||
- Database migrations: `0` down.
|
||||
- Production state unchanged: `1` organization, `1` API key, `0` customers,
|
||||
plans, subscriptions, invoices and wallets.
|
||||
- Queues: `0` pending, `0` retries, `0` dead.
|
||||
- PDF service: HTTP `200`.
|
||||
- Object-storage HEAD through Active Storage succeeded.
|
||||
- Running HiveGrid production API to Lago call: authenticated HTTP `200`.
|
||||
- `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains active.
|
||||
- Clock starts all `26` schedules and completes jobs with `error=nil`.
|
||||
- No API, general-worker or billing-worker error was found after deployment.
|
||||
|
||||
## Canonical billing catalog
|
||||
|
||||
Production contains the same controlled `BILL-11` catalog as Staging:
|
||||
|
||||
- Version: `2026-07-29.1`
|
||||
- SHA-256 digest:
|
||||
`a8ba1affd93e1b870d016284457594988c7fc5f58fdf70c3d57be80b3c915b2c`
|
||||
- Public packages: Core / Swarm / Colony with stable technical codes
|
||||
`starter` / `pro` / `business`
|
||||
- Two monthly-billed variants per package, including the 12-month SEPA
|
||||
commitment
|
||||
- Four fixed recurring storage packages
|
||||
- German VAT tax code `de-vat-19` at `19 %`, exclusive
|
||||
- No usage metering
|
||||
|
||||
The Production apply and follow-up verify completed on 2026-07-29. The verify
|
||||
returned no changed actions and the digest exactly matches Staging. Redacted
|
||||
pre-change snapshot:
|
||||
|
||||
```text
|
||||
tmp/billing-catalog-snapshots/production-2026-07-29T06-52-23-908Z.json
|
||||
```
|
||||
|
||||
Operator sequence:
|
||||
|
||||
```bash
|
||||
NODE_ENV=production pnpm billing:catalog --target=production --plan
|
||||
NODE_ENV=production pnpm billing:catalog \
|
||||
--target=production \
|
||||
--apply \
|
||||
--confirm=production:2026-07-29.1
|
||||
NODE_ENV=production pnpm billing:catalog --target=production --verify
|
||||
```
|
||||
|
||||
The catalog is active, but real Production payment side effects are not.
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remains mandatory
|
||||
until all Mollie/Legal gates below are closed. Production must not receive a
|
||||
test-mode Mollie key.
|
||||
|
||||
2026-07-29 `BILL-11` closeout proof:
|
||||
|
||||
- The read-only Production verify completed with catalog version
|
||||
`2026-07-29.1`, the same Staging digest and only `noop` actions.
|
||||
- Gitea promotion run `11477` promoted the live Staging revision
|
||||
`549f708d9809e9ba839c15366943918679a034d3` without rebuilding it.
|
||||
- Core, Admin, Portal and API health/readiness plus MCP, Realtime and all five
|
||||
worker groups are healthy on that revision.
|
||||
- `PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` remained active
|
||||
after promotion and no Mollie API key is configured in Production. No
|
||||
Production payment or customer invoice was created by this proof.
|
||||
|
||||
## Access And Domain
|
||||
|
||||
Default: no public Lago domain.
|
||||
@@ -121,10 +277,25 @@ Mollie live activation is blocked until:
|
||||
approved.
|
||||
- Live webhook endpoint and return URLs are set for the production host.
|
||||
- A live low-value payment and refund rehearsal is approved by an operator.
|
||||
- Mollie has explicitly raised the SEPA Direct Debit transaction limit above
|
||||
the highest gross monthly invoice. The default documented limit is
|
||||
`EUR 1,000`; Colony is already above it before optional seats or storage.
|
||||
|
||||
Until then, production must stay on manual billing or disabled self-service
|
||||
checkout.
|
||||
|
||||
The first SEPA authorization uses a customer-present payment method and
|
||||
`sequenceType=first`; it does not attempt a direct-debit payment before a
|
||||
mandate exists. Later Lago invoices use `sequenceType=recurring` with the
|
||||
stored Mollie customer and mandate. See Mollie's
|
||||
[recurring-payments guide](https://docs.mollie.com/gu/docs/recurring-payments)
|
||||
and [SEPA Direct Debit limits](https://docs.mollie.com/docs/sepa-direct-debit).
|
||||
|
||||
The current production gate is
|
||||
`PRODUCT_ACQUISITION_PROVIDER_SIDE_EFFECT_REQUIRED=false` on
|
||||
`worker-general`. This disables the durable provider subscriber even under
|
||||
required RLS. Do not replace it with a test-mode Mollie key.
|
||||
|
||||
## Rollback
|
||||
|
||||
Rollback has two layers:
|
||||
@@ -139,12 +310,12 @@ Lago datasets.
|
||||
|
||||
## Go-Live Checklist
|
||||
|
||||
- [ ] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
||||
- [ ] Dedicated production DB, Redis and storage are configured.
|
||||
- [ ] Production secrets are set and staging secrets are absent.
|
||||
- [ ] Backup job is configured.
|
||||
- [ ] Restore drill is completed and documented.
|
||||
- [x] `lago-production` Coolify resource exists in `HiveGrid` / `production`.
|
||||
- [x] Dedicated production DB, Redis and storage are configured.
|
||||
- [x] Production secrets are set and staging secrets are absent.
|
||||
- [x] Backup job is configured.
|
||||
- [x] Restore drill is completed and documented.
|
||||
- [ ] Mollie live profile is verified.
|
||||
- [ ] Rechnungssteller, tax and imprint data are approved.
|
||||
- [ ] Web/API/worker health checks pass with production Lago config.
|
||||
- [x] Web/API/worker health checks pass with production Lago config.
|
||||
- [ ] Manual rollback path is tested from image tag and Lago backup.
|
||||
Reference in new issue
Block a user