toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track HiveGrid Cloud planning reference

Browse Source
This commit is contained in:
david committed 2026-06-26 20:48:01 +02:00
1 parent b3d5ea6764
commit e341c5938a
5784 files changed
+1104929 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

hivegrid-cloud/docs/ops/asset-cdn.md
+257
View File
@@ -0,0 +1,257 @@
# Asset CDN Operations Runbook
When A7/B2/#51 is reactivated, front the boilerplate's static assets
(`/assets/*`, `/fonts/*`, `/images/*`) with a CDN so the origin no longer pays
the bandwidth + TLS handshake cost for every visitor and so users far from the
Frankfurt DC see sub-second first paints.
**Default provider:** Bunny.net (EU-based, cheap, simple).
**Documented alternative:** Cloudflare.
Related task: [`docs/improvement-plan/03-scaling-infrastructure/asset-cdn.md`](../improvement-plan/03-scaling-infrastructure/asset-cdn.md).
**Rollout status:** CDN provisioning and `ASSET_CDN_URL` flips remain parked
with A7/B2/#51. The deploy workflow already includes a no-op-safe Bunny purge
step for future activation; keep this runbook as the reactivation guide until
`docs/improvement-plan/OPERATOR-CHECKLIST.md` and
`docs/improvement-plan/PROGRESS.md` move #51 out of backlog.
## Architecture
```
Browser
│
├─→ cdn.{domain}/assets/app-3f4a1b.js ──→ Bunny edge (cache hit: ~1 ms)
│ │
│ └─ origin pull on miss:
│ app.{domain}/assets/app-3f4a1b.js
│
└─→ app.{domain}/ (HTML, API, /healthz) ──→ origin server (no CDN detour)
```
Only static directories travel through the CDN. HTML, API responses, and
`/healthz`/`/readyz` go direct to origin so authenticated content and
liveness probes are never edge-cached.
## Cache strategy
| Path pattern | `Cache-Control` | Why |
| ----------------------------- | ------------------------------------------ | --------------------------------------------------------------------- |
| `/assets/*` (Vite-hashed) | `public, max-age=31536000, immutable` | Filenames embed a content hash — safe to cache forever. |
| `/fonts/*`, `/images/*` | `public, max-age=86400` | Static brand assets without hash; 1-day TTL is the longest safe value.|
| HTML documents (loader/SSR) | `no-store` (unless loader overrides) | May contain CSRF tokens, session-derived markup, or feature flags. |
| `/api/*`, `/healthz`, `/readyz` | bypass cache (edge rule) | Authenticated and/or liveness-critical — caching corrupts behaviour. |
The headers above are set by `server/server.ts` (`getStaticHeaders`) for the
static directories and by `apps/web/app/entry.server.tsx` for the SSR HTML
response. Loaders may set their own `Cache-Control` to opt out of `no-store`
(e.g. marketing landing pages choosing `public, max-age=300`).
## Provider comparison
| Concern | Bunny.net (default) | Cloudflare (alternative) |
| --------------------- | --------------------------------------- | ------------------------------------------ |
| HQ | Slovenia (EU) | United States |
| Pricing baseline | ~€1-5/month for small SaaS | Free tier sufficient for small sites |
| DSGVO posture | EU jurisdiction, simple DPA | Requires Schrems-II analysis for EU users |
| DNS migration | Not required — single CNAME suffices | Free tier requires full DNS migration |
| Image optimisation | Bunny Optimizer add-on (~$9.50/mo) | Cloudflare Images ($5/mo + per-image fee) |
| Cache purge API | Simple POST with `AccessKey` header | Token-scoped API; per-zone purge endpoints |
| Origin-shield option | Yes (single optional PoP in front of origin) | Yes (Argo Tunnel / Origin Rules) |
**Choose Bunny.net unless** the operator already runs all DNS through
Cloudflare and wants a single console — then Cloudflare is the lower-friction
choice.
## Bunny.net setup (default, when A7/B2/#51 is reactivated)
Do not execute these steps while A7/B2/#51 remains parked. They are the
reactivation checklist for the future CDN rollout. None of this is automated by
the deploy pipeline.
1. **Create the Bunny account.**
Sign up at <https://bunny.net>, add a payment method, generate an
*account-level API key* in *Account → API*. Store it as the `BUNNY_API_KEY`
secret in the Gitea / Coolify environment.
2. **Create a pull zone per environment.**
- Staging pull zone:
- Name: `cdn-staging-{project}`
- Origin URL: `https://staging.{domain}` (the app origin)
- Pricing tier: *Standard Network* (high volume HDD/Volume can be selected later)
- Production pull zone:
- Name: `cdn-{project}`
- Origin URL: `https://app.{domain}`
Note the numeric **pull-zone ID** from the URL — store it as
`BUNNY_PULL_ZONE_ID` (one per environment).
3. **Add a custom hostname.**
In the pull zone, *Hostnames → Add Custom Hostname*:
- Staging: `cdn-staging.{domain}`
- Production: `cdn.{domain}`
Create the matching DNS records at your DNS provider:
```
cdn-staging.{domain} CNAME cdn-staging-{project}.b-cdn.net
cdn.{domain} CNAME cdn-{project}.b-cdn.net
```
Wait until the records resolve (`dig cdn-staging.{domain}`), then enable
*Force SSL* — Bunny provisions a Let's Encrypt certificate automatically
(~5 minutes).
4. **Configure edge rules.**
- **Bypass cache for `/api/*`** (matching: URL path starts with `/api/`).
- **Bypass cache for `/healthz`, `/readyz`** (URL exact match).
- **Force cache for `/assets/*`** (URL path starts with `/assets/`,
`Override Cache Time = 1 year`) — defensive, since the origin already
emits `immutable`, but protects against future regressions.
5. **Enable origin shield (recommended).**
*Configuration → Origin Shield → Enabled* with the closest PoP to the
origin DC (Frankfurt for Hetzner FRA). Reduces origin load when the edge
cold-misses.
6. **Set the runtime env vars in Coolify.**
| Env var | Where to set | Example |
| --------------------- | ------------------------------------- | ------------------------------------ |
| `ASSET_CDN_URL` | Web app (build-time + runtime) | `https://cdn-staging.{domain}` |
| `BUNNY_PULL_ZONE_ID` | Gitea secret + Coolify env (deploy) | `<pull-zone-id>` |
| `BUNNY_API_KEY` | Gitea secret + Coolify env (deploy) | `<bunny-api-key>` |
`ASSET_CDN_URL` must be present at **build time** — Vite bakes the CDN
origin into asset URLs via `apps/web/vite.config.ts` (`resolveAssetBase`).
7. **Trigger a deploy** and run the smoke tests below.
## Smoke tests (when A7/B2/#51 is reactivated)
Do not run these as active acceptance checks while CDN rollout remains parked.
```bash
# 1. Asset is served via Bunny
curl -sI https://cdn-staging.{domain}/assets/<hash>.js | head -10
# Expect: HTTP/2 200
# server: BunnyCDN-DE1 (or whichever PoP)
# cache-control: public, max-age=31536000, immutable
# x-cache: HIT (after the first request warms it)
# 2. HTML is NOT cached by Bunny
curl -sI https://staging.{domain}/ | head -10
# Expect: cache-control: no-store
# no `server: BunnyCDN` header
# 3. /fonts and /images get the 1-day TTL
curl -sI https://cdn-staging.{domain}/fonts/inter.woff2 | grep cache-control
# Expect: cache-control: public, max-age=86400
# 4. Purge API is reachable
curl -sX POST "https://api.bunny.net/pullzone/${BUNNY_PULL_ZONE_ID}/purgeCache" \
-H "AccessKey: ${BUNNY_API_KEY}" -o /dev/null -w "%{http_code}\n"
# Expect: 204
```
## Cache-purge script
`scripts/cdn/bunny-purge.ts` wraps the Bunny purge API and is wired into
the `Deploy` workflow as the **Purge asset CDN** step (post-deploy). It
runs unconditionally — if `BUNNY_API_KEY` or `BUNNY_PULL_ZONE_ID` is unset
it logs and exits 0, so the step is safe before the operator finishes
the setup above.
```bash
# Manual whole-zone purge
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
pnpm exec tsx scripts/cdn/bunny-purge.ts
# Targeted purge (single asset)
BUNNY_PULL_ZONE_ID=<pull-zone-id> BUNNY_API_KEY=<bunny-api-key> \
ASSET_CDN_URL=https://cdn.{domain} \
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/app-3f4a1b.js
# Multiple paths (comma-separated or repeated --paths)
pnpm exec tsx scripts/cdn/bunny-purge.ts --paths /assets/a.js,/assets/b.js
# Dry-run (prints the plan, makes no API call)
pnpm exec tsx scripts/cdn/bunny-purge.ts --dry-run
```
Exit codes:
| Code | Meaning |
| ---- | -------------------------------------------------------------------- |
| `0` | Purge succeeded, or skipped because credentials were intentionally absent. |
| `1` | Bunny returned a non-2xx response, or invalid CLI arguments. |
## Cloudflare setup (alternative)
Only follow this section if you decided against Bunny.net in the table above.
1. Add the apex/subdomain to Cloudflare (free plan suffices for small sites).
Either migrate DNS fully or proxy only the `cdn.{domain}` CNAME.
2. Create a *Page Rule* for `cdn.{domain}/assets/*` with:
*Cache Level → Cache Everything*, *Edge Cache TTL → 1 month*.
3. Create a *Page Rule* for `cdn.{domain}/api/*` with:
*Cache Level → Bypass*.
4. Generate an API token (`Zone → Cache Purge`) — store as `CF_API_TOKEN`.
You'll need a sibling purge script (out of scope for this runbook;
the Bunny script is the only one wired into the workflow today).
5. Set `ASSET_CDN_URL=https://cdn.{domain}` in Coolify.
Cloudflare's default cache TTLs and "Smart Routing" can override the
`Cache-Control` headers the origin sends. Pin explicit *Cache Rules* on
the zone if behaviour diverges from the table above.
## Image optimisation (opt-in, Phase 2)
We deliberately do NOT enable image optimisation on day one — it adds cost
and operational surface area without a measured benefit on a small footprint.
**Enable Bunny Optimizer when any of these trigger:**
- Outbound image bandwidth from the CDN exceeds ~10 GB/month, OR
- User uploads (avatars, tenant logos, post images) cross ~1 000/month, OR
- Lighthouse's *Largest Contentful Paint* sits above 2.5 s and the audit
attributes the largest portion to images.
Steps to enable:
1. In the Bunny pull zone, *Optimizer → Enable* (~$9.50/month).
2. Tag image-heavy URLs with the size hint, e.g.
`https://cdn.{domain}/uploads/avatar.png?width=200&format=webp`.
3. Adjust the web app's image components to emit the `?width=&format=`
query string and a `<picture>` block with WebP/AVIF fallbacks.
4. Re-run Lighthouse against the staging URL and confirm LCP improvement.
## Rollback
| Failure mode | Action |
| ----------------------------------------------------- | ---------------------------------------------------------------------------- |
| CDN edge serves stale/corrupt assets | Run `pnpm exec tsx scripts/cdn/bunny-purge.ts` (whole-zone purge). |
| CDN provider outage / TLS failure | Unset `ASSET_CDN_URL` in Coolify and redeploy — Vite emits origin-relative paths next build. |
| Misconfigured `Cache-Control` shipped (e.g. `immutable` on `/fonts/`) | Push a fix, deploy, then whole-zone purge. |
| Need to fully cut over to the alternative provider | Provision the new pull zone, switch `ASSET_CDN_URL` + `BUNNY_*` secrets, redeploy. Old `cdn.{domain}` CNAME can stay until the new one stabilises. |
No dedicated feature flag is needed — `ASSET_CDN_URL` *is* the flag. Empty
or unset → no CDN detour, anywhere.
## Security notes
- **CDN-as-MITM:** Bunny / Cloudflare see every asset request including
Referrer headers. Static brand assets are fine; never route
`/uploads/<user-content>` through a public-cache zone — those paths
inherit the `no-store` default and stay on the origin.
- **CORS:** When fonts/images load via `fetch` (e.g. preload-on-demand),
the CDN must forward `Access-Control-Allow-Origin: https://app.{domain}`
from the origin. Bunny forwards CORS headers transparently; verify in
the smoke tests if you add cross-origin font loading.
- **CSP:** The Content-Security-Policy in `server/server.ts` keeps a
`base-uri 'self'; frame-ancestors 'self'` policy. When you switch on
the CDN, extend `script-src`, `style-src`, `font-src`, and `img-src`
to include the CDN hostname (separately tracked).
- **Origin shield + IP allowlist:** Bunny's Origin Shield gives a single
outgoing IP range. Optionally whitelist those IPs at the origin
firewall so attackers cannot bypass the CDN by hitting the origin
directly.
- **Cache poisoning:** Only assets that emit `public, max-age=…` go to the
edge. Loader-rendered HTML defaults to `no-store`; loaders that need
caching must opt in explicitly and set the right `Vary` headers.
hivegrid-cloud/docs/ops/backup-cli.md
+173
View File
@@ -0,0 +1,173 @@
# Backup CLI Runbook
Operator guide for `@toolctrl/cli backup:*`. The CLI ships in `packages/cli` and
is callable as `toolctrl-cli backup <subcommand>` once the workspace is
installed.
## Quick reference
| Command | Purpose |
| ------------------ | ------------------------------------------------------------- |
| `backup:create` | Dump Postgres (+ Storage, + ClickHouse) and upload to S3. |
| `backup:list` | List backups in the bucket with classification + encryption. |
| `backup:rotate` | Apply daily/weekly/monthly retention. |
| `backup:restore` | Download, decrypt, extract; pg_restore against a target DSN. |
| `backup:verify` | Restore into a `verify_<uuid>` schema and run sanity checks. |
## 1. Bucket + IAM setup
The backup bucket is intentionally separate from the application storage
bucket so a compromised app credential cannot pivot to the backups.
```bash
# MinIO example (staging)
mc mb staging-minio/toolctrl-backups
mc anonymous set none staging-minio/toolctrl-backups
# AWS S3 example
aws s3api create-bucket --bucket toolctrl-backups \
--create-bucket-configuration LocationConstraint=us-east-1
aws s3api put-bucket-encryption --bucket toolctrl-backups \
--server-side-encryption-configuration \
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-public-access-block --bucket toolctrl-backups \
--public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
```
Create a dedicated IAM user / access key pair with `PutObject`,
`GetObject`, `ListBucket`, `DeleteObject` on the backup bucket only. Use those
values for `BACKUP_S3_ACCESS_KEY_ID` + `BACKUP_S3_SECRET_ACCESS_KEY` (do **not**
re-use the app `STORAGE_*` credentials).
For long-term retention, enable Object Lock with COMPLIANCE mode and / or MFA
delete so a rotated IAM key cannot purge history.
## 2. Encryption keys
The default encryption method is [age](https://age-encryption.org).
```bash
# Generate a key pair on a trusted workstation
age-keygen -o ~/.age-toolctrl-backup.key
# Output:
# Public key: age1example...
# Identity (private key) is at ~/.age-toolctrl-backup.key
# Set the public key in the deployment env
echo 'BACKUP_AGE_PUBLIC_KEY=age1example...' >> /etc/toolctrl/backup.env
```
**Store the identity file (private key) somewhere the running app does not have
access to** — a password manager, a tamper-evident envelope, a hardware token.
The public key alone is what the create flow needs; the private key only
appears in restore / verify environments via `BACKUP_AGE_IDENTITY_FILE`.
If the operator already has a GPG fleet, set `BACKUP_ENCRYPTION=gpg` and
`BACKUP_GPG_RECIPIENT=<key id>` instead.
> **Warning:** Lose the identity file and the backups are unrecoverable. The
> CLI cannot bypass age encryption.
## 3. Environment template
```env
# Backup bucket
BACKUP_S3_ENDPOINT=https://s3.your-provider.com
BACKUP_S3_REGION=us-east-1
BACKUP_S3_BUCKET=toolctrl-backups
BACKUP_S3_ACCESS_KEY_ID=AKIA...
BACKUP_S3_SECRET_ACCESS_KEY=...
BACKUP_S3_FORCE_PATH_STYLE=true # MinIO/R2; AWS uses host-style
# Encryption (age default)
BACKUP_ENCRYPTION=age
BACKUP_AGE_PUBLIC_KEY=age1example...
# Restore-side only:
# BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key
# Retention
BACKUP_RETENTION_DAILY=7
BACKUP_RETENTION_WEEKLY=4
BACKUP_RETENTION_MONTHLY=12
# Optional
BACKUP_INCLUDE_CLICKHOUSE=false
```
Postgres source is picked up from `DIRECT_URL` (preferred — bypasses the
PgBouncer pool so prepared statements work) or `DATABASE_URL` as a fallback.
## 4. Daily cron
```cron
# /etc/cron.d/toolctrl-backup
SHELL=/bin/sh
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=ops@example.com
0 3 * * * deploy /opt/toolctrl/scripts/cron/backup-daily.sh >> /var/log/toolctrl-backup.log 2>&1
```
Example wrapper:
```bash
#!/usr/bin/env bash
set -euo pipefail
cd /opt/toolctrl
node ./packages/cli/bin/toolctrl-cli.mjs backup create
node ./packages/cli/bin/toolctrl-cli.mjs backup rotate
```
For docker-compose deployments:
```yaml
backup-cron:
image: alpine:latest
volumes:
- ./scripts/cron/backup-daily.sh:/script.sh:ro
command: ["sh", "-c", "echo '0 3 * * * /script.sh' | crontab - && crond -f"]
depends_on: [postgres]
env_file: .env
```
Wire failures into your alerting channel (Sentry, Slack, PagerDuty) — an
untested backup is no backup, but an unalerted failed backup is the same
risk.
## 5. Disaster recovery drill
Run **at least monthly**:
```bash
# 1. Pick yesterday's backup
toolctrl-cli backup list
# 2. Verify in-place (creates verify_<uuid>, drops it after)
toolctrl-cli backup verify backup-2026-05-17T030000Z-aabbccdd.tar.gz.age
# 3. End-to-end DR drill (restore into a scratch host)
DATABASE_URL=postgres://dr-test/restored \
BACKUP_AGE_IDENTITY_FILE=/secrets/age-backup.key \
toolctrl-cli backup restore backup-2026-05-17T030000Z-aabbccdd.tar.gz.age --yes
```
Document the drill outcome in the incident channel or your internal ops ledger.
If the drill becomes a repository-maintained runbook, add the Markdown file
first and link it from this section.
## 6. Troubleshooting
| Symptom | Likely cause | Fix |
| ------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `pg_dump exited with code 1` | `DIRECT_URL` missing or pgbouncer in the connection. | Set `DIRECT_URL` to the port-5432 DSN, not the pooler. |
| `BACKUP_AGE_PUBLIC_KEY is required` | `BACKUP_ENCRYPTION=age` but no public key in env. | Generate via `age-keygen` and set the value. |
| `age: error: failed to read header: parsing recipient` | Identity file does not match the public key. | Decrypt with the matching identity; never delete the key store. |
| `clickhouse-client: command not found` | `BACKUP_INCLUDE_CLICKHOUSE=true` but no client on PATH. | Either install the client or set the flag to `false`. |
| `backup:verify` reports "no tables present" | Restore ran against an empty DSN. | Confirm `DATABASE_URL` points to a non-empty DB and that pg_restore succeeded above. |
## 7. Sizing
Backup size grows ≈ `DB dump size` + `Σ(storage object sizes)` (no
compression on objects, only on the dump). Track via Prometheus
`s3_object_size_bytes` and alert when daily delta is below historic minimum
(a silent-fail symptom).
hivegrid-cloud/docs/ops/pgbouncer.md
+186
View File
@@ -0,0 +1,186 @@
# PgBouncer Operations Runbook
PgBouncer multiplexes Prisma's per-process connection pools onto a small set
of real Postgres backends. The dev compose stack and the staging Coolify
stack both terminate application traffic at `pgbouncer:6432`; Prisma Migrate
keeps a separate path through `postgres:5432` (`DIRECT_URL`).
Related task: [`docs/improvement-plan/03-scaling-infrastructure/pgbouncer.md`](../improvement-plan/03-scaling-infrastructure/pgbouncer.md).
## Topology
```
+--------------+ +-------------+ +----------+
| apps (web, | 6432 | pgbouncer | 5432 | postgres |
| api, worker) +--------->+ transaction +-------->+ 16-alp. |
| Prisma client| | pool, 25/db | | |
+------+-------+ +-------------+ +-----^----+
| |
| DIRECT_URL (5432) — only for prisma migrate ---+
v
prisma migrate / db push / db pull
```
- `DATABASE_URL = postgres://...@pgbouncer:6432/<db>?pgbouncer=true&connection_limit=10`
- `pgbouncer=true` tells Prisma to skip prepared-statement caching.
- `connection_limit=10` caps Prisma's internal pool — PgBouncer pools again
on top, doubling the budget without this would defeat the purpose.
- `DIRECT_URL = postgres://...@postgres:5432/<db>` — read by
`prisma.config.ts` for `migrate`, `db push`, `db pull`, `db seed`.
## Local development
```bash
# Boot Postgres + PgBouncer side by side. The compose service is intentionally
# Docker-network-only — no host port mapping for 6432.
docker compose up postgres pgbouncer
# In another shell:
DIRECT_URL="postgres://user:pass@localhost:5432/boilerplate" \
DATABASE_URL="postgres://user:pass@localhost:6432/boilerplate?pgbouncer=true&connection_limit=10" \
pnpm dev
```
If you do not have a Postgres service in `docker-compose.yml` yet, point both
URLs at your existing local Postgres on port 5432 (skip PgBouncer entirely).
`prisma.config.ts` falls back to `DATABASE_URL` when `DIRECT_URL` is unset.
## Coolify (staging) deploy procedure
The pooler must run on the same Docker network as Postgres. The current
operator-owned deployment source is the Coolify service/network config; the
archived [`STAGING-ACCESS.md`](../improvement-plan/archive/STAGING-ACCESS.md)
is historical inventory only and must not override live Coolify state.
1. **Generate `userlist.txt`.** Postgres on staging is initialised with
`password_encryption=scram-sha-256`, so the SCRAM hash already lives in
`pg_authid.rolpassword`. From the Postgres container:
```bash
psql -U postgres -tAc \
"SELECT '\"'||rolname||'\" \"'||rolpassword||'\"'
FROM pg_authid
WHERE rolname IN ('boilerplate','pgb_admin');" > userlist.txt
chmod 600 userlist.txt
```
Drop the file into a Coolify volume mount that the pgbouncer container
will see at `/etc/pgbouncer/userlist.txt`. **Never commit a real hash.**
2. **Create the Coolify service.**
- Image: `edoburu/pgbouncer:latest`
- Network: `coolify` (must match Postgres)
- Mounts:
- `infra/pgbouncer/pgbouncer.ini` → `/etc/pgbouncer/pgbouncer.ini:ro`
- `userlist.txt` (from step 1) → `/etc/pgbouncer/userlist.txt:ro`
- No public port mapping — Traefik should not see 6432.
- Healthcheck:
`pg_isready -h 127.0.0.1 -p 6432 -U pgb_admin`
3. **Wait for the pooler to be `healthy`.** Verify from inside the network:
```bash
docker exec <pgbouncer-id> psql -h 127.0.0.1 -p 6432 -U pgb_admin pgbouncer -c "SHOW STATS;"
```
4. **Switch app env vars.** In Coolify, on `web`, `api`, `worker`:
- `DATABASE_URL=postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>?pgbouncer=true&connection_limit=10`
- `DIRECT_URL=postgres://<database-user>:${PG_PASS}@<postgres-service>:5432/<database>`
Use service and database names from the current Coolify project state; do
not copy archived staging inventory into a live cutover.
5. **Restart apps last.** Apps must boot only _after_ the pooler reports
healthy; otherwise the first Prisma query fails and the container loops.
## Migration runbook (production cutover)
1. Provision PgBouncer in Coolify but leave app env vars unchanged.
2. Verify `SHOW STATS` and `SHOW POOLS` respond.
3. Smoke-test from a one-off container:
```bash
docker run --rm --network coolify \
postgres:16-alpine psql \
"postgres://<database-user>:${PG_PASS}@<pgbouncer-service>:6432/<database>" \
-c "SELECT 1;"
```
4. Roll out `DIRECT_URL` to all apps first (no behaviour change because
`prisma.config.ts` falls back to `DATABASE_URL`).
5. Roll out the new `DATABASE_URL` (with `pgbouncer=true&connection_limit=10`).
Coolify will recreate containers — they must reconnect to the pooler.
6. Watch `pgbouncer` logs for `WARNING|ERROR|FATAL`.
7. Confirm `prisma migrate status` from a CI container still passes (uses
`DIRECT_URL`).
## Tuning knobs
| Knob | Default | When to change |
| --------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `pool_mode` | `transaction` | Stay on `transaction` for web traffic. Long-lived advisory locks or `LISTEN/NOTIFY` need `session` mode on a separate logical database. |
| `default_pool_size` | `25` | Rule of thumb: `postgres.max_connections / number_of_app_pools / 1.5`. Hetzner Postgres is provisioned with `max_connections=200`, so 25 per logical DB × app stack still leaves headroom. |
| `max_client_conn` | `1000` | Raises the upper bound of _client_-side connections PgBouncer will accept. Bumping past 1000 has memory cost (~12 KB/conn). |
| `reserve_pool_size` | `5` | Burst buffer when the main pool is exhausted. Increase only if `SHOW POOLS` shows recurring `cl_waiting`. |
| `server_idle_timeout` | `600` | Lower if Postgres is rapidly recycled (cloud-managed). Higher = fewer reconnects. |
| `query_timeout` | `0` | Leave at `0`. Per-query timeouts live in Prisma / API layer; PgBouncer killing a long report query is hard to debug. |
| `server_reset_query` | `DISCARD ALL` | Required for transaction pooling — prevents tenant state from leaking between transactions on the same backend. Do not weaken. |
Grafana dashboard lives under
`infra/grafana/dashboards/` (provisioned by Phase 02 `#35`).
Panels of interest: pool wait time p95, server connection count vs.
`default_pool_size`, total clients vs. `max_client_conn`.
## Troubleshooting
### `FATAL: prepared statement "s0" already exists`
PgBouncer in transaction pool mode reused a server-side connection that
still has the prepared statement cached. Confirm `DATABASE_URL` has
`?pgbouncer=true` and Prisma 5+ is in use. Without the flag, Prisma keeps
local prepared statements and the next transaction lands on a different
backend.
### `error: too many clients already`
Postgres ran out of connections. Either:
- `default_pool_size` × number-of-app-pools > `max_connections` — lower
pool size or raise `max_connections` on Postgres.
- Prisma's `connection_limit` is unset, so each app process opens its own
default pool (CPU + 1) against PgBouncer. Set `connection_limit=10` (or
lower) in the query string of `DATABASE_URL`.
### `cl_waiting > 0` persistently in `SHOW POOLS`
Application sustained pool exhaustion. Options:
- raise `default_pool_size` (and double-check `max_connections` headroom),
- introduce a read replica (see `03-scaling-infrastructure/read-write-split.md`),
- inspect slow queries — long-running transactions hold backends.
### Migrations fail with "prepared statement" or DDL errors
`DIRECT_URL` is wrong or missing. `prisma migrate` must hit Postgres
directly. Verify `prisma.config.ts` reads `DIRECT_URL`:
```bash
DATABASE_URL=postgres://...@pgbouncer:6432/db?pgbouncer=true \
DIRECT_URL=postgres://...@postgres:5432/db \
pnpm exec prisma migrate status
```
### Healthcheck flapping
`pg_isready` returns non-zero whenever PgBouncer cannot reach Postgres.
Check Postgres health first; PgBouncer surfaces an empty server pool as
unhealthy.
## Rollback
To bypass PgBouncer without code changes:
1. In Coolify, change `DATABASE_URL` to point at Postgres directly
(port 5432, drop `?pgbouncer=true&connection_limit=10`).
2. Restart `web`, `api`, `worker`.
3. Stop the `pgbouncer` Coolify service.
`DIRECT_URL` stays valid — it never depended on the pooler.
hivegrid-cloud/docs/ops/prod-rollout-hivegrid.md
+196
View File
@@ -0,0 +1,196 @@
# Prod-Rollout HiveGrid — Runbook (dedizierter Hetzner-Account)
> Umsetzung von **AD-24** (Prod unter eigenem Hetzner-Account, self-contained). Stand 2026-06-19.
> **Go-Gates:** Jeder Schritt mit 🟥 ist kostenwirksam / schwer reversibel und wird **vor** dem `create`
> einzeln mit dem Owner (Raffael) bestätigt. 🟩 = read-only / vorbereitend.
## 0. Voraussetzungen (🟩 vorhanden)
- Hetzner-API-Token (Prod): `~/.cli-tokens/hcloud-hivegrid-prod` — verifiziert (Account leer).
- S3 (Prod): `~/.cli-tokens/prod-s3-bucket-hetzner-credentials` — Auth verifiziert (Projekt p15040939, fsn1), Bucket noch nicht angelegt.
- Cloudflare (Zone `hivegrid.de`): `~/.cli-tokens/cloudflare-token` (Global Key + Email → `X-Auth-Email`/`X-Auth-Key`).
- Gitea/CI: Deploy-Pipeline `.gitea/workflows/deploy.yml` (Promote-Pfad → Prod).
- Entscheidungen: Ingress = **Cloudflare-Proxy + Coolify/Traefik** (kein Hetzner-LB). Region **fsn1**, x86 **cx**-Linie.
## 1. Netzwerk-Fundament (🟥 Provisionierung, aber günstig/reversibel)
1. **SSH-Key** im Prod-Account hinterlegen (aktuell 0) — Public-Key des Deploy-/Admin-Keys.
2. **Private Network** (Hetzner Cloud Network), z.B. `10.0.0.0/16`, Subnetz fsn1.
3. **Firewall**:
- eingehend `80,443/tcp` → control/web/api (bzw. nur über Cloudflare-IP-Ranges, wenn proxied).
- `22/tcp` SSH nur von Admin-/Bastion-IP.
- `db` + `services`: **kein** eingehender Public-Traffic, nur über das Private Network.
## 2. Server provisionieren (🟥 kostenwirksam — Go pro Batch)
Alle in **fsn1**, im Private Network, mit Firewall + SSH-Key. cx-Linie (x86):
| VM | Typ | Specs | Zweck |
|---|---|---|---|
| `hivegrid-control` | cx33 | 4/8/80 | Coolify + Orchestrierung/Bastion |
| `hivegrid-web` | cx33 | 4/8/80 | Web (SSR) |
| `hivegrid-api` | cx33 | 4/8/80 | API |
| `hivegrid-worker` | cx43 | 8/16/160 | 5 Worker-Queue-Gruppen + Meili + ClamAV + mCaptcha |
| `hivegrid-db` | cx33 | 4/8/80 | Postgres + PgBouncer + Redis |
4. **Volume** (~50 GB) an `hivegrid-db` für Postgres-Daten (nicht auf Boot-Disk).
5. (optional) Server-Backups für `hivegrid-db` aktivieren.
## 3. Coolify auf control (🟥)
6. Coolify auf `hivegrid-control` installieren (eigenständige Prod-Instanz).
7. `web/api/worker/db` als **remote Server** in Coolify einhängen (über Private Network).
8. Container-Registry-Zugang (Gitea-Registry) + Cosign-Keys in Coolify hinterlegen.
## 4. Daten- + Zentral-Dienste (🟥)
9. **db-VM:** Postgres (+ DB `hivegrid`, DB `mcaptcha`), **PgBouncer** (Transaction-Pooling, Port 6432), Redis. `DATABASE_URL` → PgBouncer, `DIRECT_URL` → 5432.
10. **worker+services-VM:** Meilisearch (Master-Key), ClamAV (clamd), mCaptcha (zeigt auf db-Postgres `mcaptcha` + Redis).
11. **DEK-Master-Key** (AD-8) generieren + sicher hinterlegen (nicht ins Repo).
12. **S3-Bucket** `hivegrid-prod` (fsn1) anlegen + Versioning; `prod-s3-bucket-hetzner-credentials` um `bucket=hivegrid-prod` ergänzen.
## 5. App-Env + Deploy (🟥)
13. Prod-Env je App setzen (analog Staging-Keys, aber **Prod-Werte**): `DATABASE_URL/DIRECT_URL`, `REDIS/BULLMQ_REDIS_URL`, `MEILI_URL/MEILI_API_KEY`, `MCAPTCHA_*`, `CLAMAV_HOST/PORT`, `SESSION/CRYPTO/JWT`-Secrets, `APP_BASE_DOMAIN`/Host-Vars (Prod-Domains), Sentry `SENTRY_ENVIRONMENT=production`. **`WEB_NOINDEX` NICHT setzen** (Prod soll indexierbar sein).
14. Migrationen: `prisma migrate deploy` als Pre-Deployment-Command (wie Staging).
15. Promote-Deploy über die Pipeline (oder Coolify) → Images ausrollen.
## 6. Ingress / DNS (🟥 outward-facing)
16. **Cloudflare-Records** (Zone `hivegrid.de`):
- `app.` , `api.` , `captcha.` → Prod-IPs, **proxied** (orange).
- `admin.` → Prod-IP, **dns-only** + Firewall/Cloudflare-Access-Restriktion.
- **Kein** Wildcard.
17. Coolify/Traefik: Domains je App + Let's-Encrypt am Origin; Cloudflare SSL-Modus **Full (strict)**.
## 7. Endabnahme (🟩 verifizieren)
- `app./api.` `/healthz` + `/readyz` = 200 (DB erreichbar, Migration angewandt).
- Login inkl. 2FA-Gate; mCaptcha greift; Suche (Meili) antwortet; File-Upload-Scan (ClamAV) ok.
- **Prod ist indexierbar:** `/login` trägt **kein** `X-Robots-Tag: noindex`; `robots.txt` allow-all.
- Sentry-Events landen unter `environment=production`.
## 8. Rollback / Notfall
- Deploy-Rollback: vorheriges Image-Tag in Coolify re-deployen.
- DB: Volume-Snapshot / Backup einspielen (forward-only Migrationen → Inverse-SQL als Kommentar je Migration).
- DNS: TTL niedrig halten während des Cutovers.
## Inventar (provisioniert 2026-06-19, fsn1)
| Ressource | Name | ID | Public IP | Private IP |
|---|---|---|---|---|
| Server control | `hivegrid-prod-control` (cx33) | 143044798 | 157.90.243.228 | 10.0.0.2 |
| Server web | `hivegrid-prod-web` (cx33) | 143044800 | 167.233.109.142 | 10.0.0.3 |
| Server api | `hivegrid-prod-api` (cx33) | 143044801 | 167.233.110.100 | 10.0.0.4 |
| Server worker+services | `hivegrid-prod-worker` (cx43) | 143044803 | 49.13.24.190 | 10.0.0.5 |
| Server db | `hivegrid-prod-db` (cx33) | 143044805 | 178.105.208.72 | 10.0.0.6 |
| Volume (db-Daten) | `hivegrid-prod-db-data` 50 GB | 106081942 | — | `/dev/disk/by-id/scsi-0HC_Volume_106081942` |
| Network | `hivegrid-prod-net` 10.0.0.0/16 | 12347309 | — | — |
| Firewall app (22/80/443) | `hivegrid-prod-fw-app` | 11162316 | — | — |
| Firewall db (22) | `hivegrid-prod-fw-db` | 11162317 | — | — |
| SSH-Key | `hivegrid-prod-node1` (`~/.ssh/node1`) | 113985925 | — | — |
**Offen / Härtung:** SSH derzeit von `0.0.0.0/0` (key-only) erreichbar → später auf Admin-/Bastion-IP bzw. nur über `control` einschränken. Öffentliche 80/443 später auf Cloudflare-IP-Ranges begrenzen.
## Coolify (Schritt 3) — erledigt 2026-06-19
- Coolify **v4.1.2** auf `control` installiert (alle Container healthy).
- **Root-Admin headless** angelegt (`raffael@tool-ctrl.de`) via RootUserSeeder (`docker exec -e ROOT_USER_*` — Passwort-Policy: mixedCase+Ziffer+Symbol+uncompromised).
- **API aktiviert** + **API-Token** erzeugt (Sanctum, mit `team_id`; Coolifys `createToken`-Override braucht Team-Kontext, daher direkt über `tokens()->create`). `GET /api/v1/servers` → 200.
- Zugang (Secrets in `~/.cli-tokens/coolify-prod`, **nicht** im Repo): UI `http://coolify.hivegrid.de:8000` bzw. `http://157.90.243.228:8000`.
- ⚠ Härtung offen: Coolify-FQDN auf `https://coolify.hivegrid.de` (443) umstellen + Port 8000 wieder schließen.
## Coolify-Remote-Server + DB (Schritt 3–4) — erledigt 2026-06-19
- **Docker 29.6.0** auf web/api/worker/db installiert (get.docker.com), in Coolify als Remote-Server eingehängt → alle **`usable=true`**. node1-Key in Coolify (`privkey_uuid`). UUIDs in `~/.cli-tokens/coolify-prod`.
- Coolify-Projekt **HiveGrid** (`m74l9z6an57cw0316pir9rrb`) / Environment **production** (`h3woz1vm70yf6ajpzj0ch2fz`).
- **Postgres 16** (managed) + **Redis 7** (managed) auf db-VM. Creds + URLs in `~/.cli-tokens/hivegrid-prod-services`.
- **Cross-Server-Konnektivität (Designentscheidung):** Coolify-managed DBs laufen im Docker-Netz der db-VM; die App-Server (web/api/worker) erreichen sie über das **private Hetzner-Netz** (`10.0.0.6:5432/6379`), nicht über Container-DNS. DBs sind via `public_port` auf dem Host gebunden (`0.0.0.0`), aber **public durch fw-db (nur 22) dicht** — verifiziert: web→db privat OPEN, public DICHT.
**Offen (nächste Schritte):**
- Meilisearch + ClamAV + mCaptcha als Coolify-**Applications/Services** auf worker-VM (kein managed-DB-Typ), exposed aufs private Netz. mCaptcha nutzt Postgres-DB `mcaptcha` + Redis.
- PgBouncer vor Postgres (Compose auf db-VM) → `DATABASE_URL` über Pooler, `DIRECT_URL` direkt.
- DEK-Master-Key + S3-Bucket `hivegrid-prod`.
- App-Deploy (web/api/worker) nach **mitapp-Muster: image-retag / build-once-deploy-via-pull** (nicht rebuild) + `migrate deploy`.
- DNS `app./api./admin./captcha. → Prod-IPs` (Cloudflare) + Sentry (ToolCTRL-Instanz, prod-DSNs).
## Zentral-Dienste + Storage (Schritt 4) — erledigt 2026-06-19
- **Meilisearch v1.10 + ClamAV 1.4** als Coolify-Compose-Service `hivegrid-prod-search-av` auf worker-VM, privat exposed (`10.0.0.5:7700` / `:3310`). Verifiziert: Meili von api-VM cross-server `{"status":"available"}`, ClamAV healthy. Master-Key in `~/.cli-tokens/hivegrid-prod-services`.
- **mCaptcha-Stack** (mcaptcha + eigene Postgres + mcaptcha/cache) als Coolify-Service `hivegrid-prod-mcaptcha` auf worker-VM, FQDN `captcha.hivegrid.de` (DNS dns-only → worker, LE-Cert via Traefik im Hochlauf). Secrets gespeichert. **Folgeschritt:** Account+Sitekey in der mCaptcha-Instanz anlegen → neue `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` in App-Env (Code-/Env-Anpassung).
- **S3-Bucket** `hivegrid-prod` (fsn1) angelegt + Versioning; `prod-s3-bucket-hetzner-credentials` ergänzt.
## App-Deploy — Muster verstanden (mitapp / build-once-retag)
- Image-Schema: `git.tool-ctrl.de/toolctrl-labs/hivegrid/{web,api,worker}:<commit-sha>`, build_pack `dockerimage`, Port **8080**.
- **Promote = retag, kein Rebuild:** Prod deployt denselben SHA-Tag, der getestet/gestaged ist (aktuell `9ae3963d…` = live auf Staging → Image existiert in der Gitea-Registry).
- **7 Coolify-Apps** (nicht 3!): `web`, `api`, `worker-general`, `worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin` — web/api aus `/web`+`/api`-Image, die 5 Worker aus `/worker`-Image mit unterschiedlichem `WORKER_QUEUE_GROUP`.
- Deploy-Trigger: `scripts/coolify-deploy.mjs` (PATCH `docker_registry_image_tag` → deploy), `migrate deploy` als Pre-Deployment-Command.
- **Sentry (ToolCTRL-Instanz):** gleiche DSNs wie Staging, nur `SENTRY_ENVIRONMENT=production` (Estate-Regel: Staging↔Prod = Tag, kein eigenes Projekt).
**Offen (Go-Live-Phase):** Registry-Pull-Credential in Prod-Coolify · 7 Apps anlegen (Docker-Image) · App-Env Prod (aus Staging-Env transformiert: Prod-IPs/Domains, frische Secrets, prod-Sentry-Tag, **kein** WEB_NOINDEX) · erster Promote-Deploy + `migrate deploy` · DNS `app./api./admin.` · Endabnahme · mCaptcha-Sitekey + PgBouncer (Optimierung) als Folgeschritte.
## App-Deploy + Go-Live (Schritt 5–6) — LIVE 2026-06-19
- **Registry-Pull:** `docker login git.tool-ctrl.de` auf web/api/worker (User `gitea-admin`; Härtung: scoped Bot-PAT). Image `hivegrid/{web,api,worker}:9ae3963d…` (= Staging-Stand) gepullt.
- **7 Coolify-Apps** (Docker-Image): `hivegrid-web`, `-api`, `-worker-{general,mail,import,webhook,plugin}` (Worker je `WORKER_QUEUE_GROUP`). UUIDs in `~/.cli-tokens/coolify-prod`.
- **Prod-Env** aus Staging transformiert (Bug gefixt: bei doppelten build/runtime-Keys nicht-leeren Wert bevorzugen — sonst Mailjet u.a. leer → Worker-Crash). Frische, app-konsistente Secrets in `~/.cli-tokens/hivegrid-prod-app-secrets`. Demo-Seed AUS, `WEB_NOINDEX` raus (Prod indexierbar).
- **Postgres-Image korrigiert:** `postgres:16-alpine` → `pgvector/pgvector:pg16` (Migration `04_enable_pgvector` brauchte die Extension). Migration im laufenden web-Container ausgeführt → **alle Migrationen angewandt** (inkl. P2 `UserSession`+`TrustedDevice`).
- **DNS** (Cloudflare): `app.`/`admin.` → web-VM, `api.` → api-VM. SSL-Mode strict. `app`/`api` **proxied** (Edge-TLS/WAF), `admin` dns-only (Härtung folgt), `captcha` dns-only bis Cert.
- **Endabnahme grün:** `app.hivegrid.de` + `api.hivegrid.de` `/healthz` + `/readyz` = 200 (DB-backed), `/login` rendert „Anmelden | HiveGrid", **kein** X-Robots-Tag (indexierbar). Alle 7 Container healthy.
## Fix: Admin-SSO-Login-Loop (Cookie-Domain bei Multi-Host) — 2026-06-19
**Symptom:** SSO-Login loopte (Audit zeigte wiederholt `admin.login.sso.success`, aber Folge-Request nicht eingeloggt). **Ursache:** `resolveCookieDomain(APP_BASE_DOMAIN)` band das Session-Cookie an `app.hivegrid.de` → galt **nicht** für `admin.hivegrid.de` (getrennte Hosts). Auf Staging unsichtbar, weil dort app+admin denselben Host teilen. **Fix:** `APP_BASE_DOMAIN=hivegrid.de` (Parent-Domain) auf allen Apps → Cookie-Domain `hivegrid.de` gilt für app./admin./api.; abgeleitete Hosts bleiben korrekt (`getDerivedHosts` → app./admin./api.hivegrid.de). Verifiziert: `Set-Cookie: Domain=hivegrid.de`. **Lesson:** Multi-Host-Prod braucht `APP_BASE_DOMAIN` = Registrable-Domain, nicht den app-Host.
## Härtung/Abschluss-Runde — 2026-06-19 (nach SSO-Bestätigung)
- ✅ **Admin-Fallback aus:** `ADMIN_FALLBACK_LOGIN_ENABLED=false` → SSO (Authentik) ist einziger Admin-Pfad.
- ✅ **Sentry aktiviert:** `SENTRY_DSN_WEB/API/WORKER` aus zentraler Instanz gesetzt (`environment=production`), alle 7 Apps redeployed + healthy. (Waren vorher leer → Prod lief ohne Error-Tracking.)
- ✅ **mCaptcha komplett scharf:** Root-Cause des fehlenden LE-Certs war `service ... port is missing` (Traefik kannte Ziel-Port nicht — Compose-Service ohne Port-Deklaration). Fix: Compose mit korrekter Coolify-Magic `- SERVICE_FQDN_MCAPTCHA_7000=captcha.hivegrid.de` + `expose: 7000` → Host+Port-Label generiert, **LE-Cert geholt** (`captcha.hivegrid.de` HTTP 302, gültiges Cert). Account `hivegrid` + Captcha-Widget via mCaptcha-API angelegt → `MCAPTCHA_SITEKEY_LOGIN` + `MCAPTCHA_ACCOUNT_SECRET` + `MCAPTCHA_INSTANCE_URL` auf web gesetzt + redeployed. Creds in `~/.cli-tokens/hivegrid-prod-services`. `CAPTCHA_FAIL_OPEN=true` bewusst belassen (captcha aktiv, aber mCaptcha-Ausfall sperrt nicht den Login).
## CI-Promote verdrahtet + getestet — 2026-06-19
- **Gitea-Repo-Variablen (Production-Slot):** `COOLIFY_API_URL`, `COOLIFY_PROJECT_UUID`, `COOLIFY_ENVIRONMENT_NAME/UUID`, `COOLIFY_RESOURCE_UUIDS` (7 Apps), `COOLIFY_PROVISION_WORKER_GROUPS=false` + Secret `COOLIFY_TOKEN`. (Multi-Server ok: Resource-Auflösung ist server-agnostisch per Image-Name+Queue-Group; Auto-Provisioning aus, da Apps existieren.)
- **App-Tags auf `main` umgestellt** + `:main`-Alias in Registry erstellt (= aktueller SHA). Promote aliased Staging-Image → `:main` → deployt.
- **Worker-Apps korrekt konfiguriert** (vom CI-Drift-Check erzwungen, macht Prod robuster): Health-Ports 9101–9105 + `/readyz`-Health-Checks; Worker-ENV (WORKER_ENABLED/HEALTH_PORT/ADMIN_PORT/CONCURRENCY je Gruppe; general: OUTBOX_DISPATCHER_*/WORKFLOW_TRIGGER_OUTBOX_*; mail: TRANSACTIONAL_MAIL_OUTBOX_REQUIRED/MARKETING_SEQUENCES_ENABLED).
- **Test-Promote (runs 8436–8438):** iterativ alle Drifts gefixt; **8438 deployt Prod erfolgreich** (production worker groups + web-health + dependent deploys grün; Prod app/api `/readyz` 200, alle 7 Container healthy, `migrate status` = up to date).
- ⚠ **Ein Step bleibt rot:** „Verify production migration status after web deploy" läuft auf dem **CI-Runner** und braucht DB-Zugang — die Prod-DB liegt aber im **privaten Netz des separaten Prod-Accounts** (`10.0.0.6`), für den Runner strukturell unerreichbar. **Kein Prod-Problem** (Migration lief als Pre-Deploy + `migrate status` im Container = up to date). **Kein Prod-Problem** (Migration lief als Pre-Deploy + `migrate status` im Container = up to date).
- **Container-Verify-Versuch (revertiert `b833c82e`):** Step auf SSH-Jump (Runner→control→web-Container) umgeschrieben → scheiterte am **Preflight-Wiring-Guard** `scripts/verify-gitea-deploy-workflow.mjs` (erzwingt exakte CI-Runner-Step-Struktur + verbietet secret-backed env in `echo`/SSH). Der Guard läuft im **Preflight bei jedem Push** → die Änderung machte **die geteilte Pipeline rot** (auch Staging). **Sofort revertiert**, Preflight wieder grün. **Konsequenz:** Jede Änderung am Step erfordert, **zwei bewusste CI-Guards (Struktur + Secret-Leak) + Tests** umzuschreiben → invasiv, **mit Owner abzustimmen**. Optionen: (A) Guard so anpassen, dass der Step bei privater DB nicht erzwungen wird; (B) Container-SSH-Verify voll umsetzen **mit Host-Key-Pinning** (Security-Review: kein `StrictHostKeyChecking=no` in committetem CI-Code) + Guards/Tests; (C) so lassen — Promote deployt erfolgreich, nur der redundante Schluss-Check ist bei Prod rot.
## CI-Promote KOMPLETT GRÜN — 2026-06-20 (Run 8453)
Der erste echte Prod-Promote deckte **drei latente Bugs im nie-gelaufenen Production-Pfad** der geteilten `deploy.yml` auf (Prod gab es vorher nicht). Alle gefixt, lokal/manuell verifiziert, mit committeten Zwischenständen:
1. **migration-status-verify (war rot):** CI-Runner erreicht die private Prod-DB nicht (separater Account, AD-24). Fix: `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE=true` (eingebauter Script-Mechanismus, erkennt P1001 → skip) + `DATABASE_URL`-Secret. Struktur unverändert → Wiring-Guard + Tests grün.
2. **production-dependent-health worker-check (war rot):** Step nutzte `remote_ssh`, **definierte es aber nie** (Copy-Paste-Lücke ggü. Staging) → „worker container not found". Fix: SSH-Setup + `remote_ssh` mit **ProxyCommand** (nicht ProxyJump — Key-Passthrough) über control-Bastion zur worker-VM (Multi-Server), gepinnte Host-Keys, Secret via Zwischen-var (umgeht Secret-Guard). Lokal verifiziert: Container gefunden + health-probe `ready`.
3. **web-health-revision-mismatch (war rot):** `/healthz` über `app.hivegrid.de` lieferte eine **40 min alte Cloudflare-gecachte** revision (`cf-cache-status: HIT`) statt der frisch deployten. Fix: **CF-Cache-Rule** `/healthz`+`/readyz` → bypass (Health-Endpoints nie cachen) + Purge. Kein deploy.yml-Eingriff.
- **Gitea-Config:** Prod-Coolify-Variablen + Secrets (`COOLIFY_TOKEN`, `DATABASE_URL`, `COOLIFY_PROD_SSH_KEY/KNOWN_HOSTS`, Host/Worker-IP-vars). Apps auf Tag `main`, `:main`-Alias in Registry.
- **Ergebnis:** Promote 8453 **komplett grün** (web-health + migration + dependent-health alle success). Prod läuft auf `94959b31`, app/api `/readyz` = ready. **Künftige Prod-Deploys:** `deploy.yml` workflow_dispatch `action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`.
## Härtungs-/Service-Runde 2 — 2026-06-20
- ✅ **mCaptcha scharf + proxied:** `captcha.hivegrid.de` jetzt Cloudflare-proxied; Sitekey + Account-Secret bereits gesetzt (`CAPTCHA_FAIL_OPEN=true` als Safety belassen).
- ✅ **PgBouncer (alle 7 Apps):** Coolify-Compose-Service `hivegrid-prod-pgbouncer` auf db-VM (`edoburu/pgbouncer:v1.23.1-p3`, transaction-pool, scram, `6432→5432`). Apps: `DATABASE_URL`→`10.0.0.6:6432?pgbouncer=true&connection_limit=10`, `DIRECT_URL`→`10.0.0.6:5432`. Kanarienvogel web `/readyz`=ready verifiziert, dann api+5 Worker nachgezogen — alle healthy. Creds in `~/.cli-tokens/hivegrid-prod-services`.
- ✅ **Coolify-Port 8000 zu (ohne Tunnel):** Coolify hinter den **schon laufenden Traefik** (coolify-proxy) gehängt via **Dynamic-Config-Datei** `/data/coolify/proxy/dynamic/coolify-ui.yaml` (`Host(coolify.hivegrid.de)`→`http://coolify:8080`, certresolver letsencrypt) — file-watch, **kein coolify-Restart, kein Selbst-Aussperr-Risiko**. LE-Cert geholt, `https://coolify.hivegrid.de/api/v1`=200. Firewall fw-app: **8000 entfernt** (jetzt 22/80/443). CI `COOLIFY_API_URL`→`https://coolify.hivegrid.de/api/v1`. Verifiziert: :8000 dicht, https-API 200, SSH+app/api weiter ok. (CF-Tunnel wurde erwogen, vom Owner abgelehnt → Traefik-Weg.)
- ✅ **Sentry gelöst (DSN-Host war falsch):** Die alte DSN nutzte `sentry.boiler.saas-ctrl.com` (→Staging-VM, 503) — toter Host, **richtiger Key/Projekt**. Über node1-SSH zum Dedi (`176.9.111.179`, self-hosted Sentry) die Projekte/Keys aus der Sentry-Postgres geholt: org `sentry`, Projekte `web=4`/`api=2`/`worker=3`. Korrekte DSNs (`https://<key>@sentry.saas-ctrl.com/<id>`) gesetzt auf web/api/worker + `SENTRY_ENVIRONMENT=production`. **Verifiziert:** Test-Envelope von der web-VM → HTTP 200; Container hat `…@sentry.saas-ctrl.com/4`; alle 7 healthy. Creds in `~/.cli-tokens/hivegrid-prod-services`. **Hinweis:** nutzt die generischen boilerplate-Projekte (web/api/worker, getrennt per `environment`-Tag) — dedizierte `hivegrid-*`-Sentry-Projekte wären Estate-Standard-Verfeinerung (Folgeschritt).
**Bewusst NICHT blind ausgeführt (echte Aufgaben, kein Einzeiler — Risiko/Begründung):**
- **Registry-Pull-Bot:** scoped Bot-PAT statt `gitea-admin` (Least-Privilege; aktuell funktional).
- **CF-Tunnel-Rest:** der bei Coolify-Port erwogene CF-Tunnel wurde verworfen; ein leerer Tunnel-Eintrag `hivegrid-prod-coolify` ohne Connector kann in Cloudflare gelöscht werden (API-Delete gab 400, harmlos da ohne cloudflared inaktiv).
## Offene Folgeschritte (nicht-blockierend)
- **mCaptcha:** LE-Cert für `captcha.hivegrid.de` verifizieren, dann proxied; Account+Sitekey anlegen → `MCAPTCHA_*` ins App-Env (bis dahin `CAPTCHA_FAIL_OPEN=true`).
- **Sentry:** Events unter `environment=production` verifizieren (DSN aus Staging übernommen).
- **Härtung:** SSH auf Bastion/Admin-IP einschränken · `admin.hivegrid.de` per Cloudflare-Access/IP schützen · scoped Registry-Bot statt `gitea-admin` · Coolify-FQDN auf https/443 + Port 8000 schließen.
- **Optimierung:** PgBouncer vor Postgres · Coolify Pre-Deploy-Migrate-Mechanik fürs CI-Promote klären (lief manuell, Coolify-pre-deploy griff nicht).
- **CI-Promote:** `deploy.yml`/Gitea-Variablen auf das Prod-Coolify verdrahten (für künftige `promote`-Deploys ohne manuelle Schritte).
## Härtung (Schritt 7) — teilweise erledigt 2026-06-19
- ✅ **SSH-Angriffsfläche reduziert:** neue FW `hivegrid-prod-fw-web` (nur 80/443) auf web/api/worker; **public SSH dort dicht**, Zugang nur über `control` (Bastion) + privates Netz. `control` behält 22/80/443/8000. `db`: **kein** public Port (fw-db rules geleert), nur privat. Verifiziert: control-SSH ok, web-SSH public dicht, web via control ok, app/api öffentlich ok, Coolify-Deploys (privates Netz) laufen weiter.
- ✅ Demo-Seed AUS, `WEB_NOINDEX` raus (prod indexierbar).
**Offene Härtung/Folgeschritte (priorisiert):**
1. **mCaptcha public-FQDN:** Traefik liefert noch Default-Cert für `captcha.hivegrid.de` (Compose-Service-Domain-Labels nicht generiert; fqdn per tinker reichte nicht). Container laufen intern. Domain am besten über Coolify-**UI** am `mcaptcha`-Service setzen → LE-Cert → proxied. Dann **Account+Sitekey** in der mCaptcha-Admin-UI anlegen → `MCAPTCHA_SITEKEY_*`/`ACCOUNT_SECRET` ins App-Env, `CAPTCHA_FAIL_OPEN=false`. Bis dahin: fail-open (kein Captcha-Zwang).
2. **Admin-SSO (Authentik) — eingerichtet 2026-06-19:** OIDC-Provider + App `hivegrid-admin-prod` in `auth.tool-ctrl.de` (Provider pk 31, explicit-consent-Flow, PKCE, signing-key gesetzt). Scopes openid/email/profile + **eigenes `groups`-Mapping** (`HiveGrid groups`) → liefert den `groups`-Claim für die Admin-Autorisierung (`toolctrl-admin`). `AUTHENTIK_ADMIN_CLIENT_ID/SECRET`, `AUTHENTIK_ISSUER_URL`, `AUTHENTIK_ADMIN_REDIRECT_URI`, `AUTHENTIK_ADMIN_POST_LOGOUT_REDIRECT_URI` auf web gesetzt + redeployed (Creds in `~/.cli-tokens/hivegrid-prod-authentik`). **Verifiziert:** `/auth/admin/login` → 302 zu Authentik mit korrekter redirect_uri/PKCE; SSO-Button sichtbar; OIDC-Discovery grün; raffael/akeshan/dejan/arthur in `toolctrl-admin`. **Noch zu tun (interaktiv/Browser):** einmal per SSO einloggen + prüfen, ob der erste Login einen AdminUser JIT-provisioniert (sonst AdminUser in Prod-DB seeden). **Erst nach erfolgreichem SSO-Login** `ADMIN_FALLBACK_LOGIN_ENABLED=false` setzen (aktuell `true` als Backup, allowlist raffael/akeshan). `admin.hivegrid.de` dns-only; optional Cloudflare-Access davor.
3. **Coolify-Port 8000** schließen, sobald Coolify hinter `https://coolify.hivegrid.de` (443) läuft (Port aktuell offen, key+token-geschützt; für laufende API-Schritte noch gebraucht).
4. **Registry:** scoped Bot-PAT statt `gitea-admin` für den Image-Pull-Login.
5. **CI-Promote:** `deploy.yml`/Gitea-Variablen aufs Prod-Coolify verdrahten (Deploys liefen manuell; Coolify-Pre-Deploy-Migrate griff nicht → Migration manuell im Container).
6. **Sentry** Events unter `environment=production` verifizieren · **PgBouncer** (Pooling) ergänzen.
---
## Gesamt-Status (Stand 2026-06-20) — ✅ VOLLSTÄNDIG LIVE + GEHÄRTET
`app.`/`api.`/`admin.hivegrid.de` öffentlich (HTTPS), DB-backed (Postgres 16/pgvector + **PgBouncer** + Redis), P2-Set + 5 Worker-Gruppen, Revision `94959b31`.
**Alles erledigt:**
- ✅ **CI-Promote komplett grün** (Run 8453) — `workflow_dispatch action=promote, staged_sha=<:staging-SHA>, backup_confirmed=true`. Drei latente Multi-Server-Promote-Bugs gefixt (migration-status via `DB_MIGRATE_STATUS_ALLOW_UNREACHABLE`, dependent-health via ProxyCommand-SSH, web-health via CF-Cache-Bypass).
- ✅ **SSH-gehärtet** — control = Bastion, web/api/worker/db public-SSH dicht.
- ✅ **Admin-SSO** über Authentik (`auth.tool-ctrl.de`, OIDC, groups-Claim → `toolctrl-admin`).
- ✅ **mCaptcha** scharf + proxied (`captcha.hivegrid.de`).
- ✅ **PgBouncer** vor Postgres, alle 7 Apps über `6432`.
- ✅ **Coolify-Port zu** — nur noch `https://coolify.hivegrid.de` (Traefik+LE), `:8000` firewalled.
- ✅ **Sentry funktional** — `sentry.saas-ctrl.com` (alte `boiler.`-DSN war tot/503).
**Estate-Doku nachgezogen:** `infra/` README (Host-Inventar + DNS) + **ADR-016** (dedizierter Account + eigenes Coolify) + **LESSONS #13–#17** + onboard-pro-saas-Runbook.
**Rest-Folgeschritte (Least-Privilege, kein Druck):** scoped Registry-Bot statt `gitea-admin` · dedizierte `hivegrid-*`-Sentry-Projekte (statt geteilter boilerplate-Projekte) · Sentry-DSN-Host-Fix auch auf **Staging** nachziehen (dort steckt dieselbe tote `boiler.`-DSN) · leeren CF-Tunnel `hivegrid-prod-coolify` löschen.
hivegrid-cloud/docs/ops/production-build-warnings.md
+42
View File
@@ -0,0 +1,42 @@
# Production Build Warnings
`pnpm run build` is allowed to finish with known Vite/Rollup warning classes.
Treat this file as the operator reference before opening a deploy blocker for
build output that still exits with status `0`.
Last observed: 2026-05-28 via `pnpm run build`.
## Known Warning Classes
| Warning | Current interpretation | Escalate when |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Source maps are enabled in production` | Expected only when `TOOLCTRL_BUILD_SOURCEMAPS=1` is set for the Sentry release lane. Default web and bundle-budget builds should not emit this warning. Do not put secrets in source code; environment values must stay in `@toolctrl/config` and server-only boundaries. | Source maps expose hard-coded credentials, a default build emits maps unexpectedly, or a deploy target starts publishing server-only artifacts outside the intended build output. |
| `Generated an empty chunk` | Expected for React Router resource/API routes that have no client runtime. Empty route chunks are checked by `pnpm run verify:bundle-size`; unexpected names should be added only with an explicit rationale. | A new empty chunk is not covered by the bundle-size gate or appears for a user-facing UI route that should ship client code. |
| AWS/Smithy circular chunk warnings | Mitigated in the web Vite build by grouping `@aws-sdk/*` and `@smithy/*` into `vendor-aws-sdk`. The 2026-05-28 build emits `build/server/assets/vendor-aws-sdk-*.js` and no longer prints the circular Smithy/AWS reexport warning. | The circular warning returns, runtime AWS/S3 paths fail after build, the warning points to first-party modules, or the SDK versions/change set introduce a new broken execution order. |
| `dynamically imported ... but also statically imported` | Known Vite server-build warning where a lazy import cannot move a shared server module into a separate chunk because another route imports it statically. Current examples include DB, config, admin SSO audit, subscription entitlements, admin fallback helpers, and password reset helpers. | The warning starts involving a large client-only dependency, moves code into the default client shell, or a new performance slice claims a lazy split that this warning proves ineffective. |
## Recheck Commands
```text
pnpm run build
pnpm run verify:bundle-size
pnpm run verify:docs-links
```
To inspect only warning lines from a local build log:
```text
pnpm run build > /tmp/toolctrl-build.log 2>&1
rg -n 'Source maps are enabled|Generated an empty chunk|@smithy|@aws-sdk|vendor-aws-sdk|dynamically imported|dynamic import will not move' /tmp/toolctrl-build.log
```
## Triage Notes
- Do not change runtime build configuration just to silence one of the known
warnings above.
- Prefer one warning class per follow-up PR.
- Keep unknown warning classes visible. If a warning is suppressed or accepted,
add or update a regression gate next to the owner.
- For bundle concerns, use `pnpm run verify:bundle-size` as the first hard
signal. The raw `pnpm run build` chunk list is diagnostic, not the budget
gate.
hivegrid-cloud/docs/ops/streaming-exports.md
+97
View File
@@ -0,0 +1,97 @@
# Streaming Exports — Operator Runbook
This document describes the streaming export pipeline shipped in Phase 03
task #46. It is the operator-facing complement to the detail plan
(`docs/improvement-plan/03-scaling-infrastructure/streaming-exports.md`).
## Why streaming
The previous export pipeline buffered every row of the target dataset in
worker memory, serialised it into a single string or `Uint8Array`, wrote
the result to a temporary file on disk, and only then uploaded the file
to S3. For large exports (>50k rows) this caused three operational
problems:
- worker OOM crashes on 200k+ row XLSX exports;
- user-visible latency in the multi-minute range because nothing
reached S3 until the very end;
- doubled I/O on the worker host (local write → S3 PUT).
The streaming pipeline replaces both the buffering and the tempfile with
per-format streamers and an `@aws-sdk/lib-storage` multipart upload.
## Per-format strategy
| Format | Streaming strategy | Hard cap | Notes |
|---|---|---|---|
| CSV | Async iterator → Node writable | – | `serializeExportTableToCsvStream` |
| JSON (NDJSON) | Async iterator → one JSON object per line | – | Default JSON mode |
| JSON (Pretty) | In-memory buffer of normalised rows | 10_000 | `ExportPrettyJsonTooLargeError` above the cap |
| XLSX | ExcelJS `WorkbookWriter` when `estimatedRowCount > 50_000`, in-memory workbook otherwise | 500_000 | `serializeExportTableToXlsxStream` |
| PDF | `@react-pdf/renderer` → `renderToStream` | 5_000 | `serializeExportTableToPdfStream` |
The format hard caps protect against format-specific failure modes (XLSX
stream-writer is fragile above 500k rows; PDF cost grows super-linearly).
They apply even to Enterprise tenants.
## Plan-based row caps
`packages/exports/src/limits.ts` exposes `resolveExportRowCap(plan, format)`
and `assertExportRowCap(rowCount, plan, format)`. The plan caps are:
| Plan | Cap (CSV/JSON) | XLSX cap | PDF cap |
|---|---|---|---|
| free | 10_000 | 10_000 | 5_000 |
| pro | 100_000 | 100_000 | 5_000 |
| enterprise | unbounded | 500_000 | 5_000 |
Throws `ExportRowCapExceededError` (code `EXPORT_ROW_LIMIT_PLAN`) when
the requested row count exceeds the resolved cap. Both the up-front
estimate and the streaming mid-pass enforce the cap so callers cannot
overshoot by under-reporting the estimate.
## S3 multipart upload sink
`createS3MultipartUploadSink({ client, bucket, key, contentType })`
returns `{ writeStream, completionPromise }`. The factory:
- uses `@aws-sdk/lib-storage`'s `Upload` class with `partSize: 8 MB`
and `queueSize: 4` (overridable);
- destroys the write stream when the upload rejects so producers see
the failure via the normal stream `error` event;
- relies on the AWS SDK to call `AbortMultipartUpload` on failure —
there is no orphaned multipart cleanup task to schedule.
## Progress tracking
The worker job (`apps/worker/src/jobs/export.ts`) writes
`export:progress:<exportJobId>` into the shared store every
`progressFlushInterval` rows (default 5_000) and again on success. The
payload is `{ processedRows, totalEstimate, updatedAt }` and the TTL is
1 hour.
The API exposes `GET /v1/exports/:exportJobId/progress` which simply
proxies the snapshot back. Front-ends poll every 2-5 s until the
artefact download URL becomes available.
## Backward compatibility
The in-memory façades (`serializeExportTableToCsv`,
`serializeExportTableToJson`, `serializeExportTableToXlsx`,
`serializeExportTableToPdf`) are intentionally preserved and continue to
work without code changes. They are still suitable for small synchronous
exports (e.g. the existing `Rows_Export.server.ts` page-bounded CSV
download in the web app).
New code that touches the worker pipeline should prefer the streaming
variants and the row-cap helper.
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Job stuck at 0 processed rows | Worker cannot reach the shared store | Check `REDIS_URL` and the shared-store warning emitted at boot; the worker continues but progress is invisible |
| `EXPORT_ROW_LIMIT_PLAN` returned | Tenant requested more rows than their plan allows | Upgrade plan or split the export |
| `EXPORT_ROW_LIMIT_EXCEEDED` returned | Tenant requested more than the format hard cap | Switch to CSV/NDJSON for the bulk export |
| Multipart upload visible in S3 console as `incomplete` | Job failed mid-stream; SDK abort raced with check | AWS abandons the upload after `LifecycleRule` for `AbortIncompleteMultipartUpload` (recommended: 7 days). Make sure the bucket has one configured |
| 200k-row CSV export takes minutes | Row source `iterateRows` is not actually cursored | Verify the row source uses `prisma.findMany({ cursor, take: 1000 })` rather than a single unbounded query |
hivegrid-cloud/docs/ops/stripe-meters-migration.md
+189
View File
@@ -0,0 +1,189 @@
# Stripe Meters API — Operator Runbook
**Audience:** operators wiring usage-based billing after the Stripe SDK
13 → 19 migration (B-001, Task #28).
**Why:** Stripe 19 removed the legacy Usage-Records API
(`subscriptionItems.createUsageRecord` /
`subscriptionItems.listUsageRecordSummaries`) and `Recurring.aggregate_usage`
in favor of the **Billing-Meters API**
(`billing.meterEvents.create` / `billing.meters.listEventSummaries`).
The boilerplate worker (`ai-cost-stripe-report`) still drives overage
reporting from `AICostEvent` rows; only the Stripe wire format changed.
Operators must configure a Billing-Meter once and migrate existing
subscriptions to the new meter-backed price.
---
## Step 1 — Create the Billing-Meter
In the Stripe Dashboard (or via the API):
1. **Billing → Meters → Create meter**.
2. Configure:
- **Display name:** `AI overage cents`
- **Event name:** `ai-overage` *(the boilerplate hardcodes this id;
change it only if you also override `aiOverageStripeMeterId` /
`aiOverageMeterEventName` per plan).*
- **Aggregation:** `Sum`
- **Customer mapping:** map the event payload key
`stripe_customer_id` to the Stripe customer id (default — leave
as-is).
- **Value settings:** event payload key `value`, type
`numeric`. The reporter submits the rounded overage block-count
in cents.
3. Copy the resulting **Meter id** (`mtr_…`). You will store this in
`Tenant.aiQuotaOverrideJson.aiOverageStripeMeterId` (per-tenant)
or in the plan default (per-plan, future admin UI).
CLI alternative:
```bash
stripe billing meters create \
--display-name="AI overage cents" \
--event-name="ai-overage" \
--default-aggregation[formula]="sum" \
--customer-mapping[event_payload_key]="stripe_customer_id" \
--customer-mapping[type]="by_id" \
--value-settings[event_payload_key]="value"
```
---
## Step 2 — Create the metered product + price
1. **Products → Create product** → "AI Overage".
2. **Add price**:
- **Pricing model:** `Usage-based`
- **Meter:** select the meter created in Step 1 (`mtr_…`).
- **Per-unit cost:** the per-cent charge (typically `0.01` USD
because the meter aggregates cents).
- **Billing period:** `Monthly` (match the tenant's plan).
3. Copy the resulting **Price id** (`price_…`).
---
## Step 3 — Migrate active subscriptions
For every active subscription that had a metered usage-record line in
SDK 13.x:
1. **Add a new subscription item** referencing the new price from
Step 2. Stripe API:
```bash
stripe subscription_items create \
--subscription="sub_..." \
--price="price_..." \
--quantity=0
```
2. **Remove the old subscription item** that referenced the
pre-migration metered price (or archive its price so no new
subscriptions can attach it).
3. Confirm: `stripe subscriptions retrieve sub_... --expand=items.data.price`
shows the new price and the meter id in `price.recurring.meter`.
The boilerplate's overage configuration uses `aiOverageStripeMeterId`
as the sole identifier (the legacy `aiOverageStripeSubscriptionItemId`
field was retired with the Stripe-19 cleanup). Existing tenant JSON
overrides that still carry the legacy key are silently ignored by the
parser.
---
## Step 4 — Wire the boilerplate
1. **Per-tenant override** (admin route
`/admin/settings/pricing/ai-overage`):
```json
{
"aiOverageStripeMeterId": "mtr_test_abc...",
"aiMarkupMultiplier": 2.0,
"aiOverageBlockCents": 10
}
```
2. **Per-plan default** — edit `packages/entitlements/src/ai-overage.ts`
`standardAIOverageDefaults` and bake the meter id into the plan
blueprint (operator-friendly admin UI is a future task).
3. **Worker bootstrap** — ensure the Stripe-19 client is constructed
with `apiVersion: "2025-10-29.clover"` and the worker injects a
`StripeMeterEventSubmitter` whose `submit()` calls
```ts
stripe.billing.meterEvents.create(
{
event_name: "ai-overage",
payload: {
stripe_customer_id: tenant.stripeCustomerId,
value: String(roundedBlocks),
},
identifier: idempotencyKey,
timestamp,
},
{ idempotencyKey },
);
```
See `apps/worker/src/jobs/ai-cost-stripe-report.ts` for the
pure-by-deps interface (`StripeMeterEventSubmitter`).
---
## Step 5 — Verify with a test event
1. Toggle `AI_OVERAGE_SUBMISSION_ENABLED=true`.
2. Manually emit a test event:
```bash
stripe billing meter_events create \
--event-name="ai-overage" \
--payload[stripe_customer_id]="cus_test123" \
--payload[value]="100" \
--identifier="manual-test-$(date +%s)"
```
3. Confirm aggregation:
```bash
stripe billing meters list_event_summaries mtr_... \
--customer="cus_test123" \
--start-time=$(date -v-1d +%s) \
--end-time=$(date +%s)
```
You should see a non-zero `aggregated_value` for the meter event
you just submitted.
4. End-to-end: run `pnpm --filter @toolctrl/worker test
ai-cost-stripe-report` to confirm the worker keeps passing its 13
unit tests against the new wire.
---
## Rollback
The Meters API is **not trivially reversible** — once events are
submitted, the legacy Usage-Records API cannot ingest them. The safe
rollback path is to leave already-submitted events in place and revert
the SDK to 13.x via Git revert (commit
`4b2350f0c5a098d3c2dd0778f087cac5910ec2f9` and its follow-ups). The
operator-side meter + price can stay in place — Stripe ignores them
when no subscription item references them.
---
## Open items
- Admin UI per-plan meter-id editor (currently only per-tenant via
`aiQuotaOverrideJson`).
- Auto-creation of the meter on first operator-bootstrap (currently
manual via Dashboard / CLI).
- CLI helper `toolctrl-cli ai-cost stripe-init` to scaffold the
meter + product + price idempotently — listed in
`docs/improvement-plan/05-business-features/ai-cost-phase-3.md`
follow-ups.
hivegrid-cloud/docs/ops/tenant-snapshot.md
+253
View File
@@ -0,0 +1,253 @@
# Tenant Snapshot — Operations Runbook
`tenant:snapshot` and `tenant:restore` ship inside `packages/cli` and let
operators clone a single tenant's Postgres rows + S3 storage objects into a
PII-anonymized archive, then replay that archive into another environment.
This runbook is the operator-facing companion to the implementation spec at
`docs/improvement-plan/03-scaling-infrastructure/tenant-snapshot.md`.
---
## When to use it
| Use case | Profile | Encryption | Restore target |
| --- | --- | --- | --- |
| **Bug repro** ("Customer X hit a workflow bug") | `full-anonymized` | `age` | local dev box |
| **QA fixture** (representative Pro-tier tenant) | `full-anonymized` | `age` | CI seed |
| **Sales demo** (realistic but anonymized) | `minimal-anonymized` | `age` | staging |
| **Tenant migration** to a dedicated instance | `none` (super-admin only) | `age` (mandatory) | the destination environment |
The default profile is `full-anonymized`. Never use `none` for any path that
leaves your production network.
---
## Anonymization profiles
Three profiles ship today. Picking one is a tradeoff between fidelity and
privacy risk. The full mapping table lives in
`packages/cli/src/commands/tenant/anonymize.ts`; the highlights:
| Field type | `full-anonymized` | `minimal-anonymized` | `none` |
| --- | --- | --- | --- |
| Email | `user-<hash>@anonymized.test` | `user-<hash>@anonymized.test` | (untouched) |
| First / Last name | random from a curated pool | **kept** (display names usable in QA UI) | (untouched) |
| Phone number | `null` | `null` | (untouched) |
| IP address | `127.0.0.<seeded>` | `127.0.0.<seeded>` | (untouched) |
| ZIP code | first two digits kept, rest reseeded | first two digits kept, rest reseeded | (untouched) |
| Stripe customer / subscription / payment-method id | `cus_anon<8 hex>` / `sub_anon<8 hex>` / `pm_anon<8 hex>` | same as full | (untouched) |
| Webhook / SSO / domain URL | `https://anonymized.test/<8 hex>` | same as full | (untouched) |
| Secrets, API keys, refresh tokens, payloads | `[REDACTED]` | `[REDACTED]` | (untouched) |
| Avatar URL | `https://anonymized.test/avatar/<8 hex>.png` | same as full | (untouched) |
**Determinism.** All anonymizers are seeded with `sha256(salt + original)`
where `salt` is a fresh 32-byte random value generated per snapshot. Two rows
that referenced the same original email get the same anonymized email — FK
integrity inside a snapshot is preserved. The salt itself is **not** stored
anywhere; only its `sha256` lives in the manifest as a reproducibility marker.
---
## CLI usage
### Create a snapshot (default profile, age-encrypted, upload to S3)
```bash
toolctrl-cli tenant snapshot \
--tenant-id ten_abc123 \
--output s3://toolctrl-snapshots/ten_abc123/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.age
```
The CLI requires `BACKUP_AGE_PUBLIC_KEY` (recipient public key) plus
`BACKUP_S3_*` credentials. See `docs/ops/backup-cli.md` for the env-var
catalog — the snapshot command reuses the same encryption layer.
### Restore into a dev box
```bash
NODE_ENV=development toolctrl-cli tenant restore \
--tenant-id ten_devclone \
--source s3://toolctrl-snapshots/ten_abc123/2026-05-17T03:00:00Z.tar.gz.age
```
The command refuses to run under `NODE_ENV=production` unless the operator
passes `--allow-production-restore`. The confirmation prompt can be silenced
with `--yes` for non-interactive pipelines.
### List existing snapshots
```bash
toolctrl-cli tenant list-snapshots --bucket toolctrl-snapshots
```
### Super-admin escape hatch (`--profile=none`)
Raw, un-anonymized exports require an explicit confirmation flag:
```bash
toolctrl-cli tenant snapshot \
--tenant-id ten_abc123 \
--profile none \
--i-am-the-super-admin \
--output s3://toolctrl-snapshots/migration/ten_abc123.tar.gz.age
```
Without `--i-am-the-super-admin`, the command exits non-zero with a clear
error. Every snapshot — including the `none` profile — emits a structured
`[audit] TENANT_SNAPSHOT_CREATED` line to stdout. Operators are expected to
redirect that into their central audit log; set `auditFilePath` when calling
`runTenantSnapshot()` programmatically.
---
## DSGVO / GDPR notes
- **`full-anonymized` is the only profile that may leave EU production
infrastructure.** The other profiles ship un-anonymized display names
(`minimal`) or full PII (`none`).
- Snapshots are themselves sensitive (business workflows, schema layout,
org structure). Always set `BACKUP_ENCRYPTION=age` so the artifact can
only be decrypted by the operator who holds the private identity file.
- The salt is discarded after snapshot creation; only the `sha256(salt)`
travels in the manifest. There is no way to reverse-engineer the
original PII out of an anonymized row given just the snapshot file.
- If a tenant invokes their right-to-be-forgotten on the **source**
environment after a snapshot exists, the snapshot still contains the
anonymized projection (which by design contains no resolvable PII). For
`none` snapshots, the operator must manually destroy the archive.
---
## Restore drill (recommended cadence: monthly)
1. Snapshot a representative tenant from staging:
```bash
toolctrl-cli tenant snapshot --tenant-id <staging-tenant> \
--profile full-anonymized \
--output /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age
```
2. Copy to a dev box and restore into a fresh database:
```bash
NODE_ENV=development DATABASE_URL=postgres://... \
toolctrl-cli tenant restore \
--tenant-id ten_drill_$(date +%s) \
--source /snapshots/drill-$(date -u +%Y%m%d).tar.gz.age \
--yes
```
3. Smoke test in the dev UI: login as a restored user, view a workflow,
confirm subscription state shows as expected.
4. Record the drill in the change log.
---
## How globals are handled in restore
Several tenant-scoped models carry foreign keys into **global** tables
(today only `User`). Earlier snapshots only emitted tenant-scoped rows,
so restoring into a truly empty target database failed for those
models with `ForeignKeyConstraintViolation`. Closed by Task G — the
extractor now harvests every referenced global id from the tenant
slice, fetches the rows, anonymizes them with the same per-snapshot
salt, and writes them to `globals/<Model>.jsonl` inside the archive.
Models that previously broke (now restore clean against an empty
target via the globals replay):
| Model | Foreign key into |
| ----------------- | ---------------- |
| `TenantUser` | `User` |
| `TenantIpAddress` | `User` |
| `UserRole` | `User` |
| `Event` | `User` |
| `Log` | `User` |
| `SupportTicket` | `User` |
The full list of `User`-referencing tenant-scoped models is
machine-enumerated in
`packages/cli/src/commands/tenant/models.ts::GLOBAL_REFERENCED_MODELS`
and asserted complete by `models.test.ts` against every prisma
schema relation.
### Restore modes for globals (`--globals=<mode>`)
The restore command accepts one of three strategies (default
`upsert`):
| Mode | Behaviour | When to use |
| -------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `upsert` | For each id, `findUnique` → if present, leave in place; else `create`. Idempotent. | Default. Merging a snapshot into a non-empty dev/staging. |
| `skip` | `createMany({skipDuplicates:true})`. Faster but silent on non-id unique conflicts. | Bulk restores where pre-existing rows are expected. |
| `error` | Same as upsert but aborts if any id already exists on target. | Sensitive restores into a verifiably-empty DB. |
```bash
# Default — overlay snapshot rows onto whatever already exists.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz
# Sensitive restore: refuse to proceed if any global is already there.
toolctrl-cli tenant restore --tenant-id ten_new --source ./snap.tar.gz \
--globals=error
```
### Platform-catalog globals
A second category of globals — `Entity`, `EntityGroup`, `Formula`,
`Onboarding`, `PromptFlow`, `RowComment`, `WorkflowBlock` — is
referenced by tenant-scoped models but **intentionally NOT** in the
snapshot. These are seeded once per deployment via migrations/setup
scripts; copying them per-tenant would be wrong. If a tenant slice
references a platform-catalog row that does not exist on the target,
the restore fails for that specific row and you must align the target
deployment's seed first. The list is asserted by
`models.test.ts::PLATFORM_CATALOG_GLOBALS`.
### Backward compatibility
Snapshots taken before this extractor landed have no `globals` array
in their manifest. The restore detects that, logs a warning, and
falls back to the pre-Task-G behaviour where the operator must
pre-seed referenced rows manually. Use `--include-globals=false` on
`tenant:snapshot` to produce a legacy-shaped archive intentionally.
### Anonymization of global rows
`User` rows in the globals slice are scrubbed by the same anonymizer
that handles tenant rows. The `full-anonymized` profile rewrites
`email`, `firstName`, `lastName`, `phone` and `avatar`, and nulls
credentials (`passwordHash`, `verifyToken`, `githubId`, `googleId`,
`totpSecretEncrypted`, `totpPendingSecretEncrypted`,
`mfaRecoveryCodeHashes`, `deletionRequestId`) so the restored user
cannot be logged into via the original credentials. The
`minimal-anonymized` profile keeps display names but still nulls
credentials. See
`packages/cli/src/commands/tenant/anonymize.ts::FULL_ANONYMIZED_PROFILE`
for the source of truth.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `Refusing to run --profile=none without --i-am-the-super-admin` | Guardrail working as intended. | Re-run with the flag if you are a super-admin and have a documented reason. Otherwise pick a different profile. |
| `Invalid --output: expected s3://<bucket>/<key>` | Output target malformed. | Use either a local path or a fully-qualified `s3://bucket/key/...` URL. |
| `Refusing to run tenant:restore in NODE_ENV=production` | Production restore guardrail. | Set `NODE_ENV` to a non-production value, or pass `--allow-production-restore` if you genuinely want to overwrite production. |
| `Row-count mismatch in N models` warning during restore | Target DB rejected some rows (FK or unique index conflict with pre-existing data). | Pick a fresh tenant id, or pre-clean the target DB. |
| `manifest.json: missing or invalid` | Archive truncated / wrong encryption key. | Verify download size against the S3 ETag; rerun with the correct `BACKUP_AGE_IDENTITY_FILE`. |
| Storage bucket mismatch (objects landed under wrong prefix) | `--storage-bucket` or `STORAGE_BUCKET` env mismatched between source and target. | Set the target storage bucket explicitly before re-running restore. |
| `tenant:restore aborted (--globals=error): <Model> had N pre-existing row(s)` | `--globals=error` saw an id already on target. | Re-run with `--globals=upsert` (default) to overlay or `--globals=skip` to drop conflicts. |
| `Manifest carries no globals slice (legacy snapshot)` warning | Snapshot was taken before Task G landed (or with `--include-globals=false`). | Either take a fresh snapshot, or pre-seed referenced globals on target before restoring. |
---
## Internals
- Tenant-scoped models enumerated: `packages/cli/src/commands/tenant/models.ts`
(`TENANT_SCOPED_MODELS`).
- Global-snapshot models enumerated: same file
(`GLOBAL_REFERENCED_MODELS`); platform-catalog globals on
`PLATFORM_CATALOG_GLOBALS`.
- Globals extractor: `globals-runner.ts::collectAndWriteGlobals()`.
- Profiles: `packages/cli/src/commands/tenant/anonymize.ts`.
- Manifest schema: `packages/cli/src/commands/tenant/manifest.ts`.
- Snapshot entrypoint: `runTenantSnapshot()` in `snapshot.ts`.
- Restore entrypoint: `runTenantRestore()` in `restore.ts`.
- Audit entry: `writeSnapshotAudit()` in `audit.ts`.
hivegrid-cloud/docs/ops/worker-queues.md
+381
View File
@@ -0,0 +1,381 @@
# Worker queue groups — operations guide
The standalone `apps/worker` runtime is split into five queue groups. Each
group is the same image and the same code; the only difference is the
`WORKER_QUEUE_GROUP` env variable that decides which BullMQ queues the
process subscribes to.
This page is the canonical worker topology reference. Shorter worker tables
in deployment, environment, or feature docs are summaries only and should link
back here instead of redefining queue ownership.
## Why five groups
A single worker process processed every queue. That had three operational
problems:
1. **Head-of-line blocking.** A long-running import (minutes) stalled
transactional mail and notifications that ran on different queues but
the same event loop.
2. **Wrong concurrency for the workload.** Webhook delivery needs high
parallelism (I/O-bound HTTP calls); imports need low parallelism
(CPU/memory-bound). One `WORKER_CONCURRENCY` value cannot suit both.
3. **No per-workload scaling.** When mail queues backlog we couldn't scale
mail capacity without also duplicating the import worker (waste of RAM
and Redis connections).
The split fixes all three by giving each workload its own process and its
own concurrency. Add replicas to only the group that needs them.
## Production worker matrix
Production-like Coolify environments must run one persistent worker service per
queue group. These are long-lived internal services from the same worker image,
not manual emergency processes and not public web apps. `worker-mail` with
`WORKER_QUEUE_GROUP=mail` is part of the baseline topology because
transactional mail, mail outbox, form-submission webhook delivery, and
email-sequence worker contracts are intentionally isolated from general outbox
fanout.
| Coolify app | Queue group | Owns | Health port | Admin/metrics port | Required fail-loud settings |
| ---------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------ | -------------------------------------------------------------------------- |
| `worker-general` | `WORKER_QUEUE_GROUP=general` | General package queues plus outbox dispatcher, AI-cost persistence, plugin hook fanout, workflow trigger fanout, and form-submission routing | `9101` | `9201` | `OUTBOX_DISPATCHER_REQUIRED=true`, `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` |
| `worker-mail` | `WORKER_QUEUE_GROUP=mail` | Transactional mail, transactional mail outbox bridge, email-sequence subscriber contracts, and form-submission webhook bridge | `9102` | `9202` | `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` in staging/production |
| `worker-import` | `WORKER_QUEUE_GROUP=import` | Import and export queues after runtime gates are enabled | `9103` | `9203` | `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` only after services are wired |
| `worker-webhook` | `WORKER_QUEUE_GROUP=webhook` | AI chatbot and browser automation queues | `9104` | `9204` | none beyond shared worker readiness |
| `worker-plugin` | `WORKER_QUEUE_GROUP=plugin` | Isolated WASM plugin invocation queue; idle before production plugin signoff | `9105` | `9205` | Plugin runtime gates from `docs/operations/plugin-lifecycle.md` |
The detailed queue mapping below is the source of truth for package queues,
subscriber queues, concurrency, and workload profile. Update this page first
when queue ownership changes; deployment-specific docs should keep only the
short app/group/port summary and link back here.
`worker-plugin` is required as a persistent Coolify service so the deploy
topology is stable, but production plugin execution is still gated. With
`PLUGIN_RUNTIME_PRODUCTION_ENABLED` unset or `false`, the plugin worker starts
as an idle topology placeholder and must not consume `toolctrl.plugin-invoke`.
Only after the external WASM sandbox signoff in
`docs/security/plugin-runtime-signoff.md` may operators set
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true`, provide the trusted public keys, and
accept plugin invoke consumption as production-ready. The final live proof is
operator-run, inside both `worker-general` and `worker-plugin`:
`pnpm run verify:plugin-runtime-live`.
`worker-import` follows the same placeholder-first principle. Provision the
separate Coolify app with `WORKER_QUEUE_GROUP=import`, but keep
`IMPORT_QUEUE_ENABLED` and `IMPORT_QUEUE_RUNTIME_READY` unset until import,
export, and scan services are wired into the worker bootstrap. In that idle
state `/readyz` may be healthy with an empty queue list, which proves topology
without consuming `toolctrl.imports` or `toolctrl.exports`. When both runtime
gates are set, startup fails closed unless the worker has real import, export,
and scan service dependencies.
## Group → queue mapping
| Group | Package queues | Bootstrapped subscriber queues | Default concurrency | Default replicas | Workload profile |
| --------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `general` | `toolctrl.notifications`, `toolctrl.maintenance`, `toolctrl.file-scan` | Outbox dispatcher, `outbox.ai-cost`, `outbox.search-index`, `outbox.rag-index`, `outbox.audit`, `outbox.notification`, `outbox.plugin-hook`, `outbox.plugin-capability-use`, `outbox.workflow`, `outbox.form-submission-route`, `plugin-audit-retention` | 10 | 1 | Mixed short I/O, fanout, AI-cost persistence, indexing, audit persistence, plugin hook fanout, plugin capability-use audit persistence, workflow trigger fanout, form-submission routing. The `outbox.plugin-capability-use` subscriber intentionally defaults to concurrency `1` to keep capability-use hash chains linear; do not raise `PLUGIN_CAPABILITY_USE_CONCURRENCY` without store-level serialization. |
| `mail` | `toolctrl.email` | `outbox.mail`, `outbox.marketing` (email-sequence trigger only), `outbox.webhook` (form-submission delivery bridge), `email-sequence.step` | 5 | 2 | SMTP/HTTP throughput-bound; owns direct transactional mail delivery, transactional mail outbox bridging, form-submission webhook bridging, and email-sequence trigger/step subscriber contracts. Email-sequence subscribers require `MARKETING_SEQUENCES_ENABLED=true`; keep it unset or false in production until the remaining approval follow-ups in `docs/features/email-sequences.md` land and the Mailjet env pair is configured. Stale-lead/trial-expiring cron runner functions exist but are not production scheduler-registered yet; trial-expiring still needs a canonical trial-end source before that scheduler can be wired safely. The generic marketing outbound adapter/handler foundation exists in `packages/marketing-outbound` and `apps/worker/src/jobs/marketing-outbound.ts`, but it is pure handler foundation and does not claim active ownership of `outbox.marketing`; general marketing integration delivery is not bootstrapped in `apps/worker/src/index.ts` yet. |
| `import` | `toolctrl.imports`, `toolctrl.exports` after `IMPORT_QUEUE_ENABLED=true` and `IMPORT_QUEUE_RUNTIME_READY=true` | none | 1 | 1 | CPU/memory-bound, long-running; idle topology placeholder until import/export/scan services are wired |
| `webhook` | `toolctrl.ai-chatbot`, `toolctrl.browser-automation` | none | 20 | 3 | I/O-bound calls to external services |
| `plugin` | `toolctrl.plugin-invoke` | none | 5 | 2 | Third-party WASM plugin invocations, isolated from core queues. In production this group is idle until `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is set after external sandbox signoff. |
Source of truth: `packages/queue/src/groups.ts`. A test
(`groups.test.ts`) fails CI if a queue exists that isn't mapped to a
group — adding a new package queue forces a deliberate group assignment.
Outbox and auxiliary subscriber queues are wired by `apps/worker/src/index.ts`
through their dedicated bootstrap modules; update this table when adding a
new subscriber that is not represented in `packages/queue`.
`outbox.workflow` runs on the `general` group. The worker performs
live-workflow lookup and preflight checks in `apps/worker/src/jobs`, then
starts matching workflows through the selected executor: signed internal
workflow-run API by default, or hybrid/native execution when explicitly
configured. The native executor is a credential-free safe-core path: it must
not hydrate stored workflow credentials, and any `$credentials.*`,
`lookup $credentials`, or `#with $credentials` reference is a preflight gap.
That bridge avoids importing `apps/web` server modules into the worker
while keeping the existing workflow execution semantics centralized. The
same receiver also drains `workflow.execution.requested` rows produced by
manual workflow runs and tenant API workflow runs: the web action creates the
`WorkflowExecution` and outbox row in one transaction, then worker-general
starts that existing execution through the signed internal execution-start
bridge or the native safe-core executor. Waiting execution resumes publish
`workflow.execution.continue.requested`; worker-general then resumes the
existing execution through the signed internal continuation bridge or, in
native/hybrid mode, through the worker-native safe-core continuation path when
the workflow's only native preflight gap is the current `waitForInput` boundary.
Unsupported continuation graphs still fall back to the HTTP bridge in `hybrid`
mode and fail loud in `native` mode. Stream runs remain separate until their
progress contract moves to a distributed execution path. The
subscriber/bootstrap path is active from `apps/worker/src/index.ts` when the
`general` worker is configured with the workflow execution settings.
Set `OUTBOX_DISPATCHER_ENABLED=true`, `OUTBOX_DISPATCHER_REQUIRED=true`, and
`WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` on staging/production `worker-general`
once Redis plus the bridge URL/token are provisioned; in that mode a missing
dispatcher or bridge dependency fails startup instead of leaving outbox rows or
`outbox.workflow` silently undrained behind a green process.
Source-inventory tests guard that legacy `WorkflowEngine.trigger(...)`
call-sites stay absent and that only the signed internal trigger bridge may
start `type: "trigger"` executions directly. The `row.created`, `row.updated`,
and `row.deleted` route-backed and workflow-context producers now write the row
mutation, application event, and `workflow.trigger.event` outbox row in the same
transaction when a request or workflow event context is passed. Other internal
requestless mutations remain eventless by default to avoid worker feedback loops.
`worker-general` also registers the
`workflow-execution-stalled-reconcile` maintenance scheduler. It repairs only
old manual/API `WorkflowExecution` rows that are still `running`, have no block
runs, have no waiting block, and whose matching
`workflow.execution.requested` outbox event with dispatch key
`workflow-execution:<executionId>` is already `FAILED`. It does not infer
failure from `DONE` outbox rows because `DONE` only proves dispatch into BullMQ,
not workflow completion.
Use `pnpm run verify:workflow-outbox-native` as the canonical non-browser gate
before changing workflow outbox/native execution readiness. The gate composes
existing focused tests for `outbox.workflow` routing, subscriber bootstrap,
native/hybrid safe-core execution, worker health required-mode checks, and
queued manual/API/continuation producer inventory. It does not prove a live
environment is draining Redis jobs; pair it with the opt-in integration test or
operator-gated E2E drain probe for live worker readiness.
Form submissions use a two-hop worker path. The public submit route stores
the `FormSubmission` row and emits `form.submission.process`, which the
outbox dispatcher routes to `outbox.form-submission-route` on
`worker-general`. That subscriber performs tenant-scoped CRM/entity routing,
marks the submission, and publishes follow-up events. Webhook follow-ups use
`webhook.form-submission.deliver`, routed to `outbox.webhook` on
`worker-mail`, where the bridge enqueues the existing `toolctrl.email`
`deliver-webhook` job. This keeps webhook HTTP delivery in the mail/HTTP
throughput group and leaves the general group responsible only for the
database-backed routing decision.
## Env variables per group
Required (per worker container):
| Variable | Purpose |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `WORKER_QUEUE_GROUP` | `general` / `mail` / `import` / `webhook` / `plugin` |
| `WORKER_HEALTH_PORT` | TCP port for /healthz + /readyz; Dockerfile standalone default is 9100, while compose/Coolify group apps use 9101-9105 |
| `BULLMQ_REDIS_URL` | Redis target |
| `DATABASE_URL` | Postgres (via PgBouncer in staging/production) |
Required only on `worker-general`. The default execution adapter still bridges
through the web runtime; `hybrid` canaries credential-free safe core
event/manual/if/switch/log/alertUser/variable/httpRequest paths, tenant-scoped
rowGet/email blocks, and single-boundary `waitForInput` continuation resumes,
then falls back to the signed HTTP bridge for unsupported blocks or credential
references; `native` is reserved for credential-free safe-core-only deployments
and fails loud for unsupported blocks or credential references.
The signed HTTP bridge requires tenant-scoped workflow candidates; tenantless
platform workflow trigger execution is intentionally unsupported in HTTP mode
and should use native mode once the workflow is within the supported block
contract.
`scripts/coolify-ensure-worker-groups.mjs` verifies these settings on the
Coolify worker apps before deploy: every managed worker group must have
`DATABASE_URL` and `BULLMQ_REDIS_URL` configured, and `worker-general` must
also have either a complete HTTP bridge (`WORKFLOW_TRIGGER_EXECUTOR_URL` /
`APP_URL` / `WEB_ORIGIN` plus `API_ACCESS_TOKEN`) or explicit
`WORKFLOW_TRIGGER_EXECUTOR_MODE=native` / `WORKFLOW_TRIGGER_EXECUTOR_MODE=hybrid`.
| Variable | Purpose |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKFLOW_TRIGGER_EXECUTOR_MODE=http` | Explicitly selects the signed HTTP bridge. Use `hybrid` to canary worker-native credential-free safe-core execution with HTTP fallback; use `native` only for the worker-native safe-core MVP (`event`, `manual`, `if`, `switch`, `doNothing`, `log`, `alertUser`, `variable`, `httpRequest`, tenant-scoped `rowGet`/`email`, and single-boundary `waitForInput` continuation resumes). Native/hybrid preflight rejects stored workflow credential references before block execution. |
| `APP_URL`, `WEB_ORIGIN`, or `WORKFLOW_TRIGGER_EXECUTOR_URL` | Web runtime origin used by `outbox.workflow` execution bridge |
| `API_ACCESS_TOKEN` | Canonical service token used by `outbox.workflow`; value must match the web runtime's `API_ACCESS_TOKEN` because the web runtime validates the signed internal bridge against that key |
| `OUTBOX_DISPATCHER_ENABLED=true` | Starts the transactional outbox dispatcher on `worker-general` |
| `OUTBOX_DISPATCHER_REQUIRED=true` | Fail startup if the general worker cannot start the dispatcher |
| `OUTBOX_DONE_RETENTION_HOURS=720` | Daily `worker-general` maintenance keeps DONE outbox rows for 30 days before purging them |
| `OUTBOX_DONE_RETENTION_BATCH_SIZE=1000` | Max DONE outbox rows purged per daily maintenance run; large backlogs drain gradually |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_OLDER_THAN_MINUTES=30` | Five-minute `worker-general` maintenance marks only untouched manual/API workflow executions as `error` when their initial `workflow.execution.requested` outbox dispatch is already `FAILED` and older than this threshold. |
| `WORKFLOW_EXECUTION_STALLED_RECONCILE_BATCH_SIZE=100` | Max stalled workflow executions inspected per reconciliation run; large incident backlogs drain gradually. |
| `WORKFLOW_TRIGGER_OUTBOX_ENABLED=true` | Keeps the `outbox.workflow` subscriber active; set false only for controlled maintenance |
| `WORKFLOW_TRIGGER_OUTBOX_CONCURRENCY=5` | BullMQ concurrency for workflow trigger fanout |
| `WORKFLOW_TRIGGER_OUTBOX_REQUIRED=true` | Fail startup if the general worker cannot subscribe to `outbox.workflow` |
| `PLUGIN_HOOK_FANOUT_REQUIRED=true` | Optional staging/canary fail-closed switch for `outbox.plugin-hook`; production plugin runtime with `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` already requires fanout when Redis is configured |
## Coolify resources
Coolify must run queue groups as separate persistent applications that use
the same `worker` image with different `WORKER_QUEUE_GROUP` values. The
deploy workflow requires `worker-general`, `worker-mail`,
`worker-import`, `worker-webhook`, and `worker-plugin` resources in the staging and
production resource allow-lists. `worker-mail` is a permanent service, not a
manual workaround: without it, queued transactional mail and mail-backed
outbox/webhook work is not drained after deploy. Without the other group
resources, production traffic falls back into manual starts.
This is an operator-owned runtime prerequisite: the repository and static
verifiers document the required topology, but they do not by themselves prove
that a target Coolify environment currently has every worker group deployed and
healthy.
The same operator/code split applies to plugin runtime status: docs and code
require the `worker-plugin` service, while tenant-facing production execution
remains blocked on external signoff and the explicit
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` flip. A live rollout must additionally
pass `pnpm run verify:plugin-runtime-live` in both `worker-general` and
`worker-plugin` so the general fanout/audit path and isolated invoke worker are
checked separately.
Name the resources with the group marker (`worker-general`,
`worker-mail`, `worker-import`, `worker-webhook`, `worker-plugin`) or expose the
`WORKER_QUEUE_GROUP=<group>` marker in the resource metadata returned by
Coolify. Keep `WORKER_ENABLED=true` on each worker resource and assign
distinct health ports (`9101` general, `9102` mail, `9103` import,
`9104` webhook, `9105` plugin). When metrics/admin endpoints are enabled, keep them on
separate ports (`9201` general, `9202` mail, `9203` import, `9204`
webhook, `9205` plugin) so the admin server does not collide with the health server.
`scripts/coolify-ensure-worker-groups.mjs` can create missing worker
applications during the deploy workflow when provisioning is explicitly
enabled. Staging uses:
```bash
COOLIFY_STAGING_PROVISION_WORKER_GROUPS=true
COOLIFY_STAGING_PROJECT_UUID=<project uuid>
COOLIFY_STAGING_SERVER_UUID=<server uuid>
COOLIFY_STAGING_DESTINATION_UUID=<destination uuid>
COOLIFY_STAGING_ENVIRONMENT_NAME=staging
```
Production uses the same keys without `STAGING_`:
`COOLIFY_PROVISION_WORKER_GROUPS`, `COOLIFY_PROJECT_UUID`,
`COOLIFY_SERVER_UUID`, `COOLIFY_DESTINATION_UUID`, and
`COOLIFY_ENVIRONMENT_NAME`. When it creates a new group app, the script
copies the existing worker app env from an allowed non-group worker
resource or from `COOLIFY_STAGING_WORKER_ENV_SOURCE_UUID` /
`COOLIFY_WORKER_ENV_SOURCE_UUID`, then overwrites only the group-specific
settings (`WORKER_ENABLED`, `WORKER_QUEUE_GROUP`, `WORKER_HEALTH_PORT`,
`WORKER_ADMIN_PORT`, and group concurrency). Keep that source worker app
available until every group resource has its own runtime env.
Optional overrides:
| Variable | Default | Notes |
| ------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WORKER_CONCURRENCY_GENERAL` | 10 | Only read when `WORKER_QUEUE_GROUP=general` |
| `WORKER_CONCURRENCY_MAIL` | 5 | Only read when `WORKER_QUEUE_GROUP=mail` |
| `WORKER_CONCURRENCY_IMPORT` | 1 | Only read when `WORKER_QUEUE_GROUP=import` |
| `WORKER_CONCURRENCY_WEBHOOK` | 20 | Only read when `WORKER_QUEUE_GROUP=webhook` |
| `WORKER_CONCURRENCY_PLUGIN` | 5 | Only read when `WORKER_QUEUE_GROUP=plugin` |
| `WORKER_ADMIN_PORT` | Dockerfile/code default 9101; `9201`-`9205` in Coolify | Must differ from `WORKER_HEALTH_PORT` when `METRICS_ENABLED=true` |
| `WORKER_HEALTH_HOST` | `0.0.0.0` | Override when running outside Docker on multi-NIC hosts |
| `WORKER_REDIS_POOL_SIZE` | 4/4/2/8/4 (general/mail/import/webhook/plugin) | Advisory ioredis pool size |
| `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED` | `false`; managed Coolify `worker-mail` uses `true` | Mail-group only. Set to `true` in staging/production so `worker-mail` exits if the `outbox.mail` bridge cannot subscribe. The Coolify worker-group provisioner sets and drift-checks this for managed `worker-mail` apps. |
## Health endpoints
Every worker exposes two HTTP probes on `WORKER_HEALTH_PORT`:
- `GET /healthz` — liveness. Returns 200 as long as the event loop is
responsive. No external dependency calls; safe to hammer.
- `GET /readyz` — readiness. Runs a Redis ping and a Prisma `SELECT 1`
in parallel with a 2-second timeout each. Returns 200 only when both
succeed; 503 otherwise. Queue-group subscriber probes are also included
for required/expected runtime bridges such as `outbox.mail` on
`WORKER_QUEUE_GROUP=mail`.
Use `/readyz` as the Coolify / Kubernetes readiness probe and `/healthz`
as the liveness probe.
Worker health is primarily a queue-runtime check. For managed `worker-mail`
apps with `TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true`, health diagnostics also
fail loud when the required outbound provider pair `MAILJET_API_KEY` and
`MAILJET_FROM_EMAIL` is absent, using key names only. A healthy
`worker-mail` therefore proves Redis/Postgres reachability, the mail queue
process, the transactional outbox bridge, and required Mailjet env presence;
it still does not prove sender-domain verification or live provider delivery.
Managed Coolify `worker-mail` apps must keep
`TRANSACTIONAL_MAIL_OUTBOX_REQUIRED=true` so the process fails fast when
`outbox.mail` cannot subscribe; without that bridge, invite and other
`mail.transactional.*` outbox events can remain pending even if the direct
`toolctrl.email` worker is alive. The Coolify worker-group provisioner writes
and drift-checks this env value for managed mail workers.
Example:
```bash
curl -fsS http://worker-mail:9102/healthz
curl -fsS http://worker-mail:9102/readyz
```
## Connection naming
Each worker tags its ioredis connection with `worker-<group>-<pid>`
(via `apps/worker/src/connection.ts`). Run `CLIENT LIST` against Redis
to attribute traffic to a specific worker group:
```redis
CLIENT LIST
# id=42 name=worker-mail-218 ...
# id=43 name=worker-webhook-219 ...
```
## Scaling cookbook
| Symptom | Action |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Mail queue depth > 1 000 sustained | `docker compose up -d --scale worker-mail=4` |
| Import queue depth > 10 sustained | `docker compose up -d --scale worker-import=2`; review job size first — single jobs taking >10min are usually the real fix |
| AI/browser external-call queue latency p95 climbs | `docker compose up -d --scale worker-webhook=5` (each replica handles 20 parallel) |
| Form-submission webhook or mail outbox latency climbs | `docker compose up -d --scale worker-mail=4`; review SMTP/HTTP provider throttling before raising concurrency further |
| One worker stuck at 100 % CPU | Inspect `/readyz` — if it's still 200, profile inside the process; if 503, look at the `checks` payload to see which dep failed |
## Operator commands
```bash
# Inspect all worker-group logs
docker compose logs -f worker-general worker-mail worker-import worker-webhook worker-plugin
# Restart a single group
docker compose restart worker-import
# Smoke-test all five readiness endpoints
for port in 9101 9102 9103 9104 9105; do
curl -fsS http://localhost:$port/readyz || echo "Worker on $port not ready"
done
```
## Troubleshooting
**Wrong or missing worker group.** The worker Docker image defaults
`WORKER_QUEUE_GROUP` to `general` for raw image users. Coolify group apps
must still set the explicit group marker (`general`, `mail`, `import`,
`webhook`, or `plugin`) so each app drains the intended queues. A missing
Coolify override can therefore start as another `general` worker instead of a
mail/import/webhook/plugin worker; `scripts/coolify-ensure-worker-groups.mjs`
treats that as env drift. An unknown group still fails worker startup when the
worker is enabled. Disabled workers (`WORKER_ENABLED=false`) skip this check so
dev environments without Redis still boot.
**`/readyz` returns 503 with `checks.redis="error"`** — BullMQ couldn't
reach Redis. Confirm `BULLMQ_REDIS_URL` and that the redis container is
healthy. The error is intentionally short (no host names) so the probe
response never leaks credentials.
**`/readyz` returns 503 with `checks.database="error"`** — Prisma
couldn't run `SELECT 1`. Check `DATABASE_URL`, PgBouncer health, and
whether the pool is saturated.
**A queue is silent on every worker** — verify the queue is mapped to a
group in `packages/queue/src/groups.ts`. If you just added a new queue,
the `groups.test.ts` exhaustiveness check should have failed CI; if it
didn't, the queue is missing from `names.ts`.
**Outbox dispatcher missing.** The outbox dispatcher runs on the
`general` group only (see `apps/worker/src/index.ts`). If your stack
runs no `general` replica, outbox events will not dispatch — keep at
least one `general` replica.
## Rollback
Current production worker topology requires an explicit `WORKER_QUEUE_GROUP`
on each Coolify group app whenever `WORKER_ENABLED=true`; there is no
supported `WORKER_QUEUE_GROUP=all` runtime rollback path. The Dockerfile's
standalone default is `general`, so removing the marker during rollback can
silently turn a non-general app into another general worker, while an unknown
group fails startup by design. For an incident rollback, redeploy the previous
known-good image/tag or compose/deploy definition, or keep the five group apps
and roll the affected group back independently. In-flight BullMQ job state
lives in Redis, so jobs are not lost while workers are rolled.