134 lines
16 KiB
Markdown
134 lines
16 KiB
Markdown
# OPERATOR-CHECKLIST
|
||
|
||
> **READINESS_AUTHORITY: authoritative operator readiness checklist.** This is
|
||
> the canonical readiness source for operator-pending actions, live env flips,
|
||
> and deploy/operator closeout. Historical phase trackers such as
|
||
> `docs/improvement-plan/PROGRESS.md` and unresolved-risk registers such as
|
||
> `docs/improvement-plan/BLOCKERS.md` are supporting inputs only.
|
||
|
||
**Living master list of all operator-pending actions across the project.** Aggregates the "code-complete, X-pending" caveats from `PROGRESS.md`, the open items in `BLOCKERS.md`, and the `Deviations` sections of individual phase-docs.
|
||
|
||
_Last regenerated: 2026-05-29 (operator/live-state reconciliation; B-008 closed, B-012 tracked only as operator-owned hardening risk)._
|
||
|
||
Re-generate by walking every "in_progress (… pending)" row in `PROGRESS.md` + every `## Active Blockers` and `## Unresolved Risks` entry in `BLOCKERS.md` + verifying against the live Coolify-API state. Each row carries:
|
||
|
||
- **WHAT** — the operator-side action
|
||
- **WHY** — what's blocked / what flips to green when done
|
||
- **TIME** — wall-clock estimate
|
||
- **BLOCKS** — the affected task IDs
|
||
|
||
---
|
||
|
||
## Bucket snapshot
|
||
|
||
| Bucket | Open | Closed / parked | Next action |
|
||
|---|---:|---:|---|
|
||
| A — Quick wins | 2 | 10 | A6 + A9 after Stripe Meter exists; A7/#51 is backlogged, not an active CDN item |
|
||
| B — Medium | 4 | 4 | B1 Stripe Meter, B4 Lighthouse login secrets, B6 canonical native Capacitor regeneration, or B8 runner Docker address-pool hardening |
|
||
| C — Long-running | 3 | 0 | External procurement / app-store / Authentik production migration |
|
||
| D — Polish sprints | 7 | 3 | #19/D7, TOTP/D8, and #76/D3 are closed; remaining work is code sprint backlog |
|
||
|
||
---
|
||
|
||
## A — Quick wins (5–15 min each, mostly Coolify env-flips)
|
||
|
||
These are env-var changes via the Coolify UI or `coolify env-update` API. No new infrastructure required. Each one moves a task from `in_progress (code-complete, env-flip-pending)` → `completed`.
|
||
|
||
**Status 2026-05-28:** Live verification showed most A-items are already flipped. Rows marked ✅ are verified live; rows marked ⏳ remain open; rows marked 🅿 are intentionally parked/backlogged and not active operator work. B-008 is closed; B-012 is mitigated today and remains only as operator-owned Docker address-pool hardening.
|
||
|
||
| # | Action | Status | Why | Time | Blocks |
|
||
|---|---|---|---|---|---|
|
||
| A1 | Set `METRICS_TOKEN` + `METRICS_ENABLED=true` on web/api/worker | ✅ live (`/metrics` 200) | Activates `/metrics` endpoints → Grafana dashboards stop showing "no data" | — | #35 |
|
||
| A2 | Set `OTEL_EXPORTER_OTLP_ENDPOINT=https://sentry.boiler.saas-ctrl.com/api/<project>/integration/otlp` per app (api=2, worker=3, web=4) | ✅ live; per-project paths corrected and api/worker restarted, browser endpoint delivered via A11 | Traces start flowing into Sentry; closes B-005 last-mile | — | #33 |
|
||
| A3 | Set `MEILI_URL=http://boilerplate-meilisearch:7700` + `MEILI_MASTER_KEY` on web/api/worker | ✅ live | Activates FTS hot-paths (CRM contacts + rows search) — was silent SQL fallback | — | #48 |
|
||
| A4 | Set `DATABASE_URL` on web/api/worker to pgbouncer pool | ✅ live (`DATABASE_URL` via pgbouncer, `DIRECT_URL` to raw pg) | Pool pressure removed from primary; #40 closed | — | #40 |
|
||
| A5 | Set `SENTRY_AUTH_TOKEN` Gitea Actions secret + Coolify env | ✅ live on web/api plus worker-group apps using the worker DSN/env convention; Gitea secret set | Enables sourcemap upload + release-finalize in CI; closes #32 smoke gap | — | #32 |
|
||
| A6 | Set Stripe-Meter `aiOverageStripeMeterId` on first overage-enabled tenant + optional `STRIPE_AI_OVERAGE_METER_EVENT_NAME=ai-overage` env if deviating from the default | ⏳ pending (blocked on B1) | Activates Phase-3 Stripe overage reporter cron once worker bootstrap registers `runAICostStripeReportCron` | 10 min | #54 follow-up |
|
||
| A7 | CDN env flip (`ASSET_CDN_URL` + provider secret) | 🅿 backlogged with #51 — no active CDN rollout. Origin in Nürnberg + DACH audience + existing Cloudflare proxy is enough until international latency, large media traffic, or origin bandwidth pressure appears. | Would enable CDN serving for built assets if #51 is reactivated | — | #51 |
|
||
| A8 | Set Authentik OIDC client envs on web | ✅ live as `AUTHENTIK_ISSUER_URL` + `AUTHENTIK_ADMIN_{CLIENT_ID,CLIENT_SECRET,REDIRECT_URI}` — note: the original deploy on 2026-05-18 set them as `OIDC_AUTHENTIK_*` which is the WRONG namespace (code reads `AUTHENTIK_ADMIN_*` per `apps/web/app/modules/admin-sso/client.server.ts:38-49`). Corrected on 2026-05-19 via Coolify-API POST + DELETE of the 8 dead `OIDC_AUTHENTIK_*` rows. Smoke: `/auth/admin/login` 302s to Authentik with full PKCE/state/nonce chain. | Unblocks Authentik admin SSO login | — | #81, #82 |
|
||
| A9 | Set `AI_OVERAGE_SUBMISSION_ENABLED=true` on worker container | ⏳ pending (paired with A6) | Enables Stripe-Meter overage reporter cron once tenants are configured | 2 min | #54 follow-up |
|
||
| A10 | Flip worker `SENTRY_DSN` + restart worker container | ✅ live as per-app DSN: `SENTRY_DSN_API`, `SENTRY_DSN_WEB`, `SENTRY_DSN_WORKER` (code reads these per `apps/web/app/sentry.server.ts:48`); `apps/api/src/sentry.ts` and `apps/worker/src/sentry.ts` mirror the convention | Activates Sentry capture; closes #30 staging-smoke | — | #30 |
|
||
| A11 | Browser-side Sentry + OTel — VITE_* build-arg wiring | ✅ live — Dockerfile patch (commit `8ca98efb`) + deploy.yml --build-arg (commit `218e5bd4`) + Gitea repo secrets `VITE_SENTRY_DSN_WEB` / `VITE_SENTRY_ENVIRONMENT` / `VITE_OTEL_EXPORTER_OTLP_ENDPOINT`. Bundle verification on the redeployed image shows DSN-hash `1d3d4089` and OTel endpoint `/api/4/integration/otlp` inlined into `entry.client-zyydvz2o.js`. | Coolify build_pack is `dockerimage` (not Dockerfile), so VITE_* must be passed at CI build time, not via Coolify env-flips. | — | #30 (web), #33 (browser-OTel) |
|
||
| A12 | Restart api+worker containers to pick up corrected OTEL endpoint values | ✅ done 2026-05-19 — explicit redeploys triggered via Coolify `/api/v1/deploy?uuid=…`; verified inside running containers: worker `OTEL_EXPORTER_OTLP_ENDPOINT=https://sentry.boiler.saas-ctrl.com/api/3/integration/otlp`, api still `/api/2/integration/otlp`, web `/api/4/integration/otlp`. | per-project Sentry routing now correct | — | #33 |
|
||
|
||
---
|
||
|
||
## B — Medium (1–2 hrs each, requires external setup but bounded)
|
||
|
||
| # | Action | Why | Time | Blocks |
|
||
|---|---|---|---|---|
|
||
| B1 | **Stripe Meter setup (production)** — in Stripe Dashboard create a Meter named `ai_overage_cents`, event name `ai-overage`, default aggregation `sum`, customer mapping by `stripe_customer_id`. Document the meter ID in `docs/ops/stripe-meters-migration.md`. Then set the default through `/admin/settings/pricing/ai-overage` or, for tenant overrides, merge `{ "aiOverageStripeMeterId": "<meter id>" }` into `Tenant.aiQuotaOverrideJson`; do not reference retired subscription-item fields. | Required by Stripe SDK 19 — legacy `subscriptionItems.createUsageRecord` API was removed; without the meter the overage reporter throws on every call | 1 hr | #28 follow-up, #54 |
|
||
| B2 | **CDN pull-zone provisioning** | 🅿 backlogged with #51; not active operator work until CDN is reactivated | — | #51 |
|
||
| B3 | **First bcrypt-user login on staging** — completed 2026-05-24; login as the staging admin rehashed the legacy bcrypt hash to argon2 and verified the #28 dual-mode path | #28 closeout done | done | none |
|
||
| B4 | **Lighthouse-CI session cookie rotation** — code prep is committed: `.github/.gitea` Lighthouse workflows run `scripts/rotate-lighthouse-session-cookie.mjs` before LHCI and fall back to the existing `LIGHTHOUSE_SESSION_COOKIE` during rollout. Operator remaining: set repo Actions secrets `LIGHTHOUSE_LOGIN_EMAIL` and `LIGHTHOUSE_LOGIN_PASSWORD` from `~/.cli-tokens/lighthouse-bot-staging`, then trigger `lighthouse.yml` manually and confirm the log says `Minted Lighthouse session cookie RJ_session`. Local API write could not be completed from this machine because `git.tool-ctrl.de` HTTPS/SSH timed out. | Lighthouse-CI mints a fresh cookie per run instead of depending on session TTL | 10 min | #75 hardening |
|
||
| B5 | **Coolify disk-pressure alert** (B-008) — completed 2026-05-24; Coolify Discord webhook activated and disk trigger enabled | B-008 prevention done | done | none |
|
||
| B6 | **Capacitor native builds** — run `npx cap add ios && npx cap add android` from a Mac with Xcode + Android Studio installed, then commit the platform-specific files | Required before mobile binaries can be built; staging dev-server already works | 2 hr | #65 |
|
||
| B7 | **Tenant-snapshot staging verification** — completed 2026-05-25; staging real-tenant snapshot was verified, the User-global restore gap was closed by Task G, and round-trip restore into fresh pgvector DBs passed 365/365 tests | #44 closeout done | done | none |
|
||
| B8 | **Gitea runner Docker address-pool hardening** — B-012 is mitigated today, but the permanent fix is still operator-owned: enlarge Docker `default-address-pools` on `homeserver` and add a prune cron for orphaned `GITEA-ACTIONS-TASK-*` networks | Prevents future CI/deploy outages from Docker network exhaustion; not an active code closeout blocker | 30 min + maintenance window | CI/deploy reliability |
|
||
|
||
---
|
||
|
||
## C — Long-running (multi-day external dependencies)
|
||
|
||
| # | Action | Why | Time | Blocks |
|
||
|---|---|---|---|---|
|
||
| C1 | **B-006 — External WASM-sandbox security review (audit-ready 2026-05-19)** — RFP, vendor decision matrix (top pick: Cure53), threat-model, attack-surface, internal-findings + audit-package script all landed. **Operator next step:** (1) run `scripts/build-plugin-audit-package.sh` from repo root; (2) email the resulting `artifacts/plugin-audit/<date>/*.tar.gz` to Cure53 per `docs/security/plugin-audit-rfp.md` § 9 (alternates: Trail of Bits / NCC Group / SySS / secfault). 3–5 day pen-test + report, ~15–25 k EUR. After report receipt: file findings under `docs/improvement-plan/06-plugin-system/security-findings/F-NNN-….md`, fix Critical/High, request re-test, sign off via `docs/security/plugin-runtime-signoff.md`. **Resolved internally during audit-readiness prep:** F-015 Critical (worker bootstrap unwired) closed by commit `cbac8d3d` — `handlers.ts` now dispatches to `handlePluginInvokeJob` and fails-loud if `pluginInvokeDeps` is missing. F-005 Phase A + Phase B closed — install path rejects publisher keys outside `PLUGIN_TRUSTED_PUBLIC_KEYS`, signedBy/key mismatches, missing binaries, and `verifyPluginSignature` failures before creating the `Plugin` row. F-010 + F-017 closed by the self-audit pass. | Required before Phase 06 production rollout (staging unblocked) | 2–4 wk wall-time | Phase 06 prod-rollout |
|
||
| C2 | **Mobile App Store submission** — Apple Developer Program account ($99/yr), Google Play Console ($25 one-time). App Store screenshots, privacy nutrition labels, content rating, age gates. iOS TestFlight + Play Console internal-test track first | Required to ship the Capacitor app to end users | 1–2 wk (mostly waiting on Apple review) | #65 closeout |
|
||
| C3 | **Authentik production migration** — current Authentik runs on staging compose. Operator must promote to a dedicated host (Coolify or separate VPS), regenerate secrets, run blueprints, migrate the `Toolctrl Groups`/`Policies`/`Admin Panel OIDC` config | Required before #81/#82 can flip from `in_progress` → `completed` for prod | 3–5 days | #81, #82 prod |
|
||
|
||
---
|
||
|
||
## D — Polish-sprints (own-able multi-day code work, separate from prod-deploy items)
|
||
|
||
These are explicitly "deferred sub-sprints" — code-complete enough to unblock dependent work but with a follow-up sprint scoped.
|
||
|
||
| # | Action | Why | Time | Blocks |
|
||
|---|---|---|---|---|
|
||
| D1 | **#72 i18n migration sprint** — hardcoded UI/response extraction floor is now 0/0/0 (`jsx-text`, `user-facing-prop-literal`, `user-facing-response-literal`). Remaining work is the larger namespace split, locale breadth, and Playwright locale-switch proof. | `pnpm run verify:hardcoded-strings` is clean; #72 closeout still needs the legacy `translations.json` split, additional locale namespaces/coverage, and an approved staging E2E locale flow. | 2–4 days | #72 closeout |
|
||
| D2 | **#73 WCAG audit fix-sprint** — work through `packages/ui-app/test-setup/axe.ts` violation backlog. Currently infrastructure-shipped, fix-sprint deferred | A11y compliance bar | 5–7 days | #73 closeout |
|
||
| D3 | **#76 test-coverage sprint** — completed 2026-05-29. `pnpm run verify:test-coverage` passes with the current per-module gates, and `docs/testing/coverage-sprint-backlog.md` has no positive sprint-gap rows. Future work is gate maintenance and opportunistic ratcheting, not #76 closeout. | Coverage observability + regression guard | done | none |
|
||
| D4 | **Phase-06 plugin extraction sprint** — foundation + audit-readiness shipped. Settings iframe/CSP/postMessage foundation is shipped; remaining deferred: marketplace, hot-reload, output-hooks, 13 more hook-points, global crash-counter, and external-audit follow-up after C1. Closed internal findings F-005, F-010, F-015, and F-017 are not remaining sprint work; see C1 and `docs/security/plugin-internal-findings.md` for the historical finding record. | Plugin ecosystem features | 2–3 wk | #59 closeout |
|
||
| D5 | **Phase-08 Storybook a11y/visual-regression** — 21/20 stories shipped, a11y/visual-regression/test-runner deferred | Storybook coverage matrix | 1 wk | #67/#68/#69 closeout |
|
||
| D6 | **AV-Layer1 + AV-Layer2 sprints** — staging-deployed, no-upload-yet (#16) + core-complete (#17). Need first real upload to verify EICAR detection end-to-end + container scanner E2E test | Validates AV pipeline against live traffic | 1–2 wk | #16, #17 closeout |
|
||
| D7 | **Magic-link reset mail worker** — completed 2026-05-25. B-014 resolved by persistent Coolify worker applications for `general`, `mail`, `import`, `webhook`, and `plugin`; `mail` drains `toolctrl.email` without manual one-off containers | #19 closed | done | none |
|
||
| D8 | **TOTP live MFA E2E** — completed 2026-05-24: enrollment, TOTP challenge, recovery-code challenge, and disable cleanup passed on staging | #26 closed | done | none |
|
||
| D9 | **Subscription-cache + idempotency + session migration** — 5 of 6 stores migrated to shared-store (#38); session migration is deferred, and MCP remains load-balancer/config work rather than an app-store migration. | #38 closeout | 1 wk | #38 |
|
||
| D10 | **AI-cost call-site sweep beyond core 3** — chatbot is already wired; any new model invocations should pick up `costContext` proactively | Coverage of the AI-cost outbox | 2–3 days | follow-up |
|
||
|
||
---
|
||
|
||
## Re-generation procedure
|
||
|
||
```bash
|
||
# 1. Re-count PROGRESS.md statuses
|
||
grep '^| ' docs/improvement-plan/PROGRESS.md \
|
||
| awk -F'|' '{gsub(/^ *| *$/, "", $4); print $4}' \
|
||
| grep -v '^Status' \
|
||
| awk '
|
||
BEGIN { c = ip = p = bl = 0 }
|
||
/^completed/ { c++ }
|
||
/^in_progress/ { ip++ }
|
||
/^pending/ { p++ }
|
||
/^backlog/ { bl++ }
|
||
END { print "completed="c, "in_progress="ip, "pending="p, "backlog="bl }
|
||
'
|
||
|
||
# 2. Extract operator-pending caveats
|
||
grep -oE 'in_progress \([^)]*pending[^)]*\)' docs/improvement-plan/PROGRESS.md \
|
||
| sort -u
|
||
|
||
# 3. List active blockers and unresolved risks
|
||
sed -n '/^## Active Blockers/,/^## Resolved/p' docs/improvement-plan/BLOCKERS.md \
|
||
| grep -E '^### B-'
|
||
```
|
||
|
||
When all active items in sections A + B clear, core non-plugin
|
||
deploy/operator readiness is green. Parked CDN rows A7/B2/#51 are deliberate
|
||
backlog and must not be counted as completed CDN rollout. B-008 is already
|
||
closed by operator disk-pressure alerting, while B-012 remains operator-owned
|
||
runner host hardening rather than code work. Section C still contains
|
||
production-rollout gates with external dependencies, including third-party
|
||
plugin execution signoff; release readiness must also be checked against
|
||
`docs/production-checklist.md`. Section D remains an engineering backlog that
|
||
no longer contains the Magic-Link mail-worker blocker.
|