16 KiB
16 KiB
OPERATOR-CHECKLIST
READINESS_AUTHORITY: authoritative operator readiness checklist. This is the canonical readiness source for operator-pending actions, live env flips, and deploy/operator closeout. Historical phase trackers such as
docs/improvement-plan/PROGRESS.mdand unresolved-risk registers such asdocs/improvement-plan/BLOCKERS.mdare supporting inputs only.
Living master list of all operator-pending actions across the project. Aggregates the "code-complete, X-pending" caveats from PROGRESS.md, the open items in BLOCKERS.md, and the Deviations sections of individual phase-docs.
Last regenerated: 2026-05-29 (operator/live-state reconciliation; B-008 closed, B-012 tracked only as operator-owned hardening risk).
Re-generate by walking every "in_progress (… pending)" row in PROGRESS.md + every ## Active Blockers and ## Unresolved Risks entry in BLOCKERS.md + verifying against the live Coolify-API state. Each row carries:
- WHAT — the operator-side action
- WHY — what's blocked / what flips to green when done
- TIME — wall-clock estimate
- BLOCKS — the affected task IDs
Bucket snapshot
| Bucket | Open | Closed / parked | Next action |
|---|---|---|---|
| A — Quick wins | 2 | 10 | A6 + A9 after Stripe Meter exists; A7/#51 is backlogged, not an active CDN item |
| B — Medium | 4 | 4 | B1 Stripe Meter, B4 Lighthouse login secrets, B6 canonical native Capacitor regeneration, or B8 runner Docker address-pool hardening |
| C — Long-running | 3 | 0 | External procurement / app-store / Authentik production migration |
| D — Polish sprints | 7 | 3 | #19/D7, TOTP/D8, and #76/D3 are closed; remaining work is code sprint backlog |
A — Quick wins (5–15 min each, mostly Coolify env-flips)
These are env-var changes via the Coolify UI or coolify env-update API. No new infrastructure required. Each one moves a task from in_progress (code-complete, env-flip-pending) → completed.
Status 2026-05-28: Live verification showed most A-items are already flipped. Rows marked ✅ are verified live; rows marked ⏳ remain open; rows marked 🅿 are intentionally parked/backlogged and not active operator work. B-008 is closed; B-012 is mitigated today and remains only as operator-owned Docker address-pool hardening.
| # | Action | Status | Why | Time | Blocks |
|---|---|---|---|---|---|
| A1 | Set METRICS_TOKEN + METRICS_ENABLED=true on web/api/worker |
✅ live (/metrics 200) |
Activates /metrics endpoints → Grafana dashboards stop showing "no data" |
— | #35 |
| A2 | Set OTEL_EXPORTER_OTLP_ENDPOINT=https://sentry.boiler.saas-ctrl.com/api/<project>/integration/otlp per app (api=2, worker=3, web=4) |
✅ live; per-project paths corrected and api/worker restarted, browser endpoint delivered via A11 | Traces start flowing into Sentry; closes B-005 last-mile | — | #33 |
| A3 | Set MEILI_URL=http://boilerplate-meilisearch:7700 + MEILI_MASTER_KEY on web/api/worker |
✅ live | Activates FTS hot-paths (CRM contacts + rows search) — was silent SQL fallback | — | #48 |
| A4 | Set DATABASE_URL on web/api/worker to pgbouncer pool |
✅ live (DATABASE_URL via pgbouncer, DIRECT_URL to raw pg) |
Pool pressure removed from primary; #40 closed | — | #40 |
| A5 | Set SENTRY_AUTH_TOKEN Gitea Actions secret + Coolify env |
✅ live on web/api plus worker-group apps using the worker DSN/env convention; Gitea secret set | Enables sourcemap upload + release-finalize in CI; closes #32 smoke gap | — | #32 |
| A6 | Set Stripe-Meter aiOverageStripeMeterId on first overage-enabled tenant + optional STRIPE_AI_OVERAGE_METER_EVENT_NAME=ai-overage env if deviating from the default |
⏳ pending (blocked on B1) | Activates Phase-3 Stripe overage reporter cron once worker bootstrap registers runAICostStripeReportCron |
10 min | #54 follow-up |
| A7 | CDN env flip (ASSET_CDN_URL + provider secret) |
🅿 backlogged with #51 — no active CDN rollout. Origin in Nürnberg + DACH audience + existing Cloudflare proxy is enough until international latency, large media traffic, or origin bandwidth pressure appears. | Would enable CDN serving for built assets if #51 is reactivated | — | #51 |
| A8 | Set Authentik OIDC client envs on web | ✅ live as AUTHENTIK_ISSUER_URL + AUTHENTIK_ADMIN_{CLIENT_ID,CLIENT_SECRET,REDIRECT_URI} — note: the original deploy on 2026-05-18 set them as OIDC_AUTHENTIK_* which is the WRONG namespace (code reads AUTHENTIK_ADMIN_* per apps/web/app/modules/admin-sso/client.server.ts:38-49). Corrected on 2026-05-19 via Coolify-API POST + DELETE of the 8 dead OIDC_AUTHENTIK_* rows. Smoke: /auth/admin/login 302s to Authentik with full PKCE/state/nonce chain. |
Unblocks Authentik admin SSO login | — | #81, #82 |
| A9 | Set AI_OVERAGE_SUBMISSION_ENABLED=true on worker container |
⏳ pending (paired with A6) | Enables Stripe-Meter overage reporter cron once tenants are configured | 2 min | #54 follow-up |
| A10 | Flip worker SENTRY_DSN + restart worker container |
✅ live as per-app DSN: SENTRY_DSN_API, SENTRY_DSN_WEB, SENTRY_DSN_WORKER (code reads these per apps/web/app/sentry.server.ts:48); apps/api/src/sentry.ts and apps/worker/src/sentry.ts mirror the convention |
Activates Sentry capture; closes #30 staging-smoke | — | #30 |
| A11 | Browser-side Sentry + OTel — VITE_* build-arg wiring | ✅ live — Dockerfile patch (commit 8ca98efb) + deploy.yml --build-arg (commit 218e5bd4) + Gitea repo secrets VITE_SENTRY_DSN_WEB / VITE_SENTRY_ENVIRONMENT / VITE_OTEL_EXPORTER_OTLP_ENDPOINT. Bundle verification on the redeployed image shows DSN-hash 1d3d4089 and OTel endpoint /api/4/integration/otlp inlined into entry.client-zyydvz2o.js. |
Coolify build_pack is dockerimage (not Dockerfile), so VITE_* must be passed at CI build time, not via Coolify env-flips. |
— | #30 (web), #33 (browser-OTel) |
| A12 | Restart api+worker containers to pick up corrected OTEL endpoint values | ✅ done 2026-05-19 — explicit redeploys triggered via Coolify /api/v1/deploy?uuid=…; verified inside running containers: worker OTEL_EXPORTER_OTLP_ENDPOINT=https://sentry.boiler.saas-ctrl.com/api/3/integration/otlp, api still /api/2/integration/otlp, web /api/4/integration/otlp. |
per-project Sentry routing now correct | — | #33 |
B — Medium (1–2 hrs each, requires external setup but bounded)
| # | Action | Why | Time | Blocks |
|---|---|---|---|---|
| B1 | Stripe Meter setup (production) — in Stripe Dashboard create a Meter named ai_overage_cents, event name ai-overage, default aggregation sum, customer mapping by stripe_customer_id. Document the meter ID in docs/ops/stripe-meters-migration.md. Then set the default through /admin/settings/pricing/ai-overage or, for tenant overrides, merge { "aiOverageStripeMeterId": "<meter id>" } into Tenant.aiQuotaOverrideJson; do not reference retired subscription-item fields. |
Required by Stripe SDK 19 — legacy subscriptionItems.createUsageRecord API was removed; without the meter the overage reporter throws on every call |
1 hr | #28 follow-up, #54 |
| B2 | CDN pull-zone provisioning | 🅿 backlogged with #51; not active operator work until CDN is reactivated | — | #51 |
| B3 | First bcrypt-user login on staging — completed 2026-05-24; login as the staging admin rehashed the legacy bcrypt hash to argon2 and verified the #28 dual-mode path | #28 closeout done | done | none |
| B4 | Lighthouse-CI session cookie rotation — code prep is committed: .github/.gitea Lighthouse workflows run scripts/rotate-lighthouse-session-cookie.mjs before LHCI and fall back to the existing LIGHTHOUSE_SESSION_COOKIE during rollout. Operator remaining: set repo Actions secrets LIGHTHOUSE_LOGIN_EMAIL and LIGHTHOUSE_LOGIN_PASSWORD from ~/.cli-tokens/lighthouse-bot-staging, then trigger lighthouse.yml manually and confirm the log says Minted Lighthouse session cookie RJ_session. Local API write could not be completed from this machine because git.tool-ctrl.de HTTPS/SSH timed out. |
Lighthouse-CI mints a fresh cookie per run instead of depending on session TTL | 10 min | #75 hardening |
| B5 | Coolify disk-pressure alert (B-008) — completed 2026-05-24; Coolify Discord webhook activated and disk trigger enabled | B-008 prevention done | done | none |
| B6 | Capacitor native builds — run npx cap add ios && npx cap add android from a Mac with Xcode + Android Studio installed, then commit the platform-specific files |
Required before mobile binaries can be built; staging dev-server already works | 2 hr | #65 |
| B7 | Tenant-snapshot staging verification — completed 2026-05-25; staging real-tenant snapshot was verified, the User-global restore gap was closed by Task G, and round-trip restore into fresh pgvector DBs passed 365/365 tests | #44 closeout done | done | none |
| B8 | Gitea runner Docker address-pool hardening — B-012 is mitigated today, but the permanent fix is still operator-owned: enlarge Docker default-address-pools on homeserver and add a prune cron for orphaned GITEA-ACTIONS-TASK-* networks |
Prevents future CI/deploy outages from Docker network exhaustion; not an active code closeout blocker | 30 min + maintenance window | CI/deploy reliability |
C — Long-running (multi-day external dependencies)
| # | Action | Why | Time | Blocks |
|---|---|---|---|---|
| C1 | B-006 — External WASM-sandbox security review (audit-ready 2026-05-19) — RFP, vendor decision matrix (top pick: Cure53), threat-model, attack-surface, internal-findings + audit-package script all landed. Operator next step: (1) run scripts/build-plugin-audit-package.sh from repo root; (2) email the resulting artifacts/plugin-audit/<date>/*.tar.gz to Cure53 per docs/security/plugin-audit-rfp.md § 9 (alternates: Trail of Bits / NCC Group / SySS / secfault). 3–5 day pen-test + report, ~15–25 k EUR. After report receipt: file findings under docs/improvement-plan/06-plugin-system/security-findings/F-NNN-….md, fix Critical/High, request re-test, sign off via docs/security/plugin-runtime-signoff.md. Resolved internally during audit-readiness prep: F-015 Critical (worker bootstrap unwired) closed by commit cbac8d3d — handlers.ts now dispatches to handlePluginInvokeJob and fails-loud if pluginInvokeDeps is missing. F-005 Phase A + Phase B closed — install path rejects publisher keys outside PLUGIN_TRUSTED_PUBLIC_KEYS, signedBy/key mismatches, missing binaries, and verifyPluginSignature failures before creating the Plugin row. F-010 + F-017 closed by the self-audit pass. |
Required before Phase 06 production rollout (staging unblocked) | 2–4 wk wall-time | Phase 06 prod-rollout |
| C2 | Mobile App Store submission — Apple Developer Program account ($99/yr), Google Play Console ($25 one-time). App Store screenshots, privacy nutrition labels, content rating, age gates. iOS TestFlight + Play Console internal-test track first | Required to ship the Capacitor app to end users | 1–2 wk (mostly waiting on Apple review) | #65 closeout |
| C3 | Authentik production migration — current Authentik runs on staging compose. Operator must promote to a dedicated host (Coolify or separate VPS), regenerate secrets, run blueprints, migrate the Toolctrl Groups/Policies/Admin Panel OIDC config |
Required before #81/#82 can flip from in_progress → completed for prod |
3–5 days | #81, #82 prod |
D — Polish-sprints (own-able multi-day code work, separate from prod-deploy items)
These are explicitly "deferred sub-sprints" — code-complete enough to unblock dependent work but with a follow-up sprint scoped.
| # | Action | Why | Time | Blocks |
|---|---|---|---|---|
| D1 | #72 i18n migration sprint — hardcoded UI/response extraction floor is now 0/0/0 (jsx-text, user-facing-prop-literal, user-facing-response-literal). Remaining work is the larger namespace split, locale breadth, and Playwright locale-switch proof. |
pnpm run verify:hardcoded-strings is clean; #72 closeout still needs the legacy translations.json split, additional locale namespaces/coverage, and an approved staging E2E locale flow. |
2–4 days | #72 closeout |
| D2 | #73 WCAG audit fix-sprint — work through packages/ui-app/test-setup/axe.ts violation backlog. Currently infrastructure-shipped, fix-sprint deferred |
A11y compliance bar | 5–7 days | #73 closeout |
| D3 | #76 test-coverage sprint — completed 2026-05-29. pnpm run verify:test-coverage passes with the current per-module gates, and docs/testing/coverage-sprint-backlog.md has no positive sprint-gap rows. Future work is gate maintenance and opportunistic ratcheting, not #76 closeout. |
Coverage observability + regression guard | done | none |
| D4 | Phase-06 plugin extraction sprint — foundation + audit-readiness shipped. Settings iframe/CSP/postMessage foundation is shipped; remaining deferred: marketplace, hot-reload, output-hooks, 13 more hook-points, global crash-counter, and external-audit follow-up after C1. Closed internal findings F-005, F-010, F-015, and F-017 are not remaining sprint work; see C1 and docs/security/plugin-internal-findings.md for the historical finding record. |
Plugin ecosystem features | 2–3 wk | #59 closeout |
| D5 | Phase-08 Storybook a11y/visual-regression — 21/20 stories shipped, a11y/visual-regression/test-runner deferred | Storybook coverage matrix | 1 wk | #67/#68/#69 closeout |
| D6 | AV-Layer1 + AV-Layer2 sprints — staging-deployed, no-upload-yet (#16) + core-complete (#17). Need first real upload to verify EICAR detection end-to-end + container scanner E2E test | Validates AV pipeline against live traffic | 1–2 wk | #16, #17 closeout |
| D7 | Magic-link reset mail worker — completed 2026-05-25. B-014 resolved by persistent Coolify worker applications for general, mail, import, webhook, and plugin; mail drains toolctrl.email without manual one-off containers |
#19 closed | done | none |
| D8 | TOTP live MFA E2E — completed 2026-05-24: enrollment, TOTP challenge, recovery-code challenge, and disable cleanup passed on staging | #26 closed | done | none |
| D9 | Subscription-cache + idempotency + session migration — 5 of 6 stores migrated to shared-store (#38); session migration is deferred, and MCP remains load-balancer/config work rather than an app-store migration. | #38 closeout | 1 wk | #38 |
| D10 | AI-cost call-site sweep beyond core 3 — chatbot is already wired; any new model invocations should pick up costContext proactively |
Coverage of the AI-cost outbox | 2–3 days | follow-up |
Re-generation procedure
# 1. Re-count PROGRESS.md statuses
grep '^| ' docs/improvement-plan/PROGRESS.md \
| awk -F'|' '{gsub(/^ *| *$/, "", $4); print $4}' \
| grep -v '^Status' \
| awk '
BEGIN { c = ip = p = bl = 0 }
/^completed/ { c++ }
/^in_progress/ { ip++ }
/^pending/ { p++ }
/^backlog/ { bl++ }
END { print "completed="c, "in_progress="ip, "pending="p, "backlog="bl }
'
# 2. Extract operator-pending caveats
grep -oE 'in_progress \([^)]*pending[^)]*\)' docs/improvement-plan/PROGRESS.md \
| sort -u
# 3. List active blockers and unresolved risks
sed -n '/^## Active Blockers/,/^## Resolved/p' docs/improvement-plan/BLOCKERS.md \
| grep -E '^### B-'
When all active items in sections A + B clear, core non-plugin
deploy/operator readiness is green. Parked CDN rows A7/B2/#51 are deliberate
backlog and must not be counted as completed CDN rollout. B-008 is already
closed by operator disk-pressure alerting, while B-012 remains operator-owned
runner host hardening rather than code work. Section C still contains
production-rollout gates with external dependencies, including third-party
plugin execution signoff; release readiness must also be checked against
docs/production-checklist.md. Section D remains an engineering backlog that
no longer contains the Magic-Link mail-worker blocker.