Track HiveGrid Cloud planning reference

hivegrid-cloud/docs/architecture/bundle-budgets.md

@@ -0,0 +1,112 @@
+# Bundle Budgets
+
+**Status:** active
+**Owner:** web platform
+**Related:** [`scripts/bundle-budgets.json`](../../scripts/bundle-budgets.json), [`scripts/verify-bundle-size.mjs`](../../scripts/verify-bundle-size.mjs), `docs/improvement-plan/10-compliance-polish/bundle-budget.md`
+
+## Why
+
+Web bundle growth is silent unless we measure it. The boilerplate tracks two complementary signals:
+
+1. **Group budgets** (`BUDGETS` constant in `verify-bundle-size.mjs`): coarse caps for `root` / `initial_payload_js` / `vendor` / `route` / `other` / `css` / `total_js` / `manifest_js`. Hard CI gate — failure exits 1.
+2. **Per-route + per-vendor-chunk budgets** (`scripts/bundle-budgets.json`): fine-grained gzip limits per route and per vendor chunk. **Non-blocking** — surfaces as a PR-comment with a Δ-from-main column. Reviewers decide whether growth is justified.
+
+The per-route layer was added in task #71 (Phase 10) to make day-to-day creep visible without blocking legitimate feature work.
+
+`initial_payload_js` is a dedicated hard signal for the boot asset aggregate the current parser can identify without reading Rollup/Vite dependency graphs: `entry.client-*`, `root-*`, and `manifest-*`. It intentionally excludes route, vendor, and component chunks so lazy feature payloads cannot hide a shell regression in `total_js`.
+
+## Config schema
+
+`scripts/bundle-budgets.json` validated by `scripts/bundle-budgets.schema.json`:
+
+```jsonc
+{
+  "default": { "limitKb": 200, "warningKb": 180 },
+  "routes": {
+    "app/$tenant/dashboard": { "limitKb": 250, "warningKb": 225 },
+    "auth/magic-link": { "limitKb": 80, "warningKb": 75 }
+  },
+  "vendorChunks": {
+    "vendor-react": { "limitKb": 150, "warningKb": 135 }
+  }
+}
+```
+
+- All sizes are **gzip kilobytes** (the transport-relevant number).
+- `limitKb` is the hard cap; assets above this report `[EXCEED]` in CI output and the PR comment.
+- `warningKb` is the soft threshold; between `warningKb` and `limitKb` assets report `[WARN]`.
+- Routes use React Router v7 route ids in either dot- or slash-form (`auth/magic-link` and `auth.magic-link` both match).
+- Unmatched routes fall back to `default`.
+- Unmatched vendor chunks (anything starting with `vendor-` or `chunk-`) are reported under the chunk basename and also fall back to `default`.
+
+## Local workflow
+
+```bash
+# Build apps/web normally and print the per-route table:
+pnpm --filter @toolctrl/web build
+node scripts/verify-bundle-size.mjs
+
+# Only the per-route table (skips group checks):
+node scripts/verify-bundle-size.mjs --per-route-only
+
+# Machine-readable JSON (used by the PR-comment workflow):
+node scripts/verify-bundle-size.mjs --json > pr-bundle-report.json
+```
+
+`--json` emits only the non-blocking per-route/per-vendor report and exits with
+code 0 when the report was produced. Keep the hard group-budget gate on the
+plain `node scripts/verify-bundle-size.mjs` path so redirected JSON never mixes
+machine-readable output with human diagnostics.
+
+### Visualizer (treemap)
+
+To inspect why a chunk is big:
+
+```bash
+pnpm --filter @toolctrl/web run bundle:visualize
+open apps/web/build/stats.html
+```
+
+The visualizer is gated on `ANALYZE_BUNDLE=true` so the regular production build keeps its current footprint. It uses `rollup-plugin-visualizer` (treemap, gzip + brotli sizes per chunk).
+
+**Security note:** `stats.html` contains source-map references and should never be deployed publicly. Treat it as a developer-only artifact.
+
+## PR-comment workflow
+
+`.github/workflows/bundle-budget.yml` (and `.gitea/workflows/bundle-budget.yml`) run on every PR that touches `apps/web/**`, `packages/**`, or the budget config / verifier. They:
+
+1. Build `@toolctrl/web`.
+2. Run `verify-bundle-size.mjs --json`.
+3. Download the latest `bundle-baseline-main` artifact (captured by `bundle-budget-baseline.yml` on `main` pushes).
+4. Render a sticky markdown comment via `scripts/render-bundle-budget-comment.mjs`.
+5. Upsert the comment using its `<!-- bundle-budget-report -->` marker (GitHub: `marocchino/sticky-pull-request-comment`; Gitea: `scripts/post-bundle-budget-gitea-comment.mjs`).
+
+The comment renders rows in severity order (`[EXCEED]` → `[WARN]` → unbudgeted → ok), with columns: status / kind / route or chunk key / current gzip size / configured limit / Δ from main. Δ is `—` if no baseline is available.
+
+### When the baseline is missing
+
+The first PR after a fresh checkout (or after a baseline-artifact expiry) sees `Δ from main = —` for every row. This is expected — the next push to `main` re-captures the baseline.
+
+## When to raise a limit
+
+1. Confirm the growth is intentional (a new feature shipped, a dependency landed) — not a regression from accidental imports.
+2. Open the PR; the bundle-budget comment shows the Δ. Reviewers and the PR author should agree the new floor is acceptable.
+3. Edit `scripts/bundle-budgets.json` in the **same PR** that ships the growth. Bump `warningKb` to ≈ (current gzip + 5 KB) and `limitKb` to ≈ (current gzip + 20 KB).
+4. Call the change out in the PR description so the new floor is reviewable.
+
+## Anti-patterns
+
+- **Don't disable a budget by raising it to 9999 KB.** The point is to surface drift. If a route is legitimately huge, raise the limit to a tight new floor, not infinity.
+- **Don't lazy-load critical-path code.** Bundle splits help non-critical routes, but the initial payload (`entry.client-*` / `root-*` / `manifest-*`) still has its own `initial_payload_js` hard group budget that must stay tight.
+- **Don't bundle-split shared deps.** Splitting `react` into many chunks just shifts cost into chunk boot-up. Keep `vendor-react`, `vendor-core-utils` etc. as single chunks; the manualChunks mapping in `apps/web/app/utils/build/manualChunks.ts` is the source of truth.
+
+## Troubleshooting
+
+- **PR comment never shows up:** check the `Bundle Budget` workflow run — the `Build web` step may have failed (e.g. server-only import in a client export). The comment job only runs if all upstream steps succeed.
+- **`vendor-monaco-...` reports as `vendor-monaco-...-` with default budget:** rollup occasionally emits chunk hashes ending in `-`; the baseline normalizer skips that suffix and the asset falls back to `default`. Either rename via `manualChunks.ts` or add an explicit budget entry.
+- **CI run is much slower than `pnpm typecheck`:** the workflow does a full `pnpm --filter @toolctrl/web build`. That's the expected cost. If it ever exceeds ~30 minutes, profile with `ANALYZE_BUNDLE=true` locally first.
+
+## Rollback
+
+- If the per-route budget surfaces too many false alarms, lower the warning thresholds in `scripts/bundle-budgets.json` and ship a follow-up.
+- If the entire PR-comment flow is broken, set `if: false` on the `measure` job in `bundle-budget.yml` — no other CI stage depends on it.

hivegrid-cloud/docs/architecture/csrf.md

@@ -0,0 +1,121 @@
+# CSRF — Single-Read Helper Contract
+
+This document covers the runtime API contract of `validateCSRFToken` in
+`apps/web/app/utils/session.server.ts`. The complementary route-level
+inventory of which surfaces are protected lives in
+`docs/security/csrf-inventory.md` — read that file when adding a new
+mutating route.
+
+## Background — the body-already-consumed bug
+
+Before the May 2026 fix, `validateCSRFToken(request)` always tried to
+locate the request token by either reading the `x-csrf-token` header or,
+falling back, by cloning the request and re-parsing the body:
+
+```ts
+// historical implementation (broken when caller already drained the body)
+const formData = await request.clone().formData();
+```
+
+Cloning a request whose body was already consumed throws
+`TypeError: unusable` in undici. UI form-submits where the action handler
+read `await request.formData()` first and validated afterwards crashed
+in production with a 500. The bug was first surfaced during the B-009
+lighthouse-bot session-cookie capture, where the workaround was to send
+the token via the `x-csrf-token` header to bypass the body path.
+
+## API contract
+
+```ts
+export type ValidateCSRFTokenOptions = {
+  /** Already-parsed form data. When set, the helper NEVER reads the body. */
+  formData?: FormData;
+  /** Skip body fallback entirely. Header must carry the token. */
+  headerOnly?: boolean;
+};
+
+export async function validateCSRFToken(
+  request: Request,
+  options?: ValidateCSRFTokenOptions,
+): Promise<void>;
+```
+
+### Resolution order
+
+1. `x-csrf-token` / `csrf-token` request header.
+2. If `headerOnly` is `true`, stop here.
+3. `options.formData?.get("csrf")` — used when the caller has already
+   parsed the body.
+4. Body fallback: clone-and-parse the request as `FormData`,
+   `application/json`, or `URLSearchParams`. Reaching this branch on a
+   request whose body has already been consumed surfaces a CSRF-specific
+   error (`CSRF body already consumed; pass { formData } to validateCSRFToken.`)
+   instead of the opaque `TypeError: unusable` from undici.
+
+### Calling patterns
+
+```ts
+// Preferred: validate before reading the body. Header-first short-circuit
+// keeps this path zero-cost on header-bearing requests.
+export async function action({ request }: ActionFunctionArgs) {
+  await validateCSRFToken(request);
+  const form = await request.formData();
+  // ...
+}
+```
+
+```ts
+// Required when the caller parses the body first (e.g. routing based on a
+// form action discriminator). Inject the parsed FormData — never let the
+// helper attempt to re-read the consumed body.
+export async function action({ request }: ActionFunctionArgs) {
+  const form = await request.clone().formData();
+  await validateCSRFToken(request, { formData: form });
+  // ...
+}
+```
+
+```ts
+// Helpers / cross-file flows that receive a request plus pre-parsed
+// FormData from the route action must forward the FormData.
+async function loginFromRequest(request: Request, form: FormData) {
+  await validateCSRFToken(request, { formData: form });
+  // ...
+}
+```
+
+```ts
+// Bot / lighthouse / capture contexts where the body is reserved for a
+// different purpose. The token must be sent in the x-csrf-token header.
+await validateCSRFToken(request, { headerOnly: true });
+```
+
+## Migrated callsites (May 2026 fix)
+
+- `apps/web/app/root.tsx` — the locale/theme action pre-parses form data
+  for routing and now hands the FormData to `validateCSRFToken`.
+- `apps/web/app/modules/users/services/AuthService.ts` —
+  `loginFromRequest` receives `FormData` from `routes/login.tsx` and
+  `root.tsx`, and forwards it.
+
+Any new helper that accepts `(request, formData)` from a caller MUST
+forward the FormData to `validateCSRFToken`. The convention check is
+covered by `apps/web/app/utils/session.csrf-regression.test.ts`.
+
+## Verification
+
+```bash
+pnpm exec vitest run \
+  apps/web/app/utils/session.server.test.ts \
+  apps/web/app/utils/session.csrf-regression.test.ts \
+  apps/web/app/utils/session-csrf-sweep.test.ts
+```
+
+The regression file specifically asserts:
+
+- Calling the helper after `await request.formData()` succeeds when the
+  parsed body is injected.
+- Calling the helper after the body was consumed without injection raises
+  a CSRF-specific error — never the undici `TypeError: unusable`.
+- The header-token fast path still skips the body even when the body has
+  already been read.

hivegrid-cloud/docs/architecture/email-templates.md

@@ -0,0 +1,141 @@
+# Email Templates (React-Email)
+
+**Package:** `packages/mail/src/templates-react/`
+**Status:** active (Phase 10 task #75)
+**Replaces:** `packages/mail/src/templates/transactional.ts` (deprecated)
+
+## Overview
+
+Transactional emails are React components. Every template is rendered
+deterministically to `{ html, text }` via `react-dom/server` and
+delivered through the standard Mailjet/queue provider chain.
+
+We chose React components over the legacy string-template approach
+for three reasons:
+
+1. **XSS-safety by default** — React escapes interpolated values, so
+   we cannot accidentally forget an `escapeHtml` call when a new
+   prop is introduced. The old monolithic file had a recurring
+   audit finding around manual escaping.
+2. **Component reuse** — header, footer, button, info-box, and
+   eyebrow are shared across templates. Brand-theming threads
+   through a single `MailBrand` prop instead of token-replacement
+   on raw HTML strings.
+3. **Live preview** — `pnpm --filter @toolctrl/mail preview` boots
+   a local HTTP server that renders every template with realistic
+   sample data so designers can iterate without triggering live
+   email sends.
+
+## Adding a new template
+
+1. **Create the component** in `packages/mail/src/templates-react/transactional/<TemplateName>.tsx`.
+   Accept a `brand: MailBrand` prop plus the data the template needs.
+   Compose with `TransactionalScaffold` for the standard layout.
+   Avoid `dangerouslySetInnerHTML` — it is banned by code review.
+2. **Validate URLs** — every CTA must go through `<Button>`. The
+   component throws `InvalidActionUrlError` for anything other than
+   `https://` or `mailto:` URLs.
+3. **Register the template** in
+   `packages/mail/src/templates-react/render.ts`:
+   - Add the name to `ReactTransactionalTemplateName`
+   - Add the prop type to `ReactTemplatePropsMap`
+   - Add the component to the `COMPONENTS` map
+4. **Add a snapshot test** at
+   `packages/mail/src/templates-react/transactional/<TemplateName>.test.tsx`
+   covering: default render (HTML + text snapshots), tenant brand
+   override, XSS-safety on at least one user-supplied prop, and
+   URL-validation error if the template takes an `actionUrl`.
+5. **Add a preview entry** in `packages/mail/scripts/previews.ts`
+   so the new template renders in `pnpm preview`.
+
+## Brand theming
+
+Templates accept a `brand: MailBrand` prop with the following fields:
+
+| Field | Source | Notes |
+| --- | --- | --- |
+| `productName` | `@toolctrl/brand` `BrandTokens.productName` | Defaults to "ToolCTRL". |
+| `logoUrl` | `BrandTokens.assets.logoUrl` | Optional. Falls back to wordmark. |
+| `primaryColor` | `BrandTokens.mail.colors.primary` | CTA background. |
+| `primaryTextColor` | `BrandTokens.mail.colors.primaryText` | CTA foreground. |
+| `tenantName` | derived from tenant `displayName` | Used in greeting/footer copy. |
+| `supportEmail` / `supportFromName` | `BrandTokens.support.*` | Footer + sign-off. |
+| `privacyUrl` / `termsUrl` / `imprintUrl` | `BrandTokens.legal.*` | DSGVO/CAN-SPAM compliance. |
+
+`withBrandDefaults(override?)` merges a partial override into the
+platform defaults — call it once per render so missing tenant fields
+do not blank out the design.
+
+## Plain-text fallback
+
+`renderTransactionalEmail(template, props)` returns both `html` and a
+plain-text `text` variant. The text is produced from the rendered
+HTML by `htmlToPlainText` in `render-helpers.ts` — anchor URLs are
+preserved as `label (url)` so text-mode clients keep functional
+CTAs.
+
+Mailjet forwards both parts in a MIME-multipart envelope; clients
+choose which to display.
+
+## Testing
+
+- **Snapshot tests** are mandatory. Each template owns a
+  `__snapshots__/<TemplateName>.test.tsx.snap` capturing both the
+  HTML and the text output. Reviewers compare snapshot diffs against
+  the design intent.
+- **XSS-safety tests** verify that React's default escaping turns
+  `<script>` into `&lt;script&gt;` for at least one user-supplied
+  prop per template.
+- **URL-validation tests** verify that `javascript:`, `http://`,
+  `data:`, and malformed URLs are rejected when passed to `<Button>`.
+- Run `pnpm --filter @toolctrl/mail test` to execute the suite.
+
+## Live preview workflow
+
+```
+pnpm --filter @toolctrl/mail preview
+```
+
+Then open http://localhost:3001 — the index page lists every
+template and links to the rendered HTML plus a `?format=text` link
+for the plain-text fallback.
+
+Sample props live in `packages/mail/scripts/previews.ts`. Update
+them when adding a template; never commit real PII.
+
+## Render pipeline
+
+`renderTransactionalEmail(template, props)` from `@toolctrl/mail` is
+the canonical entry point. The pre-Stripe-19 string-template renderer
+(`renderTransactionalMailTemplate`) was retired in the legacy-cleanup
+pass; the host application's wrapper at
+`apps/web/app/modules/emails/utils/EmailTemplates.ts` adapts the
+legacy template-name surface onto the React-Email pipeline.
+
+| Legacy name | React template | Notes |
+| --- | --- | --- |
+| `welcome` | `welcome` | direct migration |
+| `password-reset` | `password-reset` | direct migration |
+| `user-invitation` | `invitation` | renamed for consistency |
+| `email-verification` | `magic-link` | merged into magic-link flow (30-min expiry variant) |
+| `account-setup` | `welcome` | unified onboarding (overridable subject/eyebrow/title/intro/actionLabel) |
+| — | `magic-link` | new |
+| — | `subscription-success` / `subscription-failed` | new (Stripe webhooks) |
+| — | `invoice-sent` | new (Stripe invoices) |
+| — | `tenant-notification` | new (in-app digest) |
+
+## Security notes
+
+- React's default escaping is the primary defense — manual
+  `escapeHtml` calls are no longer required and any new
+  template that performs string concatenation in HTML attributes
+  will be rejected in code review.
+- `<Button>` enforces an allow-list of URL schemes
+  (`https:`, `mailto:`) to prevent magic-link phishing via
+  `javascript:` or `data:` URLs.
+- `dangerouslySetInnerHTML` is forbidden in
+  `packages/mail/src/templates-react/`. If you need rich text,
+  build a structured prop shape and render it as JSX.
+- Brand logo URLs come from `packages/brand` which enforces an
+  https-only allow-list; the React templates never load
+  arbitrary tenant-supplied URLs.

hivegrid-cloud/docs/architecture/job-priority.md

@@ -0,0 +1,121 @@
+# Job Priority
+
+Plan-based BullMQ job priority. Lets enterprise tenants jump the queue
+over free-tier batches and pins internal/system jobs to the top.
+
+## Why
+
+Before this feature every BullMQ job ran at the default priority. A
+free-tier tenant scheduling 10 000 mail jobs would block an enterprise
+tenant's password-reset behind them. Cron and retention jobs competed
+with user workload. There was no way to express "this auth mail must
+run now" semantically.
+
+## Priority levels
+
+| Constant                 | Value | Use case                                                  |
+| ------------------------ | ----- | --------------------------------------------------------- |
+| `JOB_PRIORITY_CRITICAL`  | `1`   | Internal jobs, auth flows (password-reset, magic-link)    |
+| `JOB_PRIORITY_HIGH`      | `3`   | System-triggered workflows that aren't auth               |
+| `JOB_PRIORITY_NORMAL`    | `5`   | Default for tenant-scoped jobs without a plan lookup      |
+| `JOB_PRIORITY_LOW`       | `10`  | Marketing / bulk batches; free-tier when a lookup is set  |
+
+BullMQ treats `priority: 1` as highest. We intentionally use only 4
+distinct levels — Grafana dashboards stay legible and the priority
+sorted-set in Redis stays small.
+
+## Resolution order (in `enqueueJob`)
+
+```
+1. options.priority (explicit numeric)      → wins everything
+2. options.priorityOverride ('critical'…)   → mapped via priorityFromOverride
+3. options.tenantId === null                → CRITICAL (internal job)
+4. options.tenantId set, lookup registered  → resolveJobPriorityForTenant
+5. options.tenantId set, no lookup          → NORMAL
+6. options absent                           → no priority field set
+```
+
+## Plan → priority mapping
+
+| Plan slug    | Priority           |
+| ------------ | ------------------ |
+| `enterprise` | CRITICAL (1)       |
+| `pro`        | NORMAL (5)         |
+| `free`       | LOW (10)           |
+| unknown/null | NORMAL (5)         |
+
+The mapping lives in `packages/queue/src/priority.ts` in
+`priorityFromPlan(plan)`.
+
+## Producer guidelines
+
+1. **Always pass `tenantId`** for tenant-scoped jobs. The descriptor
+   already carries the tenantId in most cases; pass the same value to
+   the enqueue options:
+   ```ts
+   await enqueueJob(queue, descriptor, { tenantId });
+   ```
+2. **Pass `tenantId: null` for internal jobs** — retention, cleanup,
+   audit-compaction, backup triggers. They get CRITICAL automatically.
+3. **Use `priorityOverride: 'critical'` for auth flows** regardless of
+   plan: password-reset, email-verification, magic-link, user
+   invitation. A free-tier user must still receive their reset mail
+   instantly.
+4. **Use `priorityOverride: 'low'` for marketing batches.** Don't let
+   newsletter batches starve transactional mail.
+5. **Never set `priorityOverride: 'critical'` from untrusted input.**
+   The override hook is server-only.
+
+## Registering a tenant-plan lookup
+
+The queue package keeps its plan-lookup pluggable so workers don't
+have to import apps/web. Web boots the lookup during cache bootstrap:
+
+```ts
+// apps/web/app/utils/services/.server/tenantPlanLookup.server.ts
+setSubscriptionToPlanTier((subscription) => {
+  // your classifier here — e.g. map stripe product ids to plan slugs
+  return subscription?.products[0]?.subscriptionProductId === "prod_enterprise"
+    ? "enterprise"
+    : subscription?.products.length
+      ? "pro"
+      : "free";
+});
+wireTenantPlanLookup();
+```
+
+Until a classifier is wired, the resolver returns NORMAL for every
+tenant (CRITICAL still applies to internal jobs).
+
+## Feature-flag
+
+`JOB_PRIORITY_ENABLED=false` neutralises plan-based priority — every
+tenant-scoped enqueue receives NORMAL (5). Internal jobs still get
+CRITICAL because `tenantId: null` short-circuits before the flag is
+checked. Use this for emergency rollback without a code change.
+
+## Monitoring
+
+The Counter `queue_jobs_enqueued_total{queue, priority}` increments
+on every enqueue. The `priority` label is one of
+`critical|high|normal|low|other` (low-cardinality).
+
+Useful Grafana queries:
+
+```promql
+# Enqueue rate by priority for a queue
+sum by (priority) (rate(queue_jobs_enqueued_total{queue="toolctrl.email"}[5m]))
+
+# Share of critical vs total
+sum(rate(queue_jobs_enqueued_total{priority="critical"}[5m]))
+/
+sum(rate(queue_jobs_enqueued_total[5m]))
+```
+
+## Rollback
+
+1. Set `JOB_PRIORITY_ENABLED=false` in env → next enqueue runs with
+   NORMAL.
+2. To remove the feature entirely: revert the producer-call-site
+   commits; the resolver helper stays but goes unused.
+3. In-flight jobs keep the priority assigned at enqueue time.

hivegrid-cloud/docs/architecture/lighthouse.md

@@ -0,0 +1,190 @@
+# Lighthouse-CI audit and Web-Vitals reporting
+
+This document describes how the Lighthouse-CI audit is wired into the
+Toolctrl Boilerplate, how to run it locally when debugging a regression,
+and how the Web-Vitals → Sentry bridge surfaces real-user metrics.
+
+Spec: `docs/improvement-plan/10-compliance-polish/lighthouse.md`
+(Task #70). Companion artefacts: `.lighthouserc.staging.json`,
+`.lighthouserc.json`, `scripts/lighthouse-puppeteer-auth.cjs`,
+`scripts/rotate-lighthouse-session-cookie.mjs`,
+`.github/workflows/lighthouse.yml`, `.gitea/workflows/lighthouse.yml`,
+`apps/web/app/lib/observability/web-vitals.ts`,
+`apps/web/app/lib/pwa/registerServiceWorker.ts`,
+`apps/web/public/manifest.webmanifest`, `apps/web/public/sw.js`.
+
+## Where the audit runs
+
+A meaningful Lighthouse audit needs a live server, a database and seeded
+content. Reproducing that whole stack inside a CI job is brittle, so the
+CI workflow audits the **deployed staging site**
+(`https://boiler.saas-ctrl.com`) rather than a CI-local server.
+
+That makes it a **post-deploy / monitoring audit, not a per-commit merge
+gate** — staging always lags the commit under review. The workflow
+(`.gitea/workflows/lighthouse.yml`, mirrored in `.github/`) runs:
+
+- **nightly** at 03:00 UTC (`schedule`), and
+- **on demand** (`workflow_dispatch`).
+
+A red run means the deployed staging site fell below a threshold — treat
+it as a production-quality alarm, not a blocked PR.
+
+The Gitea runner image (`node:20-bookworm`) ships no browser, so the
+workflow installs `chromium` via `apt-get` and points LHCI at it with
+`CHROME_PATH=/usr/bin/chromium`. The GitHub runner (`ubuntu-latest`)
+already bundles Chrome.
+
+## Audit targets
+
+`.lighthouserc.staging.json` audits the public staging routes — the
+surface that renders without authentication:
+
+| URL                                          |
+| -------------------------------------------- |
+| `https://boiler.saas-ctrl.com/login`         |
+| `https://boiler.saas-ctrl.com/privacy-policy` |
+| `https://boiler.saas-ctrl.com/terms-and-conditions` |
+
+Only **Performance** is a hard gate (`error`). The product is a B2B
+SaaS, not a search-discoverable marketing site, so the SEO category is
+not asserted at all. Accessibility and Best-Practices stay visible as
+`warn` — accessibility is already hard-gated separately by `a11y.yml`
+(axe-core plus the `jsx-a11y` ESLint baseline, task #73), so a second
+blocking gate here would be redundant.
+
+| Category        | Threshold | Severity                  |
+| --------------- | --------- | ------------------------- |
+| Performance     | 0.95      | error                     |
+| Accessibility   | 0.95      | warn                      |
+| Best Practices  | 0.95      | warn                      |
+| PWA             | 0.90      | warn                      |
+| SEO             | —         | not asserted (irrelevant) |
+
+Numeric audits (FCP / LCP / TBT / CLS) stay at `warn` severity so trend
+regressions remain visible without failing the run.
+
+## Session cookie rotation
+
+The workflow mints a fresh staging session before every LHCI run via
+`scripts/rotate-lighthouse-session-cookie.mjs`. The script logs in as the
+dedicated `lighthouse-bot` account, extracts the CSRF token from
+`/login`, posts the login form with the CSRF header, then appends only
+the resulting session cookie to the Actions environment. It never prints
+the cookie or password.
+
+Required workflow secrets:
+
+| Secret | Purpose |
+| ------ | ------- |
+| `LIGHTHOUSE_LOGIN_EMAIL` | Email of the staging Lighthouse bot user. |
+| `LIGHTHOUSE_LOGIN_PASSWORD` | Password of the staging Lighthouse bot user. |
+
+The session cookie name is `RJ_session`; workflows also pass
+`LIGHTHOUSE_SESSION_COOKIE_NAME=RJ_session` and
+`LIGHTHOUSE_COOKIE_DOMAIN=boiler.saas-ctrl.com`.
+
+During rollout the workflow still forwards the older
+`LIGHTHOUSE_SESSION_COOKIE` secret as a fallback. If the login email or
+password secret is absent, the script reuses that pre-provisioned cookie
+without printing it. Permanent operation should use the login secrets so
+each run mints a fresh cookie.
+
+## Authenticated routes
+
+Auditing `/app/$tenant/*` and `/admin/*` needs that freshly minted
+`RJ_session` cookie. `scripts/lighthouse-puppeteer-auth.cjs` injects the
+cookie for protected URLs and is already wired into
+`.lighthouserc.staging.json`. The current staging target list still
+audits only public routes; add protected URLs once the exact tenant/admin
+coverage list is chosen. Without a protected URL in the config, the
+cookie step is a readiness/rotation guard rather than an auth coverage
+expansion.
+
+## Running Lighthouse locally
+
+`.lighthouserc.json` is the local-iteration config (URLs on
+`http://localhost:3000`); `pnpm verify:lighthouse` runs it via
+`scripts/run-lighthouse.mjs`. Bring the app up first:
+
+```sh
+# Build apps/web (writes hashed assets the runtime SW expects to cache)
+pnpm --filter @toolctrl/web build
+
+# Serve the build at http://localhost:3000.
+pnpm --filter @toolctrl/web start &
+
+# Run the full LHCI flow against the local server
+pnpm verify:lighthouse
+```
+
+To reproduce a CI staging finding locally, point LHCI at the staging
+config instead — no local server needed, just a Chrome install:
+
+```sh
+pnpm dlx @lhci/cli@0.13.x autorun --config=./.lighthouserc.staging.json
+```
+
+The run prints the temporary-public-storage report URLs in its output;
+share those when debugging a regression.
+
+## PWA manifest + service-worker
+
+`apps/web/public/manifest.webmanifest` describes the installable app.
+`apps/web/public/sw.js` is a minimal cache-first / network-first service
+worker registered from `root.tsx` via `registerServiceWorker`.
+
+Cache strategy:
+
+- `/assets/...` (hashed by React-Router build) → **cache-first**
+- `/api/...`, `/app/...`, `/admin/...` → **network-first** with cache fallback
+- everything else (marketing routes, brand assets) → **network-first**
+
+The `CACHE_VERSION` constant inside `sw.js` is the invalidation lever —
+bump it on any deploy that changes the asset graph in a backward-
+incompatible way. Old caches are dropped automatically in the activate
+handler.
+
+To debug a stuck SW, open DevTools → Application → Service Workers and
+click "Unregister". The next page load picks up the new `sw.js`.
+
+## Web-Vitals → Sentry
+
+`apps/web/app/lib/observability/web-vitals.ts` registers callbacks for
+CLS, LCP, INP, TTFB, and FCP via the `web-vitals` library. Each metric is
+captured as a Sentry message named `web-vitals.<metric>` with tags:
+
+```text
+web.vital.name   = CLS | LCP | INP | TTFB | FCP
+web.vital.rating = good | needs-improvement | poor
+```
+
+In the Sentry dashboard:
+
+1. Filter to `message:"web-vitals.LCP"` to view a metric in isolation.
+2. Group by `tags.web.vital.rating` to see the share of "poor" loads.
+3. Group by `transaction` (Sentry React-Router integration) to compare
+   metrics across routes.
+
+The helper is SSR-safe (bails on `typeof window === "undefined"`) and
+silently no-ops when Sentry isn't initialised, so the same code runs in
+dev, preview, staging, and production without extra gating.
+
+## Troubleshooting CI runs
+
+| Symptom                                            | Likely cause / fix                                                            |
+| -------------------------------------------------- | ----------------------------------------------------------------------------- |
+| `Chrome installation not found` / `Healthcheck failed!` | The runner has no browser. The Gitea workflow installs `chromium`; confirm the install step ran and `CHROME_PATH` is set. |
+| Cookie rotation step fails with missing env vars    | Add `LIGHTHOUSE_LOGIN_EMAIL` and `LIGHTHOUSE_LOGIN_PASSWORD` as repo Actions secrets. |
+| Audit reports the `/login` page for a protected URL | Cookie injection failed or the bot no longer belongs to the tenant/admin surface being audited. |
+| Performance score < 0.95 only on first run         | Cold cache; LHCI reruns automatically (`numberOfRuns=3`).                      |
+| Audit times out                                    | Lower `numberOfRuns` to 1 temporarily while iterating; restore afterwards.     |
+| All URLs fail to load                              | Staging is down or redeploying — check the `web` container on staging.         |
+
+## Updating thresholds
+
+Permanent relaxation requires a `.lighthouserc.staging.json` commit plus
+a paragraph in this document explaining the trade-off. Prefer switching
+a category from `error` to `warn` over dropping it, so regressions stay
+visible — SEO is the one deliberate exception (not relevant to a B2B
+SaaS, so not asserted at all).

hivegrid-cloud/docs/architecture/outbox-pattern.md

@@ -0,0 +1,270 @@
+# Transactional Outbox Pattern
+
+The outbox pattern keeps side-effects (mail sends, webhooks, audit logs,
+workflow triggers) **transactionally consistent** with the database writes
+that triggered them. Without it, a crash between `DB COMMIT` and "call
+mailer" leaves the DB updated but the side-effect lost.
+
+This document describes how to publish and consume outbox events in this
+codebase. See `docs/improvement-plan/03-scaling-infrastructure/outbox-pattern.md`
+for the broader design rationale.
+
+## When to publish
+
+Publish **inside** a Prisma `$transaction` block whenever a successful DB
+write should fan out an asynchronous side-effect.
+
+```ts
+import { publishOutboxEvent } from "@toolctrl/db/outbox";
+
+await prisma.$transaction(async (tx) => {
+  const subscription = await tx.subscription.update({
+    where: { id },
+    data: { status: "active" },
+  });
+
+  await publishOutboxEvent(tx, {
+    tenantId: subscription.tenantId,
+    type: "billing.subscription.updated",
+    payload: { subscriptionId: subscription.id, status: "active" },
+    dispatchKey: `stripe:${stripeEvent.id}`, // optional idempotency key
+  });
+});
+```
+
+`publishOutboxEvent` is type-constrained to require a
+`Prisma.TransactionClient` — calling it with a top-level `PrismaClient`
+fails to type-check.
+
+## Event-type naming convention
+
+Events follow `<domain>.<entity>.<verb>`, lowercase, dot-separated. The
+dispatcher routes by the first segment by default:
+
+| Prefix                   | Target BullMQ queue            |
+| ------------------------ | ------------------------------ |
+| `tenant.subscription.`   | `outbox.notification`          |
+| `mail.`                  | `outbox.mail`                  |
+| `webhook.`               | `outbox.webhook`               |
+| `notification.`          | `outbox.notification`          |
+| `audit.`                 | `outbox.audit`                 |
+| `workflow.`              | `outbox.workflow`              |
+| `search.index.`          | `outbox.search-index`          |
+| `rag.index.`             | `outbox.rag-index`             |
+| `ai.cost.`               | `outbox.ai-cost`               |
+| `marketing.lead.`        | `outbox.marketing`             |
+| `form.submission.`       | `outbox.form-submission-route` |
+| `billing.`               | `outbox.notification`          |
+| `plugin.hook.`           | `outbox.plugin-hook`           |
+| `plugin.capability-use.` | `outbox.plugin-capability-use` |
+
+Add a routing rule before introducing a new prefix — events with no
+matching rule go straight to FAILED with a Sentry alert. See
+`apps/worker/src/outbox/event-router.ts`.
+
+Workflow routing has an active receiver/subscriber path when the worker is
+configured for the `general` queue group with the workflow execution settings.
+`apps/worker/src/jobs/workflow-trigger.ts` parses `workflow.*` outbox jobs,
+applies receiver-side idempotency, fans out to matching live event-trigger
+workflows, starts queued manual `workflow.execution.requested` executions, and
+resumes queued `workflow.execution.continue.requested` continuations through the
+selected executor. HTTP mode uses the signed internal bridge; hybrid/native mode
+can resume worker-native safe-core continuations when the only native preflight
+gap is the current `waitForInput` boundary. Candidates that fail preflight are
+skipped or recorded as rejected.
+`apps/worker/src/jobs/workflow-trigger-store.ts` provides the worker-native
+Prisma lookup/preflight port. Application events publish
+`workflow.trigger.event` through `EventsService`. The route-backed row mutation
+producers for `row.created`, `row.updated`, and `row.deleted` record the row
+write, application event, and workflow outbox event in the same transaction
+before post-commit notifications/webhook callouts run. Internal mutations
+without a request context remain intentionally eventless unless the caller opts
+into an event context. Workflow row blocks pass a `workflow` event context so
+requestless `rowCreate`, `rowUpdate`, and `rowDelete` blocks still publish row
+events without enabling external webhook callouts. A source-inventory test guards the current execution
+boundary: manual and tenant API runs must publish
+`workflow.execution.requested` instead of executing inline, non-stream
+continuations must publish `workflow.execution.continue.requested`, only the
+internal workflow bridge may execute
+`type: "trigger"` directly, and the legacy `WorkflowEngine.trigger(...)`
+facade must stay absent.
+
+The worker-native block contract is intentionally explicit. `email` is the
+first-class mail block, `httpRequest` is the outbound/webhook-style block, and
+conditions are represented by `if` plus `switch`. There is no separate
+`mail`, `webhook`, `notification`, `audit`, `delay`, or `schedule` block id in
+the registry today; adding any of those requires updating
+`packages/foundation/src/workflows.ts`, its capability matrix tests, workflow
+UI i18n, and worker-native execution/preflight behavior together.
+
+Operationally, the outbox dispatcher and workflow subscriber are separate from
+mail delivery workers. Production-like Coolify stacks need a persistent
+`worker-general` (`WORKER_QUEUE_GROUP=general`) for the dispatcher and
+`outbox.workflow`, plus a persistent `worker-mail`
+(`WORKER_QUEUE_GROUP=mail`) for `outbox.mail`, `outbox.webhook`,
+`outbox.marketing`, and `toolctrl.email` work. A green outbox row does not prove
+that both services exist or that the downstream BullMQ job finished.
+
+### Non-durable exception: stream runs
+
+Workflow stream starts remain an explicit exception to the worker-durable
+execution boundary. The stream route starts through
+`WorkflowsExecutionsService.stream(...)`, creates a running execution row, and
+then calls `execute(...)` in the web process so the existing SSE/emitter
+progress contract can deliver local run updates to the browser. Web restarts can
+interrupt an in-flight stream run before the worker can resume it. Stream
+continuations are already queued through `workflow.execution.continue.requested`;
+moving stream starts to the worker requires a distributed progress channel or
+SSE broker rather than a partial outbox migration.
+
+The worker bridge has an opt-in real-Postgres regression test at
+`apps/worker/src/jobs/workflow-trigger-e2e.test.ts`. It skips without a real
+database URL and runs with `WORKFLOW_TRIGGER_E2E_DATABASE_URL`, falling back to
+`OUTBOX_E2E_DATABASE_URL` or any non-stub `DATABASE_URL`. The test publishes a
+workflow trigger outbox event, claims and routes it through the production
+dispatcher seam, then executes a live safe-core workflow through the native
+worker runner and verifies idempotent re-delivery. It also verifies the
+`workflow.executed` plugin hook path by routing `plugin.hook.workflow.executed`
+through `outbox.plugin-hook` fanout into `toolctrl.plugin-invoke`. A paired
+failure-path fixture forces an unsupported native block, verifies terminal
+`status: "error"` persistence, then routes `plugin.hook.workflow.failed`
+through the same fanout boundary into `toolctrl.plugin-invoke`.
+
+The canonical non-browser readiness gate for this path is
+`pnpm run verify:workflow-outbox-native`. It composes the existing focused
+worker, foundation, and source-inventory tests for `outbox.workflow` routing,
+workflow-trigger subscriber bootstrap, native/hybrid executor selection,
+credential-free native execution boundaries, worker health fail-loud settings,
+and queued manual/API/continuation producers. It is intentionally not a live
+drain probe; use the opt-in real-Postgres regression or the operator-gated E2E
+drain readiness check when an environment-level worker/Redis/Postgres proof is
+needed.
+
+## Dispatcher
+
+`apps/worker/src/outbox/dispatcher.ts` polls every 1 second (configurable
+via `OUTBOX_DISPATCH_INTERVAL_MS`). Each tick:
+
+1. **Atomic claim**: `UPDATE … SELECT … FOR UPDATE SKIP LOCKED RETURNING *`
+   moves up to `OUTBOX_DISPATCH_BATCH_SIZE` (default 100) PENDING rows to
+   PROCESSING with `lockedUntil = now() + 5 min`. Two dispatchers can run
+   concurrently without ever claiming the same event.
+2. **Crash recovery**: rows stuck in PROCESSING past `lockedUntil` are
+   re-claimed automatically.
+3. **Routing**: each event's `eventType` is mapped to a BullMQ queue via
+   the event router.
+4. **Enqueue + status update**: claimed rows are processed with bounded
+   concurrency (`OUTBOX_DISPATCH_CONCURRENCY`, default 10) so a large
+   batch does not serialize every Redis enqueue and status update.
+   The default BullMQ enqueue path adds receiver-level retries with exponential
+   backoff before marking the outbox row `DONE`; transient subscriber failures
+   are retried by the target queue instead of being lost after dispatch.
+   It also uses a deterministic BullMQ `jobId` of `outbox:<eventId>` so a
+   dispatcher crash after enqueue but before the Postgres `DONE` update does
+   not create a second receiver job when the same outbox row is re-claimed.
+   - success → `status = DONE`, `processedAt = now()`
+   - failure → increment `retries`, `scheduledAt = now() + backoff(retries)`
+     with backoff `[1, 2, 4, 8, 16, 32]` minutes.
+   - retries > `OUTBOX_MAX_RETRIES` (default 6) → `status = FAILED` (dead-
+     letter).
+
+`DONE` means the dispatcher successfully handed the event to the target BullMQ
+queue and recorded that handoff in Postgres. It does not mean the subscriber
+processed the job, the provider call succeeded, or a domain state machine
+reached a terminal success state. For workflow events, operators must also
+watch the `outbox.workflow` BullMQ job state and the associated
+`WorkflowExecution` status. For mail/webhook events, also watch the
+`worker-mail` subscriber queues and the `toolctrl.email` job state.
+
+The dispatcher is enabled by `OUTBOX_DISPATCHER_ENABLED=true`. Defaults to
+**off** in dev so a missing Redis URL does not crash the worker.
+
+## Retention
+
+`worker-general` registers a daily `cleanup.run` maintenance job named
+`outbox-done-retention`. It deletes only rows with `status = DONE` and a
+non-null `processedAt` older than `OUTBOX_DONE_RETENTION_HOURS` (default 720
+hours, 30 days). The deletion is bounded by `OUTBOX_DONE_RETENTION_BATCH_SIZE`
+(default 1000, clamped to 1..10000) and uses `FOR UPDATE SKIP LOCKED`, so large
+first runs drain gradually without blocking the dispatcher.
+
+FAILED and CANCELLED rows are intentionally retained for audit and incident
+review. Deleting DONE rows releases their `dispatchKey` uniqueness after the
+retention window; producers that require longer idempotency windows must use a
+domain-level idempotency key as well.
+
+## Receiver-side idempotency
+
+BullMQ delivery is at-least-once: a receiver crash between
+"job processed" and "ack" causes a redelivery. Wrap your handler in
+`withOutboxIdempotency` so duplicate deliveries are a no-op.
+
+```ts
+import { withOutboxIdempotency } from "@toolctrl/db/outbox-receiver";
+import { createIdempotencyStore } from "@toolctrl/shared-store";
+
+const idempotencyStore = createIdempotencyStore({
+  store: sharedStore,
+  prefix: "outbox",
+  ttlSeconds: 60 * 60 * 24, // 24h — longer than the longest retry window
+});
+
+await withOutboxIdempotency(idempotencyStore, job.data.outboxEventId, async () => {
+  await sendMail(job.data.payload);
+});
+```
+
+`claim()` does an atomic `SET NX EX`. If the claim fails (another worker
+already processed the event), the handler is skipped and the cached
+result returned.
+
+## Operational
+
+### Pending count alert
+
+`SELECT count(*) FROM "OutboxEvent" WHERE status = 'PENDING' AND scheduledAt < now() - interval '60 seconds'`
+
+Alert when > 1000 for > 5 minutes — that means the dispatcher is behind.
+
+### Downstream queue alert
+
+Outbox `DONE` counts are not sufficient for worker health. Alert separately on
+target BullMQ queue depth, failed jobs, and stalled jobs for `outbox.workflow`,
+`outbox.mail`, `outbox.webhook`, `outbox.marketing`, and `toolctrl.email`.
+Workflow incidents must include the `WorkflowExecution` terminal status because
+`workflow.execution.requested` can be handed to BullMQ successfully while the
+execution later fails, stalls, or remains queued behind a missing bridge.
+
+### Re-queueing failed events
+
+Use `toolctrl-cli outbox requeue` after fixing the underlying receiver.
+The CLI supports single-row and filtered bulk requeues, `--dry-run`, JSON
+output, and guardrails that avoid replaying `PENDING` / `PROCESSING`
+rows. Prefer it over raw SQL so `processedAt`, `lastError`, retry state,
+and replay warnings stay consistent.
+
+Raw SQL is an emergency-only fallback for a DBA-led incident when the CLI
+cannot run; record the exact statement and reason in the incident log.
+
+### Environment
+
+| Variable                      | Default | Purpose                                           |
+| ----------------------------- | ------- | ------------------------------------------------- |
+| `OUTBOX_DISPATCHER_ENABLED`   | `false` | Master switch for the dispatcher worker           |
+| `OUTBOX_DISPATCH_INTERVAL_MS` | `1000`  | Polling cadence (ms)                              |
+| `OUTBOX_DISPATCH_BATCH_SIZE`  | `100`   | Max events claimed per tick                       |
+| `OUTBOX_DISPATCH_CONCURRENCY` | `10`    | Max claimed events processed in parallel per tick |
+| `OUTBOX_MAX_RETRIES`          | `6`     | Retries before dead-letter                        |
+| `OUTBOX_DONE_RETENTION_HOURS` | `720`   | DONE-row replay/debug retention window            |
+| `OUTBOX_DONE_RETENTION_BATCH_SIZE` | `1000` | Max DONE rows purged per maintenance run       |
+
+## Trade-offs
+
+- **Latency**: ~1s polling tick. A LISTEN/NOTIFY variant could cut that
+  to <10ms; deferred until needed.
+- **At-least-once delivery**: receivers MUST be idempotent
+  (`withOutboxIdempotency` is the cheap path).
+- **Storage**: each successful event leaves a DONE row until the
+  `outbox-done-retention` job purges DONE rows older than
+  `OUTBOX_DONE_RETENTION_HOURS`. FAILED rows are kept for operator triage and
+  compliance review.

hivegrid-cloud/docs/architecture/plugin-hooks.md

@@ -0,0 +1,196 @@
+# Plugin Hooks & Lifecycle
+
+**Phase 06 Task #22 foundation.** Builds on the WASM sandbox runtime from #21.
+
+## What ships
+
+This task delivers the _foundation_ for plugin host integration. Marketplace
+browsing, hot-reload, per-instruction fuel, the full 15-20 hook-point catalogue
+and broader marketplace workflows are deferred to follow-up work (`#23`
+audit-docs + backlog/marketplace). Tenant plugin audit events are now exposed
+through `apps/web/app/routes/app.$tenant/settings/plugins.$id.audit.tsx`.
+
+### 1. Hook registry — `packages/plugin-runtime/src/hooks/registry.ts`
+
+A central, immutable map of hook-name → zod payload schema. Foundation
+hooks are pre-registered:
+
+| Hook name           | Outbox event type               | When it fires                                            |
+| ------------------- | ------------------------------- | -------------------------------------------------------- |
+| `tenant.created`    | `plugin.hook.tenant.created`    | `createTenant` row insert succeeded                      |
+| `user.created`      | `plugin.hook.user.created`      | `register` user row insert succeeded                     |
+| `entity.created`    | `plugin.hook.entity.created`    | `RowsApi.create` after row + log write                   |
+| `entity.updated`    | `plugin.hook.entity.updated`    | `RowsApi.update` after row + log write                   |
+| `entity.deleted`    | `plugin.hook.entity.deleted`    | `RowsApi.del` after hardDelete                           |
+| `workflow.created`  | `plugin.hook.workflow.created`  | Tenant-scoped workflow definition is created through reviewed service/import paths |
+| `workflow.updated`  | `plugin.hook.workflow.updated`  | Tenant-scoped workflow definition is updated through reviewed service paths |
+| `workflow.deleted`  | `plugin.hook.workflow.deleted`  | Tenant-scoped workflow definition is deleted through reviewed service paths |
+| `workflow.executed` | `plugin.hook.workflow.executed` | Tenant-scoped workflow execution reaches terminal status |
+| `workflow.failed`   | `plugin.hook.workflow.failed`   | Tenant-scoped workflow execution reaches terminal error status |
+
+Adding a new hook is a single `registerHook({ name, outboxEventType,
+payloadSchema })` call at module load time. The registry validates that the
+event type matches `plugin.hook.<entity>.<verb>` so the outbox dispatcher's
+existing routing rules cover it without changes.
+
+### 2. Hook emission — `packages/plugin-runtime/src/hooks/emit.ts`
+
+`emitPluginHook(tx, def, payload, publish)` validates the payload and forwards
+to the outbox publisher. The host-side wrapper
+`apps/web/app/modules/plugins/services/.server/pluginHookPublisher.ts` owns a
+one-statement Prisma transaction so callers stay a single line:
+
+```ts
+await emitPluginHookFireAndForget(PLUGIN_HOOK_TENANT_CREATED, {...}, {
+  tenantId,
+  dispatchKey: `tenant:${id}:created`,
+});
+```
+
+The wrapper is intentionally **non-fatal to the surrounding business write**:
+hook delivery failures are logged but never propagate into tenant signup, row
+CRUD, or workflow service calls. It still returns a promise, and reviewed
+workflow definition producers await it so the one-shot outbox transaction is
+attempted before the service returns. A malformed payload triggers
+`PluginHookValidationError`, which is also swallowed and logged.
+
+### 3. Plugin lifecycle service — `apps/web/app/modules/plugins/services/.server/PluginLifecycleService.ts`
+
+`installPlugin`, `enablePlugin`, `disablePlugin`, `listPlugins`. Each mutation
+writes a hash-chained audit-event row through the existing
+`createAuditEvent` helper. The service is permission-guarded at the route
+layer: read-only list/settings views require `app.plugins.use`; capability-use
+audit trails require `app.settings.auditTrails.view`; install, capability grant,
+enable, and disable require `app.plugins.manage`.
+
+### 4. Tenant admin UI — `apps/web/app/routes/app.$tenant/settings/plugins.tsx`
+
+A minimal settings page that lists installed plugins with their capabilities,
+manifest preview, and an enable/disable toggle. CSRF-validated form
+submissions go through the lifecycle service.
+
+### 5. Plugin migration applier — `packages/plugin-runtime/src/migrations.ts`
+
+`applyPluginMigration({ pluginId, sql }, deps)` parses the SQL, enforces
+`plugin_<sanitized-pluginId>_` table prefix, requires a `tenantId` column on
+every CREATE TABLE, and rejects anything outside the `CREATE TABLE / CREATE
+INDEX` allowlist. Forbidden tokens (GRANT, DROP DATABASE, COPY, pg\_\*, etc.)
+are caught at validation time so the host never runs them.
+
+### 6. Crash tracker — `packages/plugin-runtime/src/crash-tracker.ts`
+
+SharedStore-backed sliding window per `(pluginId, tenantId)`. Failures
+increment the counter (TTL = configurable window, default 10 minutes).
+Successes decrement (floored at 0). Once the threshold (default 10) is
+crossed the tracker calls the injected `disablePlugin` seam, calls the
+optional `notify` seam, and resets the counter so a re-enable starts
+fresh. Wired into `apps/worker/src/jobs/plugin-invoke.ts` via an optional
+`recordOutcomeForCrashTracking` seam; `plugin_not_found` / `plugin_disabled`
+outcomes are deliberately skipped so configuration glitches don't trigger
+auto-disable.
+
+### 7. Workflow definition hooks
+
+`apps/web/app/modules/workflowEngine/services/WorkflowsService.ts` emits
+metadata-only `workflow.created`, `workflow.updated`, and `workflow.deleted`
+hooks for tenant-scoped service-path definition writes and reviewed template
+imports. The reviewed create/update/delete/import paths await the publisher
+attempt after the workflow write succeeds; publisher failures are recorded by
+the hook publisher logger and remain non-fatal to the workflow mutation.
+Platform-scoped definitions and writes that bypass those reviewed paths remain
+outside current producer coverage.
+
+### 8. Workflow execution hook — `workflow.executed`
+
+`apps/web/app/modules/workflowEngine/services/WorkflowsExecutionsService.ts`
+emits `workflow.executed` after `updateWorkflowExecution` returns a terminal
+tenant-scoped `success` or `error` status. The payload intentionally carries
+only execution metadata (`tenantId`, `workflowId`, `workflowExecutionId`,
+`type`, `status`, `durationMs`, `errorCode`, `appliesToAllTenants`,
+`occurredAt`). `errorCode` is null for `success` and required for `error`. Raw
+error strings, workflow input, and workflow output bodies are not included
+because they can contain provider responses, URLs, or secret-like values.
+
+Failed executions also emit `workflow.failed` with the same metadata-only shape
+but a fixed `status: "error"` and required `errorCode`. The separate failure
+hook lets plugins subscribe only to failure automation while existing
+`workflow.executed` subscribers continue to receive the full terminal-status
+stream.
+
+Current producer coverage for workflow lifecycle hooks:
+
+| Lifecycle event | Hook(s) | Producer coverage | Payload boundary |
+| --------------- | ------- | ----------------- | ---------------- |
+| Tenant-scoped definition create | `workflow.created` | `WorkflowsService.create` and reviewed tenant template imports | Metadata only; no blocks, credentials, or raw input |
+| Tenant-scoped definition update | `workflow.updated` | `WorkflowsService.update` | Metadata plus changed field names only |
+| Tenant-scoped definition delete | `workflow.deleted` | `WorkflowsService.del` | Metadata only |
+| Tenant-scoped web/API/manual execution success | `workflow.executed` | `WorkflowsExecutionsService` | Terminal metadata, `errorCode: null` |
+| Tenant-scoped web/API/manual execution error | `workflow.executed`, `workflow.failed` | `WorkflowsExecutionsService` | Terminal metadata, stable `errorCode`; no raw error |
+| Worker-native trigger execution | `workflow.executed`, `workflow.failed` | `apps/worker/src/jobs/workflow-trigger-native-executor.ts` | Terminal metadata only; failures use the stable `workflow_execution_failed` code |
+
+### 9. Worker-side hook fanout — `apps/worker/src/jobs/plugin-hook-fanout.ts`
+
+The outbox dispatcher routes `plugin.hook.*` events to
+`outbox.plugin-hook` on the `general` worker group. The fanout consumer
+loads enabled plugins for the tenant, filters manifests that declare the
+hook, and enqueues one idempotent `plugin.invoke` job per subscriber on
+the isolated `toolctrl.plugin-invoke` queue. Plugin execution itself stays
+on `WORKER_QUEUE_GROUP=plugin`.
+
+### 10. Per-user enablement — `prisma/schema/plugins.prisma`
+
+Adds `PluginUserConfig { id, pluginId, userId, enabled, configJson }` with
+unique `(pluginId, userId)` and FK cascade on Plugin delete. User-scoped hook
+fanout loads `PluginUserConfig.enabled` when the validated hook payload includes
+`userId`; `enabled === false` skips that user's plugin invocation while
+tenant-default subscribers remain eligible.
+
+### 11. Operator CLI — `packages/cli/src/commands/plugins/*`
+
+```
+toolctrl-cli plugins list [--tenantId X] [--enabled true|false]
+toolctrl-cli plugins enable --tenantId X --pluginId com.acme.slack
+toolctrl-cli plugins disable --tenantId X --pluginId com.acme.slack
+toolctrl-cli plugins uninstall --tenantId X --pluginId com.acme.slack
+```
+
+## Remaining Slices
+
+Current status and operator-facing caveats live in `docs/features/plugins.md`
+and `docs/operations/plugin-lifecycle.md`. The historical implementation plan
+in `docs/improvement-plan/06-plugin-system/hooks-lifecycle.md` remains useful
+context, but it is not the current source of truth.
+
+- The remaining hook points (`onWebhookOutbound`, `onEmailSend`,
+  `afterAuthLogin`, billing, integration-connected,
+  api-key-created, chatbot-message, export-run, etc.).
+- Synchronous output hooks (`emitHookSync` with a 500ms wall-clock budget
+  for `transform` / `cancel` capabilities).
+- Dedicated marketplace/install dialog UX. The operator-side install path runs
+  through manifest verification, rejects unknown hook names, and tenant
+  settings already approve missing manifest-declared grants via checkbox plus a
+  server-verified digest.
+- Plugin secret storage and encrypted host bindings remain pending. The
+  reusable settings iframe shell already enforces opaque-origin sandboxing,
+  CSP, and a versioned postMessage protocol; manifest `ui.settings.entry`
+  metadata is routed through the server-side bundle loader, and non-secret
+  user config persists through `PluginUserConfig.configJson`.
+- Marketplace browsing + plugin signing-key rotation UI.
+- Hot-reload + per-instruction fuel limit.
+- Global (cross-tenant) crash counter for operator-level auto-disable.
+
+## Operational notes
+
+- `PLUGIN_AUTO_DISABLE_THRESHOLD` (default 10) and
+  `PLUGIN_AUTO_DISABLE_WINDOW_MS` (default 600_000) are read by the
+  worker bootstrap when constructing the crash tracker.
+- The outbox dispatcher routes `plugin.hook.*` events to
+  `outbox.plugin-hook`; that consumer fans out to `toolctrl.plugin-invoke`.
+  Keep the general worker and the separate plugin worker running in
+  production-like topology so health checks, queue wiring, and isolation are
+  exercised. Before external runtime signoff, the production `worker-plugin`
+  should remain idle/gated unless `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` is
+  intentionally enabled.
+- Audit events use action codes `plugin.installed`, `plugin.enabled`,
+  `plugin.disabled` and `plugin.crash.auto_disabled` (the last fired by
+  the crash-tracker notifier).

hivegrid-cloud/docs/architecture/plugin-system.md

@@ -0,0 +1,129 @@
+# Plugin system (Phase 06 #21 foundation)
+
+> **Runtime gate:** This is staging/foundation runtime documentation, not
+> production authorization. Production third-party plugin execution remains
+> blocked until the external WASM sandbox review is signed off in
+> [Plugin Runtime Signoff](../security/plugin-runtime-signoff.md).
+
+This document describes the WASM-sandbox foundation shipped under Phase 06 Task #21. It captures the runtime choice, manifest schema, signature trust model, capability gating, rate-limit, audit handling, and the work that is intentionally deferred to follow-up tasks.
+
+## Scope
+
+What the foundation ships:
+
+- **`@toolctrl/plugin-runtime`** — host-side runtime:
+  - Manifest schema (Zod) with closed-enum capabilities + reverse-DNS plugin IDs + SemVer versions + hex-encoded Ed25519 signature fields.
+  - Ed25519 signature verifier against the operator-controlled trusted-keys allowlist (`PLUGIN_TRUSTED_PUBLIC_KEYS` CSV).
+  - WebAssembly sandbox wrapper (`createPluginSandbox`) with deterministic memory cap + host-fn fuel budget, invoked through a killable worker-thread boundary with a wall-clock budget.
+  - Capability gateway (`createCapabilityGateway`) that wraps host-fn maps and rejects undeclared capability calls.
+- **`@toolctrl/plugin-sdk`** — TS types plugin authors build against (`HookHandler`, `HostApi`, `PluginContext`, `defineHook`) and a `hello-world` example.
+- **`Plugin` Prisma model + migration** — one row per installed plugin per tenant.
+- **`plugin` worker queue group** with the `toolctrl.plugin-invoke` BullMQ queue.
+- **`handlePluginInvokeJob`** worker handler (`apps/worker/src/jobs/plugin-invoke.ts`) that wires Plugin lookup → binary load → signature verify → rate-limit → sandbox invoke → audit.
+
+What is **deferred** (tracked in `docs/improvement-plan/06-plugin-system/wasm-sandbox.md` under `## Deviations from Plan` once that file is updated by the executing agent):
+
+- Hot-reload — operators restart the `worker-plugin` pod to pick up new binaries.
+- Marketplace integration — backlog (#24).
+- CLI build/sign/publish is shipped; see [Plugin CLI Reference](../plugins/cli-reference.md).
+- Full guided hook-lifecycle UI (#22) — install/guided consent dialogs remain
+  deferred; tenant settings now supports list-level approval for missing
+  manifest-declared capability grants with explicit review confirmation and
+  stale-manifest digest checks.
+- Audit + docs deep-dive (#23) — author docs and the capability matrix have
+  landed; remaining gaps are external audit sign-off, richer toolchain notes,
+  and high-risk host bindings beyond the current fail-closed surface.
+- E2E testcontainer harness — staging deployment + manual smoke tests in the meantime.
+- Worker-thread pool with epoch interruption — the current worker bootstrap uses one killable worker thread per invocation; a pooled runtime and true per-instruction interruption remain deferred.
+
+Operational lifecycle status, worker topology, and current production gaps are maintained in [Plugin Lifecycle Operations](../operations/plugin-lifecycle.md). Product-facing status and user/admin behavior are maintained in [Plugins](../features/plugins.md).
+
+## Runtime choice
+
+**Default: Node 22's built-in `WebAssembly` (V8) with host-fn-driven fuel charging.**
+
+Rationale: native Wasmtime bindings for Node (`@bytecodealliance/jco`, `@bytecodealliance/preview2-shim`) are still pre-1.0 and add a ~40 MB build asset. For the foundation we ship a portable wrapper around Node's built-in `WebAssembly` runtime — sufficient for hard memory limits via `WebAssembly.Memory { maximum }`, a host-fn fuel budget on capability calls, and a worker-thread wall-clock kill for non-yielding code.
+
+**Upgrade path:** `PluginSandbox` is an abstract surface — `invoke`, `close`, `fuelRemaining`. A Wasmer or Wasmtime-jco backend can be dropped in behind the same contract once either offers a stable Component-Model story on Node. This is the documented escape hatch when richer per-instruction fuel becomes a requirement.
+
+## Manifest schema
+
+A plugin ships a `plugin.toml` (or `plugin.json`) alongside the `.wasm` binary. The runtime parses it via `parsePluginManifest`, which returns a discriminated `Result`:
+
+```ts
+{
+  schemaVersion: 1,
+  id: "de.toolctrl.hello",           // reverse-DNS, lowercase
+  version: "1.0.0",                   // SemVer
+  name: "Hello",
+  entry: "plugin.wasm",
+  wasmSha256: "<64 hex chars>",       // SHA-256 of the .wasm at signing
+  signature: "<128 hex chars>",       // Ed25519 sig over the .wasm bytes
+  signedBy:  "<64 hex chars>",        // Ed25519 public key (32 bytes)
+  allowedHosts: ["api.example.com"],  // outbound HTTP allowlist
+  capabilities: [
+    { name: "log" },
+    { name: "http.outbound", scope: { allowedHosts: ["api.example.com"] } },
+  ],
+  hooks: [{ name: "onHello", export: "handleHello" }],
+}
+```
+
+The schema is `strict()` — unknown top-level fields are rejected. Unknown capability names are rejected against the closed enum in `packages/plugin-runtime/src/capabilities.ts`.
+
+## Signature trust model
+
+- Ed25519 chosen for deterministic signatures, audited Node `libcrypto` implementation, and short 64-byte signatures.
+- Operators seed `PLUGIN_TRUSTED_PUBLIC_KEYS` (CSV of hex-encoded 32-byte public keys). Plugins signed by any key outside this list are rejected with `public_key_not_trusted`.
+- `verifyPluginSignature`:
+  1. SHA-256-hashes the `.wasm` bytes and compares against `manifest.wasmSha256` — fail-closed on mismatch.
+  2. Confirms `manifest.signedBy` is in the trusted-keys allowlist — fail-closed otherwise.
+  3. Verifies `manifest.signature` over the `.wasm` bytes using `signedBy` — fail-closed otherwise.
+- All three checks must pass; the function returns a discriminated reason on failure and never throws.
+
+## Capability gating
+
+The capability gateway wraps a host-fn record so that calls to capabilities the plugin did not declare in its manifest, or that the tenant did not approve for this invocation, throw `CapabilityDeniedError`. The default fuel cost is 100 units per host-fn call; configurable per gateway. Deny paths (`not_declared_in_manifest`, `not_granted_for_tenant`, `no_host_implementation`) fire the `onDenied` callback so the worker can audit them. The plugin worker currently exposes safe synchronous `env.toolctrl_log(i32)` and `env.toolctrl_audit_record(i32)` imports through `invokeSync(...)`; higher-risk host functions remain absent and fail closed.
+
+## Rate limit
+
+The worker handler consumes a per-tenant + per-plugin sliding-window rate limit before invoking the sandbox. The limit's parameters (window size, max calls) are owned by the worker bootstrap and read from env. The handler's `consumeRateLimit` seam returns `{ allowed, remaining }`; the handler short-circuits to `rate_limit_exceeded` on a deny.
+
+## Audit
+
+`handlePluginInvokeJob` always writes an audit row, even on failure. Invocation audit persistence is fail-closed: if the audit write fails, the handler logs `plugin.invoke.audit_failed` and throws so BullMQ can retry/dead-letter instead of acknowledging a plugin invocation with no forensic trail. The audit row carries the tuple `(tenantId, pluginId, version, hookName, outcome)`.
+
+Capability-use telemetry is batched through `PluginCapabilityUseRecorder`. The production plugin worker enables the recorder's fail-closed mode; if the outbox sink is unavailable, the sandbox invocation fails and the plugin invocation is recorded as failed. Local/dev callers can keep the default soft mode when they need best-effort telemetry only.
+
+## Deferred env vars
+
+The full env-var matrix in `docs/improvement-plan/06-plugin-system/wasm-sandbox.md` is broader than the current worker bootstrap. The foundation currently wires:
+
+| Env var | Default | Used by |
+|---|---|---|
+| `PLUGIN_TRUSTED_PUBLIC_KEYS` | (empty → fail-closed) | `parseTrustedPublicKeys` |
+| `PLUGIN_MAX_FUEL` | 1000000 | sandbox fuel limit |
+| `PLUGIN_MAX_MEMORY_MB` | 64 | sandbox memory limit |
+| `PLUGIN_MAX_WALL_CLOCK_MS` | 5000 | worker-thread sandbox wall-clock kill |
+| `PLUGIN_OUTPUT_SANITIZATION_STRICTNESS` | moderate | plugin output redaction strictness |
+| `PLUGIN_BINARY_BASE_PATH` | `/app/var/plugins` | plugin binary lookup root for the worker bootstrap |
+| `PLUGIN_RATE_LIMIT_PER_MINUTE` | 100 | worker sliding-window rate limit |
+| `PLUGIN_RATE_LIMIT_WINDOW_SECONDS` | 60 | worker sliding-window rate limit |
+| `PLUGIN_AUTO_DISABLE_THRESHOLD` | 10 | per-tenant/plugin crash auto-disable threshold |
+| `PLUGIN_AUTO_DISABLE_WINDOW_MS` | 600000 | auto-disable sliding window |
+| `PLUGIN_AUDIT_RETENTION_DAYS` | 90 | `PluginCapabilityUse` retention window |
+| `PLUGIN_AUDIT_RETENTION_BATCH_SIZE` | 1000 | audit-retention cleanup batch size |
+| `PLUGIN_AUDIT_RETENTION_MAX_BATCHES` | 100 | audit-retention cleanup max batches per run |
+| `PLUGIN_AUDIT_RETENTION_CRON` | `0 3 * * *` | audit-retention scheduler pattern |
+| `PLUGIN_INSTALL_RATE_LIMIT_PER_MIN` | 5 | tenant plugin-install route rate limit |
+| `PLUGIN_SETTINGS_WRITE_RATE_LIMIT_PER_MIN` | 60 | per-user plugin settings write route rate limit |
+| `PLUGIN_SETTINGS_FRAME_VIOLATION_RATE_LIMIT_PER_MIN` | 120 | client-reported plugin settings iframe violation route rate limit |
+| `PLUGIN_MAX_INSTALLS_PER_TENANT` | 50 | tenant plugin-install cap |
+| `WORKER_CONCURRENCY_PLUGIN` | 5 | `group-runtime.ts` |
+| `WORKER_QUEUE_GROUP=plugin` | — | worker pod entry-point |
+
+Richer host-binding controls remain runtime follow-up work. Do not treat undocumented knobs as production-enforced until the worker bootstrap reads them; the operator-facing status is tracked in [Plugin Lifecycle Operations](../operations/plugin-lifecycle.md).
+
+## Why this is "foundation"
+
+The full spec describes a 4–5 week effort covering 17 implementation steps and dual-track SDKs (AssemblyScript + Rust), per-instruction fuel via Wasmtime, multi-worker-thread pools, hot-reload, marketplace, signing-key rotation. The foundation shipped here is the **structural skeleton plus minimum-viable WASM execution** — every later step is the same `PluginSandbox` / `CapabilityGateway` / `handlePluginInvokeJob` contract with a richer implementation behind it.

hivegrid-cloud/docs/architecture/rw-split.md

@@ -0,0 +1,201 @@
+# Read / Write Split (Prisma `$extends`)
+
+> Phase 03 task #41. Status: code-complete; replica provisioning is operator-owned.
+
+## TL;DR
+
+```ts
+import { db, dbRead } from "~/utils/db.server";
+
+// Default — primary (write target). Use for all writes, auth, read-after-write.
+await db.user.update({ where: { id }, data: { lastSeenAt: new Date() } });
+
+// Explicit primary — same client, communicates intent.
+import { getWriteDbClient } from "@toolctrl/db/client.server";
+
+// Replica — eventually consistent. Use for listings, dashboards, public reads.
+const users = await dbRead.user.findMany({ where: { tenantId } });
+```
+
+`dbRead` is wired to `DATABASE_READ_URL` when set. When unset (single-DB dev,
+self-hosted setups) it returns the same client as `db`, so the code path is
+identical and lag = 0.
+
+## Convention: when to use which
+
+| Use `dbRead` (replica) | Use `db` / `getWriteDbClient()` (primary) |
+| ---------------------- | ----------------------------------------- |
+| List endpoints / paginated grids | All mutations (`create`, `update`, `delete`, `upsert`) |
+| Public marketing pages (sites, portals) | Auth-resolution paths (session → user → tenant) |
+| Knowledge-base public reads + search | Stripe webhook handlers reading state they just wrote |
+| Analytics dashboards & aggregations | Outbox dispatcher reading `PENDING` events (see #42) |
+| Admin tenant / role enumerations (post-auth) | Anything inside a `db.$transaction(...)` callback |
+| Dashboard counts (`count()` / `groupBy()`) | Idempotency-key lookups (request-scoped consistency) |
+| Search-result rendering | Read-after-write inside the same request |
+
+When in doubt: leave on `db` (= primary). The replica is an optimisation,
+not a correctness requirement. **Auto-routing is intentionally not
+implemented** — the spec rejects it because it makes read-after-write
+hazards invisible.
+
+## Read-after-write hazard
+
+Streaming replication is **eventually consistent**:
+
+- Typical lag: < 1 s
+- Burst writes (bulk imports, k6 load tests): up to 5–10 s
+- A query issued on the replica milliseconds after the write hits the
+  primary will read the **pre-write** state.
+
+**Mitigation:**
+
+1. Always use `db` inside transactions and for writes.
+2. Never re-read on `dbRead` what you just wrote on `db` in the same
+   request. If you must, route the read through `db` too.
+3. Auth-critical paths (`getUserAndTenants`, session resolution) stay on
+   `db` even though they're "reads" — they gate the rest of the
+   request, and a stale read can mis-route a tenant boundary check.
+
+## Fallback semantics
+
+`getReadDbClient()`:
+
+```ts
+export function getReadDbClient(): DbClient {
+  const readUrl = process.env.DATABASE_READ_URL;
+  if (!readUrl || readUrl.length === 0) {
+    return getDbClient(); // single instance, no double connection
+  }
+  // memoised replica singleton
+  if (!global.__dbRead) {
+    global.__dbRead = createDbClient({ connectionString: readUrl });
+  }
+  return global.__dbRead;
+}
+```
+
+| State of `DATABASE_READ_URL` | `dbRead` returns |
+| ---------------------------- | --------------------------------- |
+| unset / empty                | the write client (`db === dbRead`) |
+| valid `postgres://...`       | a distinct read-only client       |
+
+This means CI, local dev, and any deploy without a replica continue to
+work without code changes. All migrations are zero-risk to roll out.
+
+## Telemetry
+
+`@toolctrl/db/telemetry.server` exposes `attachPrismaTelemetry()` — a
+`$extends({ query })` wrapper that records every operation with
+`{ client: 'read' | 'write', model, operation, durationMs, ok }`. Wire it
+in the app bootstrap by adapting the recorder to your metrics sink
+(prom-client `Histogram` / `Counter` or an OpenTelemetry `Meter`). The
+canonical counter shape is `prisma.queries.total{client,model,operation}`.
+
+## Coolify / staging setup (operator-owned)
+
+Replica provisioning is **out of scope** for the application repo — these
+steps live with the platform operator.
+
+1. **Primary Postgres** (Coolify service `postgres`). `postgresql.conf`:
+
+   ```
+   wal_level = replica
+   max_wal_senders = 3
+   wal_keep_size = 256MB
+   ```
+
+   Then:
+
+   ```sql
+   CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD '...';
+   CREATE USER read_user WITH ENCRYPTED PASSWORD '...';
+   GRANT CONNECT ON DATABASE toolctrl TO read_user;
+   GRANT USAGE ON SCHEMA public TO read_user;
+   GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_user;
+   ALTER DEFAULT PRIVILEGES IN SCHEMA public
+     GRANT SELECT ON TABLES TO read_user;
+   ```
+
+2. **Replica Postgres** (Coolify service `postgres-replica`,
+   `postgres:16-alpine`):
+
+   ```bash
+   # Init from primary
+   until pg_basebackup -h postgres -D /var/lib/postgresql/data -U replicator -P -R; do
+     sleep 5
+   done
+   ```
+
+   `postgresql.replica.conf`:
+
+   ```
+   hot_standby = on
+   primary_conninfo = 'host=postgres port=5432 user=replicator password=...'
+   ```
+
+3. **App env**:
+
+   ```sh
+   DATABASE_URL=postgres://app_user:...@pgbouncer:6432/toolctrl?pgbouncer=true&connection_limit=10
+   DATABASE_READ_URL=postgres://read_user:...@postgres-replica:5432/toolctrl?connection_limit=10
+   ```
+
+   Optionally front the replica with its own PgBouncer for connection pooling.
+
+4. **Verification:**
+
+   ```sql
+   -- On primary
+   SELECT * FROM pg_stat_replication;
+
+   -- On replica
+   SELECT pg_last_xact_replay_timestamp(), now() - pg_last_xact_replay_timestamp() AS lag;
+   ```
+
+   Grafana panel `prisma.queries.total{client="read"}` should rise on
+   dashboard / KB / portal traffic.
+
+## Rollback
+
+- **Per-call-site:** `git revert <migration-commit>` flips that module
+  back to `db` (primary). No data change.
+- **Global:** unset `DATABASE_READ_URL` and restart the app. `dbRead === db`,
+  all traffic hits the primary.
+- **Replica down:** Prisma's read client will throw connection errors.
+  Either unset `DATABASE_READ_URL` (degraded performance, no errors) or
+  add a circuit-breaker (future work, out of scope here).
+
+## Migration recipe (for future modules)
+
+1. **Find candidate read-only functions** — pure `findMany`, `findUnique`,
+   `findFirst`, `count`, `groupBy`, `aggregate` with no surrounding write.
+2. **Check the caller graph** — is this function called immediately
+   after a write that updates the same rows? If yes, leave on `db`.
+3. **Replace `db.<model>.<op>` with `dbRead.<model>.<op>`** and add a
+   one-line comment explaining the choice.
+4. **Add `dbRead` to the file's imports** alongside `db` (keep `db` for
+   the mutations still in the file).
+5. **Run** `pnpm lint --max-warnings=0` and `pnpm typecheck`.
+6. **Tests** stay green because of the fallback semantics — single-DB
+   CI doesn't see a difference.
+7. **Commit per-module** (`03-scaling/rw-split: migrate <module> ...`)
+   so a single revert can roll back one module.
+
+## What was migrated in #41
+
+- `apps/web/app/modules/crm/services/CrmService.ts` — user enumeration
+  for sync display.
+- `apps/web/app/modules/knowledgeBase/db/*.db.server.ts` — public KB
+  reads, listings, search, featured-article queries, dashboard counts.
+- `apps/web/app/modules/analytics/db/analyticsUniqueVisitors.db.server.ts`
+  — `groupBy` aggregations.
+- `apps/web/app/utils/db/tenants.db.server.ts` — admin tenant enumerations
+  (post-auth).
+- `apps/web/app/modules/sites/db/sites.db.server.ts` — read-only site
+  loaders (auth-gating `getSiteByTypeAndSiteId` left on primary).
+- `apps/web/app/modules/portals/db/portals.db.server.ts` and
+  `portalPages.db.server.ts` — listings, page lookups, dashboard counts.
+
+`appData.ts` was **not** migrated wholesale; only the secondary tenant
+listings it consumes were moved. The auth-resolution path stays on
+primary by design.

hivegrid-cloud/docs/architecture/storybook.md

@@ -0,0 +1,181 @@
+# Storybook
+
+A single Storybook instance under `apps/storybook/` aggregates component
+stories from the boilerplate UI packages.
+
+## Quickstart
+
+```bash
+pnpm install
+pnpm --filter @toolctrl/storybook storybook
+# -> http://localhost:6006
+
+# Static build
+pnpm --filter @toolctrl/storybook build-storybook
+# -> apps/storybook/storybook-static
+```
+
+## Stack
+
+- **Storybook 8.x** with `@storybook/react-vite` builder
+- React 19 + Vite 7 (matches `apps/web`)
+- Addons: `@storybook/addon-essentials`, `@storybook/addon-a11y`,
+  `@storybook/addon-interactions`, `@storybook/addon-themes`
+
+## Where stories live
+
+Stories are **co-located** with the UI packages. The `main.ts` glob picks
+them up automatically:
+
+```
+packages/ui/src/stories/*.stories.tsx
+packages/ui-app/src/stories/*.stories.tsx
+packages/ui-subframe/src/stories/*.stories.tsx
+packages/ui-mock/src/stories/*.stories.tsx
+```
+
+> **Why `packages/ui/` only today?** The i18n hardcoded-strings guard
+> whitelists `packages/(ai|cli|config|entitlements|mail|plugins|queue|realtime|ui)/`,
+> so stories under `packages/ui/src/stories/` author freely without
+> tripping the gate. Stories in `ui-app`, `ui-subframe`, or `ui-mock`
+> work too — but currently need to avoid uppercase user-facing prop
+> literals (8+ char run starting `[A-ZÄÖÜ]`) or include their own
+> reviewed-line patterns. A follow-up will widen the guard.
+
+## Authoring a story
+
+Use **Component Story Format 3 (CSF-3)**:
+
+```tsx
+import type { Meta, StoryObj } from "@storybook/react";
+import { MyComponent } from "./MyComponent";
+
+const meta: Meta<typeof MyComponent> = {
+  title: "UI / Category / MyComponent",
+  component: MyComponent,
+  tags: ["autodocs"],
+  argTypes: {
+    variant: { control: { type: "select" }, options: ["a", "b"] },
+  },
+  args: { variant: "a" },
+};
+
+export default meta;
+
+type Story = StoryObj<typeof MyComponent>;
+
+export const Default: Story = {};
+export const VariantB: Story = { args: { variant: "b" } };
+```
+
+### Conventions
+
+- **Title path**: `UI / <Category> / <Component>` (lowercase variants work
+  too if the case trips a guard).
+- **Tags**: include `"autodocs"` to get an auto-generated docs page.
+- **Render functions** when you need state or composition. Prefer plain
+  `args` for simple variants.
+- **A11y opt-out**: `parameters: { a11y: { disable: true } }` plus a
+  comment explaining why. Default policy is "fail on a11y violations".
+
+## Theming
+
+`apps/storybook/.storybook/preview.tsx` wraps every story in a theme
+decorator from `@storybook/addon-themes`. The toolbar exposes a
+`light` / `dark` toggle that swaps the `dark` class on the preview root.
+
+Stories should style themselves via the CSS variables declared in
+`apps/storybook/.storybook/preview.css` (`--color-bg`, `--color-fg`,
+`--color-primary`, …) so they flip automatically. See the **Theme
+Guide** docs page inside Storybook for the full token table.
+
+## A11y addon
+
+`@storybook/addon-a11y` runs axe-core in the story panel. The
+`preview.tsx` defaults enable the WCAG-style `color-contrast` rule.
+Violations show up in the **Accessibility** panel.
+
+## Test-runner & a11y gate
+
+`@storybook/test-runner` walks every story registered in `main.ts` and
+runs the `postVisit` hook in `.storybook/test-runner.ts` against each
+render. The hook injects `axe-playwright` and re-runs the same ruleset
+the in-canvas panel uses (`#storybook-root` as the context, WCAG-2.2-AA
+tags) — but in a real headless Chromium where `color-contrast`
+actually resolves CSS, so failures are real WCAG regressions.
+
+Run locally:
+
+```bash
+# Terminal 1
+pnpm --filter @toolctrl/storybook storybook
+
+# Terminal 2
+pnpm --filter @toolctrl/storybook test:storybook
+```
+
+The runner respects per-story opt-outs via `parameters.a11y.disable =
+true` (must come with a justifying code comment).
+
+## CI
+
+`.github/workflows/storybook.yml` and `.gitea/workflows/storybook.yml`
+run three jobs on PRs that touch `apps/storybook/` or any
+`packages/ui*/src/stories/` path:
+
+1. **build-storybook** — `pnpm --filter @toolctrl/storybook
+   build-storybook`; uploads the static site as a 14-day artifact.
+2. **test-storybook** — downloads the artifact, serves it via
+   `http-server`, waits for readiness, installs Playwright Chromium,
+   and runs `pnpm test:storybook` against `http://127.0.0.1:6006`.
+   Any axe violation fails the job, blocking the PR until fixed or
+   explicitly opted-out.
+3. **visual-regression** — Lost-Pixel diff (see below). Self-skips
+   until a baseline is committed.
+
+The main CI Quality-Gates job is unchanged.
+
+## Visual-regression (Lost-Pixel)
+
+`lost-pixel` is wired via `apps/storybook/lostpixel.config.ts`. It
+walks the built static Storybook, screenshots every story, and diffs
+each shot against a committed baseline under
+`apps/storybook/.lostpixel/baseline/`. A pixel diff above the 0.1 %
+threshold fails CI and uploads the diff PNGs as a `lost-pixel-diff`
+artifact.
+
+The `visual-regression` CI job is **baseline-gated**: its first step
+checks for `apps/storybook/.lostpixel/baseline/*.png` and, if none are
+committed yet, skips the rest of the job with a notice instead of
+failing. This keeps the job green until an operator opts in.
+
+**Bootstrap (one-time, operator):**
+
+```bash
+pnpm --filter @toolctrl/storybook build-storybook
+pnpm --filter @toolctrl/storybook visual-regression:update
+git add apps/storybook/.lostpixel/baseline
+git commit -m "test(storybook): commit Lost-Pixel baseline screenshots"
+```
+
+From the next PR on, the job blocks merges on unintended visual drift.
+`.lostpixel/current/` and `.lostpixel/difference/` are per-run output
+and git-ignored.
+
+Chromatic (hosted, paid above a free tier) remains an alternative —
+it would need `CHROMATIC_PROJECT_TOKEN` in CI secrets and is
+deliberately **not** wired.
+
+## Troubleshooting
+
+- **Story missing?** Confirm the path matches the glob in
+  `apps/storybook/.storybook/main.ts`.
+- **Story render error?** Run `pnpm --filter @toolctrl/storybook storybook`
+  locally and check the iframe error overlay.
+- **A11y violation that should be allowed?** Use
+  `parameters.a11y.config.rules` to disable a specific axe rule, or
+  `parameters.a11y.disable = true` if a story is intentionally
+  decorative.
+- **i18n guard flags story content?** Either keep user-facing prop
+  literals under 9 characters / starting lowercase, or route them
+  through translation keys (`@toolctrl/i18n`).

hivegrid-cloud/docs/architecture/test-coverage.md

@@ -0,0 +1,160 @@
+# Test Coverage Policy
+
+**Status:** completed; no-regression gate in maintenance
+**Spec:** `docs/improvement-plan/10-compliance-polish/test-coverage.md`
+**Owner:** Engineering quality / release maintenance
+
+This document explains how test coverage is measured, gated, and reported in the
+ToolCTRL boilerplate. Per-path thresholds are encoded as data in
+`scripts/coverage-targets.json` and enforced by `scripts/verify-test-coverage.mjs`.
+CI workflows run the gate on GitHub PRs and `push` to `main`, while the
+Gitea coverage workflow currently runs on pull requests. Keep
+`.github/workflows/coverage.yml`, `.gitea/workflows/coverage.yml`, and
+this policy aligned when trigger coverage changes.
+
+## Why
+
+- **Per-path thresholds** reflect risk: auth, billing, tenant-isolation,
+  shared-store, and outbox modules carry materially higher correctness and
+  security risk than view components or marketing pages. A flat
+  repository-wide gate either over-protects unimportant code or under-protects
+  the parts that matter.
+- **No-regression baseline** is active. Critical paths carry a
+  `sprintTarget` field that records the release floor. The active gate has
+  been raised to the latest passing coverage run for the covered critical
+  paths, so contributors are blocked on regressions rather than asked to chase
+  already-closed backlog rows.
+
+## Tooling
+
+- **Provider:** V8 native coverage via `@vitest/coverage-v8` and
+  `vitest@3.2.4`.
+- **Reporters:** `text-summary`, `json-summary`, `html`, `lcov`. Configured in
+  `vitest.config.ts` so `scripts/verify-test-coverage.mjs` can use a stable
+  `vitest run --coverage` invocation.
+- **Output:** `coverage/` (git-ignored). `coverage-summary.json` feeds the
+  threshold gate and the README badge; `html/` is uploaded by CI as a build
+  artifact.
+
+## Local commands
+
+```bash
+# Run vitest with coverage and enforce per-path thresholds.
+pnpm run verify:test-coverage
+
+# Print the shields.io badge URL for the current coverage-summary.json.
+pnpm coverage:badge
+
+# Render the per-path table as a markdown PR comment.
+node scripts/coverage-summary.mjs --pr-comment
+```
+
+## Critical paths
+
+The following modules carry tighter thresholds because regressions there have
+disproportionate impact. Each entry in `scripts/coverage-targets.json` with
+`"criticalPath": true` and a `sprintTarget` belongs to this list.
+
+| Module                                     | Active gate (stmt/br) | Sprint target |
+| ------------------------------------------ | --------------------- | ------------- |
+| `apps/api/src/auth/**`                     | 99 % / 92 %           | 90 % / 85 %   |
+| `apps/api/src/billing/**`                  | 99 % / 97 %           | 99 % / 95 %   |
+| `apps/api/src/isolation/**`                | 99 % / 99 %           | 95 % / 90 %   |
+| `apps/web/app/services/auth/**`            | 97 % / 88 %           | 90 % / 85 %   |
+| `apps/web/app/utils/auth/**`               | 97 % / 92 %           | 90 % / 85 %   |
+| `apps/web/app/utils/session.server.ts`     | 99 % / 98 %           | 95 % / 90 %   |
+| `apps/web/app/utils/stripe.server.ts`      | 95 % / 90 %           | 85 % / 75 %   |
+| `apps/web/app/utils/db/permissions/**`     | 99 % / 98 %           | 90 % / 80 %   |
+| `apps/web/app/utils/db/tenants.db.server.ts` | 98 % / 95 %         | 95 % / 90 %   |
+| `packages/entitlements/src/**`             | 96 % / 89 %           | 90 % / 80 %   |
+| `packages/db/src/outbox*`                  | 92 % / 88 %           | 90 % / 80 %   |
+| `packages/shared-store/src/**`             | 91 % / 90 %           | 90 % / 80 %   |
+
+The active gate is the **no-regression baseline** — it is the floor below
+which CI will fail. The sprint target records the release floor that the
+latest ratchet already meets or exceeds for every critical-path row (see
+[Maintenance and future ratchets](#maintenance-and-future-ratchets)).
+
+## How to add a coverage gate for a new module
+
+1. Open `scripts/coverage-targets.json`.
+2. Append an entry with `name`, `pathPrefixes`, `includeGlobs`, `statements`,
+   `branches`. For critical modules, also set `criticalPath: true` and a
+   `sprintTarget` object.
+3. Run `pnpm run verify:test-coverage` locally. The gate prints the actual
+   coverage; set the active gate at or slightly below that number so the
+   commit lands green.
+4. Open a PR. The Coverage workflow will post a sticky comment showing the
+   new row in the per-path table.
+
+## How to bump or lower a threshold
+
+Bumps **up** (tighter gate) — most common after writing new tests:
+
+1. Run `pnpm run verify:test-coverage` and read the actual coverage value.
+2. Edit the relevant entry in `scripts/coverage-targets.json`.
+3. Reference the commit that added the new tests in the PR description.
+
+Bumps **down** (looser gate) — should be rare and reviewed:
+
+1. Open the PR with a justification. Acceptable reasons include: code path
+   genuinely removed; module merged into a less-critical category; sprint
+   deferred for documented operational reasons.
+2. Request review from a code owner.
+3. **Never** lower a `criticalPath` gate without compensating mitigation
+   (e.g. a corresponding E2E test, a stricter integration suite). Coverage
+   floor for critical paths is a security control.
+
+## What triggers a CI failure
+
+`pnpm run verify:test-coverage` exits non-zero in any of these cases:
+
+- vitest itself failed to run (test failure, config error, OOM)
+- `coverage/coverage-summary.json` is missing (coverage reporter mis-configured)
+- any target's actual `statements%` < `target.statements` **or**
+  actual `branches%` < `target.branches`
+
+The GitHub workflow uploads `coverage/` as `coverage-report-<run_id>` even on
+failure so contributors can drill into the HTML report.
+
+## Type-only packages
+
+Packages that intentionally emit no runtime JavaScript, such as
+`@toolctrl/api-types`, must not add fake exports or `expect(true)` tests just
+to satisfy statement coverage. Give these packages a 0/0 active gate and 0/0
+sprint target, then cover their contract with `expectTypeOf` tests and
+`tsc --noEmit` typecheck.
+
+## Maintenance and future ratchets
+
+Task #76 is completed. Keep the gate useful as a no-regression control:
+
+1. Confirm `docs/testing/coverage-sprint-backlog.md` still has no
+   critical-path or standard-tier positive-gap rows before release closeouts.
+2. Keep each active `statements`/`branches` gate at or above the relevant
+   `sprintTarget`.
+3. Treat `sprintTarget` fields as explicit release-floor annotations unless a
+   future cleanup removes them consistently from docs and gate data.
+4. Run `pnpm run verify:test-coverage` after threshold changes — it must pass
+   with the new gates.
+5. Track additional Playwright/E2E coverage separately; it is not a blocker
+   for the completed per-module coverage gate.
+
+## Out of scope (deferred to follow-up tasks)
+
+- **Coverage trend dashboard** (`docs/testing/coverage-trend.md`) — nightly
+  cron-updated markdown table. Not implemented in this phase.
+- **PR-comment diff vs main baseline** — current sticky comment shows totals
+  but not delta. A future PR can add baseline-fetch + delta rendering similar
+  to bundle-budget.
+- **E2E coverage via Playwright** (5 golden-path flows) — tracked separately.
+- **Additional critical-path test writing** — see
+  `docs/testing/coverage-sprint-backlog.md` for the ordered backlog.
+
+## References
+
+- Spec: `docs/improvement-plan/10-compliance-polish/test-coverage.md`
+- Gate runner: `scripts/verify-test-coverage.mjs`
+- Targets data: `scripts/coverage-targets.json`
+- Summary/badge: `scripts/coverage-summary.mjs`
+- CI: `.github/workflows/coverage.yml`, `.gitea/workflows/coverage.yml`

hivegrid-cloud/docs/architecture/two-tier-cache.md

@@ -0,0 +1,93 @@
+# Two-Tier Cache Pattern
+
+**Status:** active
+**Owner:** platform-scaling
+**Related:** [`packages/shared-store/README.md`](../../packages/shared-store/README.md), `docs/improvement-plan/03-scaling-infrastructure/pubsub-cache-invalidation.md`
+
+## Why
+
+A SharedStore-only cache pays one Redis round-trip per read (~1-3 ms on a healthy network). Hot paths that hit the same key 10-30 times per request (entitlement checks, limit counters, UI feature gating) accumulate enough overhead that we need an in-process L1 — but an in-process L1 alone goes stale across replicas. The two-tier cache combines both:
+
+| Tier | Backing | Latency | Consistency |
+| --- | --- | --- | --- |
+| L1 | `Map` / `LRUCache` in process | sub-microsecond | per-replica |
+| L2 | `SharedStore` (Redis or memory) | 1-3 ms | cross-replica |
+| Pub/Sub | Redis `PUBLISH` on `cache:<ns>:invalidate` | < 50 ms typical fan-out | best-effort, see safety net |
+
+A mutating call invalidates its local L1+L2 and publishes the key on the namespace channel. Every replica's subscriber receives the message and drops the key from its L1 (and optionally the L2). Reads probe L1 first; on miss, fall back to L2; on L2 miss, refresh from the source-of-truth (typically Postgres) and populate both tiers.
+
+## Decision rubric
+
+Use a two-tier cache when **all three** hold:
+
+1. The key is read more than ~5x per request, or by every request (subscription, tenant, entitlements, feature flag).
+2. Source-of-truth mutations are rare relative to reads (Stripe webhook firing seconds-to-minutes apart, admin edits).
+3. Stale reads have user-visible consequences (showing the wrong plan limit, denying access after an upgrade).
+
+Otherwise, prefer:
+
+- **In-process LRU only** when consistency doesn't matter (route-stable computed values, request-scoped caches).
+- **SharedStore cache only** when the read-rate is low and consistency must be strict (idempotency keys, session lookups).
+- **No cache** when the source is already fast (cached Prisma result, in-memory blueprint).
+
+## Channel naming
+
+Channels follow `shared-store:<store-namespace>:ch:cache:<cache-namespace>:invalidate`. Concretely, the web SharedStore (`namespace: "web"`) emits subscription invalidations on `shared-store:web:ch:cache:subscription:invalidate`. The API SharedStore (`namespace: "api"`) emits on `shared-store:api:ch:cache:subscription:invalidate`.
+
+For cross-app fan-out (Stripe webhook in API → web replicas), both apps must use the same SharedStore namespace so they share a channel. The current implementation uses per-app namespaces (`api`, `web`) but both connect to the same Redis logical DB index (default `1`). The actual channel string is namespaced under the writing app's SharedStore prefix. **If you need cross-app fan-out**, either:
+
+1. Use a dedicated shared namespace (e.g. `namespace: "platform"`) for both apps' two-tier caches, or
+2. Have the publishing app PUBLISH the message via a SharedStore configured with the receiving app's namespace explicitly.
+
+The Stripe webhook → subscription pattern in this codebase publishes on `shared-store:api:ch:cache:subscription:invalidate`. For a production multi-replica web deploy to actually receive that broadcast, configure both `apps/api/src/cache/subscription-cache-publisher.ts` and `apps/web/app/utils/services/.server/subscriptionCache.server.ts` to construct their SharedStores with the same namespace (recommended: introduce `getCacheSharedStore({ namespace: "cache" })` for both apps). The current implementation broadcasts within the API process and applies the same `clearTenantSubscriptionCache` semantics on the web side via local mutation-paths that already publish; the cross-process broadcast is the work item documented under Deviations in the detail-md.
+
+## Reconnect semantics
+
+ioredis auto-reconnects on connection loss with exponential backoff. While disconnected, the subscriber is offline and any PUBLISH issued by another replica is lost forever (pub/sub has no persistence). Mitigations layered in order:
+
+1. **L1 TTL is the floor.** Default `CACHE_L1_TTL_SECONDS=300` (5 min). Any missed invalidation self-heals within five minutes because the L1 entry expires. Lower the TTL for hotter paths if stale-window tolerance shrinks; raise it for marketing-content-grade caches.
+2. **Reconnect flush.** Each two-tier cache exposes `flushLocal()` so a higher-level boot hook can `cache.flushLocal()` on a Redis reconnect event. The minimal implementation is "TTL is enough"; a more aggressive deployment can wire `subscriber.on('ready', () => cache.flushLocal())` in app boot.
+3. **Outbox fallback.** For strictly-consistent flows (billing, entitlement enforcement), the source of truth writes an outbox event (see `docs/architecture/outbox-pattern.md`); the outbox dispatcher consumes the event and re-issues the invalidation. This guarantees at-least-once delivery and survives Redis outages — at the cost of one DB round-trip and ~1 second dispatch latency.
+
+## Channel inventory
+
+| Channel | Publisher | Subscriber | Payload | Defined in |
+| --- | --- | --- | --- | --- |
+| `cache:subscription:invalidate` | `apps/api/src/billing/stripe-webhook-route.ts` (after `updateTenantSubscriptionProductFromStripe`); `apps/web/.../subscriptionService.ts` (`clearTenantSubscriptionCache`, `clearAllSubscriptionsCache`) | `apps/web/app/utils/services/.server/subscriptionCache.server.ts` → bridges into `clearCacheKey('tenantSubscription:<tenantId>')` | tenantId (string) or `"*"` for wildcard | `subscriptionCache.server.ts` + `subscription-cache-publisher.ts` |
+| `cache:tenant:invalidate` | `apps/web/.../tenant-cache.server.ts` (`publishTenantInvalidation`) | `apps/web/.../tenant-cache.server.ts` → clears `tenant:`, `tenantSimple:`, `tenantIdOrSlug:` cachified keys | tenantId (string) or `"*"` | `tenant-cache.server.ts` |
+| `cache:entitlements:invalidate` | _piggy-backs on subscription_ — entitlements derive from the subscription and recompute on the next read after the subscription L1 is flushed | _piggy-backs on subscription_ | n/a — chained | n/a (chain) |
+| `cache:feature-flags:invalidate` | `packages/feature-flags` publishers, including CLI feature-flag enable/disable/rule updates | web feature-flag cache subscribers through the shared two-tier cache bootstrap | `{ flagKey?, tenantId?, all? }` | `packages/feature-flags/src/publisher.ts` |
+
+## Implementing a new two-tier cache
+
+1. Create a `<namespace>-cache.server.ts` module per app (api/web) that wraps `createTwoTierCache({ store: getSharedStore(), namespace, ttlSeconds, pubsubEnabled, logger })`.
+2. Expose two functions: `publish<Namespace>Invalidation(key)` (for mutators) and `subscribe<Namespace>Cache()` (idempotent boot hook).
+3. The subscriber callback bridges into the existing local cache representation (`clearCacheKey()` for cachified-LRU, `Map.delete()` for ad-hoc maps).
+4. Add the boot hook to `apps/web/app/cache-bootstrap.server.ts` (or equivalent API bootstrap) so the subscriber starts before request traffic.
+5. Add publisher call-sites at every mutation that owns the SoT (database write, admin action, scheduled job).
+6. Write a co-located test that drives the two-tier helper directly against a `MemoryAdapter` and asserts the cachified-bridge fires.
+
+## Feature-Flag Cache Notes
+
+Feature-flag invalidation is now implemented through the shared two-tier cache
+publisher path. CLI feature-flag mutations publish to
+`cache:feature-flags:invalidate` after the audit write, and web subscribers
+flush L1 entries immediately instead of waiting for the TTL window. Keep new
+feature-flag mutators on the same publisher path.
+
+## Kill switch
+
+`CACHE_INVALIDATION_ENABLED=false` (env var) flips both publishers and subscribers into local-only mode. The cache still works — read-through still populates L1+L2 — but invalidate becomes a local-only delete + no PUBLISH, and subscribers do not start. The system degrades to "every replica has its own L1 that expires after `CACHE_L1_TTL_SECONDS`". Use this switch:
+
+- During a Redis incident to stop publisher errors from spamming Sentry
+- During a deployment when subscribers can't keep up with traffic patterns
+- During investigation of a stale-cache bug to confirm whether the publisher or subscriber path is at fault
+
+Always re-enable after the incident; the L1-only mode has no replica-consistency floor below the L1 TTL.
+
+## Tests
+
+- Unit tests for `createTwoTierCache` cover round-trip, fallback, cross-replica invalidation, `localOnly`, wildcard flush, pubsub kill-switch — `packages/shared-store/src/helpers/two-tier-cache.test.ts`
+- Subscription cache integration: `apps/web/app/utils/services/.server/subscriptionCache.server.test.ts`
+- Tenant cache integration: `apps/web/app/utils/db/tenant-cache.server.test.ts`
+- Stripe webhook publish path: `apps/api/src/billing/stripe-webhook-route.test.ts` (asserts `publishSubscriptionInvalidation(tenantId)` is called after the DB update)

hivegrid-cloud/docs/architecture/vector-store.md

@@ -0,0 +1,139 @@
+# Vector Store
+
+`@toolctrl/vector-store` is a thin adapter layer over two interchangeable
+nearest-neighbour stores. The interface is small enough that swapping one for
+the other does not require code changes outside `createVectorStore`; the
+identical conformance suite (`packages/vector-store/src/adapters/conformance.test.ts`)
+guarantees behavioural parity.
+
+## Why an adapter layer at all
+
+The 80% case for tenants in this stack — knowledge bases with a few thousand
+chunks, RAG-backed help-desk replies — sits comfortably inside pgvector. It
+inherits the same operational story as the rest of Postgres: backups, replicas,
+RLS, monitoring. No extra container, no extra credentials, no extra firewall
+rule.
+
+The 20% case (large RAG corpora, dedicated tenants, latency-sensitive workloads)
+benefits from Qdrant's purpose-built HNSW implementation, richer filter
+language and snapshot tooling. We keep that path open without forcing every
+deployment to run a second data store.
+
+## Adapter comparison
+
+| Concern | pgvector | Qdrant |
+| --- | --- | --- |
+| Operational cost | Same Postgres you already run | Separate container + storage volume |
+| Backups | Free — covered by `backup:create` | Snapshot API; cron script ships separately |
+| Recall@10 on cosine | 0.95+ once HNSW is built | 0.97+ with default config |
+| P95 search latency (10k items) | <30 ms | <15 ms |
+| Filter expressiveness | `payload ->> 'key' = …` JSON ops | Native payload filter DSL |
+| Multi-vector / sparse | Not yet | Native (not exposed in adapter) |
+| Failure mode on missing collection | Throws Postgres `42P01`; adapter converts to empty result | Throws 404; adapter converts to empty result |
+
+## Selection logic
+
+```ts
+import { createVectorStore } from "@toolctrl/vector-store";
+
+const store = createVectorStore({
+  namespace: "rag-prod",            // becomes a collection prefix
+  adapter: undefined,                // optional override
+  qdrantUrl: process.env.QDRANT_URL, // forward env at the boundary
+  qdrantApiKey: process.env.QDRANT_API_KEY,
+  qdrantClientFactory: ({ url, apiKey }) =>
+    new QdrantClient({ url, apiKey }),
+  prismaWrite: getWriteDbClient(),
+  prismaRead: getReadDbClient(),
+});
+```
+
+The factory picks an adapter in this order:
+
+1. `adapter` parameter when supplied (test seam, dual-running migrations).
+2. `qdrantUrl` truthy → Qdrant. Caller must also supply
+   `qdrantClientFactory`; the package does not pin a specific Qdrant SDK
+   version.
+3. Otherwise pgvector with `prismaWrite` / `prismaRead`.
+
+The package never reads `process.env` directly — it would break the
+`env-access-enforcement` boundary test. Callers thread env values in.
+
+## Tenant scoping
+
+`createVectorStore` always returns a `TenantScopedVectorStore`. The bare
+`searchVectors` method throws on this wrapper: searches must go through
+`searchForTenant({ tenantId, ... })`. The wrapper injects the tenantId as the
+first `must` filter clause and discards any caller-supplied `tenantId` clause
+before forwarding the call, so a buggy or hostile caller cannot escape the
+boundary.
+
+Writes are guarded at the type level: `VectorItem.payload` extends
+`{ tenantId: string }`, so omitting the tenant id at upsert time is a TS
+compile error.
+
+## pgvector tuning
+
+HNSW parameters live on the adapter constructor and default to pgvector 0.7's
+recommended starting point:
+
+```ts
+new PgvectorAdapter({
+  write, read,
+  hnsw: { m: 16, efConstruction: 64 },
+});
+```
+
+- `m` controls graph degree. Bump to 24 for collections >500k vectors when
+  recall lags.
+- `efConstruction` is the candidate-list size during index build. Bump to 128
+  for higher-recall corpora; build time roughly doubles.
+
+The adapter creates one table per collection (`vec_<namespace>_<name>`) with
+three indexes:
+
+- HNSW on the vector column with the metric-specific operator class.
+- GIN on the `payload` JSONB column for filter selectivity.
+- B-tree on the generated `tenant_id` column for the (almost always)
+  tenant-scoped read path.
+
+## Qdrant deployment notes
+
+- Pin to a 1.x release. Adapter is tested against `1.12`.
+- API key is mandatory whenever the URL is not localhost. The factory's caller
+  should reject `QDRANT_URL` without `QDRANT_API_KEY` at the boundary.
+- Use a dedicated PVC; Qdrant cannot recover from a wiped storage directory.
+- The adapter auto-creates collections on first upsert. Dimension mismatches
+  surface as `VectorDimensionMismatchError` so the embedding pipeline can
+  classify them for retry vs. dead-letter.
+
+## Backups
+
+- pgvector: piggy-backs on the Postgres backup pipeline (`backup:create`,
+  `backup:restore`). No additional work.
+- Qdrant: use the Snapshot API. A dedicated cron job + S3 upload script will
+  ship with task #50; until then run manual snapshots before each schema
+  rollout.
+
+## Migration paths
+
+We do not provide a direct pgvector → Qdrant data transfer. Source-of-truth is
+always the document/chunk table in Postgres, so the migration is a re-embed:
+
+1. Deploy with `VECTOR_STORE_DEFAULT_ADAPTER=qdrant` and `QDRANT_URL` set.
+2. Run `rag:reindex` (ships with #50). Embeddings are recomputed via the
+   embedding provider and written into Qdrant.
+3. Smoke-search to confirm parity, then drop the pgvector tables via
+   `vector-store:gc` once the warm-window is clear.
+
+Same procedure in reverse, with the obvious caveat that pgvector imposes a
+practical ceiling around 1M vectors per collection before HNSW build cost
+becomes painful.
+
+## Namespace conventions
+
+The factory's `namespace` argument is prefixed onto every collection name —
+`namespace="rag-prod", collection="docs"` becomes the pgvector table
+`vec_rag_prod_docs` and the Qdrant collection `rag-prod_docs`. Pick a
+namespace per workspace or per environment so multiple deployments can share
+the same store without colliding.