# E2E Skipped-Tests Loop — Handoff for the next agent

## ✅ GREEN — 4th agent (2026-06-02 ~20:15 UTC)

**Full "everything-on" chromium run against boiler (web image `944ef476`, freshly
reseeded): `203 passed / 0 failed / 5 skipped` (9.3m).** The 5 skips are correctly
operator-gated and justifiably impossible here: real-Stripe `subscription-success`
×3 (need an externally-acquired Stripe session), the registration email-verification
inverse pair (env runs with `E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0`), and the
`E2E_REQUIRE_PLUGIN_HOOK_AUDIT` hook-audit lane (operator-gated). A second clean
full run was used to confirm stability.

**The 12 remaining failures from the previous handoff, all fixed this session:**
1. **plugin cluster (49/120/173/201)** — cross-spec pollution. `plugin-hook-lifecycle`
   now targets the dedicated, fully-provisioned `com.toolctrl.e2e-lifecycle` plugin
   instead of mutating the shared `e2e-readiness` plugin's audit-missing baseline.
2. **plugin-settings 244** — the grant/revoke + restore reloaded the page before the
   in-flight single-fetch POST landed (stale read). Now waits for RR7 revalidation.
3. **plugin-settings 587** — TWO bugs. (a) PRODUCT: the settings-iframe write posted to
   `fetch(".")` (resolves to the parent plugins-list route) and expected raw JSON;
   RR7 returns a turbo-stream document for non-`.data` fetches, so every write failed.
   Fixed to post the detail route's `.data` endpoint and gate on the HTTP status.
   (b) TEST: the post-reload iframe `get` is non-deterministic (the host frame
   quarantines its window + rotates the channel on a late `load`), so persistence is
   now verified by reading the settings loader's `.data` payload after reload.
4. **admin-email-sequences ×2** — logged in as the shared tenant user (not an admin).
   Seed now provisions a dedicated `e2e.admin@example.test` (AdminUser + global
   SuperAdmin, no TOTP) and emits `E2E_ADMIN_EMAIL/PASSWORD`; spec logs in as it.
5. **mobile-shell 570 (run/manual SSR 500)** — `MonacoEditor` mounted its browser-only
   `./MonacoEditor.client` during SSR, so the streaming renderer set HTTP 500 on EVERY
   editor page (run/manual, webhook-run, **api-docs too** — the prior handoff was wrong
   that api-docs was fine; only this test asserts `status < 500`). Fixed by gating the
   editor behind a client mount (server + first client render emit an identical
   placeholder → no hydration mismatch).
6. **portal-management** — operator chose to SHIP portals. Un-ignored the portal routes
   (`app.$tenant/__portals/**`, `admin/portals.tsx`), gated the global enable behind
   `PORTALS_ENABLED` (boilerplate default stays off; **boiler Coolify env set to
   `true`**), seeded a minimal published Portal fixture (subdomain `e2e-readiness`),
   and fixed 2 pre-existing type-drift errors the route-tree typecheck surfaced.
7. **subscription:394, workflow-builder-run 528/816** — were timing flakes; green on
   clean reseeded runs (CI `retries:1` absorbs them).

**Commits (all on `main`, deployed):** `44e30fb3` (plugin fixture sep + races),
`147d92e4` (plugin settings .data fix), `011c6958` (admin user), `13edf803`+`944ef476`
(portals + type fixes), the Monaco fix, and `a9556041` (587 loader read-back, test-only).
**Coolify (boiler web):** added `PORTALS_ENABLED=true` (single row — deduped).

**Gotchas observed:** (a) a local-machine `net::ERR_INTERNET_DISCONNECTED` mid-run
cascades ~70 false failures from one point onward — re-run, it's not real. (b) loader
`.data` payloads embed the iframe HTML (~25KB) — search the FULL body, not a prefix.
(c) the plugin settings iframe quarantines its window on late `load`s; drive it via the
loader, not a post-reload `get`.

---

**Written:** 2026-06-02 00:10 UTC · **Repo HEAD at handoff:** `494f52de` (branch `main`,
also deployed to boiler staging) · **Companion docs:** `docs/testing/e2e-readiness-handoff.md`
(full bug-fix history), agent memory `boiler-e2e-remaining-tail.md`, `boiler-deploy-seed.md`,
`e2e-singlefetch-interceptor-contract.md`.

---

## 🚀 START HERE — NEXT AGENT (handoff from the 3rd agent, 2026-06-02 ~16:35 UTC)

**Current state:** `main` HEAD `d8aaeacb`, deployed to boiler (web image `3cbb12d3`; the two
commits after it are test-only/docs and don't change the runtime image). Full chromium run, freshly
reseeded: **192 pass / 11 fail / 5 skip**. The original 26 failures are mostly fixed; the 11 are the
hard tail below. The 5 skips are correctly-gated (real-Stripe success + email-verification inverse).

**What's already done (don't redo):** subscription/api-docs/golden-path/registration/workflow-builder-actions/
mobile-shell-drawer race+selector fixes; seed prunes leaked accounts; seed fully-provisions the
LIFECYCLE plugin; seed DELETES any operator-admin grant from the tenant user; host-aware admin path
`/email-sequences`. **Coolify env on boiler (set this session, persist):**
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` (web + worker-general + worker-plugin) and
`PLUGIN_HOOK_FANOUT_REQUIRED=true` (worker-general + worker-plugin). Coolify API:
`https://saas-ctrl.com/api/v1`, token `~/.cli-tokens/coolify-token`, web uuid
`mrf8lj70kmy9hqprx9dd6w7q` (see `boiler-deploy-seed.md`).

**How to run (everything-on):** `bash /tmp/e2e-cr.sh <specs?>` — it sources `.env.e2e-readiness.boiler`
and sets every opt-in + `REQUIRE_E2E_READINESS=1` (recreate it if /tmp was wiped; see §2/§4). **RESEED
before every run** (after every deploy too — /tmp plugin bundles are ephemeral):
`WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}' | grep mrf8lj | head -1"); ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 -e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js"`.
Seed source is `scripts/seed-e2e-readiness.ts`, built to that dist on deploy → **seed changes need a
deploy** (push main → ~20min; watch the web image tag flip, NOT the commit-status API). DB inspection
is now permitted (recipe in §4); you can also just psql the readiness plugin/grant state.

**The 11 remaining, in priority order (full root-cause + fixes in the "third agent" ITERATION LOG below):**

1. **[BIGGEST WIN — ~6 tests] Plugin cluster needs FIXTURE SEPARATION.** plugin-settings is **14/0 in
   isolation** but fails ~5 in the serial run as CROSS-SPEC POLLUTION: `plugin-hook-lifecycle.*` sorts
   first, mutates the SHARED `com.toolctrl.e2e-readiness` plugin (grants audit / deactivates), and the
   "enabled + audit-missing" baseline is UI-unrestorable (revoke auto-disables; `plugins.tsx:415`
   hides the enable form for disabled+missing-cap). **Fix:** seed a SECOND, fully-provisioned readiness
   plugin WITH a settings UI (clone `buildE2eReadinessPluginManifest` → new id e.g.
   `com.toolctrl.e2e-hooklifecycle`, grant ALL caps incl `hook:workflow.executed`), and repoint
   `plugin-hook-lifecycle.mobile-a11y.spec.ts` (146/201) + the lifecycle/iframe tests at it; keep the
   audit-missing `com.toolctrl.e2e-readiness` ONLY for the review/grant/deny tests (49/120/173/244).
   Then nothing mutates the audit-missing plugin's audit state. Also fix hook-lifecycle's
   `isCapabilityGranted` — after grantAndEnable the grant lands server-side (DB-confirmed) but the
   revoke-form detection misses it (inspect the rendered DOM for the audit revoke control).
2. **[2 tests] admin-email-sequences** — seed a DEDICATED admin user (`e2e.admin@example.test`,
   `AdminUser` + global `SuperAdmin` role, reuse `E2E_READINESS_USER_PASSWORD`); emit `E2E_ADMIN_EMAIL`;
   make `admin-email-sequences.mobile-a11y.spec.ts` log in as THAT user (not the tenant user). DO NOT
   grant the shared tenant user operator-admin — it redirects to the admin host on login and breaks
   auth-flow/registration (the seed now actively deletes that grant). The host-aware `/email-sequences`
   path is already wired and the cookie reaches the admin host (verified).
3. **[1 test] run/manual SSR 500 (`mobile-authenticated-shell.spec.ts:570`)** — product bug: the page
   renders but `entry.server.tsx:121` sets HTTP 500 on a swallowed streaming error. Add temporary
   onError logging (even pre-shell), deploy, reproduce the direct GET of `wf_123/run/manual`, capture
   the error (suspect a deferred/Suspense boundary), fix. SPA nav works; only direct SSR 500s.
4. **[1 test] portal-management** — PRODUCT DECISION (ask the operator): `__portals` is build-disabled
   in `routes.ts` `defaultDisabledPluginRouteFilePatterns`. Either ship the portals module, or accept
   not-shipped (then stop emitting the portal env paths and make the test skip).
5. **[1 test, likely flake] workflow-builder-run:816** — webhook manual-run result doesn't render in
   15s (worker-drain/SSE timing; target `https://postman-echo.com/post`). Bump the result wait or fix
   the streaming; CI `retries:1` may already absorb it.

**Gotchas:** (a) the deploy `coolify-ensure-worker-groups.mjs` drift check FAILS "Stage or promote" if
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` on a worker without `PLUGIN_HOOK_FANOUT_REQUIRED=true` — both
are already set, keep them paired. (b) Re-trigger a stalled/flaky deploy with an empty commit. (c)
a11y `color-contrast` failures on toasts/loaders mid-fade are transient (CI retries absorb; `axe.ts`
settles animations). (d) Don't weaken/skip to fake green (§3). Reserve the last ~20 min for the
handoff + memory update.

---

## ⏱️ ITERATION LOG — 2026-06-02 (third agent — root-caused & mostly fixed)

**Commits `894ba2a5` + `8bd11f9c` (empty re-trigger) DEPLOYED to boiler (web image `8bd11f9c`,
reseeded). Plugin runtime env + companion set on Coolify. Outcome: 26 originally-failing tests →
~6 remaining, all genuine product/design issues (root-caused below).**

**The previous "account-leakage" leading hypothesis was mostly WRONG** (verified by reading
all 26 traces, per §3). The 26 split into distinct, mostly-independent causes. Status per cluster:

CONFIRMED GREEN post-deploy (full a11y-strict env, chromium): plugin-settings full suite (14/0,
incl. the 244/344/380/473/587 cluster), plugin hook-audit (worker fanout env), lifecycle-plugin
enable/disable (431), registration family (35/1-skip, leak pruned), subscription ×3+nav,
workflow-builder-actions (incl. delete-block), api-docs, golden-path, mobile-shell drawer.

**ADMIN GRANT REVERTED (commit `5ed35fb5`):** granting the shared e2e-readiness tenant user
`AdminUser` + global SuperAdmin DID make admin-email-sequences ×2 pass, but it is an OPERATOR
admin → on login the app redirects to the **admin host** (`admin.boiler…/dashboard`), which broke
`auth-flow` + `registration` login-redirect assertions AND added a "Switch to admin" affordance +
teal `#00bba7` 2.42:1 text to the roles page (tenant-roles-permissions a11y ×3). Net regression, so
reverted. The host-aware `/email-sequences` PATH fix is kept (correct). **admin-email-sequences
now needs a DEDICATED admin user** (seed a separate `e2e.admin@…` AdminUser+SuperAdmin and point
the spec at it via new creds) — the shared tenant user cannot be both.

- **GREEN (validated individually on boiler, chromium):**
  - workflow-builder ×2 (+3 siblings) — one-shot `count()` raced the React Flow node mount.
  - subscription ×3 (`:80/:144/:711`) — SPA-nav race + the `:144` error test used a raw-JSON
    interceptor that breaks RR7 single-fetch; switched to `singleFetchActionError`.
  - readiness-golden-path (1) — mobile settings/nav links live in the "more" popover + a
    section `<Select>` combobox, not visible `<a>`s; helper now opens both + `evaluateAll`
    href scan (the old per-node loop hung on the streaming workflow page).
  - tenant-api-docs ×2 — waited for the lazy Monaco editor to attach before `count()`.
  - registration family: 35 pass / 1 skip. nav-link asserts now open the mobile sidebar Sheet;
    delete control located by `data-ui-id="ui.settings.account.delete"` (text is "Delete account");
    email-verification inverse test `test.skip`s under EXPECTS=0; bb379 retry-opens the popover.
  - plugin tests: 12/20 green after the **env fix** (see below). The .check() multi-checkbox
    strict-mode crash is fixed.
- **ENV FIX (Coolify, applied + redeployed):** `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` set on
  boiler **web + worker-general + worker-plugin** (was unset; NODE_ENV=production → every plugin
  enable/grant 403'd → tests couldn't self-restore → cascade). User-approved.
- **SEED FIX (in commit `894ba2a5`, needs the deploy+reseed to take effect):**
  - prune leaked test-created tenants (registration/account-create) + throwaway reg users, so the
    readiness/no-tenant users stop accumulating the multi-account picker.
  - grant operator-admin (`AdminUser`) + the **global SuperAdmin role** (tenantId=null) so admin
    pages resolve. **Admin path changed to `/email-sequences`** (host-aware: app host 308-redirects
    to the admin host, which page.goto follows; the readiness cookie spans the app-base domain —
    verified the probe reaches `admin.boiler…/email-sequences`). Was `/admin/email-sequences` (404
    on the app host).
  - fully provision the **lifecycle plugin** (capabilities + `hook:workflow.executed` consent) so a
    disabled plugin isn't trapped behind the review-before-enable gate.

### STILL OPEN — ~5 tests, all genuine product/design issues (root-caused, with fixes)

✅ **FIXED this iteration — plugin-settings readiness cluster (5: 244/344/380/473/587).** ROOT was:
the readiness plugin's seed baseline is ENABLED + `audit` MISSING (so review/grant tests have a cap
to review), but that's UI-unreachable once disabled — `revokePluginCapabilities` AUTO-DISABLES
(`:614`) and `plugins.tsx:415` hides the enable form for a disabled+missing-cap plugin. Test 244
(grant→revoke audit) auto-disabled it and the restore couldn't get back, cascading to 344/380/473/
587. FIX (committed): `restorePluginCapabilityState` now prioritises leaving the plugin ENABLED via
the review→grantAndEnable flow (accepting the unreachable enabled+audit-missing baseline; reseed
restores it between runs), and `.check()` targets all review checkboxes. plugin-settings now 14/0
**in isolation**.

⚠️ **BUT in the FULL serial run (CI), plugin-settings still shows ~5 failures — and they are
CROSS-SPEC POLLUTION, not the original cause.** `plugin-hook-lifecycle.*` sorts BEFORE
`plugin-settings.*`, so its mutating tests (146 grants audit; 201 grant/deactivate) run first on the
SHARED `com.toolctrl.e2e-readiness` plugin and leave it audit-GRANTED (or disabled) — then
plugin-settings' review/grant tests (49/120/173) fail with "fixture has no missing capability" because
audit is no longer missing. The "enabled + audit-missing" baseline is UI-unrestorable (revoke
auto-disables), so NO per-test restore can undo this in a single seeded serial run. **The only clean
fix is FIXTURE SEPARATION:** give the hook-lifecycle / lifecycle-toggle / iframe tests a DEDICATED
fully-provisioned plugin, and reserve the audit-missing readiness plugin solely for the review/grant
tests so nothing mutates its audit state. (This was a wash this iteration — failures shifted from
{344/380/473} to {49/120/173} — so the committed restore fix only helps once fixtures are split.)

1. **plugin-hook-lifecycle `:201` "deactivate then re-enable" (1).** STILL RED, but narrowed to a
   pure UI-detection issue: 201 now provisions audit first (grantAndEnable) so the bare toggle can
   run, and the grant **succeeds server-side** (DB shows the plugin enabled + audit granted) — but
   the test's `isCapabilityGranted` (which reads the capability *revoke form*) never detects the
   granted `audit` cap after grantAndEnable, even with a 10s poll. So the grant works; the
   verification heuristic doesn't see the audit revoke form in this state. Fix = inspect the
   rendered DOM for the audit revoke control after grantAndEnable and correct the
   `capabilityRevokeForm` selector / detection (or assert via the absence of the "review" affordance
   instead). Also: the prior test 146's restore revokes audit → auto-disables, so 201 inherits a
   disabled plugin — making 146's restore re-enable too would let 201 start clean.
3. **`wf_123/run/manual` returns HTTP 500 on direct SSR load.** Page renders fully; `entry.server`
   `onError` (`:121`) sets 500 on a recovered streaming error that is swallowed (pre-shell, unlogged).
   SPA nav works (golden-path passes); only direct SSR load of this one page 500s (builder direct-load
   is fine). Breaks only mobile-shell browser-history slice (`:570`). Real product bug — to fix, add
   temporary logging to `entry.server` onError (even pre-shell) and deploy to capture the streaming
   error (suspect a deferred/Suspense boundary unique to the run-manual route).
4. **portal-management (1).** `app.$tenant/__portals/**` is statically build-disabled
   (`routes.ts` `defaultDisabledPluginRouteFilePatterns`). The test is *designed* to fail-closed
   under readiness when portals aren't mounted (its own comment). Product/feature decision: ship the
   portals module, or accept not-shipped. Not a test bug.
5. **admin-email-sequences (2).** Need a DEDICATED admin user (see the ADMIN GRANT REVERTED note
   above) — the shared tenant user can't be operator-admin without breaking tenant login. The
   `/email-sequences` host-aware path + the permission analysis (needs global SuperAdmin role for
   `admin.marketing.leads.view`, AND AdminUser for the admin layout) are already worked out; just
   apply them to a separate seeded `e2e.admin@…` user the spec logs in as.
6. **workflow-builder-run `:816` "webhook workflow manual run posts a valid execution contract".**
   Direct-loads the webhook run page and waits 15s for `ui.workflows.run.manual.executionResult`;
   the webhook workflow executes via the worker (target `https://postman-echo.com/post`, reachable)
   but the result doesn't render in time — worker-drain/SSE timing, same family as the run/manual
   streaming issue (#3). Likely needs a longer result wait or the streaming-status fix.

**Transient a11y flakes (CI `retries:1` absorbs these — not real bugs):** several runs showed
`color-contrast` violations on toast/loader elements caught MID-TRANSITION (e.g. react-hot-toast
`#707c90` on `#fefeff` = 4.18 vs 4.5 while fading; the FloatingLoader class). `tests/e2e/utils/axe.ts`
already settles animations but heavy serial load occasionally beats it. The roles-page `#00bba7`
contrast was admin-grant chrome ("Switch to admin") and clears with the admin-grant removal.

### Coolify env changes applied this iteration (boiler, user-approved)
- `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` on web + worker-general + worker-plugin.
- `PLUGIN_HOOK_FANOUT_REQUIRED=true` on worker-general + worker-plugin (REQUIRED companion — the
  deploy's `coolify-ensure-worker-groups.mjs` drift check fails the "Stage or promote" job without
  it whenever PLUGIN_RUNTIME_PRODUCTION_ENABLED=true on a worker group).

---

## ⏱️ ITERATION LOG — 2026-06-02 (second agent, "everything-on" attempt)

**Status at end of this iteration: NOT green. 26 real, persistent failures isolated and
root-caused (leading hypothesis below). No code pushed — diagnosis-only handoff.**

**Why I stopped:** I was commissioned at `00:13 UTC`. The machine then slept ~8h between
background runs completing and my re-invocation (verified via `date -u`: jumped 00:51 → 09:02
UTC on consecutive turns, with only ~40 min of actual compute used). By the literal 7h
wall-clock rule (deadline ~07:13 UTC) I was already past budget when I got the triage results,
so per §0 I pivoted to a clean honest handoff instead of starting 15–40 min deploy/run cycles
that could get suspended mid-flight. **Next agent: you likely have full budget again — pick up
from "Concrete next steps" below.**

### What I ran
1. Fresh re-seed → full "everything-on" run (all opt-ins + `REQUIRE_E2E_READINESS=1` +
   `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1` with target `https://postman-echo.com/post`,
   `E2E_REQUIRE_PLUGIN_HOOK_AUDIT=1`). Result: **358 pass / 260 fail / 6 skip** (25.6m).
2. **208 of the 260 failures were a LOCAL ENV bug, not product/test:**
   `browserType.launch: Executable doesn't exist … webkit-####/pw_run.sh`. The `mobile-safari`
   project needs WebKit installed locally. **FIXED for the env:** ran `npx playwright install
   webkit` (CI does `playwright install --with-deps chromium webkit`; the local box only had
   chromium). Do this first on any fresh box or you'll chase 200 phantom failures.
3. Re-ran the 12 affected spec files **chromium-only, fresh seed, stable container** to strip
   the webkit noise and the run-1 mid-run container restart (web container id changed
   `…-000832351032` → `…-003251708343` during run 1 — Coolify recycled it, which also inflated
   run-1 failures). Clean result: **65 pass / 26 fail / 3 skip** (7.9m). These 26 are the REAL,
   reproducible failures.

### The 26 persistent failures (chromium; ×3 under the full browser matrix)
Grouped by root symptom (all throw because `REQUIRE_E2E_READINESS=1` turns "fixture missing"
into a hard fail):

- **Tenant feature surfaces appear absent though the seed populates them** (these tests PASSED
  in the 180-green baseline — so this is state regression, not the un-skip):
  - `admin-email-sequences:23,62` — `/admin/email-sequences` returns **404** ("Not Found").
  - `portal-management:16` — "Portal route … not mounted (portals module disabled)"; page
    snapshot actually shows the **"Select account — You belong to 13 accounts"** picker.
  - `readiness-golden-path:22` — "Subscription settings link was not rendered".
  - `subscription-smoke:80,138,699` — "No enabled checkout submit button rendered".
  - `tenant-api-docs-logs:70,100` — "Tenant API docs editor is not visible".
  - `mobile-authenticated-shell:563` — shell history slice (same family).
- **Workflow builder fixture lost its blocks:** `workflow-builder-actions:106,128` — "Workflow
  builder fixture needs at least one persisted block". (Note: the seed DOES delete+recreate
  `wf_123` incl. blocks at scripts/seed-e2e-readiness.ts:913, so a fresh seed *should* fix this —
  needs confirming it isn't an active-account-context render issue like the cluster above.)
- **Plugin lifecycle controls unavailable / not restorable:** `plugin-settings:244,337,373,424,
  466,580` + `plugin-hook-lifecycle:146,201` — "Plugin fixture has no manageable lifecycle
  controls", "Unable to grant/restore … lifecycle state". (Seed deletes plugin grants at :872;
  plugin binaries live in ephemeral `/tmp/e2e-plugins`.)
- **Newly-enabled registration/account real flows:** `registration:162,605,709,840,888` —
  created/!provisioned accounts hit a **"Subscribe"** gate (snapshot shows
  `/subscribe/e2-e-mobile-####` and no `ui.nav.app.dashboard`), i.e. a freshly created tenant
  has **no subscription/entitlement** so its dashboard legitimately differs from the seeded
  tenant — the test assumes they look alike. `registration:245` (email-verification) is a
  **clear test bug**: under `E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0` (boiler's mode) its
  inverse-pair guard calls `skipOrFailWhenE2eFixtureMissing(...)` which THROWS under readiness
  instead of skipping. It should skip (env genuinely can't exercise email-verification).

### Leading hypothesis (NOT yet proven — verify before fixing 26 things)
**Account leakage corrupts the e2e.readiness user's active-account resolution.** The
`E2E_ALLOW_ACCOUNT_CREATE` / registration flows create accounts owned by `e2e.readiness@example.test`
and **never delete them** (no `finally` restore). The user now belongs to **13 accounts**
(seen verbatim in the portal snapshot). The seed is per-feature destructive but does **NOT**
prune these leaked accounts or memberships. If login/`defaultTenantId` no longer resolves
cleanly to e2e-readiness, every surface that depends on the *active* account context
(subscription link, portal, admin, API-docs editor, workflow block render) appears empty even
though the e2e-readiness account itself is correctly seeded — which matches the exact symptom
set above (explicit-slug pages mostly work; context-dependent features don't). The direct DB
query to confirm account count/`defaultTenantId` was **blocked by the auto-mode classifier**
(production credential read) — next agent should get that permission or confirm via a trace.

### Concrete next steps (in order)
1. `npx playwright install webkit` on the runner (kills 208 phantom failures).
2. **Confirm/refute the leakage hypothesis.** Either (a) query the DB for how many accounts
   `e2e.readiness@example.test` belongs to and its `defaultTenantId` (recipe in §4 — needs the
   classifier to allow it), or (b) log in as the tenant in a trace and see if the active account
   is e2e-readiness. If leaked accounts are the cause:
   - Make the seed **prune** the e2e.readiness user's non-e2e-readiness `tenantUser` memberships
     and delete the orphaned leaked accounts (a destructive reset path keyed off
     `E2E_LOCAL_READINESS_SEED=1`), so a re-seed heals the corruption. Then re-run.
   - AND make `E2E_ALLOW_ACCOUNT_CREATE` / no-tenant tests **self-restore** (delete the account
     they create in a `finally`) per §1, so they stop leaking.
3. Fix `registration:245` email-verification: skip (don't throw) when
   `E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION` ≠ 1 — it's environment-determined, not a missing
   fixture (allowed under §3 "justifiably impossible").
4. For the created-account "Subscribe gate" failures (`registration:840,888,...`): decide whether
   the seed should grant the freshly-created tenant an entitlement, or the test should expect the
   subscribe-gated dashboard. Read the trace — don't assume.
5. Re-run the 12 specs chromium-only after each fix; only do a full matrix run + deploy once
   chromium is clean. Webhook real-run target `https://postman-echo.com/post` is reachable from
   staging (verified 200) and is the right disposable sink.

### Verified-good this iteration (don't re-investigate)
- Webhook target `https://postman-echo.com/post` reachable from the staging web container (POST
  → 200). Use it for `E2E_WORKFLOW_WEBHOOK_TARGET_URL`.
- Re-seed works and reports "E2E readiness fixtures seeded." (the trailing `EACCES … .env` is
  harmless). Helper scripts left at `/tmp/e2e-reseed.sh`, `/tmp/e2e-run.sh` (full matrix),
  `/tmp/e2e-run-chromium.sh` (fast triage) — they set the complete opt-in env.
- The 3 real-Stripe `subscription-success` tests (`subscription-smoke:198,260,603`) use
  `skipWhenE2eOperatorOptInMissing` (never fail-closed) and need an externally-acquired Stripe
  session path — they are **separate from the 28** and correctly stay skipped here.

---

## 0. Mission (read this twice)

The current full e2e run is **180 pass / 0 fail / 28 skip** against boiler
(`boiler.saas-ctrl.com`). The 28 skips are **operator-gated** tests that perform real,
destructive side effects and only run when explicit opt-in env flags are set.

**Your job: turn the 28 skipped tests ON, run the whole suite with every opt-in enabled
(a "disposable" / `REQUIRE_E2E_READINESS=1` run), and make ALL of them pass — honestly.**
Setup cleanly, run, read the results, fix the real cause (product code OR test), deploy if you
touched product code, re-seed, re-test. Loop until everything is green. Do not stop between
iterations. Do not ask for permission to continue — keep going.

### TIME BUDGET — check the real clock, do not trust your sense of time

- **You have 7 hours of real wall-clock time from the moment you were commissioned.**
- The FIRST thing you do every iteration: run `date -u` and read the actual time. Agents in a
  loop lose track of elapsed time — the clock is the only truth. Record your start time on the
  first iteration; your deadline = start + 7h.
- Re-check `date -u` at the top of every loop iteration and after every long wait (deploys take
  ~15–20 min each; a full suite run is ~10–14 min). Budget accordingly.
- **Reserve the last ~20 minutes** to commit everything, push, and update this handoff + the
  memory file with the final state — even if not 100% green. A clean honest handoff beats an
  abandoned run.
- Today's date drifts during long sessions; a `<system-reminder>` may announce a date change.
  Always prefer `date -u` output over your assumption.

---

## 1. The 28 skips and the flags that gate them

The skips are deliberate `skipOrFailWhenE2eOptInMissing(...)` / `skipUnless...` /
`skipIfUnavailable` gates. Under `REQUIRE_E2E_READINESS=1` those guards THROW instead of
skipping (fail-closed) — that is the mode the CI readiness lane uses and the mode you must make
pass. The opt-in env vars (grep the specs for the exact set; current list):

| Flag | Unlocks | Side effect / needs |
|------|---------|---------------------|
| `E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1` | plugin enable/disable + lifecycle persist (~6) | really toggles plugin enabled state — must self-restore |
| `E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1` | plugin settings write persist (1) | persists a setting + capability grant |
| `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1` + `E2E_WORKFLOW_WEBHOOK_TARGET_URL=<url>` | webhook workflow run (3) | posts to a REAL external URL — you must provide a reachable sink (e.g. a webhook.site or an in-cluster echo). Without a target these cannot pass. |
| `E2E_ALLOW_REGISTRATION_CREATE=1` | registration creates a real account (2) | creates users/tenants in the DB |
| `E2E_ALLOW_ACCOUNT_CREATE=1` | account creation (1) | creates a tenant |
| `E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1` | no-tenant account path (1) | creates a user |
| `E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1` | Stripe checkout return surface (2) | renders the post-checkout surface; check whether it needs Stripe test keys / a synthetic session |
| `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1` | already set in your runs | needs the general worker draining the outbox |
| `REQUIRE_E2E_READINESS=1` | makes every fixture/opt-in guard FAIL instead of skip | the fail-closed mode you must satisfy |

The authoritative source of truth for the exact flags + how they are wired is
`.gitea/workflows/e2e-readiness.yml` (the readiness lane, `workflow_dispatch` with these
operator inputs) — **read it first** and mirror its env exactly. Grep:
`grep -rhoE 'E2E_ALLOW_[A-Z_]+|skipOrFail[A-Za-z]+|skipUnless[A-Za-z]+|REQUIRE_E2E_READINESS' tests/e2e/*.spec.ts | sort -u`

**Important caveat:** enabling these makes the suite NON-idempotent (real grants, real
accounts, real toggles persist). Tests that mutate MUST self-restore (capture baseline →
restore in `finally`), and you must re-seed between full runs. If a newly-unskipped test isn't
self-restoring, that is part of the work — make it restore, do not leave leakage.

---

## 2. The work loop (never stop between steps)

```
loop:
  date -u                                  # 0. read the real clock; stop if < 25 min left
  source .env + set ALL opt-in flags + REQUIRE_E2E_READINESS=1
  re-seed staging (REQUIRED after every deploy AND before every full run — /tmp is ephemeral)
  run the full suite (single worker) OR the targeted newly-failing tests
  read results:
     - 0 fail  -> run the FULL suite once more to confirm stability; if green twice, DONE
     - failures -> for EACH: open the trace, find the REAL cause (below), fix it
  if you changed PRODUCT code (apps/web, packages/*): commit, push main, wait for deploy, re-seed
  if you changed only TEST code: re-run locally (no deploy needed)
  goto loop
```

Running commands (from repo root `/Users/raffaelsheikh/Documents/GitHub/boilerplate`):

```bash
# full run with EVERYTHING on (the readiness-equivalent mode)
set -a; source .env.e2e-readiness.boiler; set +a
export E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1 E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1 \
  E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1 E2E_ALLOW_REGISTRATION_CREATE=1 E2E_ALLOW_ACCOUNT_CREATE=1 \
  E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1 E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1 \
  REQUIRE_E2E_READINESS=1
# (webhook flag only once you have a real target URL — see §1)
npx playwright test --workers=1 --reporter=line > /tmp/run.log 2>&1
# single test by title:
npx playwright test --workers=1 --reporter=line -g "<exact test title>"
```

Long runs and deploys: launch them with `run_in_background: true` so you are notified on
completion instead of blocking; chain deploy→reseed→test in one background script (see §4) so
one notification covers a whole iteration.

---

## 3. Diagnose the REAL cause — do not cheat

Hard-won lesson from the previous session: **a read-only "look at the code" diagnosis is
frequently WRONG.** Always open the failing run's Playwright trace and read the actual page
snapshot + console + network. The previous effort found that several "test bugs" were real
product bugs (a dropdown that crashed the whole builder, an SSE loader that sent a blank
stream, an iframe channel race) and several "product bugs" were test bugs — only the traces
told the truth.

```bash
d=$(ls -dt test-results/*<slug>* | head -1)         # the failing test's result dir
mkdir -p /tmp/tr && unzip -oq "$d/trace.zip" -d /tmp/tr
# real console/page errors (NOT strings inside bundled JS in resources/):
python3 - <<'PY'
import json,glob
for f in glob.glob('/tmp/tr/*.trace'):
  for line in open(f,errors='ignore'):
    try: o=json.loads(line)
    except: continue
    if o.get('type') in ('console','pageerror') and (o.get('messageType')=='error' or o.get('type')=='pageerror'):
      print(o.get('text') or (o.get('error') or {}).get('message',''))
PY
# the page snapshot (what was actually on screen) is in:
cat "$d/error-context.md"
# axe violations: the failing node's `target` selector + `html` is in the assertion diff.
```

**No-cheating rules (non-negotiable):**
- Do **not** skip, `.fixme`, `xfail`, or delete a test to make the suite "green". The point is
  the opposite — un-skip.
- Do **not** weaken an assertion to hide a real failure (e.g. dropping a status check, loosening
  a regex to match anything, removing an a11y scan). You MAY relax an assertion only when the
  trace proves the test's expectation was wrong (e.g. a legitimate non-mutating audit POST the
  test mis-counted) — and document why.
- Do **not** disable an opt-in flag to dodge a test. If a test genuinely cannot run (e.g. webhook
  with no possible target), say so explicitly in the handoff with the reason — don't pretend.
- Do **not** loosen timeouts to mask a real hang. Longer timeouts are correct only for
  genuinely-slow real operations (the worker outbox poll is 90s; the per-test timeout is already
  120s in `playwright.config.ts`).
- Fix the ROOT. If a transient UI element trips an a11y scan, fix the wait/the element — don't
  delete the scan. If a real run flakes on worker timing, give the worker real time — don't fake
  the status.
- Distinguish a real flake from a real bug: re-run a suspect test **3–4× in isolation** after a
  fresh re-seed. Passes in isolation but fails in the serial run ⇒ cross-test state/order/rate
  issue (fix the leak or the helper). Fails in isolation too ⇒ real bug.

---

## 4. Environment cookbook (everything you need)

**Targets / access** (all verified working this session):
- App: `https://boiler.saas-ctrl.com`. Env file: `.env.e2e-readiness.boiler` (repo root).
- SSH: `ssh boilerplate-staging` (root@46.225.5.93, key `~/.ssh/node1`).
- Web container name pattern: `mrf8lj...` (app UUID `mrf8lj70kmy9hqprx9dd6w7q`). General worker
  (drains outbox + runs native executor): `wzshv933n2nhy0cvsxu8bqmm`.
- Gitea API: `https://git.tool-ctrl.de/api/v1`, repo `toolctrl-labs/boilerplate`, token
  `~/.cli-tokens/gitea` (the `gitea-ci-pat` lacks actions:read — use `gitea`).

**Re-seed (REQUIRED after every web deploy AND before every full run — `/tmp` is ephemeral):**
```bash
WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}\t{{.Image}}' | grep /web: | cut -f1 | head -1")
ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 \
  -e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js"
```
The seed source is `scripts/seed-e2e-readiness.ts` (built into that dist on deploy). It also runs
`syncDefaultRolesAndPermissions`, so a re-seed applies permission changes. The `EACCES ... .env`
line at the end is harmless (it just can't write the env file); the DB seeding succeeds.

**Deploying product-code changes:** push to `main` → the Gitea `deploy.yml` builds + swaps the
web/api/worker images. Deploys run **sequentially** (a new push queues behind the current one)
and take ~15–20 min. Ground truth that a deploy landed = the staging web image tag changes:
```bash
ssh boilerplate-staging "docker ps --format '{{.Image}}\t{{.Status}}' | grep /web: | head -1"
```
The Gitea **commit-status** API is unreliable here (sticks on "pending"); trust the image tag.

**The deploy step is intermittently flaky.** The `Stage or promote Docker images` job sometimes
fails transiently (registry/runner) while `Deploy preflight gates` passes — staging then stays on
the old image. Detect by: image tag not changing ~20 min after push. Re-trigger with an empty
commit: `git commit --allow-empty -m "chore: re-trigger deploy" && git push origin HEAD:main`.
**Use a self-healing background script** (proven this session) that: waits for the expected short
SHA on the image tag (poll ~21 min), and on timeout pushes an empty commit + waits again (cap 3
tries), then re-seeds and runs. This is the only way to run unattended overnight without a human
re-triggering. NOTE: an empty commit changes HEAD's SHA, so track the new SHA after each push.

**DB inspection (throwaway psql on the coolify network; strip the query string; dollar-quote
literals to dodge ssh quoting):**
```bash
ssh boilerplate-staging 'WEB=$(docker ps --format "{{.Names}}" | grep mrf8lj | head -1); \
  DBURL=$(docker exec $WEB printenv DATABASE_URL | sed "s/?.*//"); \
  docker run --rm --network coolify postgres:16-alpine psql "$DBURL" -t \
  -c "select ... where name=\$\$some.value\$\$;"'
```
Columns are quoted camelCase. Worker logs / outbox lastError: see `boiler-deploy-seed.md`.

---

## 5. Known patterns & gotchas (so you don't re-learn them)

- **RR7 single-fetch:** `<Form method=post>` actions POST to `<path>.data` and decode a
  **turbo-stream** body. A Playwright interceptor must match `.data` AND return turbo-stream, not
  `application/json`. Helpers in `tests/e2e/utils/navigation.ts`: `singleFetchActionError`,
  `singleFetchActionData`. See `e2e-singlefetch-interceptor-contract.md`.
- **Login rate limit:** 10 logins / 60s per IP+email (`AuthService.ts`). The test helper
  (`tests/e2e/utils/auth.ts`) caches the session cookie **per email** and reuses it — keep that
  intact. New tests that log in as new users (registration/account-create) WILL hammer this;
  if they flake on "stayed on /login", that's the rate limit — back off / reuse sessions, do not
  raise the production limit.
- **Re-seed before every run.** Plugin binary + state live in ephemeral `/tmp`; mutation tests
  drift state. Skipping a re-seed causes false failures (plugin "disabled", settings page blank).
- **Native vs http workflow executor:** staging uses native mode (`WORKFLOW_TRIGGER_EXECUTOR_MODE=native`,
  Coolify env). Do NOT un-gate `internal/api/__workflows/**` in routes.ts (a test enforces it).
- **Coolify env:** POST and PATCH both CREATE env rows → dupes. GET uuids and DELETE extras.
- **a11y scans:** `expectNoCriticalA11yViolations` now waits for the FloatingLoader overlay to
  clear before scanning (transient mid-fade contrast). Keep that.
- **Flakes shift:** with everything on and one serial worker against a shared remote tenant, the
  failing test changes run-to-run (login rate-limit, worker timing, real-SSE timing). Fix roots,
  and confirm with repeated isolation runs; CI uses `retries: 1`.
- **The CI readiness lane is the ultimate arbiter** (`.gitea/workflows/e2e-readiness.yml`,
  `workflow_dispatch`, in-cluster, retries). If you can trigger it via the Gitea API with the
  operator inputs, that is the cleanest "everything on" verification — but it needs the same
  flags and a real webhook target.

---

## 6. Definition of done

1. A full suite run with ALL opt-in flags + `REQUIRE_E2E_READINESS=1` set, **freshly re-seeded**,
   ends with **0 failed and 0 skipped** (every test either passes or is justifiably impossible —
   and if impossible, documented with the concrete reason, not hidden).
2. **Confirm stability:** the green run reproduces — run the full suite twice clean, or re-run any
   test that flaked 3–4× in isolation, all green.
3. Every fix is committed to `main` with a clear message and (for product code) deployed; the
   working tree is clean (ignore the pre-existing `packages/db/generated/prisma/**` churn).
4. Update THIS file and the memory `boiler-e2e-remaining-tail.md` with the final numbers, what you
   changed, and anything genuinely blocked (e.g. webhook target unavailable) with the reason.
5. No test was skipped, weakened, or deleted to manufacture green (§3). If you ran out of the
   7-hour budget, write down exactly where you stopped, the last run's numbers, and the next step.

Start by running `date -u`, reading `.gitea/workflows/e2e-readiness.yml`, then doing one full
"everything on" re-seeded run to see the real starting failure set. Then loop. Good luck.