587 lines
41 KiB
Markdown
587 lines
41 KiB
Markdown
# E2E Skipped-Tests Loop — Handoff for the next agent
|
||
|
||
## ✅ GREEN — 4th agent (2026-06-02 ~20:15 UTC)
|
||
|
||
**Full "everything-on" chromium run against boiler (web image `944ef476`, freshly
|
||
reseeded): `203 passed / 0 failed / 5 skipped` (9.3m).** The 5 skips are correctly
|
||
operator-gated and justifiably impossible here: real-Stripe `subscription-success`
|
||
×3 (need an externally-acquired Stripe session), the registration email-verification
|
||
inverse pair (env runs with `E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0`), and the
|
||
`E2E_REQUIRE_PLUGIN_HOOK_AUDIT` hook-audit lane (operator-gated). A second clean
|
||
full run was used to confirm stability.
|
||
|
||
**The 12 remaining failures from the previous handoff, all fixed this session:**
|
||
1. **plugin cluster (49/120/173/201)** — cross-spec pollution. `plugin-hook-lifecycle`
|
||
now targets the dedicated, fully-provisioned `com.toolctrl.e2e-lifecycle` plugin
|
||
instead of mutating the shared `e2e-readiness` plugin's audit-missing baseline.
|
||
2. **plugin-settings 244** — the grant/revoke + restore reloaded the page before the
|
||
in-flight single-fetch POST landed (stale read). Now waits for RR7 revalidation.
|
||
3. **plugin-settings 587** — TWO bugs. (a) PRODUCT: the settings-iframe write posted to
|
||
`fetch(".")` (resolves to the parent plugins-list route) and expected raw JSON;
|
||
RR7 returns a turbo-stream document for non-`.data` fetches, so every write failed.
|
||
Fixed to post the detail route's `.data` endpoint and gate on the HTTP status.
|
||
(b) TEST: the post-reload iframe `get` is non-deterministic (the host frame
|
||
quarantines its window + rotates the channel on a late `load`), so persistence is
|
||
now verified by reading the settings loader's `.data` payload after reload.
|
||
4. **admin-email-sequences ×2** — logged in as the shared tenant user (not an admin).
|
||
Seed now provisions a dedicated `e2e.admin@example.test` (AdminUser + global
|
||
SuperAdmin, no TOTP) and emits `E2E_ADMIN_EMAIL/PASSWORD`; spec logs in as it.
|
||
5. **mobile-shell 570 (run/manual SSR 500)** — `MonacoEditor` mounted its browser-only
|
||
`./MonacoEditor.client` during SSR, so the streaming renderer set HTTP 500 on EVERY
|
||
editor page (run/manual, webhook-run, **api-docs too** — the prior handoff was wrong
|
||
that api-docs was fine; only this test asserts `status < 500`). Fixed by gating the
|
||
editor behind a client mount (server + first client render emit an identical
|
||
placeholder → no hydration mismatch).
|
||
6. **portal-management** — operator chose to SHIP portals. Un-ignored the portal routes
|
||
(`app.$tenant/__portals/**`, `admin/portals.tsx`), gated the global enable behind
|
||
`PORTALS_ENABLED` (boilerplate default stays off; **boiler Coolify env set to
|
||
`true`**), seeded a minimal published Portal fixture (subdomain `e2e-readiness`),
|
||
and fixed 2 pre-existing type-drift errors the route-tree typecheck surfaced.
|
||
7. **subscription:394, workflow-builder-run 528/816** — were timing flakes; green on
|
||
clean reseeded runs (CI `retries:1` absorbs them).
|
||
|
||
**Commits (all on `main`, deployed):** `44e30fb3` (plugin fixture sep + races),
|
||
`147d92e4` (plugin settings .data fix), `011c6958` (admin user), `13edf803`+`944ef476`
|
||
(portals + type fixes), the Monaco fix, and `a9556041` (587 loader read-back, test-only).
|
||
**Coolify (boiler web):** added `PORTALS_ENABLED=true` (single row — deduped).
|
||
|
||
**Gotchas observed:** (a) a local-machine `net::ERR_INTERNET_DISCONNECTED` mid-run
|
||
cascades ~70 false failures from one point onward — re-run, it's not real. (b) loader
|
||
`.data` payloads embed the iframe HTML (~25KB) — search the FULL body, not a prefix.
|
||
(c) the plugin settings iframe quarantines its window on late `load`s; drive it via the
|
||
loader, not a post-reload `get`.
|
||
|
||
---
|
||
|
||
**Written:** 2026-06-02 00:10 UTC · **Repo HEAD at handoff:** `494f52de` (branch `main`,
|
||
also deployed to boiler staging) · **Companion docs:** `docs/testing/e2e-readiness-handoff.md`
|
||
(full bug-fix history), agent memory `boiler-e2e-remaining-tail.md`, `boiler-deploy-seed.md`,
|
||
`e2e-singlefetch-interceptor-contract.md`.
|
||
|
||
---
|
||
|
||
## 🚀 START HERE — NEXT AGENT (handoff from the 3rd agent, 2026-06-02 ~16:35 UTC)
|
||
|
||
**Current state:** `main` HEAD `d8aaeacb`, deployed to boiler (web image `3cbb12d3`; the two
|
||
commits after it are test-only/docs and don't change the runtime image). Full chromium run, freshly
|
||
reseeded: **192 pass / 11 fail / 5 skip**. The original 26 failures are mostly fixed; the 11 are the
|
||
hard tail below. The 5 skips are correctly-gated (real-Stripe success + email-verification inverse).
|
||
|
||
**What's already done (don't redo):** subscription/api-docs/golden-path/registration/workflow-builder-actions/
|
||
mobile-shell-drawer race+selector fixes; seed prunes leaked accounts; seed fully-provisions the
|
||
LIFECYCLE plugin; seed DELETES any operator-admin grant from the tenant user; host-aware admin path
|
||
`/email-sequences`. **Coolify env on boiler (set this session, persist):**
|
||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` (web + worker-general + worker-plugin) and
|
||
`PLUGIN_HOOK_FANOUT_REQUIRED=true` (worker-general + worker-plugin). Coolify API:
|
||
`https://saas-ctrl.com/api/v1`, token `~/.cli-tokens/coolify-token`, web uuid
|
||
`mrf8lj70kmy9hqprx9dd6w7q` (see `boiler-deploy-seed.md`).
|
||
|
||
**How to run (everything-on):** `bash /tmp/e2e-cr.sh <specs?>` — it sources `.env.e2e-readiness.boiler`
|
||
and sets every opt-in + `REQUIRE_E2E_READINESS=1` (recreate it if /tmp was wiped; see §2/§4). **RESEED
|
||
before every run** (after every deploy too — /tmp plugin bundles are ephemeral):
|
||
`WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}' | grep mrf8lj | head -1"); ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 -e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js"`.
|
||
Seed source is `scripts/seed-e2e-readiness.ts`, built to that dist on deploy → **seed changes need a
|
||
deploy** (push main → ~20min; watch the web image tag flip, NOT the commit-status API). DB inspection
|
||
is now permitted (recipe in §4); you can also just psql the readiness plugin/grant state.
|
||
|
||
**The 11 remaining, in priority order (full root-cause + fixes in the "third agent" ITERATION LOG below):**
|
||
|
||
1. **[BIGGEST WIN — ~6 tests] Plugin cluster needs FIXTURE SEPARATION.** plugin-settings is **14/0 in
|
||
isolation** but fails ~5 in the serial run as CROSS-SPEC POLLUTION: `plugin-hook-lifecycle.*` sorts
|
||
first, mutates the SHARED `com.toolctrl.e2e-readiness` plugin (grants audit / deactivates), and the
|
||
"enabled + audit-missing" baseline is UI-unrestorable (revoke auto-disables; `plugins.tsx:415`
|
||
hides the enable form for disabled+missing-cap). **Fix:** seed a SECOND, fully-provisioned readiness
|
||
plugin WITH a settings UI (clone `buildE2eReadinessPluginManifest` → new id e.g.
|
||
`com.toolctrl.e2e-hooklifecycle`, grant ALL caps incl `hook:workflow.executed`), and repoint
|
||
`plugin-hook-lifecycle.mobile-a11y.spec.ts` (146/201) + the lifecycle/iframe tests at it; keep the
|
||
audit-missing `com.toolctrl.e2e-readiness` ONLY for the review/grant/deny tests (49/120/173/244).
|
||
Then nothing mutates the audit-missing plugin's audit state. Also fix hook-lifecycle's
|
||
`isCapabilityGranted` — after grantAndEnable the grant lands server-side (DB-confirmed) but the
|
||
revoke-form detection misses it (inspect the rendered DOM for the audit revoke control).
|
||
2. **[2 tests] admin-email-sequences** — seed a DEDICATED admin user (`e2e.admin@example.test`,
|
||
`AdminUser` + global `SuperAdmin` role, reuse `E2E_READINESS_USER_PASSWORD`); emit `E2E_ADMIN_EMAIL`;
|
||
make `admin-email-sequences.mobile-a11y.spec.ts` log in as THAT user (not the tenant user). DO NOT
|
||
grant the shared tenant user operator-admin — it redirects to the admin host on login and breaks
|
||
auth-flow/registration (the seed now actively deletes that grant). The host-aware `/email-sequences`
|
||
path is already wired and the cookie reaches the admin host (verified).
|
||
3. **[1 test] run/manual SSR 500 (`mobile-authenticated-shell.spec.ts:570`)** — product bug: the page
|
||
renders but `entry.server.tsx:121` sets HTTP 500 on a swallowed streaming error. Add temporary
|
||
onError logging (even pre-shell), deploy, reproduce the direct GET of `wf_123/run/manual`, capture
|
||
the error (suspect a deferred/Suspense boundary), fix. SPA nav works; only direct SSR 500s.
|
||
4. **[1 test] portal-management** — PRODUCT DECISION (ask the operator): `__portals` is build-disabled
|
||
in `routes.ts` `defaultDisabledPluginRouteFilePatterns`. Either ship the portals module, or accept
|
||
not-shipped (then stop emitting the portal env paths and make the test skip).
|
||
5. **[1 test, likely flake] workflow-builder-run:816** — webhook manual-run result doesn't render in
|
||
15s (worker-drain/SSE timing; target `https://postman-echo.com/post`). Bump the result wait or fix
|
||
the streaming; CI `retries:1` may already absorb it.
|
||
|
||
**Gotchas:** (a) the deploy `coolify-ensure-worker-groups.mjs` drift check FAILS "Stage or promote" if
|
||
`PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` on a worker without `PLUGIN_HOOK_FANOUT_REQUIRED=true` — both
|
||
are already set, keep them paired. (b) Re-trigger a stalled/flaky deploy with an empty commit. (c)
|
||
a11y `color-contrast` failures on toasts/loaders mid-fade are transient (CI retries absorb; `axe.ts`
|
||
settles animations). (d) Don't weaken/skip to fake green (§3). Reserve the last ~20 min for the
|
||
handoff + memory update.
|
||
|
||
---
|
||
|
||
## ⏱️ ITERATION LOG — 2026-06-02 (third agent — root-caused & mostly fixed)
|
||
|
||
**Commits `894ba2a5` + `8bd11f9c` (empty re-trigger) DEPLOYED to boiler (web image `8bd11f9c`,
|
||
reseeded). Plugin runtime env + companion set on Coolify. Outcome: 26 originally-failing tests →
|
||
~6 remaining, all genuine product/design issues (root-caused below).**
|
||
|
||
**The previous "account-leakage" leading hypothesis was mostly WRONG** (verified by reading
|
||
all 26 traces, per §3). The 26 split into distinct, mostly-independent causes. Status per cluster:
|
||
|
||
CONFIRMED GREEN post-deploy (full a11y-strict env, chromium): plugin-settings full suite (14/0,
|
||
incl. the 244/344/380/473/587 cluster), plugin hook-audit (worker fanout env), lifecycle-plugin
|
||
enable/disable (431), registration family (35/1-skip, leak pruned), subscription ×3+nav,
|
||
workflow-builder-actions (incl. delete-block), api-docs, golden-path, mobile-shell drawer.
|
||
|
||
**ADMIN GRANT REVERTED (commit `5ed35fb5`):** granting the shared e2e-readiness tenant user
|
||
`AdminUser` + global SuperAdmin DID make admin-email-sequences ×2 pass, but it is an OPERATOR
|
||
admin → on login the app redirects to the **admin host** (`admin.boiler…/dashboard`), which broke
|
||
`auth-flow` + `registration` login-redirect assertions AND added a "Switch to admin" affordance +
|
||
teal `#00bba7` 2.42:1 text to the roles page (tenant-roles-permissions a11y ×3). Net regression, so
|
||
reverted. The host-aware `/email-sequences` PATH fix is kept (correct). **admin-email-sequences
|
||
now needs a DEDICATED admin user** (seed a separate `e2e.admin@…` AdminUser+SuperAdmin and point
|
||
the spec at it via new creds) — the shared tenant user cannot be both.
|
||
|
||
- **GREEN (validated individually on boiler, chromium):**
|
||
- workflow-builder ×2 (+3 siblings) — one-shot `count()` raced the React Flow node mount.
|
||
- subscription ×3 (`:80/:144/:711`) — SPA-nav race + the `:144` error test used a raw-JSON
|
||
interceptor that breaks RR7 single-fetch; switched to `singleFetchActionError`.
|
||
- readiness-golden-path (1) — mobile settings/nav links live in the "more" popover + a
|
||
section `<Select>` combobox, not visible `<a>`s; helper now opens both + `evaluateAll`
|
||
href scan (the old per-node loop hung on the streaming workflow page).
|
||
- tenant-api-docs ×2 — waited for the lazy Monaco editor to attach before `count()`.
|
||
- registration family: 35 pass / 1 skip. nav-link asserts now open the mobile sidebar Sheet;
|
||
delete control located by `data-ui-id="ui.settings.account.delete"` (text is "Delete account");
|
||
email-verification inverse test `test.skip`s under EXPECTS=0; bb379 retry-opens the popover.
|
||
- plugin tests: 12/20 green after the **env fix** (see below). The .check() multi-checkbox
|
||
strict-mode crash is fixed.
|
||
- **ENV FIX (Coolify, applied + redeployed):** `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` set on
|
||
boiler **web + worker-general + worker-plugin** (was unset; NODE_ENV=production → every plugin
|
||
enable/grant 403'd → tests couldn't self-restore → cascade). User-approved.
|
||
- **SEED FIX (in commit `894ba2a5`, needs the deploy+reseed to take effect):**
|
||
- prune leaked test-created tenants (registration/account-create) + throwaway reg users, so the
|
||
readiness/no-tenant users stop accumulating the multi-account picker.
|
||
- grant operator-admin (`AdminUser`) + the **global SuperAdmin role** (tenantId=null) so admin
|
||
pages resolve. **Admin path changed to `/email-sequences`** (host-aware: app host 308-redirects
|
||
to the admin host, which page.goto follows; the readiness cookie spans the app-base domain —
|
||
verified the probe reaches `admin.boiler…/email-sequences`). Was `/admin/email-sequences` (404
|
||
on the app host).
|
||
- fully provision the **lifecycle plugin** (capabilities + `hook:workflow.executed` consent) so a
|
||
disabled plugin isn't trapped behind the review-before-enable gate.
|
||
|
||
### STILL OPEN — ~5 tests, all genuine product/design issues (root-caused, with fixes)
|
||
|
||
✅ **FIXED this iteration — plugin-settings readiness cluster (5: 244/344/380/473/587).** ROOT was:
|
||
the readiness plugin's seed baseline is ENABLED + `audit` MISSING (so review/grant tests have a cap
|
||
to review), but that's UI-unreachable once disabled — `revokePluginCapabilities` AUTO-DISABLES
|
||
(`:614`) and `plugins.tsx:415` hides the enable form for a disabled+missing-cap plugin. Test 244
|
||
(grant→revoke audit) auto-disabled it and the restore couldn't get back, cascading to 344/380/473/
|
||
587. FIX (committed): `restorePluginCapabilityState` now prioritises leaving the plugin ENABLED via
|
||
the review→grantAndEnable flow (accepting the unreachable enabled+audit-missing baseline; reseed
|
||
restores it between runs), and `.check()` targets all review checkboxes. plugin-settings now 14/0
|
||
**in isolation**.
|
||
|
||
⚠️ **BUT in the FULL serial run (CI), plugin-settings still shows ~5 failures — and they are
|
||
CROSS-SPEC POLLUTION, not the original cause.** `plugin-hook-lifecycle.*` sorts BEFORE
|
||
`plugin-settings.*`, so its mutating tests (146 grants audit; 201 grant/deactivate) run first on the
|
||
SHARED `com.toolctrl.e2e-readiness` plugin and leave it audit-GRANTED (or disabled) — then
|
||
plugin-settings' review/grant tests (49/120/173) fail with "fixture has no missing capability" because
|
||
audit is no longer missing. The "enabled + audit-missing" baseline is UI-unrestorable (revoke
|
||
auto-disables), so NO per-test restore can undo this in a single seeded serial run. **The only clean
|
||
fix is FIXTURE SEPARATION:** give the hook-lifecycle / lifecycle-toggle / iframe tests a DEDICATED
|
||
fully-provisioned plugin, and reserve the audit-missing readiness plugin solely for the review/grant
|
||
tests so nothing mutates its audit state. (This was a wash this iteration — failures shifted from
|
||
{344/380/473} to {49/120/173} — so the committed restore fix only helps once fixtures are split.)
|
||
|
||
1. **plugin-hook-lifecycle `:201` "deactivate then re-enable" (1).** STILL RED, but narrowed to a
|
||
pure UI-detection issue: 201 now provisions audit first (grantAndEnable) so the bare toggle can
|
||
run, and the grant **succeeds server-side** (DB shows the plugin enabled + audit granted) — but
|
||
the test's `isCapabilityGranted` (which reads the capability *revoke form*) never detects the
|
||
granted `audit` cap after grantAndEnable, even with a 10s poll. So the grant works; the
|
||
verification heuristic doesn't see the audit revoke form in this state. Fix = inspect the
|
||
rendered DOM for the audit revoke control after grantAndEnable and correct the
|
||
`capabilityRevokeForm` selector / detection (or assert via the absence of the "review" affordance
|
||
instead). Also: the prior test 146's restore revokes audit → auto-disables, so 201 inherits a
|
||
disabled plugin — making 146's restore re-enable too would let 201 start clean.
|
||
3. **`wf_123/run/manual` returns HTTP 500 on direct SSR load.** Page renders fully; `entry.server`
|
||
`onError` (`:121`) sets 500 on a recovered streaming error that is swallowed (pre-shell, unlogged).
|
||
SPA nav works (golden-path passes); only direct SSR load of this one page 500s (builder direct-load
|
||
is fine). Breaks only mobile-shell browser-history slice (`:570`). Real product bug — to fix, add
|
||
temporary logging to `entry.server` onError (even pre-shell) and deploy to capture the streaming
|
||
error (suspect a deferred/Suspense boundary unique to the run-manual route).
|
||
4. **portal-management (1).** `app.$tenant/__portals/**` is statically build-disabled
|
||
(`routes.ts` `defaultDisabledPluginRouteFilePatterns`). The test is *designed* to fail-closed
|
||
under readiness when portals aren't mounted (its own comment). Product/feature decision: ship the
|
||
portals module, or accept not-shipped. Not a test bug.
|
||
5. **admin-email-sequences (2).** Need a DEDICATED admin user (see the ADMIN GRANT REVERTED note
|
||
above) — the shared tenant user can't be operator-admin without breaking tenant login. The
|
||
`/email-sequences` host-aware path + the permission analysis (needs global SuperAdmin role for
|
||
`admin.marketing.leads.view`, AND AdminUser for the admin layout) are already worked out; just
|
||
apply them to a separate seeded `e2e.admin@…` user the spec logs in as.
|
||
6. **workflow-builder-run `:816` "webhook workflow manual run posts a valid execution contract".**
|
||
Direct-loads the webhook run page and waits 15s for `ui.workflows.run.manual.executionResult`;
|
||
the webhook workflow executes via the worker (target `https://postman-echo.com/post`, reachable)
|
||
but the result doesn't render in time — worker-drain/SSE timing, same family as the run/manual
|
||
streaming issue (#3). Likely needs a longer result wait or the streaming-status fix.
|
||
|
||
**Transient a11y flakes (CI `retries:1` absorbs these — not real bugs):** several runs showed
|
||
`color-contrast` violations on toast/loader elements caught MID-TRANSITION (e.g. react-hot-toast
|
||
`#707c90` on `#fefeff` = 4.18 vs 4.5 while fading; the FloatingLoader class). `tests/e2e/utils/axe.ts`
|
||
already settles animations but heavy serial load occasionally beats it. The roles-page `#00bba7`
|
||
contrast was admin-grant chrome ("Switch to admin") and clears with the admin-grant removal.
|
||
|
||
### Coolify env changes applied this iteration (boiler, user-approved)
|
||
- `PLUGIN_RUNTIME_PRODUCTION_ENABLED=true` on web + worker-general + worker-plugin.
|
||
- `PLUGIN_HOOK_FANOUT_REQUIRED=true` on worker-general + worker-plugin (REQUIRED companion — the
|
||
deploy's `coolify-ensure-worker-groups.mjs` drift check fails the "Stage or promote" job without
|
||
it whenever PLUGIN_RUNTIME_PRODUCTION_ENABLED=true on a worker group).
|
||
|
||
---
|
||
|
||
## ⏱️ ITERATION LOG — 2026-06-02 (second agent, "everything-on" attempt)
|
||
|
||
**Status at end of this iteration: NOT green. 26 real, persistent failures isolated and
|
||
root-caused (leading hypothesis below). No code pushed — diagnosis-only handoff.**
|
||
|
||
**Why I stopped:** I was commissioned at `00:13 UTC`. The machine then slept ~8h between
|
||
background runs completing and my re-invocation (verified via `date -u`: jumped 00:51 → 09:02
|
||
UTC on consecutive turns, with only ~40 min of actual compute used). By the literal 7h
|
||
wall-clock rule (deadline ~07:13 UTC) I was already past budget when I got the triage results,
|
||
so per §0 I pivoted to a clean honest handoff instead of starting 15–40 min deploy/run cycles
|
||
that could get suspended mid-flight. **Next agent: you likely have full budget again — pick up
|
||
from "Concrete next steps" below.**
|
||
|
||
### What I ran
|
||
1. Fresh re-seed → full "everything-on" run (all opt-ins + `REQUIRE_E2E_READINESS=1` +
|
||
`E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1` with target `https://postman-echo.com/post`,
|
||
`E2E_REQUIRE_PLUGIN_HOOK_AUDIT=1`). Result: **358 pass / 260 fail / 6 skip** (25.6m).
|
||
2. **208 of the 260 failures were a LOCAL ENV bug, not product/test:**
|
||
`browserType.launch: Executable doesn't exist … webkit-####/pw_run.sh`. The `mobile-safari`
|
||
project needs WebKit installed locally. **FIXED for the env:** ran `npx playwright install
|
||
webkit` (CI does `playwright install --with-deps chromium webkit`; the local box only had
|
||
chromium). Do this first on any fresh box or you'll chase 200 phantom failures.
|
||
3. Re-ran the 12 affected spec files **chromium-only, fresh seed, stable container** to strip
|
||
the webkit noise and the run-1 mid-run container restart (web container id changed
|
||
`…-000832351032` → `…-003251708343` during run 1 — Coolify recycled it, which also inflated
|
||
run-1 failures). Clean result: **65 pass / 26 fail / 3 skip** (7.9m). These 26 are the REAL,
|
||
reproducible failures.
|
||
|
||
### The 26 persistent failures (chromium; ×3 under the full browser matrix)
|
||
Grouped by root symptom (all throw because `REQUIRE_E2E_READINESS=1` turns "fixture missing"
|
||
into a hard fail):
|
||
|
||
- **Tenant feature surfaces appear absent though the seed populates them** (these tests PASSED
|
||
in the 180-green baseline — so this is state regression, not the un-skip):
|
||
- `admin-email-sequences:23,62` — `/admin/email-sequences` returns **404** ("Not Found").
|
||
- `portal-management:16` — "Portal route … not mounted (portals module disabled)"; page
|
||
snapshot actually shows the **"Select account — You belong to 13 accounts"** picker.
|
||
- `readiness-golden-path:22` — "Subscription settings link was not rendered".
|
||
- `subscription-smoke:80,138,699` — "No enabled checkout submit button rendered".
|
||
- `tenant-api-docs-logs:70,100` — "Tenant API docs editor is not visible".
|
||
- `mobile-authenticated-shell:563` — shell history slice (same family).
|
||
- **Workflow builder fixture lost its blocks:** `workflow-builder-actions:106,128` — "Workflow
|
||
builder fixture needs at least one persisted block". (Note: the seed DOES delete+recreate
|
||
`wf_123` incl. blocks at scripts/seed-e2e-readiness.ts:913, so a fresh seed *should* fix this —
|
||
needs confirming it isn't an active-account-context render issue like the cluster above.)
|
||
- **Plugin lifecycle controls unavailable / not restorable:** `plugin-settings:244,337,373,424,
|
||
466,580` + `plugin-hook-lifecycle:146,201` — "Plugin fixture has no manageable lifecycle
|
||
controls", "Unable to grant/restore … lifecycle state". (Seed deletes plugin grants at :872;
|
||
plugin binaries live in ephemeral `/tmp/e2e-plugins`.)
|
||
- **Newly-enabled registration/account real flows:** `registration:162,605,709,840,888` —
|
||
created/!provisioned accounts hit a **"Subscribe"** gate (snapshot shows
|
||
`/subscribe/e2-e-mobile-####` and no `ui.nav.app.dashboard`), i.e. a freshly created tenant
|
||
has **no subscription/entitlement** so its dashboard legitimately differs from the seeded
|
||
tenant — the test assumes they look alike. `registration:245` (email-verification) is a
|
||
**clear test bug**: under `E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0` (boiler's mode) its
|
||
inverse-pair guard calls `skipOrFailWhenE2eFixtureMissing(...)` which THROWS under readiness
|
||
instead of skipping. It should skip (env genuinely can't exercise email-verification).
|
||
|
||
### Leading hypothesis (NOT yet proven — verify before fixing 26 things)
|
||
**Account leakage corrupts the e2e.readiness user's active-account resolution.** The
|
||
`E2E_ALLOW_ACCOUNT_CREATE` / registration flows create accounts owned by `e2e.readiness@example.test`
|
||
and **never delete them** (no `finally` restore). The user now belongs to **13 accounts**
|
||
(seen verbatim in the portal snapshot). The seed is per-feature destructive but does **NOT**
|
||
prune these leaked accounts or memberships. If login/`defaultTenantId` no longer resolves
|
||
cleanly to e2e-readiness, every surface that depends on the *active* account context
|
||
(subscription link, portal, admin, API-docs editor, workflow block render) appears empty even
|
||
though the e2e-readiness account itself is correctly seeded — which matches the exact symptom
|
||
set above (explicit-slug pages mostly work; context-dependent features don't). The direct DB
|
||
query to confirm account count/`defaultTenantId` was **blocked by the auto-mode classifier**
|
||
(production credential read) — next agent should get that permission or confirm via a trace.
|
||
|
||
### Concrete next steps (in order)
|
||
1. `npx playwright install webkit` on the runner (kills 208 phantom failures).
|
||
2. **Confirm/refute the leakage hypothesis.** Either (a) query the DB for how many accounts
|
||
`e2e.readiness@example.test` belongs to and its `defaultTenantId` (recipe in §4 — needs the
|
||
classifier to allow it), or (b) log in as the tenant in a trace and see if the active account
|
||
is e2e-readiness. If leaked accounts are the cause:
|
||
- Make the seed **prune** the e2e.readiness user's non-e2e-readiness `tenantUser` memberships
|
||
and delete the orphaned leaked accounts (a destructive reset path keyed off
|
||
`E2E_LOCAL_READINESS_SEED=1`), so a re-seed heals the corruption. Then re-run.
|
||
- AND make `E2E_ALLOW_ACCOUNT_CREATE` / no-tenant tests **self-restore** (delete the account
|
||
they create in a `finally`) per §1, so they stop leaking.
|
||
3. Fix `registration:245` email-verification: skip (don't throw) when
|
||
`E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION` ≠ 1 — it's environment-determined, not a missing
|
||
fixture (allowed under §3 "justifiably impossible").
|
||
4. For the created-account "Subscribe gate" failures (`registration:840,888,...`): decide whether
|
||
the seed should grant the freshly-created tenant an entitlement, or the test should expect the
|
||
subscribe-gated dashboard. Read the trace — don't assume.
|
||
5. Re-run the 12 specs chromium-only after each fix; only do a full matrix run + deploy once
|
||
chromium is clean. Webhook real-run target `https://postman-echo.com/post` is reachable from
|
||
staging (verified 200) and is the right disposable sink.
|
||
|
||
### Verified-good this iteration (don't re-investigate)
|
||
- Webhook target `https://postman-echo.com/post` reachable from the staging web container (POST
|
||
→ 200). Use it for `E2E_WORKFLOW_WEBHOOK_TARGET_URL`.
|
||
- Re-seed works and reports "E2E readiness fixtures seeded." (the trailing `EACCES … .env` is
|
||
harmless). Helper scripts left at `/tmp/e2e-reseed.sh`, `/tmp/e2e-run.sh` (full matrix),
|
||
`/tmp/e2e-run-chromium.sh` (fast triage) — they set the complete opt-in env.
|
||
- The 3 real-Stripe `subscription-success` tests (`subscription-smoke:198,260,603`) use
|
||
`skipWhenE2eOperatorOptInMissing` (never fail-closed) and need an externally-acquired Stripe
|
||
session path — they are **separate from the 28** and correctly stay skipped here.
|
||
|
||
---
|
||
|
||
## 0. Mission (read this twice)
|
||
|
||
The current full e2e run is **180 pass / 0 fail / 28 skip** against boiler
|
||
(`boiler.saas-ctrl.com`). The 28 skips are **operator-gated** tests that perform real,
|
||
destructive side effects and only run when explicit opt-in env flags are set.
|
||
|
||
**Your job: turn the 28 skipped tests ON, run the whole suite with every opt-in enabled
|
||
(a "disposable" / `REQUIRE_E2E_READINESS=1` run), and make ALL of them pass — honestly.**
|
||
Setup cleanly, run, read the results, fix the real cause (product code OR test), deploy if you
|
||
touched product code, re-seed, re-test. Loop until everything is green. Do not stop between
|
||
iterations. Do not ask for permission to continue — keep going.
|
||
|
||
### TIME BUDGET — check the real clock, do not trust your sense of time
|
||
|
||
- **You have 7 hours of real wall-clock time from the moment you were commissioned.**
|
||
- The FIRST thing you do every iteration: run `date -u` and read the actual time. Agents in a
|
||
loop lose track of elapsed time — the clock is the only truth. Record your start time on the
|
||
first iteration; your deadline = start + 7h.
|
||
- Re-check `date -u` at the top of every loop iteration and after every long wait (deploys take
|
||
~15–20 min each; a full suite run is ~10–14 min). Budget accordingly.
|
||
- **Reserve the last ~20 minutes** to commit everything, push, and update this handoff + the
|
||
memory file with the final state — even if not 100% green. A clean honest handoff beats an
|
||
abandoned run.
|
||
- Today's date drifts during long sessions; a `<system-reminder>` may announce a date change.
|
||
Always prefer `date -u` output over your assumption.
|
||
|
||
---
|
||
|
||
## 1. The 28 skips and the flags that gate them
|
||
|
||
The skips are deliberate `skipOrFailWhenE2eOptInMissing(...)` / `skipUnless...` /
|
||
`skipIfUnavailable` gates. Under `REQUIRE_E2E_READINESS=1` those guards THROW instead of
|
||
skipping (fail-closed) — that is the mode the CI readiness lane uses and the mode you must make
|
||
pass. The opt-in env vars (grep the specs for the exact set; current list):
|
||
|
||
| Flag | Unlocks | Side effect / needs |
|
||
|------|---------|---------------------|
|
||
| `E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1` | plugin enable/disable + lifecycle persist (~6) | really toggles plugin enabled state — must self-restore |
|
||
| `E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1` | plugin settings write persist (1) | persists a setting + capability grant |
|
||
| `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1` + `E2E_WORKFLOW_WEBHOOK_TARGET_URL=<url>` | webhook workflow run (3) | posts to a REAL external URL — you must provide a reachable sink (e.g. a webhook.site or an in-cluster echo). Without a target these cannot pass. |
|
||
| `E2E_ALLOW_REGISTRATION_CREATE=1` | registration creates a real account (2) | creates users/tenants in the DB |
|
||
| `E2E_ALLOW_ACCOUNT_CREATE=1` | account creation (1) | creates a tenant |
|
||
| `E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1` | no-tenant account path (1) | creates a user |
|
||
| `E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1` | Stripe checkout return surface (2) | renders the post-checkout surface; check whether it needs Stripe test keys / a synthetic session |
|
||
| `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1` | already set in your runs | needs the general worker draining the outbox |
|
||
| `REQUIRE_E2E_READINESS=1` | makes every fixture/opt-in guard FAIL instead of skip | the fail-closed mode you must satisfy |
|
||
|
||
The authoritative source of truth for the exact flags + how they are wired is
|
||
`.gitea/workflows/e2e-readiness.yml` (the readiness lane, `workflow_dispatch` with these
|
||
operator inputs) — **read it first** and mirror its env exactly. Grep:
|
||
`grep -rhoE 'E2E_ALLOW_[A-Z_]+|skipOrFail[A-Za-z]+|skipUnless[A-Za-z]+|REQUIRE_E2E_READINESS' tests/e2e/*.spec.ts | sort -u`
|
||
|
||
**Important caveat:** enabling these makes the suite NON-idempotent (real grants, real
|
||
accounts, real toggles persist). Tests that mutate MUST self-restore (capture baseline →
|
||
restore in `finally`), and you must re-seed between full runs. If a newly-unskipped test isn't
|
||
self-restoring, that is part of the work — make it restore, do not leave leakage.
|
||
|
||
---
|
||
|
||
## 2. The work loop (never stop between steps)
|
||
|
||
```
|
||
loop:
|
||
date -u # 0. read the real clock; stop if < 25 min left
|
||
source .env + set ALL opt-in flags + REQUIRE_E2E_READINESS=1
|
||
re-seed staging (REQUIRED after every deploy AND before every full run — /tmp is ephemeral)
|
||
run the full suite (single worker) OR the targeted newly-failing tests
|
||
read results:
|
||
- 0 fail -> run the FULL suite once more to confirm stability; if green twice, DONE
|
||
- failures -> for EACH: open the trace, find the REAL cause (below), fix it
|
||
if you changed PRODUCT code (apps/web, packages/*): commit, push main, wait for deploy, re-seed
|
||
if you changed only TEST code: re-run locally (no deploy needed)
|
||
goto loop
|
||
```
|
||
|
||
Running commands (from repo root `/Users/raffaelsheikh/Documents/GitHub/boilerplate`):
|
||
|
||
```bash
|
||
# full run with EVERYTHING on (the readiness-equivalent mode)
|
||
set -a; source .env.e2e-readiness.boiler; set +a
|
||
export E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1 E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1 \
|
||
E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1 E2E_ALLOW_REGISTRATION_CREATE=1 E2E_ALLOW_ACCOUNT_CREATE=1 \
|
||
E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1 E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1 \
|
||
REQUIRE_E2E_READINESS=1
|
||
# (webhook flag only once you have a real target URL — see §1)
|
||
npx playwright test --workers=1 --reporter=line > /tmp/run.log 2>&1
|
||
# single test by title:
|
||
npx playwright test --workers=1 --reporter=line -g "<exact test title>"
|
||
```
|
||
|
||
Long runs and deploys: launch them with `run_in_background: true` so you are notified on
|
||
completion instead of blocking; chain deploy→reseed→test in one background script (see §4) so
|
||
one notification covers a whole iteration.
|
||
|
||
---
|
||
|
||
## 3. Diagnose the REAL cause — do not cheat
|
||
|
||
Hard-won lesson from the previous session: **a read-only "look at the code" diagnosis is
|
||
frequently WRONG.** Always open the failing run's Playwright trace and read the actual page
|
||
snapshot + console + network. The previous effort found that several "test bugs" were real
|
||
product bugs (a dropdown that crashed the whole builder, an SSE loader that sent a blank
|
||
stream, an iframe channel race) and several "product bugs" were test bugs — only the traces
|
||
told the truth.
|
||
|
||
```bash
|
||
d=$(ls -dt test-results/*<slug>* | head -1) # the failing test's result dir
|
||
mkdir -p /tmp/tr && unzip -oq "$d/trace.zip" -d /tmp/tr
|
||
# real console/page errors (NOT strings inside bundled JS in resources/):
|
||
python3 - <<'PY'
|
||
import json,glob
|
||
for f in glob.glob('/tmp/tr/*.trace'):
|
||
for line in open(f,errors='ignore'):
|
||
try: o=json.loads(line)
|
||
except: continue
|
||
if o.get('type') in ('console','pageerror') and (o.get('messageType')=='error' or o.get('type')=='pageerror'):
|
||
print(o.get('text') or (o.get('error') or {}).get('message',''))
|
||
PY
|
||
# the page snapshot (what was actually on screen) is in:
|
||
cat "$d/error-context.md"
|
||
# axe violations: the failing node's `target` selector + `html` is in the assertion diff.
|
||
```
|
||
|
||
**No-cheating rules (non-negotiable):**
|
||
- Do **not** skip, `.fixme`, `xfail`, or delete a test to make the suite "green". The point is
|
||
the opposite — un-skip.
|
||
- Do **not** weaken an assertion to hide a real failure (e.g. dropping a status check, loosening
|
||
a regex to match anything, removing an a11y scan). You MAY relax an assertion only when the
|
||
trace proves the test's expectation was wrong (e.g. a legitimate non-mutating audit POST the
|
||
test mis-counted) — and document why.
|
||
- Do **not** disable an opt-in flag to dodge a test. If a test genuinely cannot run (e.g. webhook
|
||
with no possible target), say so explicitly in the handoff with the reason — don't pretend.
|
||
- Do **not** loosen timeouts to mask a real hang. Longer timeouts are correct only for
|
||
genuinely-slow real operations (the worker outbox poll is 90s; the per-test timeout is already
|
||
120s in `playwright.config.ts`).
|
||
- Fix the ROOT. If a transient UI element trips an a11y scan, fix the wait/the element — don't
|
||
delete the scan. If a real run flakes on worker timing, give the worker real time — don't fake
|
||
the status.
|
||
- Distinguish a real flake from a real bug: re-run a suspect test **3–4× in isolation** after a
|
||
fresh re-seed. Passes in isolation but fails in the serial run ⇒ cross-test state/order/rate
|
||
issue (fix the leak or the helper). Fails in isolation too ⇒ real bug.
|
||
|
||
---
|
||
|
||
## 4. Environment cookbook (everything you need)
|
||
|
||
**Targets / access** (all verified working this session):
|
||
- App: `https://boiler.saas-ctrl.com`. Env file: `.env.e2e-readiness.boiler` (repo root).
|
||
- SSH: `ssh boilerplate-staging` (root@46.225.5.93, key `~/.ssh/node1`).
|
||
- Web container name pattern: `mrf8lj...` (app UUID `mrf8lj70kmy9hqprx9dd6w7q`). General worker
|
||
(drains outbox + runs native executor): `wzshv933n2nhy0cvsxu8bqmm`.
|
||
- Gitea API: `https://git.tool-ctrl.de/api/v1`, repo `toolctrl-labs/boilerplate`, token
|
||
`~/.cli-tokens/gitea` (the `gitea-ci-pat` lacks actions:read — use `gitea`).
|
||
|
||
**Re-seed (REQUIRED after every web deploy AND before every full run — `/tmp` is ephemeral):**
|
||
```bash
|
||
WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}\t{{.Image}}' | grep /web: | cut -f1 | head -1")
|
||
ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 \
|
||
-e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js"
|
||
```
|
||
The seed source is `scripts/seed-e2e-readiness.ts` (built into that dist on deploy). It also runs
|
||
`syncDefaultRolesAndPermissions`, so a re-seed applies permission changes. The `EACCES ... .env`
|
||
line at the end is harmless (it just can't write the env file); the DB seeding succeeds.
|
||
|
||
**Deploying product-code changes:** push to `main` → the Gitea `deploy.yml` builds + swaps the
|
||
web/api/worker images. Deploys run **sequentially** (a new push queues behind the current one)
|
||
and take ~15–20 min. Ground truth that a deploy landed = the staging web image tag changes:
|
||
```bash
|
||
ssh boilerplate-staging "docker ps --format '{{.Image}}\t{{.Status}}' | grep /web: | head -1"
|
||
```
|
||
The Gitea **commit-status** API is unreliable here (sticks on "pending"); trust the image tag.
|
||
|
||
**The deploy step is intermittently flaky.** The `Stage or promote Docker images` job sometimes
|
||
fails transiently (registry/runner) while `Deploy preflight gates` passes — staging then stays on
|
||
the old image. Detect by: image tag not changing ~20 min after push. Re-trigger with an empty
|
||
commit: `git commit --allow-empty -m "chore: re-trigger deploy" && git push origin HEAD:main`.
|
||
**Use a self-healing background script** (proven this session) that: waits for the expected short
|
||
SHA on the image tag (poll ~21 min), and on timeout pushes an empty commit + waits again (cap 3
|
||
tries), then re-seeds and runs. This is the only way to run unattended overnight without a human
|
||
re-triggering. NOTE: an empty commit changes HEAD's SHA, so track the new SHA after each push.
|
||
|
||
**DB inspection (throwaway psql on the coolify network; strip the query string; dollar-quote
|
||
literals to dodge ssh quoting):**
|
||
```bash
|
||
ssh boilerplate-staging 'WEB=$(docker ps --format "{{.Names}}" | grep mrf8lj | head -1); \
|
||
DBURL=$(docker exec $WEB printenv DATABASE_URL | sed "s/?.*//"); \
|
||
docker run --rm --network coolify postgres:16-alpine psql "$DBURL" -t \
|
||
-c "select ... where name=\$\$some.value\$\$;"'
|
||
```
|
||
Columns are quoted camelCase. Worker logs / outbox lastError: see `boiler-deploy-seed.md`.
|
||
|
||
---
|
||
|
||
## 5. Known patterns & gotchas (so you don't re-learn them)
|
||
|
||
- **RR7 single-fetch:** `<Form method=post>` actions POST to `<path>.data` and decode a
|
||
**turbo-stream** body. A Playwright interceptor must match `.data` AND return turbo-stream, not
|
||
`application/json`. Helpers in `tests/e2e/utils/navigation.ts`: `singleFetchActionError`,
|
||
`singleFetchActionData`. See `e2e-singlefetch-interceptor-contract.md`.
|
||
- **Login rate limit:** 10 logins / 60s per IP+email (`AuthService.ts`). The test helper
|
||
(`tests/e2e/utils/auth.ts`) caches the session cookie **per email** and reuses it — keep that
|
||
intact. New tests that log in as new users (registration/account-create) WILL hammer this;
|
||
if they flake on "stayed on /login", that's the rate limit — back off / reuse sessions, do not
|
||
raise the production limit.
|
||
- **Re-seed before every run.** Plugin binary + state live in ephemeral `/tmp`; mutation tests
|
||
drift state. Skipping a re-seed causes false failures (plugin "disabled", settings page blank).
|
||
- **Native vs http workflow executor:** staging uses native mode (`WORKFLOW_TRIGGER_EXECUTOR_MODE=native`,
|
||
Coolify env). Do NOT un-gate `internal/api/__workflows/**` in routes.ts (a test enforces it).
|
||
- **Coolify env:** POST and PATCH both CREATE env rows → dupes. GET uuids and DELETE extras.
|
||
- **a11y scans:** `expectNoCriticalA11yViolations` now waits for the FloatingLoader overlay to
|
||
clear before scanning (transient mid-fade contrast). Keep that.
|
||
- **Flakes shift:** with everything on and one serial worker against a shared remote tenant, the
|
||
failing test changes run-to-run (login rate-limit, worker timing, real-SSE timing). Fix roots,
|
||
and confirm with repeated isolation runs; CI uses `retries: 1`.
|
||
- **The CI readiness lane is the ultimate arbiter** (`.gitea/workflows/e2e-readiness.yml`,
|
||
`workflow_dispatch`, in-cluster, retries). If you can trigger it via the Gitea API with the
|
||
operator inputs, that is the cleanest "everything on" verification — but it needs the same
|
||
flags and a real webhook target.
|
||
|
||
---
|
||
|
||
## 6. Definition of done
|
||
|
||
1. A full suite run with ALL opt-in flags + `REQUIRE_E2E_READINESS=1` set, **freshly re-seeded**,
|
||
ends with **0 failed and 0 skipped** (every test either passes or is justifiably impossible —
|
||
and if impossible, documented with the concrete reason, not hidden).
|
||
2. **Confirm stability:** the green run reproduces — run the full suite twice clean, or re-run any
|
||
test that flaked 3–4× in isolation, all green.
|
||
3. Every fix is committed to `main` with a clear message and (for product code) deployed; the
|
||
working tree is clean (ignore the pre-existing `packages/db/generated/prisma/**` churn).
|
||
4. Update THIS file and the memory `boiler-e2e-remaining-tail.md` with the final numbers, what you
|
||
changed, and anything genuinely blocked (e.g. webhook target unavailable) with the reason.
|
||
5. No test was skipped, weakened, or deleted to manufacture green (§3). If you ran out of the
|
||
7-hour budget, write down exactly where you stopped, the last run's numbers, and the next step.
|
||
|
||
Start by running `date -u`, reading `.gitea/workflows/e2e-readiness.yml`, then doing one full
|
||
"everything on" re-seeded run to see the real starting failure set. Then loop. Good luck.
|