toolctrl-labs/HiveGrid-Wireframes
9
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3cb93a81bea50927864585d1fafe0af4c4772a09
HiveGrid-Wireframes/hivegrid-cloud/docs/testing/e2e-skipped-tests-loop-handoff.md
T

41 KiB
Raw Blame History

E2E Skipped-Tests Loop — Handoff for the next agent

✅ GREEN — 4th agent (2026-06-02 ~20:15 UTC)

Full "everything-on" chromium run against boiler (web image 944ef476, freshly reseeded): 203 passed / 0 failed / 5 skipped (9.3m). The 5 skips are correctly operator-gated and justifiably impossible here: real-Stripe subscription-success ×3 (need an externally-acquired Stripe session), the registration email-verification inverse pair (env runs with E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0), and the E2E_REQUIRE_PLUGIN_HOOK_AUDIT hook-audit lane (operator-gated). A second clean full run was used to confirm stability.

The 12 remaining failures from the previous handoff, all fixed this session:

  1. plugin cluster (49/120/173/201) — cross-spec pollution. plugin-hook-lifecycle now targets the dedicated, fully-provisioned com.toolctrl.e2e-lifecycle plugin instead of mutating the shared e2e-readiness plugin's audit-missing baseline.
  2. plugin-settings 244 — the grant/revoke + restore reloaded the page before the in-flight single-fetch POST landed (stale read). Now waits for RR7 revalidation.
  3. plugin-settings 587 — TWO bugs. (a) PRODUCT: the settings-iframe write posted to fetch(".") (resolves to the parent plugins-list route) and expected raw JSON; RR7 returns a turbo-stream document for non-.data fetches, so every write failed. Fixed to post the detail route's .data endpoint and gate on the HTTP status. (b) TEST: the post-reload iframe get is non-deterministic (the host frame quarantines its window + rotates the channel on a late load), so persistence is now verified by reading the settings loader's .data payload after reload.
  4. admin-email-sequences ×2 — logged in as the shared tenant user (not an admin). Seed now provisions a dedicated e2e.admin@example.test (AdminUser + global SuperAdmin, no TOTP) and emits E2E_ADMIN_EMAIL/PASSWORD; spec logs in as it.
  5. mobile-shell 570 (run/manual SSR 500) — MonacoEditor mounted its browser-only ./MonacoEditor.client during SSR, so the streaming renderer set HTTP 500 on EVERY editor page (run/manual, webhook-run, api-docs too — the prior handoff was wrong that api-docs was fine; only this test asserts status < 500). Fixed by gating the editor behind a client mount (server + first client render emit an identical placeholder → no hydration mismatch).
  6. portal-management — operator chose to SHIP portals. Un-ignored the portal routes (app.$tenant/__portals/**, admin/portals.tsx), gated the global enable behind PORTALS_ENABLED (boilerplate default stays off; boiler Coolify env set to true), seeded a minimal published Portal fixture (subdomain e2e-readiness), and fixed 2 pre-existing type-drift errors the route-tree typecheck surfaced.
  7. subscription:394, workflow-builder-run 528/816 — were timing flakes; green on clean reseeded runs (CI retries:1 absorbs them).

Commits (all on main, deployed): 44e30fb3 (plugin fixture sep + races), 147d92e4 (plugin settings .data fix), 011c6958 (admin user), 13edf803+944ef476 (portals + type fixes), the Monaco fix, and a9556041 (587 loader read-back, test-only). Coolify (boiler web): added PORTALS_ENABLED=true (single row — deduped).

Gotchas observed: (a) a local-machine net::ERR_INTERNET_DISCONNECTED mid-run cascades ~70 false failures from one point onward — re-run, it's not real. (b) loader .data payloads embed the iframe HTML (~25KB) — search the FULL body, not a prefix. (c) the plugin settings iframe quarantines its window on late loads; drive it via the loader, not a post-reload get.

Written: 2026-06-02 00:10 UTC · Repo HEAD at handoff: 494f52de (branch main, also deployed to boiler staging) · Companion docs: docs/testing/e2e-readiness-handoff.md (full bug-fix history), agent memory boiler-e2e-remaining-tail.md, boiler-deploy-seed.md, e2e-singlefetch-interceptor-contract.md.

🚀 START HERE — NEXT AGENT (handoff from the 3rd agent, 2026-06-02 ~16:35 UTC)

Current state: main HEAD d8aaeacb, deployed to boiler (web image 3cbb12d3; the two commits after it are test-only/docs and don't change the runtime image). Full chromium run, freshly reseeded: 192 pass / 11 fail / 5 skip. The original 26 failures are mostly fixed; the 11 are the hard tail below. The 5 skips are correctly-gated (real-Stripe success + email-verification inverse).

What's already done (don't redo): subscription/api-docs/golden-path/registration/workflow-builder-actions/ mobile-shell-drawer race+selector fixes; seed prunes leaked accounts; seed fully-provisions the LIFECYCLE plugin; seed DELETES any operator-admin grant from the tenant user; host-aware admin path /email-sequences. Coolify env on boiler (set this session, persist): PLUGIN_RUNTIME_PRODUCTION_ENABLED=true (web + worker-general + worker-plugin) and PLUGIN_HOOK_FANOUT_REQUIRED=true (worker-general + worker-plugin). Coolify API: https://saas-ctrl.com/api/v1, token ~/.cli-tokens/coolify-token, web uuid mrf8lj70kmy9hqprx9dd6w7q (see boiler-deploy-seed.md).

How to run (everything-on): bash /tmp/e2e-cr.sh <specs?> — it sources .env.e2e-readiness.boiler and sets every opt-in + REQUIRE_E2E_READINESS=1 (recreate it if /tmp was wiped; see §2/§4). RESEED before every run (after every deploy too — /tmp plugin bundles are ephemeral): WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}' | grep mrf8lj | head -1"); ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 -e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js". Seed source is scripts/seed-e2e-readiness.ts, built to that dist on deploy → seed changes need a deploy (push main → ~20min; watch the web image tag flip, NOT the commit-status API). DB inspection is now permitted (recipe in §4); you can also just psql the readiness plugin/grant state.

The 11 remaining, in priority order (full root-cause + fixes in the "third agent" ITERATION LOG below):

  1. [BIGGEST WIN — ~6 tests] Plugin cluster needs FIXTURE SEPARATION. plugin-settings is 14/0 in isolation but fails ~5 in the serial run as CROSS-SPEC POLLUTION: plugin-hook-lifecycle.* sorts first, mutates the SHARED com.toolctrl.e2e-readiness plugin (grants audit / deactivates), and the "enabled + audit-missing" baseline is UI-unrestorable (revoke auto-disables; plugins.tsx:415 hides the enable form for disabled+missing-cap). Fix: seed a SECOND, fully-provisioned readiness plugin WITH a settings UI (clone buildE2eReadinessPluginManifest → new id e.g. com.toolctrl.e2e-hooklifecycle, grant ALL caps incl hook:workflow.executed), and repoint plugin-hook-lifecycle.mobile-a11y.spec.ts (146/201) + the lifecycle/iframe tests at it; keep the audit-missing com.toolctrl.e2e-readiness ONLY for the review/grant/deny tests (49/120/173/244). Then nothing mutates the audit-missing plugin's audit state. Also fix hook-lifecycle's isCapabilityGranted — after grantAndEnable the grant lands server-side (DB-confirmed) but the revoke-form detection misses it (inspect the rendered DOM for the audit revoke control).
  2. [2 tests] admin-email-sequences — seed a DEDICATED admin user (e2e.admin@example.test, AdminUser + global SuperAdmin role, reuse E2E_READINESS_USER_PASSWORD); emit E2E_ADMIN_EMAIL; make admin-email-sequences.mobile-a11y.spec.ts log in as THAT user (not the tenant user). DO NOT grant the shared tenant user operator-admin — it redirects to the admin host on login and breaks auth-flow/registration (the seed now actively deletes that grant). The host-aware /email-sequences path is already wired and the cookie reaches the admin host (verified).
  3. [1 test] run/manual SSR 500 (mobile-authenticated-shell.spec.ts:570) — product bug: the page renders but entry.server.tsx:121 sets HTTP 500 on a swallowed streaming error. Add temporary onError logging (even pre-shell), deploy, reproduce the direct GET of wf_123/run/manual, capture the error (suspect a deferred/Suspense boundary), fix. SPA nav works; only direct SSR 500s.
  4. [1 test] portal-management — PRODUCT DECISION (ask the operator): __portals is build-disabled in routes.ts defaultDisabledPluginRouteFilePatterns. Either ship the portals module, or accept not-shipped (then stop emitting the portal env paths and make the test skip).
  5. [1 test, likely flake] workflow-builder-run:816 — webhook manual-run result doesn't render in 15s (worker-drain/SSE timing; target https://postman-echo.com/post). Bump the result wait or fix the streaming; CI retries:1 may already absorb it.

Gotchas: (a) the deploy coolify-ensure-worker-groups.mjs drift check FAILS "Stage or promote" if PLUGIN_RUNTIME_PRODUCTION_ENABLED=true on a worker without PLUGIN_HOOK_FANOUT_REQUIRED=true — both are already set, keep them paired. (b) Re-trigger a stalled/flaky deploy with an empty commit. (c) a11y color-contrast failures on toasts/loaders mid-fade are transient (CI retries absorb; axe.ts settles animations). (d) Don't weaken/skip to fake green (§3). Reserve the last ~20 min for the handoff + memory update.

⏱️ ITERATION LOG — 2026-06-02 (third agent — root-caused & mostly fixed)

Commits 894ba2a5 + 8bd11f9c (empty re-trigger) DEPLOYED to boiler (web image 8bd11f9c, reseeded). Plugin runtime env + companion set on Coolify. Outcome: 26 originally-failing tests → ~6 remaining, all genuine product/design issues (root-caused below).

The previous "account-leakage" leading hypothesis was mostly WRONG (verified by reading all 26 traces, per §3). The 26 split into distinct, mostly-independent causes. Status per cluster:

CONFIRMED GREEN post-deploy (full a11y-strict env, chromium): plugin-settings full suite (14/0, incl. the 244/344/380/473/587 cluster), plugin hook-audit (worker fanout env), lifecycle-plugin enable/disable (431), registration family (35/1-skip, leak pruned), subscription ×3+nav, workflow-builder-actions (incl. delete-block), api-docs, golden-path, mobile-shell drawer.

ADMIN GRANT REVERTED (commit 5ed35fb5): granting the shared e2e-readiness tenant user AdminUser + global SuperAdmin DID make admin-email-sequences ×2 pass, but it is an OPERATOR admin → on login the app redirects to the admin host (admin.boiler…/dashboard), which broke auth-flow + registration login-redirect assertions AND added a "Switch to admin" affordance + teal #00bba7 2.42:1 text to the roles page (tenant-roles-permissions a11y ×3). Net regression, so reverted. The host-aware /email-sequences PATH fix is kept (correct). admin-email-sequences now needs a DEDICATED admin user (seed a separate e2e.admin@… AdminUser+SuperAdmin and point the spec at it via new creds) — the shared tenant user cannot be both.

  • GREEN (validated individually on boiler, chromium):
    • workflow-builder ×2 (+3 siblings) — one-shot count() raced the React Flow node mount.
    • subscription ×3 (:80/:144/:711) — SPA-nav race + the :144 error test used a raw-JSON interceptor that breaks RR7 single-fetch; switched to singleFetchActionError.
    • readiness-golden-path (1) — mobile settings/nav links live in the "more" popover + a section <Select> combobox, not visible <a>s; helper now opens both + evaluateAll href scan (the old per-node loop hung on the streaming workflow page).
    • tenant-api-docs ×2 — waited for the lazy Monaco editor to attach before count().
    • registration family: 35 pass / 1 skip. nav-link asserts now open the mobile sidebar Sheet; delete control located by data-ui-id="ui.settings.account.delete" (text is "Delete account"); email-verification inverse test test.skips under EXPECTS=0; bb379 retry-opens the popover.
    • plugin tests: 12/20 green after the env fix (see below). The .check() multi-checkbox strict-mode crash is fixed.
  • ENV FIX (Coolify, applied + redeployed): PLUGIN_RUNTIME_PRODUCTION_ENABLED=true set on boiler web + worker-general + worker-plugin (was unset; NODE_ENV=production → every plugin enable/grant 403'd → tests couldn't self-restore → cascade). User-approved.
  • SEED FIX (in commit 894ba2a5, needs the deploy+reseed to take effect):
    • prune leaked test-created tenants (registration/account-create) + throwaway reg users, so the readiness/no-tenant users stop accumulating the multi-account picker.
    • grant operator-admin (AdminUser) + the global SuperAdmin role (tenantId=null) so admin pages resolve. Admin path changed to /email-sequences (host-aware: app host 308-redirects to the admin host, which page.goto follows; the readiness cookie spans the app-base domain — verified the probe reaches admin.boiler…/email-sequences). Was /admin/email-sequences (404 on the app host).
    • fully provision the lifecycle plugin (capabilities + hook:workflow.executed consent) so a disabled plugin isn't trapped behind the review-before-enable gate.

STILL OPEN — ~5 tests, all genuine product/design issues (root-caused, with fixes)

✅ FIXED this iteration — plugin-settings readiness cluster (5: 244/344/380/473/587). ROOT was: the readiness plugin's seed baseline is ENABLED + audit MISSING (so review/grant tests have a cap to review), but that's UI-unreachable once disabled — revokePluginCapabilities AUTO-DISABLES (:614) and plugins.tsx:415 hides the enable form for a disabled+missing-cap plugin. Test 244 (grant→revoke audit) auto-disabled it and the restore couldn't get back, cascading to 344/380/473/ 587. FIX (committed): restorePluginCapabilityState now prioritises leaving the plugin ENABLED via the review→grantAndEnable flow (accepting the unreachable enabled+audit-missing baseline; reseed restores it between runs), and .check() targets all review checkboxes. plugin-settings now 14/0 in isolation.

⚠️ BUT in the FULL serial run (CI), plugin-settings still shows ~5 failures — and they are CROSS-SPEC POLLUTION, not the original cause. plugin-hook-lifecycle.* sorts BEFORE plugin-settings.*, so its mutating tests (146 grants audit; 201 grant/deactivate) run first on the SHARED com.toolctrl.e2e-readiness plugin and leave it audit-GRANTED (or disabled) — then plugin-settings' review/grant tests (49/120/173) fail with "fixture has no missing capability" because audit is no longer missing. The "enabled + audit-missing" baseline is UI-unrestorable (revoke auto-disables), so NO per-test restore can undo this in a single seeded serial run. The only clean fix is FIXTURE SEPARATION: give the hook-lifecycle / lifecycle-toggle / iframe tests a DEDICATED fully-provisioned plugin, and reserve the audit-missing readiness plugin solely for the review/grant tests so nothing mutates its audit state. (This was a wash this iteration — failures shifted from {344/380/473} to {49/120/173} — so the committed restore fix only helps once fixtures are split.)

  1. plugin-hook-lifecycle :201 "deactivate then re-enable" (1). STILL RED, but narrowed to a pure UI-detection issue: 201 now provisions audit first (grantAndEnable) so the bare toggle can run, and the grant succeeds server-side (DB shows the plugin enabled + audit granted) — but the test's isCapabilityGranted (which reads the capability revoke form) never detects the granted audit cap after grantAndEnable, even with a 10s poll. So the grant works; the verification heuristic doesn't see the audit revoke form in this state. Fix = inspect the rendered DOM for the audit revoke control after grantAndEnable and correct the capabilityRevokeForm selector / detection (or assert via the absence of the "review" affordance instead). Also: the prior test 146's restore revokes audit → auto-disables, so 201 inherits a disabled plugin — making 146's restore re-enable too would let 201 start clean.
  2. wf_123/run/manual returns HTTP 500 on direct SSR load. Page renders fully; entry.server onError (:121) sets 500 on a recovered streaming error that is swallowed (pre-shell, unlogged). SPA nav works (golden-path passes); only direct SSR load of this one page 500s (builder direct-load is fine). Breaks only mobile-shell browser-history slice (:570). Real product bug — to fix, add temporary logging to entry.server onError (even pre-shell) and deploy to capture the streaming error (suspect a deferred/Suspense boundary unique to the run-manual route).
  3. portal-management (1). app.$tenant/__portals/** is statically build-disabled (routes.ts defaultDisabledPluginRouteFilePatterns). The test is designed to fail-closed under readiness when portals aren't mounted (its own comment). Product/feature decision: ship the portals module, or accept not-shipped. Not a test bug.
  4. admin-email-sequences (2). Need a DEDICATED admin user (see the ADMIN GRANT REVERTED note above) — the shared tenant user can't be operator-admin without breaking tenant login. The /email-sequences host-aware path + the permission analysis (needs global SuperAdmin role for admin.marketing.leads.view, AND AdminUser for the admin layout) are already worked out; just apply them to a separate seeded e2e.admin@… user the spec logs in as.
  5. workflow-builder-run :816 "webhook workflow manual run posts a valid execution contract". Direct-loads the webhook run page and waits 15s for ui.workflows.run.manual.executionResult; the webhook workflow executes via the worker (target https://postman-echo.com/post, reachable) but the result doesn't render in time — worker-drain/SSE timing, same family as the run/manual streaming issue (#3). Likely needs a longer result wait or the streaming-status fix.

Transient a11y flakes (CI retries:1 absorbs these — not real bugs): several runs showed color-contrast violations on toast/loader elements caught MID-TRANSITION (e.g. react-hot-toast #707c90 on #fefeff = 4.18 vs 4.5 while fading; the FloatingLoader class). tests/e2e/utils/axe.ts already settles animations but heavy serial load occasionally beats it. The roles-page #00bba7 contrast was admin-grant chrome ("Switch to admin") and clears with the admin-grant removal.

Coolify env changes applied this iteration (boiler, user-approved)

  • PLUGIN_RUNTIME_PRODUCTION_ENABLED=true on web + worker-general + worker-plugin.
  • PLUGIN_HOOK_FANOUT_REQUIRED=true on worker-general + worker-plugin (REQUIRED companion — the deploy's coolify-ensure-worker-groups.mjs drift check fails the "Stage or promote" job without it whenever PLUGIN_RUNTIME_PRODUCTION_ENABLED=true on a worker group).

⏱️ ITERATION LOG — 2026-06-02 (second agent, "everything-on" attempt)

Status at end of this iteration: NOT green. 26 real, persistent failures isolated and root-caused (leading hypothesis below). No code pushed — diagnosis-only handoff.

Why I stopped: I was commissioned at 00:13 UTC. The machine then slept ~8h between background runs completing and my re-invocation (verified via date -u: jumped 00:51 → 09:02 UTC on consecutive turns, with only ~40 min of actual compute used). By the literal 7h wall-clock rule (deadline ~07:13 UTC) I was already past budget when I got the triage results, so per §0 I pivoted to a clean honest handoff instead of starting 15–40 min deploy/run cycles that could get suspended mid-flight. Next agent: you likely have full budget again — pick up from "Concrete next steps" below.

What I ran

  1. Fresh re-seed → full "everything-on" run (all opt-ins + REQUIRE_E2E_READINESS=1 + E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1 with target https://postman-echo.com/post, E2E_REQUIRE_PLUGIN_HOOK_AUDIT=1). Result: 358 pass / 260 fail / 6 skip (25.6m).
  2. 208 of the 260 failures were a LOCAL ENV bug, not product/test: browserType.launch: Executable doesn't exist … webkit-####/pw_run.sh. The mobile-safari project needs WebKit installed locally. FIXED for the env: ran npx playwright install webkit (CI does playwright install --with-deps chromium webkit; the local box only had chromium). Do this first on any fresh box or you'll chase 200 phantom failures.
  3. Re-ran the 12 affected spec files chromium-only, fresh seed, stable container to strip the webkit noise and the run-1 mid-run container restart (web container id changed …-000832351032 → …-003251708343 during run 1 — Coolify recycled it, which also inflated run-1 failures). Clean result: 65 pass / 26 fail / 3 skip (7.9m). These 26 are the REAL, reproducible failures.

The 26 persistent failures (chromium; ×3 under the full browser matrix)

Grouped by root symptom (all throw because REQUIRE_E2E_READINESS=1 turns "fixture missing" into a hard fail):

  • Tenant feature surfaces appear absent though the seed populates them (these tests PASSED in the 180-green baseline — so this is state regression, not the un-skip):
    • admin-email-sequences:23,62 — /admin/email-sequences returns 404 ("Not Found").
    • portal-management:16 — "Portal route … not mounted (portals module disabled)"; page snapshot actually shows the "Select account — You belong to 13 accounts" picker.
    • readiness-golden-path:22 — "Subscription settings link was not rendered".
    • subscription-smoke:80,138,699 — "No enabled checkout submit button rendered".
    • tenant-api-docs-logs:70,100 — "Tenant API docs editor is not visible".
    • mobile-authenticated-shell:563 — shell history slice (same family).
  • Workflow builder fixture lost its blocks: workflow-builder-actions:106,128 — "Workflow builder fixture needs at least one persisted block". (Note: the seed DOES delete+recreate wf_123 incl. blocks at scripts/seed-e2e-readiness.ts:913, so a fresh seed should fix this — needs confirming it isn't an active-account-context render issue like the cluster above.)
  • Plugin lifecycle controls unavailable / not restorable: plugin-settings:244,337,373,424, 466,580 + plugin-hook-lifecycle:146,201 — "Plugin fixture has no manageable lifecycle controls", "Unable to grant/restore … lifecycle state". (Seed deletes plugin grants at :872; plugin binaries live in ephemeral /tmp/e2e-plugins.)
  • Newly-enabled registration/account real flows: registration:162,605,709,840,888 — created/!provisioned accounts hit a "Subscribe" gate (snapshot shows /subscribe/e2-e-mobile-#### and no ui.nav.app.dashboard), i.e. a freshly created tenant has no subscription/entitlement so its dashboard legitimately differs from the seeded tenant — the test assumes they look alike. registration:245 (email-verification) is a clear test bug: under E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0 (boiler's mode) its inverse-pair guard calls skipOrFailWhenE2eFixtureMissing(...) which THROWS under readiness instead of skipping. It should skip (env genuinely can't exercise email-verification).

Leading hypothesis (NOT yet proven — verify before fixing 26 things)

Account leakage corrupts the e2e.readiness user's active-account resolution. The E2E_ALLOW_ACCOUNT_CREATE / registration flows create accounts owned by e2e.readiness@example.test and never delete them (no finally restore). The user now belongs to 13 accounts (seen verbatim in the portal snapshot). The seed is per-feature destructive but does NOT prune these leaked accounts or memberships. If login/defaultTenantId no longer resolves cleanly to e2e-readiness, every surface that depends on the active account context (subscription link, portal, admin, API-docs editor, workflow block render) appears empty even though the e2e-readiness account itself is correctly seeded — which matches the exact symptom set above (explicit-slug pages mostly work; context-dependent features don't). The direct DB query to confirm account count/defaultTenantId was blocked by the auto-mode classifier (production credential read) — next agent should get that permission or confirm via a trace.

Concrete next steps (in order)

  1. npx playwright install webkit on the runner (kills 208 phantom failures).
  2. Confirm/refute the leakage hypothesis. Either (a) query the DB for how many accounts e2e.readiness@example.test belongs to and its defaultTenantId (recipe in §4 — needs the classifier to allow it), or (b) log in as the tenant in a trace and see if the active account is e2e-readiness. If leaked accounts are the cause:
    • Make the seed prune the e2e.readiness user's non-e2e-readiness tenantUser memberships and delete the orphaned leaked accounts (a destructive reset path keyed off E2E_LOCAL_READINESS_SEED=1), so a re-seed heals the corruption. Then re-run.
    • AND make E2E_ALLOW_ACCOUNT_CREATE / no-tenant tests self-restore (delete the account they create in a finally) per §1, so they stop leaking.
  3. Fix registration:245 email-verification: skip (don't throw) when E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION ≠ 1 — it's environment-determined, not a missing fixture (allowed under §3 "justifiably impossible").
  4. For the created-account "Subscribe gate" failures (registration:840,888,...): decide whether the seed should grant the freshly-created tenant an entitlement, or the test should expect the subscribe-gated dashboard. Read the trace — don't assume.
  5. Re-run the 12 specs chromium-only after each fix; only do a full matrix run + deploy once chromium is clean. Webhook real-run target https://postman-echo.com/post is reachable from staging (verified 200) and is the right disposable sink.

Verified-good this iteration (don't re-investigate)

  • Webhook target https://postman-echo.com/post reachable from the staging web container (POST → 200). Use it for E2E_WORKFLOW_WEBHOOK_TARGET_URL.
  • Re-seed works and reports "E2E readiness fixtures seeded." (the trailing EACCES … .env is harmless). Helper scripts left at /tmp/e2e-reseed.sh, /tmp/e2e-run.sh (full matrix), /tmp/e2e-run-chromium.sh (fast triage) — they set the complete opt-in env.
  • The 3 real-Stripe subscription-success tests (subscription-smoke:198,260,603) use skipWhenE2eOperatorOptInMissing (never fail-closed) and need an externally-acquired Stripe session path — they are separate from the 28 and correctly stay skipped here.

0. Mission (read this twice)

The current full e2e run is 180 pass / 0 fail / 28 skip against boiler (boiler.saas-ctrl.com). The 28 skips are operator-gated tests that perform real, destructive side effects and only run when explicit opt-in env flags are set.

Your job: turn the 28 skipped tests ON, run the whole suite with every opt-in enabled (a "disposable" / REQUIRE_E2E_READINESS=1 run), and make ALL of them pass — honestly. Setup cleanly, run, read the results, fix the real cause (product code OR test), deploy if you touched product code, re-seed, re-test. Loop until everything is green. Do not stop between iterations. Do not ask for permission to continue — keep going.

TIME BUDGET — check the real clock, do not trust your sense of time

  • You have 7 hours of real wall-clock time from the moment you were commissioned.
  • The FIRST thing you do every iteration: run date -u and read the actual time. Agents in a loop lose track of elapsed time — the clock is the only truth. Record your start time on the first iteration; your deadline = start + 7h.
  • Re-check date -u at the top of every loop iteration and after every long wait (deploys take ~15–20 min each; a full suite run is ~10–14 min). Budget accordingly.
  • Reserve the last ~20 minutes to commit everything, push, and update this handoff + the memory file with the final state — even if not 100% green. A clean honest handoff beats an abandoned run.
  • Today's date drifts during long sessions; a <system-reminder> may announce a date change. Always prefer date -u output over your assumption.

1. The 28 skips and the flags that gate them

The skips are deliberate skipOrFailWhenE2eOptInMissing(...) / skipUnless... / skipIfUnavailable gates. Under REQUIRE_E2E_READINESS=1 those guards THROW instead of skipping (fail-closed) — that is the mode the CI readiness lane uses and the mode you must make pass. The opt-in env vars (grep the specs for the exact set; current list):

Flag Unlocks Side effect / needs
E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1 plugin enable/disable + lifecycle persist (~6) really toggles plugin enabled state — must self-restore
E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1 plugin settings write persist (1) persists a setting + capability grant
E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1 + E2E_WORKFLOW_WEBHOOK_TARGET_URL=<url> webhook workflow run (3) posts to a REAL external URL — you must provide a reachable sink (e.g. a webhook.site or an in-cluster echo). Without a target these cannot pass.
E2E_ALLOW_REGISTRATION_CREATE=1 registration creates a real account (2) creates users/tenants in the DB
E2E_ALLOW_ACCOUNT_CREATE=1 account creation (1) creates a tenant
E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1 no-tenant account path (1) creates a user
E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1 Stripe checkout return surface (2) renders the post-checkout surface; check whether it needs Stripe test keys / a synthetic session
E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1 already set in your runs needs the general worker draining the outbox
REQUIRE_E2E_READINESS=1 makes every fixture/opt-in guard FAIL instead of skip the fail-closed mode you must satisfy

The authoritative source of truth for the exact flags + how they are wired is .gitea/workflows/e2e-readiness.yml (the readiness lane, workflow_dispatch with these operator inputs) — read it first and mirror its env exactly. Grep: grep -rhoE 'E2E_ALLOW_[A-Z_]+|skipOrFail[A-Za-z]+|skipUnless[A-Za-z]+|REQUIRE_E2E_READINESS' tests/e2e/*.spec.ts | sort -u

Important caveat: enabling these makes the suite NON-idempotent (real grants, real accounts, real toggles persist). Tests that mutate MUST self-restore (capture baseline → restore in finally), and you must re-seed between full runs. If a newly-unskipped test isn't self-restoring, that is part of the work — make it restore, do not leave leakage.

2. The work loop (never stop between steps)

loop:
  date -u                                  # 0. read the real clock; stop if < 25 min left
  source .env + set ALL opt-in flags + REQUIRE_E2E_READINESS=1
  re-seed staging (REQUIRED after every deploy AND before every full run — /tmp is ephemeral)
  run the full suite (single worker) OR the targeted newly-failing tests
  read results:
     - 0 fail  -> run the FULL suite once more to confirm stability; if green twice, DONE
     - failures -> for EACH: open the trace, find the REAL cause (below), fix it
  if you changed PRODUCT code (apps/web, packages/*): commit, push main, wait for deploy, re-seed
  if you changed only TEST code: re-run locally (no deploy needed)
  goto loop

Running commands (from repo root /Users/raffaelsheikh/Documents/GitHub/boilerplate):

# full run with EVERYTHING on (the readiness-equivalent mode)
set -a; source .env.e2e-readiness.boiler; set +a
export E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1 E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1 \
  E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1 E2E_ALLOW_REGISTRATION_CREATE=1 E2E_ALLOW_ACCOUNT_CREATE=1 \
  E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1 E2E_ALLOW_SYNTHETIC_CHECKOUT_RETURN_SURFACE=1 \
  REQUIRE_E2E_READINESS=1
# (webhook flag only once you have a real target URL — see §1)
npx playwright test --workers=1 --reporter=line > /tmp/run.log 2>&1
# single test by title:
npx playwright test --workers=1 --reporter=line -g "<exact test title>"

Long runs and deploys: launch them with run_in_background: true so you are notified on completion instead of blocking; chain deploy→reseed→test in one background script (see §4) so one notification covers a whole iteration.

3. Diagnose the REAL cause — do not cheat

Hard-won lesson from the previous session: a read-only "look at the code" diagnosis is frequently WRONG. Always open the failing run's Playwright trace and read the actual page snapshot + console + network. The previous effort found that several "test bugs" were real product bugs (a dropdown that crashed the whole builder, an SSE loader that sent a blank stream, an iframe channel race) and several "product bugs" were test bugs — only the traces told the truth.

d=$(ls -dt test-results/*<slug>* | head -1)         # the failing test's result dir
mkdir -p /tmp/tr && unzip -oq "$d/trace.zip" -d /tmp/tr
# real console/page errors (NOT strings inside bundled JS in resources/):
python3 - <<'PY'
import json,glob
for f in glob.glob('/tmp/tr/*.trace'):
  for line in open(f,errors='ignore'):
    try: o=json.loads(line)
    except: continue
    if o.get('type') in ('console','pageerror') and (o.get('messageType')=='error' or o.get('type')=='pageerror'):
      print(o.get('text') or (o.get('error') or {}).get('message',''))
PY
# the page snapshot (what was actually on screen) is in:
cat "$d/error-context.md"
# axe violations: the failing node's `target` selector + `html` is in the assertion diff.

No-cheating rules (non-negotiable):

  • Do not skip, .fixme, xfail, or delete a test to make the suite "green". The point is the opposite — un-skip.
  • Do not weaken an assertion to hide a real failure (e.g. dropping a status check, loosening a regex to match anything, removing an a11y scan). You MAY relax an assertion only when the trace proves the test's expectation was wrong (e.g. a legitimate non-mutating audit POST the test mis-counted) — and document why.
  • Do not disable an opt-in flag to dodge a test. If a test genuinely cannot run (e.g. webhook with no possible target), say so explicitly in the handoff with the reason — don't pretend.
  • Do not loosen timeouts to mask a real hang. Longer timeouts are correct only for genuinely-slow real operations (the worker outbox poll is 90s; the per-test timeout is already 120s in playwright.config.ts).
  • Fix the ROOT. If a transient UI element trips an a11y scan, fix the wait/the element — don't delete the scan. If a real run flakes on worker timing, give the worker real time — don't fake the status.
  • Distinguish a real flake from a real bug: re-run a suspect test 3–4× in isolation after a fresh re-seed. Passes in isolation but fails in the serial run ⇒ cross-test state/order/rate issue (fix the leak or the helper). Fails in isolation too ⇒ real bug.

4. Environment cookbook (everything you need)

Targets / access (all verified working this session):

  • App: https://boiler.saas-ctrl.com. Env file: .env.e2e-readiness.boiler (repo root).
  • SSH: ssh boilerplate-staging (root@46.225.5.93, key ~/.ssh/node1).
  • Web container name pattern: mrf8lj... (app UUID mrf8lj70kmy9hqprx9dd6w7q). General worker (drains outbox + runs native executor): wzshv933n2nhy0cvsxu8bqmm.
  • Gitea API: https://git.tool-ctrl.de/api/v1, repo toolctrl-labs/boilerplate, token ~/.cli-tokens/gitea (the gitea-ci-pat lacks actions:read — use gitea).

Re-seed (REQUIRED after every web deploy AND before every full run — /tmp is ephemeral):

WEB=$(ssh boilerplate-staging "docker ps --format '{{.Names}}\t{{.Image}}' | grep /web: | cut -f1 | head -1")
ssh boilerplate-staging "docker exec -w /tmp -e E2E_SEED_ALLOW_PRODUCTION=1 -e E2E_LOCAL_READINESS_SEED=1 \
  -e PLUGIN_BINARY_BASE_PATH=/tmp/e2e-plugins $WEB node /myapp/packages/db/dist/e2e-readiness-seed.server.js"

The seed source is scripts/seed-e2e-readiness.ts (built into that dist on deploy). It also runs syncDefaultRolesAndPermissions, so a re-seed applies permission changes. The EACCES ... .env line at the end is harmless (it just can't write the env file); the DB seeding succeeds.

Deploying product-code changes: push to main → the Gitea deploy.yml builds + swaps the web/api/worker images. Deploys run sequentially (a new push queues behind the current one) and take ~15–20 min. Ground truth that a deploy landed = the staging web image tag changes:

ssh boilerplate-staging "docker ps --format '{{.Image}}\t{{.Status}}' | grep /web: | head -1"

The Gitea commit-status API is unreliable here (sticks on "pending"); trust the image tag.

The deploy step is intermittently flaky. The Stage or promote Docker images job sometimes fails transiently (registry/runner) while Deploy preflight gates passes — staging then stays on the old image. Detect by: image tag not changing ~20 min after push. Re-trigger with an empty commit: git commit --allow-empty -m "chore: re-trigger deploy" && git push origin HEAD:main. Use a self-healing background script (proven this session) that: waits for the expected short SHA on the image tag (poll ~21 min), and on timeout pushes an empty commit + waits again (cap 3 tries), then re-seeds and runs. This is the only way to run unattended overnight without a human re-triggering. NOTE: an empty commit changes HEAD's SHA, so track the new SHA after each push.

DB inspection (throwaway psql on the coolify network; strip the query string; dollar-quote literals to dodge ssh quoting):

ssh boilerplate-staging 'WEB=$(docker ps --format "{{.Names}}" | grep mrf8lj | head -1); \
  DBURL=$(docker exec $WEB printenv DATABASE_URL | sed "s/?.*//"); \
  docker run --rm --network coolify postgres:16-alpine psql "$DBURL" -t \
  -c "select ... where name=\$\$some.value\$\$;"'

Columns are quoted camelCase. Worker logs / outbox lastError: see boiler-deploy-seed.md.

5. Known patterns & gotchas (so you don't re-learn them)

  • RR7 single-fetch: <Form method=post> actions POST to <path>.data and decode a turbo-stream body. A Playwright interceptor must match .data AND return turbo-stream, not application/json. Helpers in tests/e2e/utils/navigation.ts: singleFetchActionError, singleFetchActionData. See e2e-singlefetch-interceptor-contract.md.
  • Login rate limit: 10 logins / 60s per IP+email (AuthService.ts). The test helper (tests/e2e/utils/auth.ts) caches the session cookie per email and reuses it — keep that intact. New tests that log in as new users (registration/account-create) WILL hammer this; if they flake on "stayed on /login", that's the rate limit — back off / reuse sessions, do not raise the production limit.
  • Re-seed before every run. Plugin binary + state live in ephemeral /tmp; mutation tests drift state. Skipping a re-seed causes false failures (plugin "disabled", settings page blank).
  • Native vs http workflow executor: staging uses native mode (WORKFLOW_TRIGGER_EXECUTOR_MODE=native, Coolify env). Do NOT un-gate internal/api/__workflows/** in routes.ts (a test enforces it).
  • Coolify env: POST and PATCH both CREATE env rows → dupes. GET uuids and DELETE extras.
  • a11y scans: expectNoCriticalA11yViolations now waits for the FloatingLoader overlay to clear before scanning (transient mid-fade contrast). Keep that.
  • Flakes shift: with everything on and one serial worker against a shared remote tenant, the failing test changes run-to-run (login rate-limit, worker timing, real-SSE timing). Fix roots, and confirm with repeated isolation runs; CI uses retries: 1.
  • The CI readiness lane is the ultimate arbiter (.gitea/workflows/e2e-readiness.yml, workflow_dispatch, in-cluster, retries). If you can trigger it via the Gitea API with the operator inputs, that is the cleanest "everything on" verification — but it needs the same flags and a real webhook target.

6. Definition of done

  1. A full suite run with ALL opt-in flags + REQUIRE_E2E_READINESS=1 set, freshly re-seeded, ends with 0 failed and 0 skipped (every test either passes or is justifiably impossible — and if impossible, documented with the concrete reason, not hidden).
  2. Confirm stability: the green run reproduces — run the full suite twice clean, or re-run any test that flaked 3–4× in isolation, all green.
  3. Every fix is committed to main with a clear message and (for product code) deployed; the working tree is clean (ignore the pre-existing packages/db/generated/prisma/** churn).
  4. Update THIS file and the memory boiler-e2e-remaining-tail.md with the final numbers, what you changed, and anything genuinely blocked (e.g. webhook target unavailable) with the reason.
  5. No test was skipped, weakened, or deleted to manufacture green (§3). If you ran out of the 7-hour budget, write down exactly where you stopped, the last run's numbers, and the next step.

Start by running date -u, reading .gitea/workflows/e2e-readiness.yml, then doing one full "everything on" re-seeded run to see the real starting failure set. Then loop. Good luck.

Reference in new issue View Git Blame Copy Permalink