154 lines
16 KiB
Markdown
154 lines
16 KiB
Markdown
# E2E Readiness Fixtures
|
|
|
|
Use the local readiness seed when a fresh database should exercise the authenticated Playwright suite instead of silently skipping plugin, workflow, billing, and dashboard paths. The same fail-closed contract is wired into `.github/workflows/e2e-readiness.yml` and `.gitea/workflows/e2e-readiness.yml` for scheduled or manually dispatched staging/readiness runs.
|
|
|
|
This is a local test-readiness runbook. It does not replace the manual,
|
|
redacted staging evidence contract in [Staging, Live, and UAT Readiness](../staging-live-uat.md), and it does not mark the plugin runtime production-ready before the external security sign-off described in [Plugins](../features/plugins.md).
|
|
These fixtures prove local authenticated browser/a11y coverage for the named UI
|
|
paths; they do not prove production readiness for plugin execution, workflow
|
|
workers, provider delivery, staging UAT, or operator-owned infrastructure.
|
|
Run the browser command below only as an operator-approved local/readiness
|
|
browser run; it is intentionally not part of automatic docs/static readiness
|
|
verification.
|
|
If `E2E_BASE_URL` is unset, empty, or only whitespace, the Playwright wrapper
|
|
probes the local default `http://localhost:3000`.
|
|
|
|
```bash
|
|
E2E_LOCAL_READINESS_SEED=1 pnpm run seed:e2e-readiness
|
|
set -a; source .env.e2e-readiness; set +a
|
|
pnpm run verify:a11y-coverage
|
|
REQUIRE_E2E_A11Y=1 REQUIRE_AUTH_E2E=1 REQUIRE_E2E_READINESS=1 E2E_A11Y_STRICT=1 E2E_BROWSER_MATRIX=1 E2E_ALLOW_WORKFLOW_RUN=1 pnpm run verify:e2e-a11y
|
|
```
|
|
|
|
The seed is idempotent, refuses `NODE_ENV=production`, and writes `.env.e2e-readiness` with deterministic credentials, `E2E_NO_TENANT_EMAIL` / `E2E_NO_TENANT_PASSWORD` for the disposable no-tenant account-creation flow, `E2E_PLUGIN_READONLY_EMAIL` / `E2E_PLUGIN_READONLY_PASSWORD`, account slug/name, tenant dashboard/settings/profile paths, governance/policies path, API overview/keys/docs/logs paths, admin email sequence path, plugin settings path, workflow builder/API-run/manual-run paths, a separate webhook workflow builder/manual-run path, and a waiting-workflow execution path. The seeded workflow ids are fixed (`wf_123`, `wf_webhook_123`, and `wf_waiting_123`) so the local env file and scheduled `.github`/`.gitea` readiness workflows target the same staging paths. The seed also creates one successful manual execution (`wfe_123`) with a successful log block run (`wfbe_log_123`) so fresh strict-readiness databases can render workflow execution list/detail mobile and a11y checks before any mutating run drains through a worker. It also creates a paused wait-for-input execution (`wfe_waiting_123`) so waiting-state execution details and continuation form contracts can be inspected deterministically without claiming that the worker completed the continuation. The no-tenant fixture user is reset with no default tenant, no memberships, and no roles so `E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1` exercises the real account onboarding path instead of reusing the seeded tenant. The read-only plugin user joins the seeded tenant, has that tenant as its default tenant, and is reset to only the default `User` app role, which includes `app.plugins.use` but not `app.plugins.manage`. It also writes `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=0` plus an empty `E2E_WORKFLOW_WEBHOOK_TARGET_URL` so a sourced readiness env never accidentally calls an external endpoint. It also writes `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=0`, `E2E_WORKFLOW_OUTBOX_DRAIN_EXPECTED_STATUS=success`, `E2E_WORKFLOW_OUTBOX_DRAIN_ASSERTION=operator-gated`, and an operator-gated readiness note because successful workflow execution requires the general worker/outbox path to be running. It also seeds one active owned subscription plan for billing/cancellation checks and a second public unowned upgrade plan so checkout-catalog smokes always have an enabled submit button without mutating the active subscription. It also seeds deterministic admin email sequences, one non-secret API key, and one `/v1/e2e-readiness` API log row so authenticated API overview, docs, logs, governance, email-sequence dashboard, and key-management mobile/a11y checks can run without production telemetry. The default manual workflow is log-only and enables `E2E_ALLOW_WORKFLOW_RUN=1`; the API-run page stays form-contract tested without invoking the backend API directly, and the webhook workflow stays contract-tested by default so it can validate HTTP/Webhook payload wiring without external secrets. The plugin fixture intentionally has one granted capability and one missing capability so consent, revoke, manifest, and settings iframe checks have meaningful state.
|
|
|
|
Workflow worker/outbox-drain readiness is now available as an explicit operator-gated browser probe. Keep `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN` unset or `0` for normal local, staging, and scheduled readiness runs unless the general worker is running and subscribed to the workflow/outbox path. When an operator sets `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1`, the workflow mobile/a11y spec requires seeded tenant credentials plus `E2E_WORKFLOW_RUN_MANUAL_PATH`, submits the log-only manual workflow, follows the returned execution details link, and polls until the execution reaches `E2E_WORKFLOW_OUTBOX_DRAIN_EXPECTED_STATUS` (`success` by default). This probe intentionally waits for the domain `WorkflowExecution`, not merely for the `OutboxEvent` row to become `DONE`: outbox `DONE` only proves handoff to BullMQ, while `outbox.workflow` job state and `WorkflowExecution.status` prove whether the worker bridge actually ran the workflow. A drained execution that ends in `error` is a failed readiness signal unless an operator explicitly sets `E2E_WORKFLOW_OUTBOX_DRAIN_EXPECTED_STATUS=error` for a negative-path probe. Use `E2E_WORKFLOW_OUTBOX_DRAIN_TIMEOUT_MS` to tune the default 90-second poll window for slower readiness environments.
|
|
|
|
Plugin `workflow.executed` hook readiness is a separate operator-gated browser probe layered on top of the workflow outbox drain. Keep `E2E_REQUIRE_PLUGIN_HOOK_AUDIT` unset or `0` for normal local, staging, and scheduled readiness runs; the plugin hook lifecycle spec then only exercises the non-mutating audit-trail view and the mutation-gated deactivate→re-enable lifecycle flow. Setting `E2E_REQUIRE_PLUGIN_HOOK_AUDIT=1` requires a running general worker, the plugin runtime, and outbox drain, so it reuses the same workflow outbox-drain readiness gate (it internally calls `requireWorkflowOutboxDrainReadiness`, which still needs `E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1`). When enabled, the spec requires `E2E_TENANT_EMAIL`, `E2E_TENANT_PASSWORD`, `E2E_PLUGIN_SETTINGS_PATH`, and `E2E_WORKFLOW_RUN_MANUAL_PATH`; any missing value fails with a clear configuration error instead of skipping. The probe grants the seeded readiness plugin's `audit` capability (mutation-gated additionally behind `E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1`), runs the seeded log-only manual workflow to fire `workflow.executed`, waits for the worker to drain the outbox and the execution to terminate, then polls the plugin capability-use audit trail until a success row appears for the hook-driven `audit`/`log` capability. It restores the seeded baseline (revoking the `audit` grant; re-running the readiness seed also resets capability state) so reruns stay idempotent. Use `E2E_PLUGIN_HOOK_AUDIT_TIMEOUT_MS` to tune the default 90-second audit poll window for slower readiness environments. This operator opt-in is owned by the readiness lane, not the seed, so the seed's `buildE2eReadinessEnv` does not emit `E2E_REQUIRE_PLUGIN_HOOK_AUDIT` or `E2E_PLUGIN_HOOK_AUDIT_TIMEOUT_MS`.
|
|
|
|
Real webhook/HTTP execution is a separate opt-in. Keep `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN` unset or `0` for normal local, staging, and scheduled readiness runs; the Playwright spec intercepts the form POST and verifies the payload contract only. For a disposable operator-approved run that may call an HTTP endpoint, set both `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1` and `E2E_WORKFLOW_WEBHOOK_TARGET_URL=<operator-owned-url>` after sourcing `.env.e2e-readiness`. When that opt-in is enabled, the wrapper and spec require `E2E_TENANT_EMAIL`, `E2E_TENANT_PASSWORD`, `E2E_WORKFLOW_WEBHOOK_BUILDER_PATH`, `E2E_WORKFLOW_WEBHOOK_RUN_MANUAL_PATH`, and `E2E_WORKFLOW_WEBHOOK_TARGET_URL`; any missing value fails with a clear configuration error instead of skipping.
|
|
|
|
The scheduled GitHub/Gitea readiness workflows keep those operator-owned
|
|
probes off by default. Manual `workflow_dispatch` exposes explicit inputs for
|
|
plugin lifecycle mutation, plugin settings writes, workflow outbox drain, and
|
|
real webhook execution; the wrapper fails before Playwright starts when a
|
|
selected input is missing its required path, credential, expected status, or
|
|
target URL contract.
|
|
|
|
Mutating provisioning checks stay opt-in and are separate from normal strict
|
|
readiness. Set `REQUIRE_E2E_MUTATING_READINESS=1` together with the relevant
|
|
`E2E_ALLOW_*` opt-ins only for disposable local/readiness databases where the
|
|
suite may create a new user, tenant, no-tenant account, or mutate plugin state.
|
|
Keep those values unset for non-mutating staging smoke runs.
|
|
|
|
For a disposable local database where all mutating readiness flows are
|
|
operator-approved, use the named wrapper so every required opt-in stays visible
|
|
and versioned together:
|
|
|
|
```bash
|
|
E2E_LOCAL_READINESS_SEED=1 pnpm run seed:e2e-readiness
|
|
set -a; source .env.e2e-readiness; set +a
|
|
pnpm run verify:e2e-mutating-readiness:local
|
|
```
|
|
|
|
`verify:e2e-mutating-readiness:local` sets
|
|
`REQUIRE_E2E_MUTATING_READINESS=1`,
|
|
`E2E_ALLOW_REGISTRATION_CREATE=1`, `E2E_ALLOW_ACCOUNT_CREATE=1`,
|
|
`E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1`,
|
|
`E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1`, and
|
|
`E2E_ALLOW_PLUGIN_SETTINGS_WRITE=1`. Do not use it against shared staging or
|
|
production databases.
|
|
Registration provisioning has two explicit branches: keep
|
|
`E2E_REGISTRATION_EXPECTS_EMAIL_VERIFICATION=0` when the lane must prove that a
|
|
new user reaches a tenant dashboard and can log back in; set it to `1` only for
|
|
a disposable verification-required lane that proves signup stops at the email
|
|
verification confirmation instead of entering the app shell.
|
|
When `E2E_ALLOW_NO_TENANT_ACCOUNT_CREATE=1` is set, the wrapper also fails
|
|
before Playwright starts unless `E2E_NO_TENANT_EMAIL` and
|
|
`E2E_NO_TENANT_PASSWORD` are present. The local readiness seed writes those
|
|
values, so the named wrapper should be run after sourcing `.env.e2e-readiness`.
|
|
|
|
When the local general worker is also running and subscribed to the workflow
|
|
outbox path, use the worker-drain wrapper instead:
|
|
|
|
```bash
|
|
E2E_LOCAL_READINESS_SEED=1 pnpm run seed:e2e-readiness
|
|
set -a; source .env.e2e-readiness; set +a
|
|
pnpm run verify:e2e-mutating-readiness:worker-local
|
|
```
|
|
|
|
`verify:e2e-mutating-readiness:worker-local` delegates to the disposable local
|
|
mutating readiness lane and additionally sets
|
|
`E2E_REQUIRE_WORKFLOW_OUTBOX_DRAIN=1`. That makes the browser workflow run poll
|
|
the resulting `WorkflowExecution.status` until it reaches
|
|
`E2E_WORKFLOW_OUTBOX_DRAIN_EXPECTED_STATUS` (`success` by default), so the lane
|
|
proves the worker bridge drained the workflow instead of only proving that the
|
|
web runtime accepted a manual run request. Do not use it against shared staging
|
|
or production databases.
|
|
|
|
Plugin lifecycle persistence checks also stay opt-in. Default plugin settings
|
|
readiness verifies mobile/a11y, consent, revoke forms, lifecycle forms,
|
|
manifest disclosure, and settings iframe contracts without changing the seeded
|
|
plugin state. Set `E2E_ALLOW_PLUGIN_LIFECYCLE_MUTATION=1` only for disposable
|
|
local/readiness databases where the suite may persist capability grant/revoke
|
|
and plugin enable/disable state, then leave it unset for non-mutating staging
|
|
smoke runs.
|
|
|
|
`REQUIRE_E2E_READINESS=1` intentionally requires the mobile browser matrix, strict a11y mode, real log-only workflow-run opt-in, and every seeded path or credential consumed by the non-mutating strict specs: dashboard, account/profile settings, governance/policies, subscription, API overview/keys/docs/logs, portal analytics/settings, admin email sequences, plugin settings, read-only plugin credentials, workflow engine index, workflow builder/API-run/manual-run, webhook builder/manual-run, and the local plugin binary base path. That keeps staging/readiness runs from silently exercising only the desktop Chromium lane, skipping account/profile continuity, API docs/logs, portal management coverage, plugin permission coverage, workflow engine coverage, email-sequence dashboard coverage, or governance coverage, or ignoring moderate accessibility regressions. It does not require `E2E_ALLOW_WORKFLOW_WEBHOOK_RUN=1`; real webhook delivery stays operator-owned.
|
|
|
|
Subscription checkout success remains a separate strict opt-in because it
|
|
requires an externally acquired Stripe checkout session return path. Normal
|
|
readiness runs stay non-mutating and do not require
|
|
`E2E_TENANT_SUBSCRIPTION_SUCCESS_PATH`. In an operator-approved lane that has a
|
|
valid success return URL, enable the manual `subscription_success` dispatch
|
|
input and provide `subscription_success_path`; the workflow exports
|
|
`E2E_REQUIRE_SUBSCRIPTION_SUCCESS=1` together with
|
|
`E2E_TENANT_SUBSCRIPTION_SUCCESS_PATH`. The wrapper fails before Playwright
|
|
starts if that path is missing, and the subscription success smoke fails
|
|
instead of silently skipping if the configured path is unavailable or resolves
|
|
to an invalid checkout session. The same `E2E_REQUIRE_SUBSCRIPTION_SUCCESS`
|
|
lane now also verifies reconciliation followthrough: after the checkout return
|
|
renders, it follows the dashboard link, opens the subscription settings page,
|
|
and asserts the reconciled active-subscription state survives both a fresh
|
|
page reload and a full logout plus re-login (a fresh appData load). Because
|
|
reconciliation is loader-side (pull-based), the reconciled state is read back
|
|
through the real subscription settings loader rather than the success-page
|
|
render alone, and the flow only navigates app routes (dashboard, settings,
|
|
login/logout) so it never triggers a real Stripe network call. This stays a
|
|
strict opt-in for the same external-blocker reason: it needs a real Stripe
|
|
checkout return path, so the lane skips when `E2E_REQUIRE_SUBSCRIPTION_SUCCESS`
|
|
is off and fails loudly when it is required but `E2E_TENANT_SUBSCRIPTION_SUCCESS_PATH`
|
|
or tenant credentials are missing.
|
|
|
|
The static a11y coverage gate also enforces named browser readiness surfaces,
|
|
not only raw file counts. `pnpm run verify:a11y-coverage` must keep explicit
|
|
coverage for registration-to-dashboard, plugin settings, tenant API docs/logs,
|
|
tenant API keys, audit logs, audit export, privacy self-service, tenant members,
|
|
tenant notification settings, tenant chatbot settings, AI usage, workflow
|
|
builder/run, portal management, admin email sequences, subscription smoke, authenticated mobile shell,
|
|
authenticated mobile locale persistence, public auth, and auth-locale flows. The required
|
|
files must contain the expected guard calls, not only exist: mobile/readiness
|
|
flows need `expectNoHorizontalOverflow`, and critical user surfaces also need
|
|
`expectNoCriticalA11yViolations`. The authenticated mobile locale flow must
|
|
switch locales, reload the dashboard, navigate through the mobile shell to
|
|
profile settings, and restore the original locale in `finally`.
|
|
Registration/account
|
|
readiness also keeps a non-mutating mobile login redirect replay for
|
|
`/new-account`, verifies the account creation form contract is present, and
|
|
returns to the seeded dashboard without creating another tenant. The public
|
|
mobile surface is part of that named coverage: landing, legal, error,
|
|
feature-disabled, and footer selector flows must cover narrow and 320px compact
|
|
viewports, no horizontal overflow, and critical axe checks.
|
|
|
|
The scheduled readiness workflows intentionally stay separate from main CI. They
|
|
install Chromium and WebKit, enable the mobile browser matrix, require staging
|
|
credentials via `E2E_TENANT_EMAIL` / `E2E_TENANT_PASSWORD`, require a read-only
|
|
plugin user via `E2E_PLUGIN_READONLY_EMAIL` /
|
|
`E2E_PLUGIN_READONLY_PASSWORD`, and run
|
|
`pnpm run verify:e2e-a11y` with the non-mutating readiness `REQUIRE_*` flags set. Missing credentials,
|
|
missing seeded paths, missing browsers, unreachable staging, or moderate
|
|
accessibility issues fail the lane instead of producing a SKIP.
|
|
Do not set `SKIP_E2E_A11Y=1` in required readiness, auth, mutating readiness,
|
|
or subscription-success lanes; the wrapper treats that combination as a
|
|
configuration failure.
|