Test Coverage Policy

Status: completed; no-regression gate in maintenance Spec: docs/improvement-plan/10-compliance-polish/test-coverage.md Owner: Engineering quality / release maintenance

This document explains how test coverage is measured, gated, and reported in the ToolCTRL boilerplate. Per-path thresholds are encoded as data in scripts/coverage-targets.json and enforced by scripts/verify-test-coverage.mjs. CI workflows run the gate on GitHub PRs and push to main, while the Gitea coverage workflow currently runs on pull requests. Keep .github/workflows/coverage.yml, .gitea/workflows/coverage.yml, and this policy aligned when trigger coverage changes.

Why

Per-path thresholds reflect risk: auth, billing, tenant-isolation, shared-store, and outbox modules carry materially higher correctness and security risk than view components or marketing pages. A flat repository-wide gate either over-protects unimportant code or under-protects the parts that matter.
No-regression baseline is active. Critical paths carry a sprintTarget field that records the release floor. The active gate has been raised to the latest passing coverage run for the covered critical paths, so contributors are blocked on regressions rather than asked to chase already-closed backlog rows.

Tooling

Provider: V8 native coverage via @vitest/coverage-v8 and vitest@3.2.4.
Reporters: text-summary, json-summary, html, lcov. Configured in vitest.config.ts so scripts/verify-test-coverage.mjs can use a stable vitest run --coverage invocation.
Output: coverage/ (git-ignored). coverage-summary.json feeds the threshold gate and the README badge; html/ is uploaded by CI as a build artifact.

Local commands

# Run vitest with coverage and enforce per-path thresholds.
pnpm run verify:test-coverage

# Print the shields.io badge URL for the current coverage-summary.json.
pnpm coverage:badge

# Render the per-path table as a markdown PR comment.
node scripts/coverage-summary.mjs --pr-comment

Critical paths

The following modules carry tighter thresholds because regressions there have disproportionate impact. Each entry in scripts/coverage-targets.json with "criticalPath": true and a sprintTarget belongs to this list.

Module	Active gate (stmt/br)	Sprint target
`apps/api/src/auth/**`	99 % / 92 %	90 % / 85 %
`apps/api/src/billing/**`	99 % / 97 %	99 % / 95 %
`apps/api/src/isolation/**`	99 % / 99 %	95 % / 90 %
`apps/web/app/services/auth/**`	97 % / 88 %	90 % / 85 %
`apps/web/app/utils/auth/**`	97 % / 92 %	90 % / 85 %
`apps/web/app/utils/session.server.ts`	99 % / 98 %	95 % / 90 %
`apps/web/app/utils/stripe.server.ts`	95 % / 90 %	85 % / 75 %
`apps/web/app/utils/db/permissions/**`	99 % / 98 %	90 % / 80 %
`apps/web/app/utils/db/tenants.db.server.ts`	98 % / 95 %	95 % / 90 %
`packages/entitlements/src/**`	96 % / 89 %	90 % / 80 %
`packages/db/src/outbox*`	92 % / 88 %	90 % / 80 %
`packages/shared-store/src/**`	91 % / 90 %	90 % / 80 %

The active gate is the no-regression baseline — it is the floor below which CI will fail. The sprint target records the release floor that the latest ratchet already meets or exceeds for every critical-path row (see Maintenance and future ratchets).

How to add a coverage gate for a new module

Open scripts/coverage-targets.json.
Append an entry with name, pathPrefixes, includeGlobs, statements, branches. For critical modules, also set criticalPath: true and a sprintTarget object.
Run pnpm run verify:test-coverage locally. The gate prints the actual coverage; set the active gate at or slightly below that number so the commit lands green.
Open a PR. The Coverage workflow will post a sticky comment showing the new row in the per-path table.

How to bump or lower a threshold

Bumps up (tighter gate) — most common after writing new tests:

Run pnpm run verify:test-coverage and read the actual coverage value.
Edit the relevant entry in scripts/coverage-targets.json.
Reference the commit that added the new tests in the PR description.

Bumps down (looser gate) — should be rare and reviewed:

Open the PR with a justification. Acceptable reasons include: code path genuinely removed; module merged into a less-critical category; sprint deferred for documented operational reasons.
Request review from a code owner.
Never lower a criticalPath gate without compensating mitigation (e.g. a corresponding E2E test, a stricter integration suite). Coverage floor for critical paths is a security control.

What triggers a CI failure

pnpm run verify:test-coverage exits non-zero in any of these cases:

vitest itself failed to run (test failure, config error, OOM)
coverage/coverage-summary.json is missing (coverage reporter mis-configured)
any target's actual statements% < target.statements or actual branches% < target.branches

The GitHub workflow uploads coverage/ as coverage-report-<run_id> even on failure so contributors can drill into the HTML report.

Type-only packages

Packages that intentionally emit no runtime JavaScript, such as @toolctrl/api-types, must not add fake exports or expect(true) tests just to satisfy statement coverage. Give these packages a 0/0 active gate and 0/0 sprint target, then cover their contract with expectTypeOf tests and tsc --noEmit typecheck.

Maintenance and future ratchets

Task #76 is completed. Keep the gate useful as a no-regression control:

Confirm docs/testing/coverage-sprint-backlog.md still has no critical-path or standard-tier positive-gap rows before release closeouts.
Keep each active statements/branches gate at or above the relevant sprintTarget.
Treat sprintTarget fields as explicit release-floor annotations unless a future cleanup removes them consistently from docs and gate data.
Run pnpm run verify:test-coverage after threshold changes — it must pass with the new gates.
Track additional Playwright/E2E coverage separately; it is not a blocker for the completed per-module coverage gate.

Out of scope (deferred to follow-up tasks)

Coverage trend dashboard (docs/testing/coverage-trend.md) — nightly cron-updated markdown table. Not implemented in this phase.
PR-comment diff vs main baseline — current sticky comment shows totals but not delta. A future PR can add baseline-fetch + delta rendering similar to bundle-budget.
E2E coverage via Playwright (5 golden-path flows) — tracked separately.
Additional critical-path test writing — see docs/testing/coverage-sprint-backlog.md for the ordered backlog.

References

Spec: docs/improvement-plan/10-compliance-polish/test-coverage.md
Gate runner: scripts/verify-test-coverage.mjs
Targets data: scripts/coverage-targets.json
Summary/badge: scripts/coverage-summary.mjs
CI: .github/workflows/coverage.yml, .gitea/workflows/coverage.yml

7.5 KiB Raw Blame History