161 lines
7.5 KiB
Markdown
161 lines
7.5 KiB
Markdown
# Test Coverage Policy
|
|
|
|
**Status:** completed; no-regression gate in maintenance
|
|
**Spec:** `docs/improvement-plan/10-compliance-polish/test-coverage.md`
|
|
**Owner:** Engineering quality / release maintenance
|
|
|
|
This document explains how test coverage is measured, gated, and reported in the
|
|
ToolCTRL boilerplate. Per-path thresholds are encoded as data in
|
|
`scripts/coverage-targets.json` and enforced by `scripts/verify-test-coverage.mjs`.
|
|
CI workflows run the gate on GitHub PRs and `push` to `main`, while the
|
|
Gitea coverage workflow currently runs on pull requests. Keep
|
|
`.github/workflows/coverage.yml`, `.gitea/workflows/coverage.yml`, and
|
|
this policy aligned when trigger coverage changes.
|
|
|
|
## Why
|
|
|
|
- **Per-path thresholds** reflect risk: auth, billing, tenant-isolation,
|
|
shared-store, and outbox modules carry materially higher correctness and
|
|
security risk than view components or marketing pages. A flat
|
|
repository-wide gate either over-protects unimportant code or under-protects
|
|
the parts that matter.
|
|
- **No-regression baseline** is active. Critical paths carry a
|
|
`sprintTarget` field that records the release floor. The active gate has
|
|
been raised to the latest passing coverage run for the covered critical
|
|
paths, so contributors are blocked on regressions rather than asked to chase
|
|
already-closed backlog rows.
|
|
|
|
## Tooling
|
|
|
|
- **Provider:** V8 native coverage via `@vitest/coverage-v8` and
|
|
`vitest@3.2.4`.
|
|
- **Reporters:** `text-summary`, `json-summary`, `html`, `lcov`. Configured in
|
|
`vitest.config.ts` so `scripts/verify-test-coverage.mjs` can use a stable
|
|
`vitest run --coverage` invocation.
|
|
- **Output:** `coverage/` (git-ignored). `coverage-summary.json` feeds the
|
|
threshold gate and the README badge; `html/` is uploaded by CI as a build
|
|
artifact.
|
|
|
|
## Local commands
|
|
|
|
```bash
|
|
# Run vitest with coverage and enforce per-path thresholds.
|
|
pnpm run verify:test-coverage
|
|
|
|
# Print the shields.io badge URL for the current coverage-summary.json.
|
|
pnpm coverage:badge
|
|
|
|
# Render the per-path table as a markdown PR comment.
|
|
node scripts/coverage-summary.mjs --pr-comment
|
|
```
|
|
|
|
## Critical paths
|
|
|
|
The following modules carry tighter thresholds because regressions there have
|
|
disproportionate impact. Each entry in `scripts/coverage-targets.json` with
|
|
`"criticalPath": true` and a `sprintTarget` belongs to this list.
|
|
|
|
| Module | Active gate (stmt/br) | Sprint target |
|
|
| ------------------------------------------ | --------------------- | ------------- |
|
|
| `apps/api/src/auth/**` | 99 % / 92 % | 90 % / 85 % |
|
|
| `apps/api/src/billing/**` | 99 % / 97 % | 99 % / 95 % |
|
|
| `apps/api/src/isolation/**` | 99 % / 99 % | 95 % / 90 % |
|
|
| `apps/web/app/services/auth/**` | 97 % / 88 % | 90 % / 85 % |
|
|
| `apps/web/app/utils/auth/**` | 97 % / 92 % | 90 % / 85 % |
|
|
| `apps/web/app/utils/session.server.ts` | 99 % / 98 % | 95 % / 90 % |
|
|
| `apps/web/app/utils/stripe.server.ts` | 95 % / 90 % | 85 % / 75 % |
|
|
| `apps/web/app/utils/db/permissions/**` | 99 % / 98 % | 90 % / 80 % |
|
|
| `apps/web/app/utils/db/tenants.db.server.ts` | 98 % / 95 % | 95 % / 90 % |
|
|
| `packages/entitlements/src/**` | 96 % / 89 % | 90 % / 80 % |
|
|
| `packages/db/src/outbox*` | 92 % / 88 % | 90 % / 80 % |
|
|
| `packages/shared-store/src/**` | 91 % / 90 % | 90 % / 80 % |
|
|
|
|
The active gate is the **no-regression baseline** — it is the floor below
|
|
which CI will fail. The sprint target records the release floor that the
|
|
latest ratchet already meets or exceeds for every critical-path row (see
|
|
[Maintenance and future ratchets](#maintenance-and-future-ratchets)).
|
|
|
|
## How to add a coverage gate for a new module
|
|
|
|
1. Open `scripts/coverage-targets.json`.
|
|
2. Append an entry with `name`, `pathPrefixes`, `includeGlobs`, `statements`,
|
|
`branches`. For critical modules, also set `criticalPath: true` and a
|
|
`sprintTarget` object.
|
|
3. Run `pnpm run verify:test-coverage` locally. The gate prints the actual
|
|
coverage; set the active gate at or slightly below that number so the
|
|
commit lands green.
|
|
4. Open a PR. The Coverage workflow will post a sticky comment showing the
|
|
new row in the per-path table.
|
|
|
|
## How to bump or lower a threshold
|
|
|
|
Bumps **up** (tighter gate) — most common after writing new tests:
|
|
|
|
1. Run `pnpm run verify:test-coverage` and read the actual coverage value.
|
|
2. Edit the relevant entry in `scripts/coverage-targets.json`.
|
|
3. Reference the commit that added the new tests in the PR description.
|
|
|
|
Bumps **down** (looser gate) — should be rare and reviewed:
|
|
|
|
1. Open the PR with a justification. Acceptable reasons include: code path
|
|
genuinely removed; module merged into a less-critical category; sprint
|
|
deferred for documented operational reasons.
|
|
2. Request review from a code owner.
|
|
3. **Never** lower a `criticalPath` gate without compensating mitigation
|
|
(e.g. a corresponding E2E test, a stricter integration suite). Coverage
|
|
floor for critical paths is a security control.
|
|
|
|
## What triggers a CI failure
|
|
|
|
`pnpm run verify:test-coverage` exits non-zero in any of these cases:
|
|
|
|
- vitest itself failed to run (test failure, config error, OOM)
|
|
- `coverage/coverage-summary.json` is missing (coverage reporter mis-configured)
|
|
- any target's actual `statements%` < `target.statements` **or**
|
|
actual `branches%` < `target.branches`
|
|
|
|
The GitHub workflow uploads `coverage/` as `coverage-report-<run_id>` even on
|
|
failure so contributors can drill into the HTML report.
|
|
|
|
## Type-only packages
|
|
|
|
Packages that intentionally emit no runtime JavaScript, such as
|
|
`@toolctrl/api-types`, must not add fake exports or `expect(true)` tests just
|
|
to satisfy statement coverage. Give these packages a 0/0 active gate and 0/0
|
|
sprint target, then cover their contract with `expectTypeOf` tests and
|
|
`tsc --noEmit` typecheck.
|
|
|
|
## Maintenance and future ratchets
|
|
|
|
Task #76 is completed. Keep the gate useful as a no-regression control:
|
|
|
|
1. Confirm `docs/testing/coverage-sprint-backlog.md` still has no
|
|
critical-path or standard-tier positive-gap rows before release closeouts.
|
|
2. Keep each active `statements`/`branches` gate at or above the relevant
|
|
`sprintTarget`.
|
|
3. Treat `sprintTarget` fields as explicit release-floor annotations unless a
|
|
future cleanup removes them consistently from docs and gate data.
|
|
4. Run `pnpm run verify:test-coverage` after threshold changes — it must pass
|
|
with the new gates.
|
|
5. Track additional Playwright/E2E coverage separately; it is not a blocker
|
|
for the completed per-module coverage gate.
|
|
|
|
## Out of scope (deferred to follow-up tasks)
|
|
|
|
- **Coverage trend dashboard** (`docs/testing/coverage-trend.md`) — nightly
|
|
cron-updated markdown table. Not implemented in this phase.
|
|
- **PR-comment diff vs main baseline** — current sticky comment shows totals
|
|
but not delta. A future PR can add baseline-fetch + delta rendering similar
|
|
to bundle-budget.
|
|
- **E2E coverage via Playwright** (5 golden-path flows) — tracked separately.
|
|
- **Additional critical-path test writing** — see
|
|
`docs/testing/coverage-sprint-backlog.md` for the ordered backlog.
|
|
|
|
## References
|
|
|
|
- Spec: `docs/improvement-plan/10-compliance-polish/test-coverage.md`
|
|
- Gate runner: `scripts/verify-test-coverage.mjs`
|
|
- Targets data: `scripts/coverage-targets.json`
|
|
- Summary/badge: `scripts/coverage-summary.mjs`
|
|
- CI: `.github/workflows/coverage.yml`, `.gitea/workflows/coverage.yml`
|