Skip to content

Commit f593fa0

Browse files
shinaokashinaoka
authored
fix: sync compiled-source coverage checker semantics (#5)
* docs: add coverage checker design * docs: add coverage checker implementation plan * fix: check coverage against compiled runtime sources
1 parent 3afb291 commit f593fa0

File tree

4 files changed

+614
-10
lines changed

4 files changed

+614
-10
lines changed

‎docs/plans/2026-03-08-compiled-source-coverage-checker-design.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Compiled Source Coverage Checker Design
2+
3+
## Context
4+
5+
`scripts/check-coverage.py` should validate the Rust source files that were actually compiled in the current `cargo llvm-cov` lane, not every `src/**/*.rs` file that exists in the repository and not only the files already present in `coverage.json`.
6+
7+
Issue #3 asks for the semantics already proven in downstream `tenferro-rs`, while preserving template-local threshold configuration and the existing `--report-only` surface.
8+
9+
## Goals
10+
11+
- Prefer dep-info from `target/llvm-cov-target/**/*.d` to derive the expected source universe.
12+
- Judge only runtime-bearing `src/**/*.rs` files that were compiled in the current lane.
13+
- Ignore module-local unit test sources under `src/**/tests/*.rs`.
14+
- Ignore declaration-only compiled Rust files that carry no runtime code.
15+
- Fall back to a repository scan only when dep-info is unavailable.
16+
- Preserve `default`, `files`, `exclude`, and `--report-only`.
17+
18+
## Non-Goals
19+
20+
- Raising default thresholds.
21+
- Adding downstream repo-specific smoke tests.
22+
- Changing CI policy or coverage thresholds outside the checker contract.
23+
24+
## Approach Options
25+
26+
### Option 1: Coverage JSON only
27+
28+
Use only entries already present in `coverage.json` and keep threshold evaluation as-is.
29+
30+
- Pros: smallest implementation.
31+
- Cons: cannot detect compiled source files omitted from the coverage report, which is the actual bug.
32+
33+
### Option 2: Dep-info first, repository scan fallback
34+
35+
Parse `target/llvm-cov-target/**/*.d` to discover compiled `.rs` files, filter them to runtime-bearing `src/**/*.rs`, and fall back to scanning the repo only when dep-info is unavailable.
36+
37+
- Pros: matches the desired contract, supports feature-gated lanes, and keeps local coverage aligned with the built lane.
38+
- Cons: requires lightweight Rust source classification logic.
39+
40+
### Option 3: Always full repository scan
41+
42+
Ignore dep-info and scan the repository for `src/**/*.rs` files, then heuristically drop declarations and module-local tests.
43+
44+
- Pros: simpler than dep-info parsing.
45+
- Cons: incorrectly includes feature-gated files not compiled in the current lane.
46+
47+
## Decision
48+
49+
Use Option 2.
50+
51+
This is the only option that satisfies the issue requirements without broadening the source universe beyond the current `cargo llvm-cov` lane.
52+
53+
## Data Flow
54+
55+
1. Load thresholds from `coverage-thresholds.json`.
56+
2. Read coverage JSON from stdin or the first positional path, while preserving `--report-only`.
57+
3. Build the expected source universe:
58+
- Parse dep-info files when available.
59+
- Resolve dep-info paths against both the dep-info parent directory and the repository root.
60+
- Keep only compiled `src/**/*.rs` files.
61+
- Exclude `src/**/tests/*.rs`.
62+
- Exclude declaration-only files with no runtime-bearing code.
63+
- Fall back to repository scan when dep-info yields no files.
64+
4. Compare expected files against `coverage.json` entries, still honoring `exclude`.
65+
5. Report threshold failures plus compiled-but-missing files.
66+
67+
## Runtime-Bearing Heuristic
68+
69+
The checker needs a narrow heuristic instead of a full Rust parser.
70+
71+
- Treat functions with bodies as runtime-bearing.
72+
- Treat `const` or `static` items initialized with block expressions as runtime-bearing.
73+
- Treat declaration-only items such as traits, structs, enums, and function signatures ending in `;` as non-runtime-bearing.
74+
- Allow multi-line function signatures by buffering until either `{` or `;` appears.
75+
76+
This is intentionally conservative: it should avoid false positives for declaration-only modules while still catching compiled executable code.
77+
78+
## Testing Strategy
79+
80+
Add script tests covering:
81+
82+
- failing when a compiled `src/**/*.rs` file is absent from `coverage.json`
83+
- ignoring `src/<module>/tests/*.rs`
84+
- ignoring feature-gated files not compiled in the current lane
85+
- ignoring declaration-only compiled files
86+
- resolving repo-relative dep-info paths
87+
88+
The tests should build temporary repositories around the script so the semantics are validated without depending on the workspace layout.
89+
90+
## Risks
91+
92+
- Dep-info files may contain relative paths from different working directories.
93+
Mitigation: resolve against both the dep-info file location and repo root.
94+
- The runtime heuristic may misclassify unusual Rust constructs.
95+
Mitigation: keep the heuristic minimal and behavior-focused, matching the issue requirements only.

‎docs/plans/2026-03-08-compiled-source-coverage-checker.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Compiled Source Coverage Checker Implementation Plan
2+
3+
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
4+
5+
**Goal:** Update `scripts/check-coverage.py` so it judges the runtime-bearing Rust source files actually compiled in the current `cargo llvm-cov` lane, with regression tests for the compiled-source contract.
6+
7+
**Architecture:** Extend the checker with two source-universe builders: a dep-info-based builder for the preferred path and a repository scan fallback for missing dep-info. Keep threshold handling and `--report-only` intact, and validate semantics through temporary-repo script tests.
8+
9+
**Tech Stack:** Python 3, `unittest`, `cargo llvm-cov` dep-info semantics, Rust source text heuristics
10+
11+
---
12+
13+
### Task 1: Capture the red tests in the isolated worktree
14+
15+
**Files:**
16+
- Create: `scripts/tests/test_check_coverage.py`
17+
- Modify: `scripts/check-coverage.py`
18+
19+
**Step 1: Write the failing tests**
20+
21+
Add regression tests for:
22+
23+
- compiled `src/**/*.rs` files missing from `coverage.json`
24+
- `src/**/tests/*.rs` exclusion
25+
- feature-gated compiled-source exclusion when absent from dep-info
26+
- declaration-only compiled-source exclusion
27+
- repo-relative dep-info resolution
28+
29+
**Step 2: Run the targeted tests to verify they fail**
30+
31+
Run: `python3 -m unittest tests.test_repo_settings_scripts`
32+
33+
Expected: unchanged existing tests pass, and the new coverage-checker tests fail until the script is updated.
34+
35+
**Step 3: Keep the test fixture style minimal**
36+
37+
- Use temporary directories.
38+
- Copy the workspace script into the temp repo.
39+
- Construct only the files needed for each case.
40+
41+
**Step 4: Commit the red tests**
42+
43+
```bash
44+
git add scripts/tests/test_check_coverage.py
45+
git commit -m "test: add coverage checker regression cases"
46+
```
47+
48+
### Task 2: Implement dep-info-driven source discovery
49+
50+
**Files:**
51+
- Modify: `scripts/check-coverage.py`
52+
- Test: `scripts/tests/test_check_coverage.py`
53+
54+
**Step 1: Add the minimal helper functions**
55+
56+
Implement helpers for:
57+
58+
- runtime-bearing source detection
59+
- repository scan fallback
60+
- dep-info parsing
61+
- expected source universe assembly
62+
63+
**Step 2: Update main coverage evaluation**
64+
65+
- Preserve `default`, `files`, and `exclude`.
66+
- Preserve stdin vs path input handling.
67+
- Preserve `--report-only`.
68+
- Add compiled-but-missing files as failures with `0.0%`.
69+
70+
**Step 3: Run the new tests**
71+
72+
Run: `python3 scripts/tests/test_check_coverage.py`
73+
74+
Expected: PASS
75+
76+
**Step 4: Run a second pass against formatting and broader regressions**
77+
78+
Run: `python3 -m unittest tests.test_monitor_pr_checks`
79+
80+
Expected: PASS
81+
82+
**Step 5: Commit the implementation**
83+
84+
```bash
85+
git add scripts/check-coverage.py scripts/tests/test_check_coverage.py
86+
git commit -m "fix: check coverage against compiled runtime sources"
87+
```
88+
89+
### Task 3: Verify the workspace contract end-to-end
90+
91+
**Files:**
92+
- Verify: `scripts/check-coverage.py`
93+
- Verify: `scripts/tests/test_check_coverage.py`
94+
95+
**Step 1: Run Rust workspace verification**
96+
97+
Run: `cargo fmt --all`
98+
99+
Expected: exit 0
100+
101+
**Step 2: Run lint verification**
102+
103+
Run: `cargo clippy --workspace`
104+
105+
Expected: exit 0
106+
107+
**Step 3: Run release tests**
108+
109+
Run: `cargo test --release --workspace`
110+
111+
Expected: exit 0
112+
113+
**Step 4: Run the Python regression suite**
114+
115+
Run: `python3 scripts/tests/test_check_coverage.py`
116+
117+
Expected: PASS
118+
119+
**Step 5: Commit verification-only follow-up if the implementation changed during cleanup**
120+
121+
```bash
122+
git add scripts/check-coverage.py scripts/tests/test_check_coverage.py
123+
git commit -m "chore: finalize coverage checker verification"
124+
```
125+
126+
Only create this commit if verification required additional code changes.

0 commit comments

Comments
 (0)