237 lines
12 KiB
Markdown
237 lines
12 KiB
Markdown
# Coverage domains
|
|
|
|
Function: repository coverage ownership and CI operations.
|
|
|
|
Background lesson: [Domain gates must be owned by the repository](../../discussion/lessons/DSC-0002-runtime-edge-test-plan/LSN-0037-domain-gates-must-be-owned-by-the-repository.md).
|
|
|
|
Coverage domains assign crate source code to a responsible area. Unit,
|
|
integration and cross-layer tests keep their Cargo targets and contribute to
|
|
the domains containing the implementation they exercise. Test files do not
|
|
need to move into domain directories.
|
|
|
|
The authorities are
|
|
[runtime-edge-coverage-domains.json](../../files/config/runtime-edge-coverage-domains.json),
|
|
[the ownership auditor](../../scripts/audit-coverage-domains.py) and
|
|
[the evidence/gate helper](../../scripts/coverage-domain-evidence.sh).
|
|
The configuration retains its original runtime-edge filename and includes
|
|
the core workspace responsibilities as well.
|
|
|
|
## Ownership
|
|
|
|
Every Rust file under a repository crate src tree has exactly one owner.
|
|
Inclusive paths use repository-relative prefixes; optional exclude_paths
|
|
use the same semantics and win within their domain. A file without a domain
|
|
or with two domains fails the ownership audit. Sources with no executable
|
|
LLVM lines still need an owner.
|
|
|
|
| Domain | Source responsibility |
|
|
| --- | --- |
|
|
| system/os | OS facades, process and task services |
|
|
| system/runtime | System-side VM lifecycle, scheduling and render integration |
|
|
| system/fs | Runtime filesystem, memcard.rs and the complete memcard/ module tree |
|
|
| system/programs | Resident system programs |
|
|
| system/core | Remaining system sources, excluding the four specialized owners |
|
|
| asset/bank | HAL asset/cartridge prefixes, driver asset.rs and the complete driver asset/ module tree |
|
|
| firmware | Firmware state and transitions |
|
|
| host-dependent | Desktop host |
|
|
| host/headless | Headless host and automation implementation |
|
|
| vm/core | VM execution, values, heap, verifier, admission, intrinsics and math |
|
|
| bytecode | PBX/Registry encodings, ISA, assembler, decoder and binary admission |
|
|
| hal/core | HAL contracts and helpers, excluding existing asset/bank ownership |
|
|
| drivers/core | Peripheral/render implementations, excluding the asset driver |
|
|
| tooling | CLI and dummy/stress PBX producers |
|
|
| dev/test-support | Shared test-program construction and development infrastructure source stubs |
|
|
|
|
Existing specialized paths retain their original ownership.
|
|
Both module entry files and sibling module directories need explicit prefixes.
|
|
Memcard and asset-driver descendants, including future nested modules, retain
|
|
their specialized owner even when absent from the coverage report or reporting
|
|
zero executable lines. The corresponding residual exclusions cover both forms.
|
|
Residual domains use broad crate source prefixes with explicit exclusions,
|
|
so a new module inherits the appropriate owner. A new crate or conflicting
|
|
prefix must be assigned before the ownership audit can pass.
|
|
|
|
## How test suites contribute
|
|
|
|
VM unit/integration tests, including numeric_contract and the portable
|
|
MPFR-generated fixture, exercise vm/core. Bytecode round-trip and Registry
|
|
admission tests exercise bytecode; end-to-end loading can also exercise
|
|
vm/core. HAL/drivers suites contribute to their corresponding implementation
|
|
owners. Headless/CLI integration and development cross-layer suites can
|
|
contribute to several domains in one run.
|
|
|
|
Quality tests remain in prometeu-quality-checks/tests. They enforce the map
|
|
and drive the auditor/helper with independent source/report fixtures,
|
|
including omitted/overlapping owners, exclusions, absolute/relative filenames,
|
|
absent/empty evidence, invalid statistics and percentage regression.
|
|
Expected-owner tests use the auditor's ownership function for entry files,
|
|
every existing descendant and future nested paths. Synthetic reports also
|
|
check absent and zero-line descendants. Exclusive residual fallback is not
|
|
sufficient: it can silently transfer specialized code into the wrong gate.
|
|
The empty quality/layer library stubs inherit infrastructure ownership;
|
|
they do not create separate gates with zero executable lines.
|
|
|
|
Only crate src paths participate in this domain denominator; integration
|
|
test directories, examples, benches, external dependencies and standalone
|
|
Python oracle code are outside it. The LLVM instrumentation/report remains
|
|
the existing workspace all-features default-profile measurement. Unit tests
|
|
colocated in src retain the tool's existing reporting behavior.
|
|
|
|
The standalone MPFR precision/bit comparison remains complementary numeric
|
|
qualification. It is not replaced by line coverage and is not newly required
|
|
by make ci-domains. See [numeric qualification](NUMERIC_QUALIFICATION.md).
|
|
|
|
## Evidence and gate rules
|
|
|
|
Reports may use absolute or repository-relative source paths. The auditor
|
|
normalizes these names before assigning ownership; evidence display and
|
|
percentage checks consume its same domain statistics. This prevents path
|
|
aliases or separate matching implementations from disagreeing about totals.
|
|
The verbose file lists use normalized repository-relative names.
|
|
|
|
Each report source within scope must exist in the repository and appear once.
|
|
Every configured domain must have a positive aggregate instrumented-line
|
|
count. Missing reports, unknown sources, duplicate records, invalid counts
|
|
and empty domains fail validation. A zero baseline cannot make missing or
|
|
empty evidence pass.
|
|
|
|
The gate compares covered instrumented lines to total instrumented lines,
|
|
truncating the percentage downward to two decimals. A domain passes when
|
|
this percentage is at least baseline_percent. Counts for functions and
|
|
regions remain supporting evidence. Scenario/contract assertions continue
|
|
to govern behavior; a percentage does not replace them.
|
|
|
|
Measured coverage and the acceptance floor are separate values. The historical
|
|
margin revision gave every domain at least 5.00 percentage points of headroom
|
|
from that recorded local measurement. Its rule was
|
|
`new_floor = min(original_floor, measured_percent - 5.00)`, using the same
|
|
downward truncation. Floors already providing that margin remain unchanged.
|
|
Five percentage points means an 88.56% measurement permits an 83.56% floor;
|
|
it is not a relative five-percent reduction.
|
|
That historical margin is not a guarantee about later measurements or corrected
|
|
ownership. The current map restores the reviewed system/core floor of 87.71%
|
|
and preserves every other floor.
|
|
|
|
Floors remain fixed during CI. A failing run does not automatically lower them.
|
|
An intentional scope or floor revision requires explicit review with fresh
|
|
measured evidence. New domains also require an explicit floor supported by a
|
|
fresh measurement.
|
|
|
|
## Commands and CI
|
|
|
|
List domains, inspect source ownership and validate coverage evidence:
|
|
|
|
make coverage-domain-list
|
|
python3 scripts/audit-coverage-domains.py
|
|
make coverage-domain-audit
|
|
python3 scripts/audit-coverage-domains.py --report target/llvm-cov/report.json --json
|
|
|
|
Produce a fresh report and run all gates:
|
|
|
|
make ci-domains
|
|
|
|
The existing Jenkins configuration invokes this repository target. Its
|
|
coverage-domain-gate step also performs the ownership/evidence audit.
|
|
GitHub's workspace-test step discovers the quality regression tests; it
|
|
does not separately produce a domain coverage report.
|
|
|
|
Inspect a domain:
|
|
|
|
make coverage-domain-evidence DOMAIN=vm/core
|
|
scripts/coverage-domain-evidence.sh --check --verbose hal/core
|
|
scripts/coverage-domain-evidence.sh --check-all
|
|
|
|
Dependencies are the existing Rust/LLVM coverage tooling, Bash, Git, jq and rg,
|
|
plus Python 3 for the standard-library ownership auditor.
|
|
|
|
## Historical local baseline
|
|
|
|
The [initial measured evidence](COVERAGE_DOMAINS_EVIDENCE.json) records configuration,
|
|
source/report fingerprints, command/toolchain/host, line totals and gate
|
|
results for all 15 domains before the margin revision. It preserves the eight
|
|
earlier domain definitions and records these seven additional starting floors
|
|
on macOS/aarch64:
|
|
|
|
| New domain | Baseline lines |
|
|
| --- | ---: |
|
|
| vm/core | 89.12% |
|
|
| bytecode | 93.23% |
|
|
| hal/core | 88.56% |
|
|
| drivers/core | 84.51% |
|
|
| system/core | 92.71% |
|
|
| tooling | 64.75% |
|
|
| dev/test-support | 85.96% |
|
|
|
|
These are historical starting floors; current acceptance floors appear below.
|
|
|
|
## Historical margin revision
|
|
|
|
The [margin evidence](COVERAGE_DOMAINS_MARGIN.json) records fresh measurements,
|
|
original and revised floors, toolchain, fingerprints and executed validation.
|
|
Only domains with less than 5.00 percentage points of headroom were adjusted.
|
|
The historical evidence above remains unchanged.
|
|
|
|
| Domain | Measured lines | Original floor | Revised floor | Margin (points) |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| system/os | 82.24% | 77.24% | 77.24% | 5.00 |
|
|
| system/runtime | 81.19% | 76.50% | 76.19% | 5.00 |
|
|
| system/fs | 81.08% | 76.08% | 76.08% | 5.00 |
|
|
| system/programs | 98.05% | 93.05% | 93.05% | 5.00 |
|
|
| asset/bank | 91.76% | 86.76% | 86.76% | 5.00 |
|
|
| firmware | 88.26% | 83.20% | 83.20% | 5.06 |
|
|
| host-dependent | 45.05% | 40.05% | 40.05% | 5.00 |
|
|
| host/headless | 88.39% | 83.39% | 83.39% | 5.00 |
|
|
| vm/core | 89.29% | 89.12% | 84.29% | 5.00 |
|
|
| bytecode | 93.26% | 93.23% | 88.26% | 5.00 |
|
|
| hal/core | 88.56% | 88.56% | 83.56% | 5.00 |
|
|
| drivers/core | 84.51% | 84.51% | 79.51% | 5.00 |
|
|
| system/core | 92.71% | 92.71% | 87.71% | 5.00 |
|
|
| tooling | 64.75% | 64.75% | 59.75% | 5.00 |
|
|
| dev/test-support | 85.96% | 85.96% | 80.96% | 5.00 |
|
|
|
|
The full local `make ci-domains` validation passed with 939 tests passed and
|
|
9 ignored. `system/runtime` measured 81.26% in the initial run and 81.19%
|
|
in verification; its final floor uses the lower measurement. All gates were
|
|
rechecked against the verification report after that final adjustment.
|
|
|
|
## Current local measurement
|
|
|
|
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) records the repaired
|
|
module-tree ownership, fixed floors, normalized source paths, actual line
|
|
counts and source/config/report fingerprints. Both historical JSON records
|
|
above remain unchanged. The measurement uses the implementation working tree
|
|
identified by its base HEAD and content fingerprints.
|
|
|
|
| Domain | Covered / total lines | Measured lines | Fixed floor | Margin (points) |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| system/os | 1410 / 1711 | 82.40% | 77.24% | 5.16 |
|
|
| system/runtime | 2762 / 3309 | 83.46% | 76.19% | 7.27 |
|
|
| system/fs | 1944 / 2469 | 78.73% | 76.08% | 2.65 |
|
|
| system/programs | 556 / 567 | 98.05% | 93.05% | 5.00 |
|
|
| asset/bank | 3096 / 3420 | 90.52% | 86.76% | 3.76 |
|
|
| firmware | 1990 / 2314 | 85.99% | 83.20% | 2.79 |
|
|
| host-dependent | 1196 / 2059 | 58.08% | 40.05% | 18.03 |
|
|
| host/headless | 739 / 855 | 86.43% | 83.39% | 3.04 |
|
|
| vm/core | 8308 / 9294 | 89.39% | 84.29% | 5.10 |
|
|
| bytecode | 3085 / 3303 | 93.39% | 88.26% | 5.13 |
|
|
| hal/core | 2093 / 2369 | 88.34% | 83.56% | 4.78 |
|
|
| drivers/core | 2728 / 3059 | 89.17% | 79.51% | 9.66 |
|
|
| system/core | 1353 / 1444 | 93.69% | 87.71% | 5.98 |
|
|
| tooling | 396 / 612 | 64.70% | 59.75% | 4.95 |
|
|
| dev/test-support | 99 / 115 | 86.08% | 80.96% | 5.12 |
|
|
|
|
The fresh macOS/aarch64 `make ci-domains` run passed all 15 coverage gates
|
|
with 1,021 tests passed and 11 ignored. The 10 focused ownership/evidence
|
|
tests and strict package Clippy also passed. No floor reduction or additional
|
|
runtime behavior test was needed to pass the repaired map.
|
|
|
|
The current Makefile target still uses non-strict workspace Clippy. A separate
|
|
`cargo clippy --workspace --all-targets --all-features -- -D warnings` run failed
|
|
on existing VM `useless_vec`, `cloned_ref_to_slice_refs` and
|
|
`items_after_test_module` diagnostics. This evidence certifies the corrected
|
|
coverage map and its gates; full strict workspace qualification remains pending.
|
|
|
|
Platform-specific compilation and host scenarios can change report totals.
|
|
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
|
|
those runs against the configured floors and investigate differences.
|