Files
prometeu-runtime/docs/vm-arch/COVERAGE_DOMAINS.md
T
bquarkz 22769aa438
Intrepid/Prometeu/Runtime/pipeline/pr-master There was a failure building this commit
docs: close runtime quality review with consolidated lesson
2026-10-14 14:09:12 +01:00

341 lines
18 KiB
Markdown

# Coverage domains
Function: repository coverage ownership and CI operations.
Background lesson: [Domain gates must be owned by the repository](../../discussion/lessons/DSC-0002-runtime-edge-test-plan/LSN-0037-domain-gates-must-be-owned-by-the-repository.md).
Coverage domains assign crate source code to a responsible area. Unit,
integration and cross-layer tests keep their Cargo targets and contribute to
the domains containing the implementation they exercise. Test files do not
need to move into domain directories.
The authorities are
[runtime-edge-coverage-domains.json](../../files/config/runtime-edge-coverage-domains.json),
[the ownership auditor](../../scripts/audit-coverage-domains.py) and
[the evidence/gate helper](../../scripts/coverage-domain-evidence.sh).
The configuration retains its original runtime-edge filename and includes
the core workspace responsibilities as well.
## Ownership
Every Rust file under a repository crate src tree has exactly one owner.
Inclusive paths use repository-relative prefixes; optional exclude_paths
use the same semantics and win within their domain. A file without a domain
or with two domains fails the ownership audit. Sources with no executable
LLVM lines still need an owner.
| Domain | Source responsibility |
| --- | --- |
| system/os | OS facades, process and task services |
| system/runtime | System-side VM lifecycle, scheduling and render integration |
| system/fs | Runtime filesystem, memcard.rs and the complete memcard/ module tree |
| system/programs | Resident system programs |
| system/core | Remaining system sources, excluding the four specialized owners |
| asset/bank | HAL asset/cartridge prefixes, driver asset.rs and the complete driver asset/ module tree |
| firmware | Firmware state and transitions |
| host-dependent | Desktop host |
| host/headless | Headless host and automation implementation |
| vm/core | VM execution, values, heap, verifier, admission, intrinsics and math |
| bytecode | PBX/Registry encodings, ISA, assembler, decoder and binary admission |
| hal/core | HAL contracts and helpers, excluding existing asset/bank ownership |
| drivers/core | Peripheral/render implementations, excluding the asset driver |
| tooling | CLI and dummy/stress PBX producers |
| dev/test-support | Shared test-program construction and development infrastructure source stubs |
Existing specialized paths retain their original ownership.
Both module entry files and sibling module directories need explicit prefixes.
Memcard and asset-driver descendants, including future nested modules, retain
their specialized owner even when absent from the coverage report or reporting
zero executable lines. The corresponding residual exclusions cover both forms.
Residual domains use broad crate source prefixes with explicit exclusions,
so a new module inherits the appropriate owner. A new crate or conflicting
prefix must be assigned before the ownership audit can pass.
## How test suites contribute
VM unit/integration tests, including numeric_contract and the portable
MPFR-generated fixture, exercise vm/core. Bytecode round-trip and Registry
admission tests exercise bytecode; end-to-end loading can also exercise
vm/core. HAL/drivers suites contribute to their corresponding implementation
owners. Headless/CLI integration and development cross-layer suites can
contribute to several domains in one run.
Quality tests remain in prometeu-quality-checks/tests. They enforce the map
and drive the auditor/helper with independent source/report fixtures,
including omitted/overlapping owners, exclusions, absolute/relative filenames,
absent/empty evidence, invalid statistics and percentage regression.
Expected-owner tests use the auditor's ownership function for entry files,
every existing descendant and future nested paths. Synthetic reports also
check absent and zero-line descendants. Exclusive residual fallback is not
sufficient: it can silently transfer specialized code into the wrong gate.
The empty quality/layer library stubs inherit infrastructure ownership;
they do not create separate gates with zero executable lines.
Only crate src paths participate in this domain denominator; integration
test directories, examples, benches, external dependencies and standalone
Python oracle code are outside it. The LLVM instrumentation/report remains
the existing workspace all-features default-profile measurement. Unit tests
colocated in src retain the tool's existing reporting behavior.
The standalone MPFR precision/bit comparison remains complementary numeric
qualification. It is not replaced by line coverage and is not newly required
by make ci-domains. See [numeric qualification](NUMERIC_QUALIFICATION.md).
## Evidence and gate rules
Reports may use absolute or repository-relative source paths. The auditor
normalizes these names before assigning ownership; evidence display and
percentage checks consume its same domain statistics. This prevents path
aliases or separate matching implementations from disagreeing about totals.
The verbose file lists use normalized repository-relative names.
Each report source within scope must exist in the repository and appear once.
Every configured domain must have a positive aggregate instrumented-line
count. Missing reports, unknown sources, duplicate records, invalid counts
and empty domains fail validation. A zero baseline cannot make missing or
empty evidence pass.
The gate compares covered instrumented lines to total instrumented lines,
truncating the percentage downward to two decimals. A domain passes when
this percentage is at least baseline_percent. Counts for functions and
regions remain supporting evidence. Scenario/contract assertions continue
to govern behavior; a percentage does not replace them.
Measured coverage and the acceptance floor are separate values. The historical
margin revision gave every domain at least 5.00 percentage points of headroom
from that recorded local measurement. Its rule was
`new_floor = min(original_floor, measured_percent - 5.00)`, using the same
downward truncation. Floors already providing that margin remain unchanged.
Five percentage points means an 88.56% measurement permits an 83.56% floor;
it is not a relative five-percent reduction.
That historical margin is not a guarantee about later measurements or corrected
ownership. The current map restores the reviewed system/core floor of 87.71%
and preserves every other floor.
Floors remain fixed during CI. A failing run does not automatically lower them.
An intentional scope or floor revision requires explicit review with fresh
measured evidence. New domains also require an explicit floor supported by a
fresh measurement.
## Commands and CI
`make ci-domains` is the authoritative repository quality gate. Both
[GitHub CI](../../.github/workflows/ci.yml) and
[Jenkins](../../files/config/Jenkinsfile) provision dependencies and invoke that
same target. `make ci` is an alias. Invoke one gate per checkout at a time.
The Makefile supplies this ordered sequence to the serial evidence runner:
1. Clean LLVM coverage instrumentation from the previous run.
2. Check formatting with `cargo fmt --all -- --check`.
3. Run `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
4. Test gate failure/report handling with the Python regression suite.
5. Run `cargo test --workspace --all-targets --all-features --no-fail-fast`.
6. Run the five ignored debugger socket tests through `make test-debugger-socket`.
7. Generate fresh workspace/all-feature LLVM coverage, HTML, full JSON,
Cobertura XML and summary JSON.
8. Audit source ownership/evidence and check every fixed coverage domain floor.
The socket command is deliberately restricted:
```sh
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
```
It selects `test_debug_port_opens`, `test_debug_reconnection`,
`test_debug_refuse_second_connection`, `test_get_state_returns_response` and
`test_debug_resume_on_disconnect`. It does not select the audio-device test.
Local sandbox TCP permissions must permit bind/connect; a permission denial is
not evidence of a debugger behavior regression.
The runner executes child Make targets with `-j1` even if the outer invocation
uses `-j`. CI invokes `make ci-domains` serially. Do not run a separate Cargo,
coverage-clean or report process concurrently in the same checkout: serialized
steps inside one gate cannot protect against an independent process.
### Prerequisites and reports
Both pipelines use `bash scripts/provision-quality-gate.sh`. A Debian/Ubuntu
agent needs root or sudo for apt provisioning, and an existing Rust/rustup
installation. GitHub installs stable Rust first; Jenkins must supply a supported
Rust toolchain and rustup in CARGO_HOME/bin. The script installs rustfmt, Clippy,
llvm-tools-preview and pinned cargo-llvm-cov 0.8.5 (`--locked`).
Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev,
libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3,
jq, ripgrep, Git, curl and CA certificates. ALSA development files satisfy CPAL's
Linux build dependency; a running audio device is not needed. Non-Linux local
hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config,
Python 3, jq, rg and Git. The provisioning script verifies these commands.
`target/quality-gate/metadata.json` records revision, dirty working tree, host,
Rust/coverage-tool versions, configuration/source fingerprints, ordered step
exit codes and report hashes. Each step has a separate combined-output log.
The runner stops on the first failed Make target and returns its nonzero exit
status; the log preserves underlying Cargo diagnostics. Three regression cases
check failure logs/identity, fresh success evidence and partial-report retention
when a later gate fails.
Both pipelines discard old generated report directories before provisioning and
publish current artifacts even after gate failure. An early failed gate has
metadata/logs but no old coverage report. Failure after generation preserves the
partial current reports. Provisioning failure can leave no report at all and
still fails CI. Reports are archived from `target/quality-gate/**` and
`target/llvm-cov/**`; Jenkins also publishes current HTML when present. Artifact
publication does not turn a failed gate into success.
List domains or inspect ownership and existing evidence:
```sh
make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core
```
### Explicit manual qualification
Ordinary all-target tests and coverage leave ignored device/measurement tests
ignored. The shared target only opts in to the five socket tests above. It does
not certify the following manual workloads:
```sh
# Physical desktop audio, exact named test.
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
# Physical callback duration under host I/O pressure.
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
# Eight-hour active release soak.
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
# File-host commit latency with the production asset decoder.
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
# Explicit layout observation.
cargo test --release -p prometeu-bytecode --test registry_v1 reference_map_probe -- --ignored --exact --nocapture
```
Other ignored timing probes remain opt-in by their individual test names.
All-target execution can print ordinary benchmark observations; this does not
establish completion of ignored manual qualification. For scope and prior measured
results see [resource stability](resource-stability-qualification.md) and
[audio callback qualification](audio-callback-qualification.md).
## Historical local baseline
The [initial measured evidence](COVERAGE_DOMAINS_EVIDENCE.json) records configuration,
source/report fingerprints, command/toolchain/host, line totals and gate
results for all 15 domains before the margin revision. It preserves the eight
earlier domain definitions and records these seven additional starting floors
on macOS/aarch64:
| New domain | Baseline lines |
| --- | ---: |
| vm/core | 89.12% |
| bytecode | 93.23% |
| hal/core | 88.56% |
| drivers/core | 84.51% |
| system/core | 92.71% |
| tooling | 64.75% |
| dev/test-support | 85.96% |
These are historical starting floors; current acceptance floors appear below.
## Historical margin revision
The [margin evidence](COVERAGE_DOMAINS_MARGIN.json) records fresh measurements,
original and revised floors, toolchain, fingerprints and executed validation.
Only domains with less than 5.00 percentage points of headroom were adjusted.
The historical evidence above remains unchanged.
| Domain | Measured lines | Original floor | Revised floor | Margin (points) |
| --- | ---: | ---: | ---: | ---: |
| system/os | 82.24% | 77.24% | 77.24% | 5.00 |
| system/runtime | 81.19% | 76.50% | 76.19% | 5.00 |
| system/fs | 81.08% | 76.08% | 76.08% | 5.00 |
| system/programs | 98.05% | 93.05% | 93.05% | 5.00 |
| asset/bank | 91.76% | 86.76% | 86.76% | 5.00 |
| firmware | 88.26% | 83.20% | 83.20% | 5.06 |
| host-dependent | 45.05% | 40.05% | 40.05% | 5.00 |
| host/headless | 88.39% | 83.39% | 83.39% | 5.00 |
| vm/core | 89.29% | 89.12% | 84.29% | 5.00 |
| bytecode | 93.26% | 93.23% | 88.26% | 5.00 |
| hal/core | 88.56% | 88.56% | 83.56% | 5.00 |
| drivers/core | 84.51% | 84.51% | 79.51% | 5.00 |
| system/core | 92.71% | 92.71% | 87.71% | 5.00 |
| tooling | 64.75% | 64.75% | 59.75% | 5.00 |
| dev/test-support | 85.96% | 85.96% | 80.96% | 5.00 |
The full local `make ci-domains` validation passed with 939 tests passed and
9 ignored. `system/runtime` measured 81.26% in the initial run and 81.19%
in verification; its final floor uses the lower measurement. All gates were
rechecked against the verification report after that final adjustment.
## Current coverage measurement
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) identifies the committed
source by HEAD and content fingerprints, records the strict shared gate and
retains the earlier ownership-repair measurement as historical evidence.
Both original historical JSON records above remain unchanged. The latest
measurement uses the repaired module-tree map with all 15 fixed floors.
| Domain | Covered / total lines | Measured lines | Fixed floor | Margin (points) |
| --- | ---: | ---: | ---: | ---: |
| system/os | 1410 / 1711 | 82.40% | 77.24% | 5.16 |
| system/runtime | 2757 / 3309 | 83.31% | 76.19% | 7.12 |
| system/fs | 1946 / 2469 | 78.81% | 76.08% | 2.73 |
| system/programs | 556 / 567 | 98.05% | 93.05% | 5.00 |
| asset/bank | 3176 / 3434 | 92.48% | 86.76% | 5.72 |
| firmware | 1990 / 2314 | 85.99% | 83.20% | 2.79 |
| host-dependent | 1531 / 2393 | 63.97% | 40.05% | 23.92 |
| host/headless | 739 / 855 | 86.43% | 83.39% | 3.04 |
| vm/core | 8522 / 9526 | 89.46% | 84.29% | 5.17 |
| bytecode | 3085 / 3303 | 93.39% | 88.26% | 5.13 |
| hal/core | 2093 / 2369 | 88.34% | 83.56% | 4.78 |
| drivers/core | 2775 / 3107 | 89.31% | 79.51% | 9.80 |
| system/core | 1353 / 1444 | 93.69% | 87.71% | 5.98 |
| tooling | 396 / 612 | 64.70% | 59.75% | 4.95 |
| dev/test-support | 99 / 115 | 86.08% | 80.96% | 5.12 |
The fresh macOS/aarch64 `make ci-domains` run passed all 15 domain gates,
1,060 ordinary tests (13 ignored), five explicit socket tests and three
runner regression tests. The fresh coverage replay also passed 1,060 tests;
it is not an additional set of unique behavioral tests. All ten focused
ownership/evidence tests passed again on the committed source. No floor
was lowered. The retained ownership-repair measurement used the earlier
non-strict gate; the current measurement passes strict workspace Clippy.
Platform-specific compilation and host scenarios can change report totals.
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
those runs against the configured floors and investigate differences.
## Strict shared-gate qualification
On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict
workspace/all-target/all-feature Clippy, all 1,060 ordinary tests (13 ignored),
the five explicitly selected socket tests, three gate-runner regression tests,
and all 15 fixed coverage domain gates. The separate fresh coverage execution
also passed 1,060 tests with 13 ignored. Those repeated runs are not distinct
behavioral tests and must not be added into a unique-test count.
The baseline strict invocation failed on the concrete VM lint sources above;
the repaired sources pass without new lint allowances. Socket tests initially
failed because the sandbox denied localhost TCP operations; the authorized
outside-sandbox run passed all five. Gate failure fixtures preserve nonzero
Make status, logs and commit/config identity, discard stale coverage, and retain
fresh partial reports when a later gate fails. Interrupted runners record failure
rather than success.
GitHub's YAML was parsed locally, and both pipeline definitions were inspected
for shared provisioning, `make ci-domains` and unconditional failure-artifact
publication. No remote GitHub or Jenkins build was launched. No Linux gate ran:
the local Docker API was denied by the sandbox and the unrestricted daemon query
did not return a usable result before interruption. Local macOS coverage does not
certify Linux compilation or domain percentages. Such failures must be investigated
on that platform without lowering the existing floors.
No physical audio, eight-hour soak, packaged-installation or ignored manual
latency/layout qualification was selected by this gate. Prior manual observations
remain separate evidence. Current executable metadata and reports are emitted
under `target/quality-gate/` and `target/llvm-cov/` by every invocation.