Intrepid/Prometeu/Runtime/pipeline/pr-master There was a failure building this commit
341 lines
18 KiB
Markdown
341 lines
18 KiB
Markdown
# Coverage domains
|
|
|
|
Function: repository coverage ownership and CI operations.
|
|
|
|
Background lesson: [Domain gates must be owned by the repository](../../discussion/lessons/DSC-0002-runtime-edge-test-plan/LSN-0037-domain-gates-must-be-owned-by-the-repository.md).
|
|
|
|
Coverage domains assign crate source code to a responsible area. Unit,
|
|
integration and cross-layer tests keep their Cargo targets and contribute to
|
|
the domains containing the implementation they exercise. Test files do not
|
|
need to move into domain directories.
|
|
|
|
The authorities are
|
|
[runtime-edge-coverage-domains.json](../../files/config/runtime-edge-coverage-domains.json),
|
|
[the ownership auditor](../../scripts/audit-coverage-domains.py) and
|
|
[the evidence/gate helper](../../scripts/coverage-domain-evidence.sh).
|
|
The configuration retains its original runtime-edge filename and includes
|
|
the core workspace responsibilities as well.
|
|
|
|
## Ownership
|
|
|
|
Every Rust file under a repository crate src tree has exactly one owner.
|
|
Inclusive paths use repository-relative prefixes; optional exclude_paths
|
|
use the same semantics and win within their domain. A file without a domain
|
|
or with two domains fails the ownership audit. Sources with no executable
|
|
LLVM lines still need an owner.
|
|
|
|
| Domain | Source responsibility |
|
|
| --- | --- |
|
|
| system/os | OS facades, process and task services |
|
|
| system/runtime | System-side VM lifecycle, scheduling and render integration |
|
|
| system/fs | Runtime filesystem, memcard.rs and the complete memcard/ module tree |
|
|
| system/programs | Resident system programs |
|
|
| system/core | Remaining system sources, excluding the four specialized owners |
|
|
| asset/bank | HAL asset/cartridge prefixes, driver asset.rs and the complete driver asset/ module tree |
|
|
| firmware | Firmware state and transitions |
|
|
| host-dependent | Desktop host |
|
|
| host/headless | Headless host and automation implementation |
|
|
| vm/core | VM execution, values, heap, verifier, admission, intrinsics and math |
|
|
| bytecode | PBX/Registry encodings, ISA, assembler, decoder and binary admission |
|
|
| hal/core | HAL contracts and helpers, excluding existing asset/bank ownership |
|
|
| drivers/core | Peripheral/render implementations, excluding the asset driver |
|
|
| tooling | CLI and dummy/stress PBX producers |
|
|
| dev/test-support | Shared test-program construction and development infrastructure source stubs |
|
|
|
|
Existing specialized paths retain their original ownership.
|
|
Both module entry files and sibling module directories need explicit prefixes.
|
|
Memcard and asset-driver descendants, including future nested modules, retain
|
|
their specialized owner even when absent from the coverage report or reporting
|
|
zero executable lines. The corresponding residual exclusions cover both forms.
|
|
Residual domains use broad crate source prefixes with explicit exclusions,
|
|
so a new module inherits the appropriate owner. A new crate or conflicting
|
|
prefix must be assigned before the ownership audit can pass.
|
|
|
|
## How test suites contribute
|
|
|
|
VM unit/integration tests, including numeric_contract and the portable
|
|
MPFR-generated fixture, exercise vm/core. Bytecode round-trip and Registry
|
|
admission tests exercise bytecode; end-to-end loading can also exercise
|
|
vm/core. HAL/drivers suites contribute to their corresponding implementation
|
|
owners. Headless/CLI integration and development cross-layer suites can
|
|
contribute to several domains in one run.
|
|
|
|
Quality tests remain in prometeu-quality-checks/tests. They enforce the map
|
|
and drive the auditor/helper with independent source/report fixtures,
|
|
including omitted/overlapping owners, exclusions, absolute/relative filenames,
|
|
absent/empty evidence, invalid statistics and percentage regression.
|
|
Expected-owner tests use the auditor's ownership function for entry files,
|
|
every existing descendant and future nested paths. Synthetic reports also
|
|
check absent and zero-line descendants. Exclusive residual fallback is not
|
|
sufficient: it can silently transfer specialized code into the wrong gate.
|
|
The empty quality/layer library stubs inherit infrastructure ownership;
|
|
they do not create separate gates with zero executable lines.
|
|
|
|
Only crate src paths participate in this domain denominator; integration
|
|
test directories, examples, benches, external dependencies and standalone
|
|
Python oracle code are outside it. The LLVM instrumentation/report remains
|
|
the existing workspace all-features default-profile measurement. Unit tests
|
|
colocated in src retain the tool's existing reporting behavior.
|
|
|
|
The standalone MPFR precision/bit comparison remains complementary numeric
|
|
qualification. It is not replaced by line coverage and is not newly required
|
|
by make ci-domains. See [numeric qualification](NUMERIC_QUALIFICATION.md).
|
|
|
|
## Evidence and gate rules
|
|
|
|
Reports may use absolute or repository-relative source paths. The auditor
|
|
normalizes these names before assigning ownership; evidence display and
|
|
percentage checks consume its same domain statistics. This prevents path
|
|
aliases or separate matching implementations from disagreeing about totals.
|
|
The verbose file lists use normalized repository-relative names.
|
|
|
|
Each report source within scope must exist in the repository and appear once.
|
|
Every configured domain must have a positive aggregate instrumented-line
|
|
count. Missing reports, unknown sources, duplicate records, invalid counts
|
|
and empty domains fail validation. A zero baseline cannot make missing or
|
|
empty evidence pass.
|
|
|
|
The gate compares covered instrumented lines to total instrumented lines,
|
|
truncating the percentage downward to two decimals. A domain passes when
|
|
this percentage is at least baseline_percent. Counts for functions and
|
|
regions remain supporting evidence. Scenario/contract assertions continue
|
|
to govern behavior; a percentage does not replace them.
|
|
|
|
Measured coverage and the acceptance floor are separate values. The historical
|
|
margin revision gave every domain at least 5.00 percentage points of headroom
|
|
from that recorded local measurement. Its rule was
|
|
`new_floor = min(original_floor, measured_percent - 5.00)`, using the same
|
|
downward truncation. Floors already providing that margin remain unchanged.
|
|
Five percentage points means an 88.56% measurement permits an 83.56% floor;
|
|
it is not a relative five-percent reduction.
|
|
That historical margin is not a guarantee about later measurements or corrected
|
|
ownership. The current map restores the reviewed system/core floor of 87.71%
|
|
and preserves every other floor.
|
|
|
|
Floors remain fixed during CI. A failing run does not automatically lower them.
|
|
An intentional scope or floor revision requires explicit review with fresh
|
|
measured evidence. New domains also require an explicit floor supported by a
|
|
fresh measurement.
|
|
|
|
## Commands and CI
|
|
|
|
`make ci-domains` is the authoritative repository quality gate. Both
|
|
[GitHub CI](../../.github/workflows/ci.yml) and
|
|
[Jenkins](../../files/config/Jenkinsfile) provision dependencies and invoke that
|
|
same target. `make ci` is an alias. Invoke one gate per checkout at a time.
|
|
|
|
The Makefile supplies this ordered sequence to the serial evidence runner:
|
|
|
|
1. Clean LLVM coverage instrumentation from the previous run.
|
|
2. Check formatting with `cargo fmt --all -- --check`.
|
|
3. Run `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
|
|
4. Test gate failure/report handling with the Python regression suite.
|
|
5. Run `cargo test --workspace --all-targets --all-features --no-fail-fast`.
|
|
6. Run the five ignored debugger socket tests through `make test-debugger-socket`.
|
|
7. Generate fresh workspace/all-feature LLVM coverage, HTML, full JSON,
|
|
Cobertura XML and summary JSON.
|
|
8. Audit source ownership/evidence and check every fixed coverage domain floor.
|
|
|
|
The socket command is deliberately restricted:
|
|
|
|
```sh
|
|
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
|
|
```
|
|
|
|
It selects `test_debug_port_opens`, `test_debug_reconnection`,
|
|
`test_debug_refuse_second_connection`, `test_get_state_returns_response` and
|
|
`test_debug_resume_on_disconnect`. It does not select the audio-device test.
|
|
Local sandbox TCP permissions must permit bind/connect; a permission denial is
|
|
not evidence of a debugger behavior regression.
|
|
|
|
The runner executes child Make targets with `-j1` even if the outer invocation
|
|
uses `-j`. CI invokes `make ci-domains` serially. Do not run a separate Cargo,
|
|
coverage-clean or report process concurrently in the same checkout: serialized
|
|
steps inside one gate cannot protect against an independent process.
|
|
|
|
### Prerequisites and reports
|
|
|
|
Both pipelines use `bash scripts/provision-quality-gate.sh`. A Debian/Ubuntu
|
|
agent needs root or sudo for apt provisioning, and an existing Rust/rustup
|
|
installation. GitHub installs stable Rust first; Jenkins must supply a supported
|
|
Rust toolchain and rustup in CARGO_HOME/bin. The script installs rustfmt, Clippy,
|
|
llvm-tools-preview and pinned cargo-llvm-cov 0.8.5 (`--locked`).
|
|
|
|
Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev,
|
|
libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3,
|
|
jq, ripgrep, Git, curl and CA certificates. ALSA development files satisfy CPAL's
|
|
Linux build dependency; a running audio device is not needed. Non-Linux local
|
|
hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config,
|
|
Python 3, jq, rg and Git. The provisioning script verifies these commands.
|
|
|
|
`target/quality-gate/metadata.json` records revision, dirty working tree, host,
|
|
Rust/coverage-tool versions, configuration/source fingerprints, ordered step
|
|
exit codes and report hashes. Each step has a separate combined-output log.
|
|
The runner stops on the first failed Make target and returns its nonzero exit
|
|
status; the log preserves underlying Cargo diagnostics. Three regression cases
|
|
check failure logs/identity, fresh success evidence and partial-report retention
|
|
when a later gate fails.
|
|
|
|
Both pipelines discard old generated report directories before provisioning and
|
|
publish current artifacts even after gate failure. An early failed gate has
|
|
metadata/logs but no old coverage report. Failure after generation preserves the
|
|
partial current reports. Provisioning failure can leave no report at all and
|
|
still fails CI. Reports are archived from `target/quality-gate/**` and
|
|
`target/llvm-cov/**`; Jenkins also publishes current HTML when present. Artifact
|
|
publication does not turn a failed gate into success.
|
|
|
|
List domains or inspect ownership and existing evidence:
|
|
|
|
```sh
|
|
make coverage-domain-list
|
|
python3 scripts/audit-coverage-domains.py
|
|
make coverage-domain-audit
|
|
make coverage-domain-evidence DOMAIN=vm/core
|
|
scripts/coverage-domain-evidence.sh --check --verbose hal/core
|
|
```
|
|
|
|
### Explicit manual qualification
|
|
|
|
Ordinary all-target tests and coverage leave ignored device/measurement tests
|
|
ignored. The shared target only opts in to the five socket tests above. It does
|
|
not certify the following manual workloads:
|
|
|
|
```sh
|
|
# Physical desktop audio, exact named test.
|
|
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
|
|
# Physical callback duration under host I/O pressure.
|
|
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
|
|
# Eight-hour active release soak.
|
|
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
|
|
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
|
|
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
|
|
# File-host commit latency with the production asset decoder.
|
|
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
|
|
# Explicit layout observation.
|
|
cargo test --release -p prometeu-bytecode --test registry_v1 reference_map_probe -- --ignored --exact --nocapture
|
|
```
|
|
|
|
Other ignored timing probes remain opt-in by their individual test names.
|
|
All-target execution can print ordinary benchmark observations; this does not
|
|
establish completion of ignored manual qualification. For scope and prior measured
|
|
results see [resource stability](resource-stability-qualification.md) and
|
|
[audio callback qualification](audio-callback-qualification.md).
|
|
|
|
## Historical local baseline
|
|
|
|
The [initial measured evidence](COVERAGE_DOMAINS_EVIDENCE.json) records configuration,
|
|
source/report fingerprints, command/toolchain/host, line totals and gate
|
|
results for all 15 domains before the margin revision. It preserves the eight
|
|
earlier domain definitions and records these seven additional starting floors
|
|
on macOS/aarch64:
|
|
|
|
| New domain | Baseline lines |
|
|
| --- | ---: |
|
|
| vm/core | 89.12% |
|
|
| bytecode | 93.23% |
|
|
| hal/core | 88.56% |
|
|
| drivers/core | 84.51% |
|
|
| system/core | 92.71% |
|
|
| tooling | 64.75% |
|
|
| dev/test-support | 85.96% |
|
|
|
|
These are historical starting floors; current acceptance floors appear below.
|
|
|
|
## Historical margin revision
|
|
|
|
The [margin evidence](COVERAGE_DOMAINS_MARGIN.json) records fresh measurements,
|
|
original and revised floors, toolchain, fingerprints and executed validation.
|
|
Only domains with less than 5.00 percentage points of headroom were adjusted.
|
|
The historical evidence above remains unchanged.
|
|
|
|
| Domain | Measured lines | Original floor | Revised floor | Margin (points) |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| system/os | 82.24% | 77.24% | 77.24% | 5.00 |
|
|
| system/runtime | 81.19% | 76.50% | 76.19% | 5.00 |
|
|
| system/fs | 81.08% | 76.08% | 76.08% | 5.00 |
|
|
| system/programs | 98.05% | 93.05% | 93.05% | 5.00 |
|
|
| asset/bank | 91.76% | 86.76% | 86.76% | 5.00 |
|
|
| firmware | 88.26% | 83.20% | 83.20% | 5.06 |
|
|
| host-dependent | 45.05% | 40.05% | 40.05% | 5.00 |
|
|
| host/headless | 88.39% | 83.39% | 83.39% | 5.00 |
|
|
| vm/core | 89.29% | 89.12% | 84.29% | 5.00 |
|
|
| bytecode | 93.26% | 93.23% | 88.26% | 5.00 |
|
|
| hal/core | 88.56% | 88.56% | 83.56% | 5.00 |
|
|
| drivers/core | 84.51% | 84.51% | 79.51% | 5.00 |
|
|
| system/core | 92.71% | 92.71% | 87.71% | 5.00 |
|
|
| tooling | 64.75% | 64.75% | 59.75% | 5.00 |
|
|
| dev/test-support | 85.96% | 85.96% | 80.96% | 5.00 |
|
|
|
|
The full local `make ci-domains` validation passed with 939 tests passed and
|
|
9 ignored. `system/runtime` measured 81.26% in the initial run and 81.19%
|
|
in verification; its final floor uses the lower measurement. All gates were
|
|
rechecked against the verification report after that final adjustment.
|
|
|
|
## Current coverage measurement
|
|
|
|
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) identifies the committed
|
|
source by HEAD and content fingerprints, records the strict shared gate and
|
|
retains the earlier ownership-repair measurement as historical evidence.
|
|
Both original historical JSON records above remain unchanged. The latest
|
|
measurement uses the repaired module-tree map with all 15 fixed floors.
|
|
|
|
| Domain | Covered / total lines | Measured lines | Fixed floor | Margin (points) |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| system/os | 1410 / 1711 | 82.40% | 77.24% | 5.16 |
|
|
| system/runtime | 2757 / 3309 | 83.31% | 76.19% | 7.12 |
|
|
| system/fs | 1946 / 2469 | 78.81% | 76.08% | 2.73 |
|
|
| system/programs | 556 / 567 | 98.05% | 93.05% | 5.00 |
|
|
| asset/bank | 3176 / 3434 | 92.48% | 86.76% | 5.72 |
|
|
| firmware | 1990 / 2314 | 85.99% | 83.20% | 2.79 |
|
|
| host-dependent | 1531 / 2393 | 63.97% | 40.05% | 23.92 |
|
|
| host/headless | 739 / 855 | 86.43% | 83.39% | 3.04 |
|
|
| vm/core | 8522 / 9526 | 89.46% | 84.29% | 5.17 |
|
|
| bytecode | 3085 / 3303 | 93.39% | 88.26% | 5.13 |
|
|
| hal/core | 2093 / 2369 | 88.34% | 83.56% | 4.78 |
|
|
| drivers/core | 2775 / 3107 | 89.31% | 79.51% | 9.80 |
|
|
| system/core | 1353 / 1444 | 93.69% | 87.71% | 5.98 |
|
|
| tooling | 396 / 612 | 64.70% | 59.75% | 4.95 |
|
|
| dev/test-support | 99 / 115 | 86.08% | 80.96% | 5.12 |
|
|
|
|
The fresh macOS/aarch64 `make ci-domains` run passed all 15 domain gates,
|
|
1,060 ordinary tests (13 ignored), five explicit socket tests and three
|
|
runner regression tests. The fresh coverage replay also passed 1,060 tests;
|
|
it is not an additional set of unique behavioral tests. All ten focused
|
|
ownership/evidence tests passed again on the committed source. No floor
|
|
was lowered. The retained ownership-repair measurement used the earlier
|
|
non-strict gate; the current measurement passes strict workspace Clippy.
|
|
|
|
Platform-specific compilation and host scenarios can change report totals.
|
|
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
|
|
those runs against the configured floors and investigate differences.
|
|
|
|
## Strict shared-gate qualification
|
|
|
|
On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict
|
|
workspace/all-target/all-feature Clippy, all 1,060 ordinary tests (13 ignored),
|
|
the five explicitly selected socket tests, three gate-runner regression tests,
|
|
and all 15 fixed coverage domain gates. The separate fresh coverage execution
|
|
also passed 1,060 tests with 13 ignored. Those repeated runs are not distinct
|
|
behavioral tests and must not be added into a unique-test count.
|
|
|
|
The baseline strict invocation failed on the concrete VM lint sources above;
|
|
the repaired sources pass without new lint allowances. Socket tests initially
|
|
failed because the sandbox denied localhost TCP operations; the authorized
|
|
outside-sandbox run passed all five. Gate failure fixtures preserve nonzero
|
|
Make status, logs and commit/config identity, discard stale coverage, and retain
|
|
fresh partial reports when a later gate fails. Interrupted runners record failure
|
|
rather than success.
|
|
|
|
GitHub's YAML was parsed locally, and both pipeline definitions were inspected
|
|
for shared provisioning, `make ci-domains` and unconditional failure-artifact
|
|
publication. No remote GitHub or Jenkins build was launched. No Linux gate ran:
|
|
the local Docker API was denied by the sandbox and the unrestricted daemon query
|
|
did not return a usable result before interruption. Local macOS coverage does not
|
|
certify Linux compilation or domain percentages. Such failures must be investigated
|
|
on that platform without lowering the existing floors.
|
|
|
|
No physical audio, eight-hour soak, packaged-installation or ignored manual
|
|
latency/layout qualification was selected by this gate. Prior manual observations
|
|
remain separate evidence. Current executable metadata and reports are emitted
|
|
under `target/quality-gate/` and `target/llvm-cov/` by every invocation.
|