Files
prometeu-runtime/docs/vm-arch/COVERAGE_DOMAINS.md
T
bquarkz a2275932a5
Intrepid/Prometeu/Runtime/pipeline/pr-master There was a failure building this commit
Keep PBX at v0 and simplify Jenkins quality gate
2026-10-14 14:35:44 +01:00

18 KiB

Coverage domains

Function: repository coverage ownership and CI operations.

Background lesson: Domain gates must be owned by the repository.

Coverage domains assign crate source code to a responsible area. Unit, integration and cross-layer tests keep their Cargo targets and contribute to the domains containing the implementation they exercise. Test files do not need to move into domain directories.

The authorities are runtime-edge-coverage-domains.json, the ownership auditor and the evidence/gate helper. The configuration retains its original runtime-edge filename and includes the core workspace responsibilities as well.

Ownership

Every Rust file under a repository crate src tree has exactly one owner. Inclusive paths use repository-relative prefixes; optional exclude_paths use the same semantics and win within their domain. A file without a domain or with two domains fails the ownership audit. Sources with no executable LLVM lines still need an owner.

Domain Source responsibility
system/os OS facades, process and task services
system/runtime System-side VM lifecycle, scheduling and render integration
system/fs Runtime filesystem, memcard.rs and the complete memcard/ module tree
system/programs Resident system programs
system/core Remaining system sources, excluding the four specialized owners
asset/bank HAL asset/cartridge prefixes, driver asset.rs and the complete driver asset/ module tree
firmware Firmware state and transitions
host-dependent Desktop host
host/headless Headless host and automation implementation
vm/core VM execution, values, heap, verifier, admission, intrinsics and math
bytecode PBX/Registry encodings, ISA, assembler, decoder and binary admission
hal/core HAL contracts and helpers, excluding existing asset/bank ownership
drivers/core Peripheral/render implementations, excluding the asset driver
tooling CLI and dummy/stress PBX producers
dev/test-support Shared test-program construction and development infrastructure source stubs

Existing specialized paths retain their original ownership. Both module entry files and sibling module directories need explicit prefixes. Memcard and asset-driver descendants, including future nested modules, retain their specialized owner even when absent from the coverage report or reporting zero executable lines. The corresponding residual exclusions cover both forms. Residual domains use broad crate source prefixes with explicit exclusions, so a new module inherits the appropriate owner. A new crate or conflicting prefix must be assigned before the ownership audit can pass.

How test suites contribute

VM unit/integration tests, including numeric_contract and the portable MPFR-generated fixture, exercise vm/core. Bytecode round-trip and Registry admission tests exercise bytecode; end-to-end loading can also exercise vm/core. HAL/drivers suites contribute to their corresponding implementation owners. Headless/CLI integration and development cross-layer suites can contribute to several domains in one run.

Quality tests remain in prometeu-quality-checks/tests. They enforce the map and drive the auditor/helper with independent source/report fixtures, including omitted/overlapping owners, exclusions, absolute/relative filenames, absent/empty evidence, invalid statistics and percentage regression. Expected-owner tests use the auditor's ownership function for entry files, every existing descendant and future nested paths. Synthetic reports also check absent and zero-line descendants. Exclusive residual fallback is not sufficient: it can silently transfer specialized code into the wrong gate. The empty quality/layer library stubs inherit infrastructure ownership; they do not create separate gates with zero executable lines.

Only crate src paths participate in this domain denominator; integration test directories, examples, benches, external dependencies and standalone Python oracle code are outside it. The LLVM instrumentation/report remains the existing workspace all-features default-profile measurement. Unit tests colocated in src retain the tool's existing reporting behavior.

The standalone MPFR precision/bit comparison remains complementary numeric qualification. It is not replaced by line coverage and is not newly required by make ci-domains. See numeric qualification.

Evidence and gate rules

Reports may use absolute or repository-relative source paths. The auditor normalizes these names before assigning ownership; evidence display and percentage checks consume its same domain statistics. This prevents path aliases or separate matching implementations from disagreeing about totals. The verbose file lists use normalized repository-relative names.

Each report source within scope must exist in the repository and appear once. Every configured domain must have a positive aggregate instrumented-line count. Missing reports, unknown sources, duplicate records, invalid counts and empty domains fail validation. A zero baseline cannot make missing or empty evidence pass.

The gate compares covered instrumented lines to total instrumented lines, truncating the percentage downward to two decimals. A domain passes when this percentage is at least baseline_percent. Counts for functions and regions remain supporting evidence. Scenario/contract assertions continue to govern behavior; a percentage does not replace them.

Measured coverage and the acceptance floor are separate values. The historical margin revision gave every domain at least 5.00 percentage points of headroom from that recorded local measurement. Its rule was new_floor = min(original_floor, measured_percent - 5.00), using the same downward truncation. Floors already providing that margin remain unchanged. Five percentage points means an 88.56% measurement permits an 83.56% floor; it is not a relative five-percent reduction. That historical margin is not a guarantee about later measurements or corrected ownership. The current map restores the reviewed system/core floor of 87.71% and preserves every other floor.

Floors remain fixed during CI. A failing run does not automatically lower them. An intentional scope or floor revision requires explicit review with fresh measured evidence. New domains also require an explicit floor supported by a fresh measurement.

Commands and CI

Jenkins orchestrates the quality gate through explicit stages invoking Makefile targets. make ci-domains runs the same checks locally and in GitHub CI; make ci is an alias. Invoke one gate per checkout at a time.

The ordered checks are:

  1. Clean previous LLVM coverage instrumentation and reports.
  2. Check formatting.
  3. Run workspace Clippy with warnings denied.
  4. Run workspace tests across all targets and features.
  5. Run the five ignored debugger socket tests.
  6. Generate workspace coverage, HTML, full JSON, Cobertura and summary JSON.
  7. Audit source ownership/evidence and check every fixed coverage domain floor.

The socket command is deliberately restricted:

cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored

It selects test_debug_port_opens, test_debug_reconnection, test_debug_refuse_second_connection, test_get_state_returns_response and test_debug_resume_on_disconnect. It does not select the audio-device test. Local sandbox TCP permissions must permit bind/connect; a permission denial is not evidence of a debugger behavior regression.

The local Make recipe executes child targets serially with -j1, even if the outer invocation uses -j. Jenkins executes its stages sequentially. Do not run a separate Cargo, coverage-clean or report process concurrently in the same checkout: serialized steps inside one gate cannot protect against an independent process.

Prerequisites and reports

Jenkins uses bash scripts/check-quality-gate-prerequisites.sh to validate an already provisioned agent. The job does not install system packages, Rust components or Cargo tools and does not require root or sudo. The agent image must supply a supported Rust toolchain and rustup in CARGO_HOME/bin, rustfmt, Clippy, llvm-tools-preview and pinned cargo-llvm-cov 0.8.5.

Provision the Jenkins agent before running jobs. The standalone bash scripts/provision-quality-gate.sh helper requires root or sudo on Debian/Ubuntu and an existing Rust/rustup installation; run it during agent setup, with Rust tools installed for the job user. GitHub's disposable hosted runner still uses this helper after installing stable Rust.

Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev, libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3, jq, Git, curl and CA certificates. ALSA development files satisfy CPAL's Linux build dependency; a running audio device is not needed. Non-Linux local hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config, Python 3, jq and Git. The provisioning script verifies these commands.

Jenkins owns stage status, console logs, build identity and artifact fingerprints. GitHub retains step logs and uploaded coverage artifacts. Local execution streams command output and stops at the first failed target; no separate Python runner or metadata report is required.

Both pipelines discard old coverage reports before prerequisite checks or setup, and archive current target/llvm-cov/** reports even after failure. Early failures may leave no report; later failures retain partial current reports. Jenkins also publishes HTML when present. Artifact publication does not mask a failed gate.

List domains or inspect ownership and existing evidence:

make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core

Explicit manual qualification

Ordinary all-target tests and coverage leave ignored device/measurement tests ignored. The shared target only opts in to the five socket tests above. It does not certify the following manual workloads:

# Physical desktop audio, exact named test.
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
# Physical callback duration under host I/O pressure.
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
# Eight-hour active release soak.
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
# File-host commit latency with the production asset decoder.
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
# Explicit layout observation.
cargo test --release -p prometeu-bytecode --test registry_v0 reference_map_probe -- --ignored --exact --nocapture

Other ignored timing probes remain opt-in by their individual test names. All-target execution can print ordinary benchmark observations; this does not establish completion of ignored manual qualification. For scope and prior measured results see resource stability and audio callback qualification.

Historical local baseline

The initial measured evidence records configuration, source/report fingerprints, command/toolchain/host, line totals and gate results for all 15 domains before the margin revision. It preserves the eight earlier domain definitions and records these seven additional starting floors on macOS/aarch64:

New domain Baseline lines
vm/core 89.12%
bytecode 93.23%
hal/core 88.56%
drivers/core 84.51%
system/core 92.71%
tooling 64.75%
dev/test-support 85.96%

These are historical starting floors; current acceptance floors appear below.

Historical margin revision

The margin evidence records fresh measurements, original and revised floors, toolchain, fingerprints and executed validation. Only domains with less than 5.00 percentage points of headroom were adjusted. The historical evidence above remains unchanged.

Domain Measured lines Original floor Revised floor Margin (points)
system/os 82.24% 77.24% 77.24% 5.00
system/runtime 81.19% 76.50% 76.19% 5.00
system/fs 81.08% 76.08% 76.08% 5.00
system/programs 98.05% 93.05% 93.05% 5.00
asset/bank 91.76% 86.76% 86.76% 5.00
firmware 88.26% 83.20% 83.20% 5.06
host-dependent 45.05% 40.05% 40.05% 5.00
host/headless 88.39% 83.39% 83.39% 5.00
vm/core 89.29% 89.12% 84.29% 5.00
bytecode 93.26% 93.23% 88.26% 5.00
hal/core 88.56% 88.56% 83.56% 5.00
drivers/core 84.51% 84.51% 79.51% 5.00
system/core 92.71% 92.71% 87.71% 5.00
tooling 64.75% 64.75% 59.75% 5.00
dev/test-support 85.96% 85.96% 80.96% 5.00

The full local make ci-domains validation passed with 939 tests passed and 9 ignored. system/runtime measured 81.26% in the initial run and 81.19% in verification; its final floor uses the lower measurement. All gates were rechecked against the verification report after that final adjustment.

Current coverage measurement

The current evidence identifies the committed source by HEAD and content fingerprints, records the strict shared gate and retains the earlier ownership-repair measurement as historical evidence. Both original historical JSON records above remain unchanged. The latest measurement uses the repaired module-tree map with all 15 fixed floors.

Domain Covered / total lines Measured lines Fixed floor Margin (points)
system/os 1410 / 1711 82.40% 77.24% 5.16
system/runtime 2757 / 3309 83.31% 76.19% 7.12
system/fs 1946 / 2469 78.81% 76.08% 2.73
system/programs 556 / 567 98.05% 93.05% 5.00
asset/bank 3176 / 3434 92.48% 86.76% 5.72
firmware 1990 / 2314 85.99% 83.20% 2.79
host-dependent 1531 / 2393 63.97% 40.05% 23.92
host/headless 739 / 855 86.43% 83.39% 3.04
vm/core 8522 / 9526 89.46% 84.29% 5.17
bytecode 3085 / 3303 93.39% 88.26% 5.13
hal/core 2093 / 2369 88.34% 83.56% 4.78
drivers/core 2775 / 3107 89.31% 79.51% 9.80
system/core 1353 / 1444 93.69% 87.71% 5.98
tooling 396 / 612 64.70% 59.75% 4.95
dev/test-support 99 / 115 86.08% 80.96% 5.12

The fresh macOS/aarch64 make ci-domains run passed all 15 domain gates, 1,060 ordinary tests (13 ignored), five explicit socket tests and three runner regression tests. The fresh coverage replay also passed 1,060 tests; it is not an additional set of unique behavioral tests. All ten focused ownership/evidence tests passed again on the committed source. No floor was lowered. The retained ownership-repair measurement used the earlier non-strict gate; the current measurement passes strict workspace Clippy.

Platform-specific compilation and host scenarios can change report totals. Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate those runs against the configured floors and investigate differences.

Strict shared-gate qualification

On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict workspace/all-target/all-feature Clippy, all 1,060 ordinary tests (13 ignored), the five explicitly selected socket tests, three gate-runner regression tests, and all 15 fixed coverage domain gates. The separate fresh coverage execution also passed 1,060 tests with 13 ignored. Those repeated runs are not distinct behavioral tests and must not be added into a unique-test count.

The baseline strict invocation failed on the concrete VM lint sources above; the repaired sources pass without new lint allowances. Socket tests initially failed because the sandbox denied localhost TCP operations; the authorized outside-sandbox run passed all five. Gate failure fixtures preserve nonzero Make status, logs and commit/config identity, discard stale coverage, and retain fresh partial reports when a later gate fails. Interrupted runners record failure rather than success.

GitHub's YAML was parsed locally, and both pipeline definitions were inspected for shared provisioning, make ci-domains and unconditional failure-artifact publication. No remote GitHub or Jenkins build was launched. No Linux gate ran: the local Docker API was denied by the sandbox and the unrestricted daemon query did not return a usable result before interruption. Local macOS coverage does not certify Linux compilation or domain percentages. Such failures must be investigated on that platform without lowering the existing floors.

No physical audio, eight-hour soak, packaged-installation or ignored manual latency/layout qualification was selected by this gate. Prior manual observations remain separate evidence. Current coverage reports are emitted under target/llvm-cov/; CI retains execution logs and build status.