ci: enforce a shared strict repository quality gate

This commit is contained in:
2026-10-14 12:14:11 +01:00
parent 461b554c1a
commit 418e574ac8
14 changed files with 454 additions and 118 deletions
+131 -24
View File
@@ -119,30 +119,107 @@ fresh measurement.
## Commands and CI
List domains, inspect source ownership and validate coverage evidence:
`make ci-domains` is the authoritative repository quality gate. Both
[GitHub CI](../../.github/workflows/ci.yml) and
[Jenkins](../../files/config/Jenkinsfile) provision dependencies and invoke that
same target. `make ci` is an alias. Invoke one gate per checkout at a time.
make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
python3 scripts/audit-coverage-domains.py --report target/llvm-cov/report.json --json
The Makefile supplies this ordered sequence to the serial evidence runner:
Produce a fresh report and run all gates:
1. Clean LLVM coverage instrumentation from the previous run.
2. Check formatting with `cargo fmt --all -- --check`.
3. Run `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
4. Test gate failure/report handling with the Python regression suite.
5. Run `cargo test --workspace --all-targets --all-features --no-fail-fast`.
6. Run the five ignored debugger socket tests through `make test-debugger-socket`.
7. Generate fresh workspace/all-feature LLVM coverage, HTML, full JSON,
Cobertura XML and summary JSON.
8. Audit source ownership/evidence and check every fixed coverage domain floor.
make ci-domains
The socket command is deliberately restricted:
The existing Jenkins configuration invokes this repository target. Its
coverage-domain-gate step also performs the ownership/evidence audit.
GitHub's workspace-test step discovers the quality regression tests; it
does not separately produce a domain coverage report.
```sh
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
```
Inspect a domain:
It selects `test_debug_port_opens`, `test_debug_reconnection`,
`test_debug_refuse_second_connection`, `test_get_state_returns_response` and
`test_debug_resume_on_disconnect`. It does not select the audio-device test.
Local sandbox TCP permissions must permit bind/connect; a permission denial is
not evidence of a debugger behavior regression.
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core
scripts/coverage-domain-evidence.sh --check-all
The runner executes child Make targets with `-j1` even if the outer invocation
uses `-j`. CI invokes `make ci-domains` serially. Do not run a separate Cargo,
coverage-clean or report process concurrently in the same checkout: serialized
steps inside one gate cannot protect against an independent process.
Dependencies are the existing Rust/LLVM coverage tooling, Bash, Git, jq and rg,
plus Python 3 for the standard-library ownership auditor.
### Prerequisites and reports
Both pipelines use `bash scripts/provision-quality-gate.sh`. A Debian/Ubuntu
agent needs root or sudo for apt provisioning, and an existing Rust/rustup
installation. GitHub installs stable Rust first; Jenkins must supply a supported
Rust toolchain and rustup in CARGO_HOME/bin. The script installs rustfmt, Clippy,
llvm-tools-preview and pinned cargo-llvm-cov 0.8.5 (`--locked`).
Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev,
libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3,
jq, ripgrep, Git, curl and CA certificates. ALSA development files satisfy CPAL's
Linux build dependency; a running audio device is not needed. Non-Linux local
hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config,
Python 3, jq, rg and Git. The provisioning script verifies these commands.
`target/quality-gate/metadata.json` records revision, dirty working tree, host,
Rust/coverage-tool versions, configuration/source fingerprints, ordered step
exit codes and report hashes. Each step has a separate combined-output log.
The runner stops on the first failed Make target and returns its nonzero exit
status; the log preserves underlying Cargo diagnostics. Three regression cases
check failure logs/identity, fresh success evidence and partial-report retention
when a later gate fails.
Both pipelines discard old generated report directories before provisioning and
publish current artifacts even after gate failure. An early failed gate has
metadata/logs but no old coverage report. Failure after generation preserves the
partial current reports. Provisioning failure can leave no report at all and
still fails CI. Reports are archived from `target/quality-gate/**` and
`target/llvm-cov/**`; Jenkins also publishes current HTML when present. Artifact
publication does not turn a failed gate into success.
List domains or inspect ownership and existing evidence:
```sh
make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core
```
### Explicit manual qualification
Ordinary all-target tests and coverage leave ignored device/measurement tests
ignored. The shared target only opts in to the five socket tests above. It does
not certify the following manual workloads:
```sh
# Physical desktop audio, exact named test.
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
# Physical callback duration under host I/O pressure.
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
# Eight-hour active release soak.
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
# File-host commit latency with the production asset decoder.
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
# Explicit layout observation.
cargo test --release -p prometeu-bytecode --test registry_v1 reference_map_probe -- --ignored --exact --nocapture
```
Other ignored timing probes remain opt-in by their individual test names.
All-target execution can print ordinary benchmark observations; this does not
establish completion of ignored manual qualification. For scope and prior measured
results see [resource stability](resource-stability-qualification.md) and
[audio callback qualification](audio-callback-qualification.md).
## Historical local baseline
@@ -194,9 +271,9 @@ The full local `make ci-domains` validation passed with 939 tests passed and
in verification; its final floor uses the lower measurement. All gates were
rechecked against the verification report after that final adjustment.
## Current local measurement
## Ownership-repair measurement
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) records the repaired
The [ownership-repair evidence](COVERAGE_DOMAINS_CURRENT.json) records the repaired
module-tree ownership, fixed floors, normalized source paths, actual line
counts and source/config/report fingerprints. Both historical JSON records
above remain unchanged. The measurement uses the implementation working tree
@@ -225,12 +302,42 @@ with 1,021 tests passed and 11 ignored. The 10 focused ownership/evidence
tests and strict package Clippy also passed. No floor reduction or additional
runtime behavior test was needed to pass the repaired map.
The current Makefile target still uses non-strict workspace Clippy. A separate
`cargo clippy --workspace --all-targets --all-features -- -D warnings` run failed
on existing VM `useless_vec`, `cloned_ref_to_slice_refs` and
`items_after_test_module` diagnostics. This evidence certifies the corrected
coverage map and its gates; full strict workspace qualification remains pending.
At the time of that ownership measurement the target used non-strict workspace
Clippy, and a separate strict invocation failed on VM `useless_vec`,
`cloned_ref_to_slice_refs` and `items_after_test_module` diagnostics. That historical
record qualifies the repaired coverage map, not the later strict gate. The lint
sources have since been corrected without blanket suppression or floor changes.
Platform-specific compilation and host scenarios can change report totals.
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
those runs against the configured floors and investigate differences.
## Strict shared-gate qualification
On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict
workspace/all-target/all-feature Clippy, all 1,039 ordinary tests (13 ignored),
the five explicitly selected socket tests, three gate-runner regression tests,
and all 15 fixed coverage domain gates. The separate fresh coverage execution
also passed 1,039 tests with 13 ignored. Those repeated runs are not distinct
behavioral tests and must not be added into a unique-test count.
The baseline strict invocation failed on the concrete VM lint sources above;
the repaired sources pass without new lint allowances. Socket tests initially
failed because the sandbox denied localhost TCP operations; the authorized
outside-sandbox run passed all five. Gate failure fixtures preserve nonzero
Make status, logs and commit/config identity, discard stale coverage, and retain
fresh partial reports when a later gate fails. Interrupted runners record failure
rather than success.
GitHub's YAML was parsed locally, and both pipeline definitions were inspected
for shared provisioning, `make ci-domains` and unconditional failure-artifact
publication. No remote GitHub or Jenkins build was launched. No Linux gate ran:
the local Docker API was denied by the sandbox and the unrestricted daemon query
did not return a usable result before interruption. Local macOS coverage does not
certify Linux compilation or domain percentages. Such failures must be investigated
on that platform without lowering the existing floors.
No physical audio, eight-hour soak, packaged-installation or ignored manual
latency/layout qualification was selected by this gate. Prior manual observations
remain separate evidence. Current executable metadata and reports are emitted
under `target/quality-gate/` and `target/llvm-cov/` by every invocation.
@@ -29,11 +29,17 @@ cargo test --workspace --features prometeu-host-headless/resource-diagnostics
cargo test --release --workspace --features prometeu-host-headless/resource-diagnostics
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --frames 216000
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
cargo test --release -p prometeu-vm --features resource-diagnostics --lib -- --ignored --nocapture
cargo test --release -p prometeu-vm --features resource-diagnostics --lib sealed_image_load_probe -- --ignored --nocapture
cargo test --release -p prometeu-vm --features resource-diagnostics --lib storage_and_gc_probe -- --ignored --nocapture
cargo test --release -p prometeu-bytecode reference_map_probe -- --ignored --nocapture
cargo test -p prometeu-host-desktop-winit --lib -- --ignored
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
```
The reproduction commands name manual probes individually. The original recorded
desktop qualification grouped ignored tests; current automation selects only the
five debugger socket cases and leaves device audio explicit.
The `resource-diagnostics` feature adds engineering observations through Rust
only. It adds no guest syscall. The eight-hour example is a manual command;
the repository CI, release workflow and Jenkins pipeline do not invoke it. Normal builds omit timing histograms, additional