ci: enforce a shared strict repository quality gate
This commit is contained in:
@@ -119,30 +119,107 @@ fresh measurement.
|
||||
|
||||
## Commands and CI
|
||||
|
||||
List domains, inspect source ownership and validate coverage evidence:
|
||||
`make ci-domains` is the authoritative repository quality gate. Both
|
||||
[GitHub CI](../../.github/workflows/ci.yml) and
|
||||
[Jenkins](../../files/config/Jenkinsfile) provision dependencies and invoke that
|
||||
same target. `make ci` is an alias. Invoke one gate per checkout at a time.
|
||||
|
||||
make coverage-domain-list
|
||||
python3 scripts/audit-coverage-domains.py
|
||||
make coverage-domain-audit
|
||||
python3 scripts/audit-coverage-domains.py --report target/llvm-cov/report.json --json
|
||||
The Makefile supplies this ordered sequence to the serial evidence runner:
|
||||
|
||||
Produce a fresh report and run all gates:
|
||||
1. Clean LLVM coverage instrumentation from the previous run.
|
||||
2. Check formatting with `cargo fmt --all -- --check`.
|
||||
3. Run `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
|
||||
4. Test gate failure/report handling with the Python regression suite.
|
||||
5. Run `cargo test --workspace --all-targets --all-features --no-fail-fast`.
|
||||
6. Run the five ignored debugger socket tests through `make test-debugger-socket`.
|
||||
7. Generate fresh workspace/all-feature LLVM coverage, HTML, full JSON,
|
||||
Cobertura XML and summary JSON.
|
||||
8. Audit source ownership/evidence and check every fixed coverage domain floor.
|
||||
|
||||
make ci-domains
|
||||
The socket command is deliberately restricted:
|
||||
|
||||
The existing Jenkins configuration invokes this repository target. Its
|
||||
coverage-domain-gate step also performs the ownership/evidence audit.
|
||||
GitHub's workspace-test step discovers the quality regression tests; it
|
||||
does not separately produce a domain coverage report.
|
||||
```sh
|
||||
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
|
||||
```
|
||||
|
||||
Inspect a domain:
|
||||
It selects `test_debug_port_opens`, `test_debug_reconnection`,
|
||||
`test_debug_refuse_second_connection`, `test_get_state_returns_response` and
|
||||
`test_debug_resume_on_disconnect`. It does not select the audio-device test.
|
||||
Local sandbox TCP permissions must permit bind/connect; a permission denial is
|
||||
not evidence of a debugger behavior regression.
|
||||
|
||||
make coverage-domain-evidence DOMAIN=vm/core
|
||||
scripts/coverage-domain-evidence.sh --check --verbose hal/core
|
||||
scripts/coverage-domain-evidence.sh --check-all
|
||||
The runner executes child Make targets with `-j1` even if the outer invocation
|
||||
uses `-j`. CI invokes `make ci-domains` serially. Do not run a separate Cargo,
|
||||
coverage-clean or report process concurrently in the same checkout: serialized
|
||||
steps inside one gate cannot protect against an independent process.
|
||||
|
||||
Dependencies are the existing Rust/LLVM coverage tooling, Bash, Git, jq and rg,
|
||||
plus Python 3 for the standard-library ownership auditor.
|
||||
### Prerequisites and reports
|
||||
|
||||
Both pipelines use `bash scripts/provision-quality-gate.sh`. A Debian/Ubuntu
|
||||
agent needs root or sudo for apt provisioning, and an existing Rust/rustup
|
||||
installation. GitHub installs stable Rust first; Jenkins must supply a supported
|
||||
Rust toolchain and rustup in CARGO_HOME/bin. The script installs rustfmt, Clippy,
|
||||
llvm-tools-preview and pinned cargo-llvm-cov 0.8.5 (`--locked`).
|
||||
|
||||
Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev,
|
||||
libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3,
|
||||
jq, ripgrep, Git, curl and CA certificates. ALSA development files satisfy CPAL's
|
||||
Linux build dependency; a running audio device is not needed. Non-Linux local
|
||||
hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config,
|
||||
Python 3, jq, rg and Git. The provisioning script verifies these commands.
|
||||
|
||||
`target/quality-gate/metadata.json` records revision, dirty working tree, host,
|
||||
Rust/coverage-tool versions, configuration/source fingerprints, ordered step
|
||||
exit codes and report hashes. Each step has a separate combined-output log.
|
||||
The runner stops on the first failed Make target and returns its nonzero exit
|
||||
status; the log preserves underlying Cargo diagnostics. Three regression cases
|
||||
check failure logs/identity, fresh success evidence and partial-report retention
|
||||
when a later gate fails.
|
||||
|
||||
Both pipelines discard old generated report directories before provisioning and
|
||||
publish current artifacts even after gate failure. An early failed gate has
|
||||
metadata/logs but no old coverage report. Failure after generation preserves the
|
||||
partial current reports. Provisioning failure can leave no report at all and
|
||||
still fails CI. Reports are archived from `target/quality-gate/**` and
|
||||
`target/llvm-cov/**`; Jenkins also publishes current HTML when present. Artifact
|
||||
publication does not turn a failed gate into success.
|
||||
|
||||
List domains or inspect ownership and existing evidence:
|
||||
|
||||
```sh
|
||||
make coverage-domain-list
|
||||
python3 scripts/audit-coverage-domains.py
|
||||
make coverage-domain-audit
|
||||
make coverage-domain-evidence DOMAIN=vm/core
|
||||
scripts/coverage-domain-evidence.sh --check --verbose hal/core
|
||||
```
|
||||
|
||||
### Explicit manual qualification
|
||||
|
||||
Ordinary all-target tests and coverage leave ignored device/measurement tests
|
||||
ignored. The shared target only opts in to the five socket tests above. It does
|
||||
not certify the following manual workloads:
|
||||
|
||||
```sh
|
||||
# Physical desktop audio, exact named test.
|
||||
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
|
||||
# Physical callback duration under host I/O pressure.
|
||||
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
|
||||
# Eight-hour active release soak.
|
||||
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
|
||||
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
|
||||
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
|
||||
# File-host commit latency with the production asset decoder.
|
||||
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
|
||||
# Explicit layout observation.
|
||||
cargo test --release -p prometeu-bytecode --test registry_v1 reference_map_probe -- --ignored --exact --nocapture
|
||||
```
|
||||
|
||||
Other ignored timing probes remain opt-in by their individual test names.
|
||||
All-target execution can print ordinary benchmark observations; this does not
|
||||
establish completion of ignored manual qualification. For scope and prior measured
|
||||
results see [resource stability](resource-stability-qualification.md) and
|
||||
[audio callback qualification](audio-callback-qualification.md).
|
||||
|
||||
## Historical local baseline
|
||||
|
||||
@@ -194,9 +271,9 @@ The full local `make ci-domains` validation passed with 939 tests passed and
|
||||
in verification; its final floor uses the lower measurement. All gates were
|
||||
rechecked against the verification report after that final adjustment.
|
||||
|
||||
## Current local measurement
|
||||
## Ownership-repair measurement
|
||||
|
||||
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) records the repaired
|
||||
The [ownership-repair evidence](COVERAGE_DOMAINS_CURRENT.json) records the repaired
|
||||
module-tree ownership, fixed floors, normalized source paths, actual line
|
||||
counts and source/config/report fingerprints. Both historical JSON records
|
||||
above remain unchanged. The measurement uses the implementation working tree
|
||||
@@ -225,12 +302,42 @@ with 1,021 tests passed and 11 ignored. The 10 focused ownership/evidence
|
||||
tests and strict package Clippy also passed. No floor reduction or additional
|
||||
runtime behavior test was needed to pass the repaired map.
|
||||
|
||||
The current Makefile target still uses non-strict workspace Clippy. A separate
|
||||
`cargo clippy --workspace --all-targets --all-features -- -D warnings` run failed
|
||||
on existing VM `useless_vec`, `cloned_ref_to_slice_refs` and
|
||||
`items_after_test_module` diagnostics. This evidence certifies the corrected
|
||||
coverage map and its gates; full strict workspace qualification remains pending.
|
||||
At the time of that ownership measurement the target used non-strict workspace
|
||||
Clippy, and a separate strict invocation failed on VM `useless_vec`,
|
||||
`cloned_ref_to_slice_refs` and `items_after_test_module` diagnostics. That historical
|
||||
record qualifies the repaired coverage map, not the later strict gate. The lint
|
||||
sources have since been corrected without blanket suppression or floor changes.
|
||||
|
||||
Platform-specific compilation and host scenarios can change report totals.
|
||||
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
|
||||
those runs against the configured floors and investigate differences.
|
||||
|
||||
## Strict shared-gate qualification
|
||||
|
||||
On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict
|
||||
workspace/all-target/all-feature Clippy, all 1,039 ordinary tests (13 ignored),
|
||||
the five explicitly selected socket tests, three gate-runner regression tests,
|
||||
and all 15 fixed coverage domain gates. The separate fresh coverage execution
|
||||
also passed 1,039 tests with 13 ignored. Those repeated runs are not distinct
|
||||
behavioral tests and must not be added into a unique-test count.
|
||||
|
||||
The baseline strict invocation failed on the concrete VM lint sources above;
|
||||
the repaired sources pass without new lint allowances. Socket tests initially
|
||||
failed because the sandbox denied localhost TCP operations; the authorized
|
||||
outside-sandbox run passed all five. Gate failure fixtures preserve nonzero
|
||||
Make status, logs and commit/config identity, discard stale coverage, and retain
|
||||
fresh partial reports when a later gate fails. Interrupted runners record failure
|
||||
rather than success.
|
||||
|
||||
GitHub's YAML was parsed locally, and both pipeline definitions were inspected
|
||||
for shared provisioning, `make ci-domains` and unconditional failure-artifact
|
||||
publication. No remote GitHub or Jenkins build was launched. No Linux gate ran:
|
||||
the local Docker API was denied by the sandbox and the unrestricted daemon query
|
||||
did not return a usable result before interruption. Local macOS coverage does not
|
||||
certify Linux compilation or domain percentages. Such failures must be investigated
|
||||
on that platform without lowering the existing floors.
|
||||
|
||||
No physical audio, eight-hour soak, packaged-installation or ignored manual
|
||||
latency/layout qualification was selected by this gate. Prior manual observations
|
||||
remain separate evidence. Current executable metadata and reports are emitted
|
||||
under `target/quality-gate/` and `target/llvm-cov/` by every invocation.
|
||||
|
||||
@@ -29,11 +29,17 @@ cargo test --workspace --features prometeu-host-headless/resource-diagnostics
|
||||
cargo test --release --workspace --features prometeu-host-headless/resource-diagnostics
|
||||
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --frames 216000
|
||||
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
|
||||
cargo test --release -p prometeu-vm --features resource-diagnostics --lib -- --ignored --nocapture
|
||||
cargo test --release -p prometeu-vm --features resource-diagnostics --lib sealed_image_load_probe -- --ignored --nocapture
|
||||
cargo test --release -p prometeu-vm --features resource-diagnostics --lib storage_and_gc_probe -- --ignored --nocapture
|
||||
cargo test --release -p prometeu-bytecode reference_map_probe -- --ignored --nocapture
|
||||
cargo test -p prometeu-host-desktop-winit --lib -- --ignored
|
||||
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
|
||||
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
|
||||
```
|
||||
|
||||
The reproduction commands name manual probes individually. The original recorded
|
||||
desktop qualification grouped ignored tests; current automation selects only the
|
||||
five debugger socket cases and leaves device audio explicit.
|
||||
|
||||
The `resource-diagnostics` feature adds engineering observations through Rust
|
||||
only. It adds no guest syscall. The eight-hour example is a manual command;
|
||||
the repository CI, release workflow and Jenkins pipeline do not invoke it. Normal builds omit timing histograms, additional
|
||||
|
||||
Reference in New Issue
Block a user