9 Commits
60 changed files with 5804 additions and 535 deletions
+17 -7
View File
@@ -15,15 +15,25 @@ jobs:
- uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
components: rustfmt, clippy, llvm-tools-preview
- uses: Swatinem/rust-cache@v2
- name: Format check
run: cargo fmt -- --check
- name: Discard reports from previous runs
run: rm -rf target/quality-gate target/llvm-cov
- name: Clippy
run: cargo clippy --workspace --all-features
- name: Provision quality-gate tools and native dependencies
run: bash scripts/provision-quality-gate.sh
- name: Test
run: cargo test --workspace --all-targets --all-features --no-fail-fast
- name: Repository quality gate
run: make ci-domains
- name: Publish gate evidence (including failures)
if: always()
uses: actions/upload-artifact@v4
with:
name: quality-gate-${{ runner.os }}-${{ github.sha }}
path: |
target/quality-gate/**
target/llvm-cov/**
if-no-files-found: warn
+15 -7
View File
@@ -1,19 +1,19 @@
.PHONY: fmt fmt-check clippy test-local test-debugger-socket clean coverage coverage-xml coverage-json coverage-report-json coverage-domain-list coverage-domain-evidence coverage-domain-audit coverage-domain-gate test ci ci-domains cobertura
.PHONY: fmt fmt-check clippy test-local test-debugger-socket clean coverage coverage-xml coverage-json coverage-report-json coverage-domain-list coverage-domain-evidence coverage-domain-audit coverage-domain-gate test ci ci-domains test-quality-gate cobertura
fmt:
cargo fmt
cargo fmt --all
fmt-check:
cargo fmt -- --check
cargo fmt --all -- --check
clippy:
cargo clippy --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
test-local:
cargo test --workspace --all-targets --all-features --no-fail-fast
test-debugger-socket:
cargo test -p prometeu-host-desktop-winit --lib -- --ignored
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
clean:
cargo llvm-cov clean --workspace
@@ -47,6 +47,14 @@ coverage-domain-gate:
./scripts/coverage-domain-evidence.sh --check-all
test: fmt-check clippy test-local test-debugger-socket
ci: clean fmt-check clippy coverage coverage-report-json
ci-domains: clean fmt-check clippy coverage coverage-report-json coverage-xml coverage-json coverage-domain-gate
# Keep the step list here: CI providers consume the repository contract.
# The runner executes each step serially even if the caller supplies make -j.
CI_DOMAIN_STEPS = clean fmt-check clippy test-quality-gate test-local test-debugger-socket coverage coverage-report-json coverage-xml coverage-json coverage-domain-audit coverage-domain-gate
ci: ci-domains
ci-domains:
python3 scripts/run-quality-gate.py $(CI_DOMAIN_STEPS)
test-quality-gate:
python3 -B -m unittest discover -s scripts/tests -p 'test_quality_gate.py'
cobertura: coverage-xml coverage-json
+14 -19
View File
@@ -107,19 +107,19 @@ Format, lint, and test the workspace:
make test
```
Run the CI-oriented non-domain gate:
Run the shared strict CI gate (`make ci` is an alias):
```bash
make ci
```
Run the CI-oriented domain coverage gate:
The authoritative target used by both GitHub and Jenkins is:
```bash
make ci-domains
```
The debugger socket tests bind localhost sockets and are marked ignored in Rust test metadata; `make test` includes them through the dedicated `test-debugger-socket` target.
The five debugger socket tests bind localhost sockets and are marked ignored in Rust test metadata; both `make test` and `make ci-domains` select only `runner::tests` through `test-debugger-socket`. Physical audio and other manual probes remain opt-in. See [quality-gate prerequisites and commands](docs/vm-arch/COVERAGE_DOMAINS.md#commands-and-ci).
## Coverage Governance
@@ -128,20 +128,14 @@ The repository uses a two-layer runtime-edge coverage model:
- a mandatory CI gate based on canonical runtime-edge domains;
- domain-oriented review evidence for the canonical runtime-edge domains.
Current canonical domains:
- `system/runtime`
- `fs`
- `asset/bank`
- `firmware`
- `host-dependent`
- `host/headless`
The fifteen canonical domains and fixed floors are defined in
[the repository configuration](files/config/runtime-edge-coverage-domains.json).
Use `make coverage-domain-list` or the [ownership documentation](docs/vm-arch/COVERAGE_DOMAINS.md)
to inspect the current map.
Global coverage artifacts:
```bash
make ci
make cobertura
make ci-domains
```
@@ -151,6 +145,7 @@ This produces:
- `target/llvm-cov/report.json`
- `target/llvm-cov/summary.json`
- `target/llvm-cov/cobertura.xml`
- `target/quality-gate/metadata.json` and per-step logs
Domain-oriented evidence helpers:
@@ -161,12 +156,12 @@ make coverage-domain-evidence DOMAIN=host-dependent
make coverage-domain-evidence DOMAIN=host/headless
```
Per-domain baselines currently start around `60`.
That does not waive test obligations. It means the project is adopting explicit domain evidence immediately while tightening quantitative expectations incrementally over time.
`make ci-domains` is the CI entrypoint used by Jenkins.
It runs the standard coverage pipeline, produces the HTML/XML/JSON artifacts, and fails when any canonical domain falls below its configured `baseline_percent`.
The CI gate currently compares domain `lines` coverage against each domain baseline.
`make ci-domains` runs formatting, strict workspace/all-target/all-feature Clippy,
all-target tests, the five debugger socket tests, fresh coverage and every domain
ownership/evidence gate. Both CI providers archive available evidence on failure.
A failing run never lowers a configured floor; local and CI checks use the same
repository-owned commands. Reports identify their revision, source/configuration
and actual execution platform.
When a PR changes the observable contract of a canonical domain, it is expected to:
+26 -226
View File
@@ -1,4 +1,5 @@
#![allow(clippy::collapsible_if)]
mod decode;
mod resumable;
use crate::memory_banks::{GlyphBankPoolInstaller, SceneBankPoolInstaller, SoundBankPoolInstaller};
use prometeu_hal::AssetBridge;
@@ -634,62 +635,13 @@ impl AssetManager {
fn decode_glyph_bank_layout(
entry: &AssetEntry,
) -> Result<(TileSize, usize, usize, usize, usize, usize), String> {
let meta = entry.metadata_as_glyph_bank()?;
let tile_size = match meta.tile_size {
8 => TileSize::Size8,
16 => TileSize::Size16,
32 => TileSize::Size32,
_ => return Err(format!("Invalid tile_size: {}", meta.tile_size)),
};
let palette_count = meta.palette_count as usize;
if !(1..=GLYPH_BANK_MAX_PALETTE_COUNT_V1).contains(&palette_count) {
return Err(format!("Invalid palette_count: {}", meta.palette_count));
}
let width = meta.width as usize;
let height = meta.height as usize;
let logical_pixels = width.checked_mul(height).ok_or("GlyphBank dimensions overflow")?;
let serialized_pixel_bytes = logical_pixels.div_ceil(2);
let palette_bytes = palette_count
.checked_mul(GLYPH_BANK_PALETTE_BYTES_PER_PALETTE)
.ok_or("GlyphBank palette size overflow")?;
let serialized_size = serialized_pixel_bytes
.checked_add(palette_bytes)
.ok_or("GlyphBank serialized size overflow")?;
let decoded_size =
logical_pixels.checked_add(palette_bytes).ok_or("GlyphBank decoded size overflow")?;
if entry.size != serialized_size as u64 {
return Err(format!(
"Invalid GLYPHBANK serialized size: expected {}, got {}",
serialized_size, entry.size
));
}
if entry.decoded_size != decoded_size as u64 {
return Err(format!(
"Invalid GLYPHBANK decoded_size: expected {}, got {}",
decoded_size, entry.decoded_size
));
}
Ok((tile_size, width, height, serialized_pixel_bytes, palette_count, palette_bytes))
decode::glyph_layout(entry)
}
fn unpack_glyph_bank_pixels(packed_pixels: &[u8], logical_pixels: usize) -> Vec<u8> {
let mut pixel_indices = Vec::with_capacity(logical_pixels);
for &packed in packed_pixels {
if pixel_indices.len() < logical_pixels {
pixel_indices.push(packed >> 4);
}
if pixel_indices.len() < logical_pixels {
pixel_indices.push(packed & 0x0f);
}
}
pixel_indices
let mut pixels = Vec::with_capacity(logical_pixels);
decode::append_pixels(&mut pixels, packed_pixels, logical_pixels);
pixels
}
fn op_mode_for(entry: &AssetEntry) -> Result<AssetOpMode, String> {
@@ -1260,17 +1212,7 @@ impl AssetManager {
let palette_data = &buffer[packed_pixel_bytes..packed_pixel_bytes + palette_bytes];
let mut palettes = vec![[Color::BLACK; GLYPH_BANK_COLORS_PER_PALETTE]; palette_count];
for (p, pal) in palettes.iter_mut().enumerate() {
for (c, slot) in pal.iter_mut().enumerate() {
let offset = (p * 16 + c) * 4;
*slot = Color::rgba(
palette_data[offset],
palette_data[offset + 1],
palette_data[offset + 2],
palette_data[offset + 3],
);
}
}
decode::write_palettes(&mut palettes, 0, palette_data);
Ok(GlyphBank { tile_size, width, height, pixel_indices, palettes })
}
@@ -1295,17 +1237,7 @@ impl AssetManager {
.map_err(|_| "Buffer too small for GLYPHBANK".to_string())?;
let mut palettes = vec![[Color::BLACK; GLYPH_BANK_COLORS_PER_PALETTE]; palette_count];
for (p, pal) in palettes.iter_mut().enumerate() {
for (c, slot) in pal.iter_mut().enumerate() {
let offset = (p * 16 + c) * 4;
*slot = Color::rgba(
palette_data[offset],
palette_data[offset + 1],
palette_data[offset + 2],
palette_data[offset + 3],
);
}
}
decode::write_palettes(&mut palettes, 0, &palette_data);
Ok(GlyphBank { tile_size, width, height, pixel_indices, palettes })
}
@@ -1340,7 +1272,7 @@ impl AssetManager {
entry: &AssetEntry,
assets_data: Arc<RwLock<AssetsPayloadSource>>,
) -> Result<SceneBank, String> {
let _ = entry.metadata_as_scene_bank()?;
decode::scene_metadata(entry)?;
let op_mode = Self::op_mode_for(entry)?;
let slice = {
let assets_data = assets_data.read().unwrap();
@@ -1367,152 +1299,25 @@ impl AssetManager {
entry: &AssetEntry,
buffer: &[u8],
) -> Result<SceneBank, String> {
let _ = entry.metadata_as_scene_bank()?;
if buffer.len() < SCENE_HEADER_BYTES_V1 {
return Err("Buffer too small for SCENE".to_string());
}
if buffer[0..4] != SCENE_PAYLOAD_MAGIC_V1 {
return Err("Invalid SCENE magic".to_string());
}
let version = u16::from_le_bytes([buffer[4], buffer[5]]);
if version != SCENE_PAYLOAD_VERSION_V1 {
return Err(format!("Unsupported SCENE version: {}", version));
}
let layer_count = u16::from_le_bytes([buffer[6], buffer[7]]) as usize;
if layer_count != SCENE_LAYER_COUNT_V1 {
return Err(format!("Invalid SCENE layer count: {}", layer_count));
}
decode::scene_metadata(entry)?;
decode::payload_size(entry, buffer.len())?;
decode::scene_header(buffer)?;
let mut offset = SCENE_HEADER_BYTES_V1;
let mut decoded_size = 0_usize;
let layers = std::array::from_fn(|_| SceneLayer {
active: false,
glyph_asset_id: 0,
tile_size: TileSize::Size8,
parallax_factor: ParallaxFactor { x: 1.0, y: 1.0 },
tilemap: TileMap { width: 0, height: 0, tiles: Vec::new() },
});
let mut layers = layers;
let mut decoded = 0;
let mut layers = decode::empty_layers();
for layer in &mut layers {
let header_end = offset
let end = offset
.checked_add(SCENE_LAYER_HEADER_BYTES_V1)
.ok_or("SCENE layer header offset overflow")?;
if header_end > buffer.len() {
return Err("Buffer too small for SCENE layer header".to_string());
}
let flags = buffer[offset];
let glyph_asset_id = i32::from_le_bytes([
buffer[offset + 1],
buffer[offset + 2],
buffer[offset + 3],
buffer[offset + 4],
]);
let tile_size_raw = buffer[offset + 5];
let tile_size = match tile_size_raw {
8 => TileSize::Size8,
16 => TileSize::Size16,
32 => TileSize::Size32,
other => return Err(format!("Invalid SCENE tile size: {}", other)),
};
let parallax_factor_x = f32::from_le_bytes([
buffer[offset + 8],
buffer[offset + 9],
buffer[offset + 10],
buffer[offset + 11],
]);
let parallax_factor_y = f32::from_le_bytes([
buffer[offset + 12],
buffer[offset + 13],
buffer[offset + 14],
buffer[offset + 15],
]);
if !parallax_factor_x.is_finite() || !parallax_factor_y.is_finite() {
return Err("Invalid SCENE parallax_factor".to_string());
}
let width = u32::from_le_bytes([
buffer[offset + 16],
buffer[offset + 17],
buffer[offset + 18],
buffer[offset + 19],
]) as usize;
let height = u32::from_le_bytes([
buffer[offset + 20],
buffer[offset + 21],
buffer[offset + 22],
buffer[offset + 23],
]) as usize;
let tile_count = u32::from_le_bytes([
buffer[offset + 24],
buffer[offset + 25],
buffer[offset + 26],
buffer[offset + 27],
]) as usize;
let expected_tile_count =
width.checked_mul(height).ok_or("SCENE tile count overflow")?;
if tile_count != expected_tile_count {
return Err(format!(
"Invalid SCENE tile count for layer: expected {}, got {}",
expected_tile_count, tile_count
));
}
offset = header_end;
let tile_bytes = tile_count
.checked_mul(SCENE_TILE_RECORD_BYTES_V1)
.ok_or("SCENE tile payload overflow")?;
let tiles_end = offset.checked_add(tile_bytes).ok_or("SCENE payload overflow")?;
if tiles_end > buffer.len() {
return Err("Buffer too small for SCENE tile data".to_string());
}
let mut tiles = Vec::with_capacity(tile_count);
for _ in 0..tile_count {
let tile_flags = buffer[offset];
let palette_id = buffer[offset + 1];
let glyph_id = u16::from_le_bytes([buffer[offset + 2], buffer[offset + 3]]);
tiles.push(Tile {
active: (tile_flags & 0b0000_0001) != 0,
glyph: Glyph { glyph_id, palette_id },
flip_x: (tile_flags & 0b0000_0010) != 0,
flip_y: (tile_flags & 0b0000_0100) != 0,
});
offset += SCENE_TILE_RECORD_BYTES_V1;
}
decoded_size = decoded_size
.checked_add(SCENE_DECODED_LAYER_OVERHEAD_BYTES_V1)
.and_then(|size| size.checked_add(tile_count * size_of::<Tile>()))
.ok_or("SCENE decoded_size overflow")?;
*layer = SceneLayer {
active: (flags & 0b0000_0001) != 0,
glyph_asset_id,
tile_size,
parallax_factor: ParallaxFactor { x: parallax_factor_x, y: parallax_factor_y },
tilemap: TileMap { width, height, tiles },
};
.ok_or("SCENE header offset overflow")?;
let header =
buffer.get(offset..end).ok_or("Buffer too small for SCENE layer header")?;
*layer = decode::scene_layer(header, end, entry.size, &mut decoded)?;
let tile_bytes =
layer.tilemap.width * layer.tilemap.height * SCENE_TILE_RECORD_BYTES_V1;
offset = end + tile_bytes; // checked by scene_layer before allocation
decode::append_tiles(&mut layer.tilemap.tiles, &buffer[end..offset]);
}
if offset != buffer.len() {
return Err("Trailing bytes in SCENE payload".to_string());
}
if entry.decoded_size != decoded_size as u64 {
return Err(format!(
"Invalid SCENE decoded_size: expected {}, got {}",
decoded_size, entry.decoded_size
));
}
decode::scene_finish(entry, offset, decoded)?;
Ok(SceneBank { layers })
}
@@ -1529,15 +1334,10 @@ impl AssetManager {
entry: &AssetEntry,
buffer: &[u8],
) -> Result<SoundBank, String> {
let meta = entry.metadata_as_sound_bank()?;
let sample_rate = meta.sample_rate;
let sample_rate = decode::sound_rate(entry)?;
decode::payload_size(entry, buffer.len())?;
let mut data = Vec::with_capacity(buffer.len() / 2);
for i in (0..buffer.len()).step_by(2) {
if i + 1 < buffer.len() {
data.push(i16::from_le_bytes([buffer[i], buffer[i + 1]]));
}
}
decode::append_pcm(&mut data, buffer);
let sample = Arc::new(Sample::new(sample_rate, data));
Ok(SoundBank::new(vec![sample]))
@@ -0,0 +1,187 @@
//! Format authority shared by complete-buffer and bounded resumable decoding.
use super::*;
pub(super) fn glyph_layout(
entry: &AssetEntry,
) -> Result<(TileSize, usize, usize, usize, usize, usize), String> {
let meta = entry.metadata_as_glyph_bank()?;
let tile_size = match meta.tile_size {
8 => TileSize::Size8,
16 => TileSize::Size16,
32 => TileSize::Size32,
_ => return Err(format!("Invalid tile_size: {}", meta.tile_size)),
};
let palette_count = meta.palette_count as usize;
if !(1..=GLYPH_BANK_MAX_PALETTE_COUNT_V1).contains(&palette_count) {
return Err(format!("Invalid palette_count: {}", meta.palette_count));
}
let width = meta.width as usize;
let height = meta.height as usize;
let logical_pixels = width.checked_mul(height).ok_or("GlyphBank dimensions overflow")?;
let serialized_pixel_bytes = logical_pixels.div_ceil(2);
let palette_bytes = palette_count
.checked_mul(GLYPH_BANK_PALETTE_BYTES_PER_PALETTE)
.ok_or("GlyphBank palette size overflow")?;
let serialized_size = serialized_pixel_bytes
.checked_add(palette_bytes)
.ok_or("GlyphBank serialized size overflow")?;
let decoded_size =
logical_pixels.checked_add(palette_bytes).ok_or("GlyphBank decoded size overflow")?;
if entry.size != serialized_size as u64 {
return Err(format!(
"Invalid GLYPHBANK serialized size: expected {}, got {}",
serialized_size, entry.size
));
}
if entry.decoded_size != decoded_size as u64 {
return Err(format!(
"Invalid GLYPHBANK decoded_size: expected {}, got {}",
decoded_size, entry.decoded_size
));
}
Ok((tile_size, width, height, serialized_pixel_bytes, palette_count, palette_bytes))
}
pub(super) fn append_pixels(pixels: &mut Vec<u8>, bytes: &[u8], logical: usize) {
for &packed in bytes {
if pixels.len() < logical {
pixels.push(packed >> 4);
}
if pixels.len() < logical {
pixels.push(packed & 15);
}
}
}
pub(super) fn write_palettes(
palettes: &mut [[Color; GLYPH_BANK_COLORS_PER_PALETTE]],
position: usize,
bytes: &[u8],
) {
for (i, rgba) in bytes.as_chunks::<4>().0.iter().enumerate() {
let at = position / 4 + i;
palettes[at / 16][at % 16] = Color::rgba(rgba[0], rgba[1], rgba[2], rgba[3]);
}
}
pub(super) fn sound_rate(entry: &AssetEntry) -> Result<u32, String> {
Ok(entry.metadata_as_sound_bank()?.sample_rate)
}
// Callers split on even chunk boundaries. Only the final odd byte is ignored.
pub(super) fn append_pcm(data: &mut Vec<i16>, bytes: &[u8]) {
data.extend(bytes.as_chunks::<2>().0.iter().map(|p| i16::from_le_bytes(*p)));
}
pub(super) fn payload_size(entry: &AssetEntry, actual: usize) -> Result<(), String> {
if actual as u64 != entry.size {
return Err("Invalid asset payload size".into());
}
Ok(())
}
pub(super) fn scene_metadata(entry: &AssetEntry) -> Result<(), String> {
entry.metadata_as_scene_bank().map(|_| ())
}
pub(super) fn empty_layers() -> [SceneLayer; SCENE_LAYER_COUNT_V1] {
std::array::from_fn(|_| SceneLayer {
active: false,
glyph_asset_id: 0,
tile_size: TileSize::Size8,
parallax_factor: ParallaxFactor { x: 1.0, y: 1.0 },
tilemap: TileMap { width: 0, height: 0, tiles: Vec::new() },
})
}
pub(super) fn scene_header(bytes: &[u8]) -> Result<(), String> {
if bytes.len() < SCENE_HEADER_BYTES_V1 {
return Err("Buffer too small for SCENE".into());
}
if bytes[..4] != SCENE_PAYLOAD_MAGIC_V1 {
return Err("Invalid SCENE magic".into());
}
let version = u16::from_le_bytes([bytes[4], bytes[5]]);
if version != SCENE_PAYLOAD_VERSION_V1 {
return Err(format!("Unsupported SCENE version: {version}"));
}
let count = u16::from_le_bytes([bytes[6], bytes[7]]) as usize;
if count != SCENE_LAYER_COUNT_V1 {
return Err(format!("Invalid SCENE layer count: {count}"));
}
Ok(())
}
pub(super) fn scene_layer(
bytes: &[u8],
position: usize,
payload_len: u64,
decoded: &mut usize,
) -> Result<SceneLayer, String> {
if bytes.len() < SCENE_LAYER_HEADER_BYTES_V1 {
return Err("Buffer too small for SCENE layer header".into());
}
let u32at = |at| u32::from_le_bytes(bytes[at..at + 4].try_into().unwrap());
let tile_size = match bytes[5] {
8 => TileSize::Size8,
16 => TileSize::Size16,
32 => TileSize::Size32,
other => return Err(format!("Invalid SCENE tile size: {other}")),
};
let x = f32::from_le_bytes(bytes[8..12].try_into().unwrap());
let y = f32::from_le_bytes(bytes[12..16].try_into().unwrap());
if !x.is_finite() || !y.is_finite() {
return Err("Invalid SCENE parallax_factor".into());
}
let width = u32at(16) as usize;
let height = u32at(20) as usize;
let count = u32at(24) as usize;
let expected = width.checked_mul(height).ok_or("SCENE tile count overflow")?;
if count != expected {
return Err(format!(
"Invalid SCENE tile count for layer: expected {expected}, got {count}"
));
}
let end = count
.checked_mul(SCENE_TILE_RECORD_BYTES_V1)
.and_then(|n| position.checked_add(n))
.ok_or("SCENE payload overflow")?;
if end as u64 > payload_len {
return Err("Buffer too small for SCENE tile data".into());
}
*decoded = decoded
.checked_add(SCENE_DECODED_LAYER_OVERHEAD_BYTES_V1)
.and_then(|n| count.checked_mul(size_of::<Tile>()).and_then(|b| n.checked_add(b)))
.ok_or("SCENE decoded_size overflow")?;
Ok(SceneLayer {
active: bytes[0] & 1 != 0,
glyph_asset_id: i32::from_le_bytes(bytes[1..5].try_into().unwrap()),
tile_size,
parallax_factor: ParallaxFactor { x, y },
tilemap: TileMap { width, height, tiles: Vec::with_capacity(count) },
})
}
pub(super) fn append_tiles(tiles: &mut Vec<Tile>, bytes: &[u8]) {
tiles.extend(bytes.as_chunks::<SCENE_TILE_RECORD_BYTES_V1>().0.iter().map(|b| Tile {
active: b[0] & 1 != 0,
glyph: Glyph { glyph_id: u16::from_le_bytes([b[2], b[3]]), palette_id: b[1] },
flip_x: b[0] & 2 != 0,
flip_y: b[0] & 4 != 0,
}));
}
pub(super) fn scene_finish(
entry: &AssetEntry,
position: usize,
decoded: usize,
) -> Result<(), String> {
payload_size(entry, position)?;
if decoded as u64 != entry.decoded_size {
return Err(format!(
"Invalid SCENE decoded_size: expected {decoded}, got {}",
entry.decoded_size
));
}
Ok(())
}
#[cfg(test)]
pub(super) mod tests;
@@ -0,0 +1,432 @@
use super::*;
use serde_json::{Value, json};
struct FileFixture(std::path::PathBuf);
impl FileFixture {
fn new(name: &str, bytes: &[u8]) -> Self {
let path = std::env::temp_dir()
.join(format!("prometeu-conformance-{}-{name}", std::process::id()));
std::fs::write(&path, bytes).unwrap();
Self(path)
}
fn source(&self, declared_len: u64) -> Arc<RwLock<AssetsPayloadSource>> {
Arc::new(RwLock::new(AssetsPayloadSource::from_file(self.0.clone(), 0, declared_len)))
}
}
impl Drop for FileFixture {
fn drop(&mut self) {
std::fs::remove_file(&self.0).unwrap();
}
}
fn pcm_entry(size: u64) -> AssetEntry {
AssetEntry {
asset_id: 1,
asset_name: "pcm".into(),
bank_type: BankType::SOUNDS,
offset: 0,
size,
decoded_size: size,
codec: AssetCodec::None,
metadata: json!({"sample_rate": 44100, "channels": 1}),
}
}
#[test]
fn pcm_chunk_edges_and_odd_tails_have_explicit_baseline_outputs() {
for len in [0, 1, 2, 3, 4095, 4096, 4097, 8191, 8192, 8193] {
let bytes: Vec<u8> = (0..len).map(|i| if i % 2 == 0 { 0x34 } else { 0x12 }).collect();
let entry = pcm_entry(len as u64);
let expected = json!([[vec![0x1234_i16; len / 2], 44100, null, null]]);
let buffer = AssetManager::decode_sound_bank_from_buffer(&entry, &bytes).unwrap();
let reader = AssetManager::decode_sound_bank_from_reader(&entry, &mut &bytes[..]).unwrap();
let source = Arc::new(RwLock::new(AssetsPayloadSource::from_bytes(bytes)));
let production = AssetManager::perform_load_sound_bank(&entry, source.clone()).unwrap();
assert_eq!(sound_snapshot(&buffer), expected, "buffer len={len}");
assert_eq!(sound_snapshot(&reader), expected, "reader len={len}");
assert_eq!(sound_snapshot(&production), expected, "legacy load len={len}");
assert_eq!(
resumable::conformance_result(&entry, &source).unwrap(),
expected,
"resumable len={len}"
);
}
}
#[test]
fn physically_truncated_pcm_file_is_rejected_by_both_modes() {
// Two actual bytes, three declared bytes: the missing byte is an odd tail.
// The approved rule rejects missing declared bytes, including an odd tail.
let fixture = FileFixture::new("truncated-pcm", &[0x34, 0x12]);
let entry = pcm_entry(3);
let source = fixture.source(3);
assert!(AssetManager::perform_load_sound_bank(&entry, source.clone()).is_err());
assert!(resumable::conformance_result(&entry, &source).is_err());
// With the declared byte physically present, both accept and ignore it.
std::fs::write(&fixture.0, [0x34, 0x12, 0xff]).unwrap();
let expected = json!([[[0x1234], 44100, null, null]]);
assert_eq!(resumable::conformance_result(&entry, &source).unwrap(), expected);
assert_eq!(
sound_snapshot(&AssetManager::perform_load_sound_bank(&entry, source.clone()).unwrap()),
expected
);
// Memory carriers reject the range before either decoder can accept it.
let memory = Arc::new(RwLock::new(AssetsPayloadSource::from_bytes(vec![0x34, 0x12])));
assert!(AssetManager::perform_load_sound_bank(&entry, memory.clone()).is_err());
assert!(resumable::conformance_result(&entry, &memory).is_err());
}
#[test]
fn physically_short_scene_file_is_rejected_by_both_modes() {
// Minimal scene: four empty layers, 12-byte header + 4 * 32-byte headers.
let mut bytes = vec![0_u8; 140];
bytes[..8].copy_from_slice(&[b'S', b'C', b'N', b'E', 1, 0, 4, 0]);
for start in [12, 44, 76, 108] {
bytes[start + 5] = 8;
}
let fixture = FileFixture::new("short-scene", &bytes);
let mut entry = AssetEntry {
asset_id: 2,
asset_name: "scene".into(),
bank_type: BankType::SCENE,
offset: 0,
size: 141,
decoded_size: 64,
codec: AssetCodec::None,
metadata: json!({}),
};
let expected = json!(vec![json!([false, 0, 8, 0.0, 0.0, 0, 0, []]); 4]);
let source = fixture.source(141);
assert!(AssetManager::perform_load_scene_bank(&entry, source.clone()).is_err());
assert!(resumable::conformance_result(&entry, &source).is_err());
// Exact physical/declared size is accepted by both.
entry.size = 140;
assert_eq!(resumable::conformance_result(&entry, &source).unwrap(), expected);
assert_eq!(
scene_snapshot(&AssetManager::perform_load_scene_bank(&entry, source.clone()).unwrap()),
expected
);
// An actual trailing byte is rejected by both.
entry.size = 141;
bytes.push(0);
std::fs::write(&fixture.0, bytes).unwrap();
assert!(AssetManager::perform_load_scene_bank(&entry, source.clone()).is_err());
assert!(resumable::conformance_result(&entry, &source).is_err());
}
// Every mode sees the same declared slice, just as it does in production.
fn check_case(name: &str, entry: &AssetEntry, bytes: Vec<u8>, expected: Option<Value>) {
let source = Arc::new(RwLock::new(AssetsPayloadSource::from_bytes(bytes)));
for mode in 0..4 {
let result = (|| -> Result<Value, String> {
if mode == 3 {
return resumable::conformance_result(entry, &source);
}
if mode == 2 {
return match entry.bank_type {
BankType::GLYPH => AssetManager::perform_load_glyph_bank(entry, source.clone())
.map(|b| glyph_snapshot(&b)),
BankType::SOUNDS => {
AssetManager::perform_load_sound_bank(entry, source.clone())
.map(|b| sound_snapshot(&b))
}
BankType::SCENE => AssetManager::perform_load_scene_bank(entry, source.clone())
.map(|b| scene_snapshot(&b)),
};
}
let slice = source
.read()
.unwrap()
.open_slice(entry.offset, entry.size)
.map_err(|e| e.to_string())?;
let bytes = slice.read_all().map_err(|e| e.to_string())?;
let mut reader = &bytes[..];
match (entry.bank_type, mode) {
(BankType::GLYPH, 0) => AssetManager::decode_glyph_bank_from_buffer(entry, &bytes)
.map(|b| glyph_snapshot(&b)),
(BankType::GLYPH, _) => {
AssetManager::decode_glyph_bank_from_reader(entry, &mut reader)
.map(|b| glyph_snapshot(&b))
}
(BankType::SOUNDS, 0) => AssetManager::decode_sound_bank_from_buffer(entry, &bytes)
.map(|b| sound_snapshot(&b)),
(BankType::SOUNDS, _) => {
AssetManager::decode_sound_bank_from_reader(entry, &mut reader)
.map(|b| sound_snapshot(&b))
}
(BankType::SCENE, 0) => AssetManager::decode_scene_bank_from_buffer(entry, &bytes)
.map(|b| scene_snapshot(&b)),
(BankType::SCENE, _) => {
AssetManager::decode_scene_bank_from_reader(entry, &mut reader)
.map(|b| scene_snapshot(&b))
}
}
})();
assert_eq!(result.ok(), expected, "{name}, mode={mode}");
}
}
#[test]
fn glyph_layout_palette_nibbles_and_chunk_boundaries() {
for pixels in [0_usize, 1, 2, 8191, 8192, 8193, 16385] {
for palettes in [1, 3, 64] {
let mut bytes = vec![0x12; pixels.div_ceil(2)];
bytes.extend(std::iter::repeat_n([0x12, 0x34, 0x56, 0x78], palettes * 16).flatten());
let entry = AssetEntry {
asset_id: 1,
asset_name: "glyph".into(),
bank_type: BankType::GLYPH,
offset: 0,
size: bytes.len() as u64,
decoded_size: (pixels + palettes * 64) as u64,
codec: AssetCodec::None,
metadata: json!({"tile_size":16,"width":pixels,"height":1,"palette_count":palettes,"palette_authored":palettes}),
};
let expected = json!([
16,
pixels,
1,
(0..pixels).map(|i| 1 + i % 2).collect::<Vec<_>>(),
vec![vec![0x12345678_u32; 16]; palettes]
]);
check_case("glyph-valid", &entry, bytes.clone(), Some(expected));
for (key, value) in [
("tile_size", json!(7)),
("palette_count", json!(0)),
("palette_count", json!(65)),
("width", json!(-1)),
("height", json!("bad")),
] {
let mut invalid = entry.clone();
invalid.metadata[key] = value;
check_case(key, &invalid, bytes.clone(), None);
}
let mut invalid = entry.clone();
invalid.decoded_size += 1;
check_case("glyph-decoded-size", &invalid, bytes.clone(), None);
let mut invalid = entry.clone();
invalid.size += 1;
let mut extra = bytes.clone();
extra.push(0);
check_case("glyph-trailing", &invalid, extra, None);
bytes.pop();
check_case("glyph-truncated", &entry, bytes, None);
}
}
}
fn empty_scene() -> (AssetEntry, Vec<u8>, Value) {
let mut bytes = vec![0_u8; 140];
bytes[..8].copy_from_slice(&[b'S', b'C', b'N', b'E', 1, 0, 4, 0]);
for start in [12, 44, 76, 108] {
bytes[start + 5] = 8;
}
let entry = AssetEntry {
asset_id: 2,
asset_name: "scene".into(),
bank_type: BankType::SCENE,
offset: 0,
size: 140,
decoded_size: 64,
codec: AssetCodec::None,
metadata: json!({}),
};
(entry, bytes, json!(vec![json!([false, 0, 8, 0.0, 0.0, 0, 0, []]); 4]))
}
#[test]
fn scene_header_ranges_nonfinite_and_truncation_corpus() {
let (entry, bytes, expected) = empty_scene();
check_case("empty-scene", &entry, bytes.clone(), Some(expected));
for end in 0..bytes.len() {
let mut e = entry.clone();
e.size = end as u64;
check_case("scene-every-truncation", &e, bytes[..end].to_vec(), None);
}
for (offset, value) in [(0, 0xff), (4, 2), (6, 3), (17, 7), (36, 1)] {
let mut invalid = bytes.clone();
invalid[offset] = value;
check_case("scene-invalid-header-field", &entry, invalid, None);
}
for offset in [20, 24] {
for value in [f32::NAN, f32::INFINITY, f32::NEG_INFINITY] {
let mut invalid = bytes.clone();
invalid[offset..offset + 4].copy_from_slice(&value.to_le_bytes());
check_case("scene-nonfinite", &entry, invalid, None);
}
}
let mut invalid = bytes.clone();
for offset in [28, 32, 36] {
invalid[offset..offset + 4].copy_from_slice(&u32::MAX.to_le_bytes());
}
check_case("scene-dimensions-overflow-or-count-mismatch", &entry, invalid, None);
let mut invalid = bytes.clone();
for offset in [28, 32, 36] {
invalid[offset..offset + 4].copy_from_slice(&1_u32.to_le_bytes());
}
// Four layer headers no longer fit after the declared tile consumes bytes.
check_case("scene-tile-range", &entry, invalid, None);
let mut sequence_metadata = entry.clone();
sequence_metadata.metadata = json!([]);
check_case(
"scene-empty-sequence-metadata",
&sequence_metadata,
bytes.clone(),
Some(empty_scene().2),
);
let mut e = entry.clone();
e.decoded_size += 1;
check_case("scene-decoded-size", &e, bytes.clone(), None);
e = entry.clone();
e.size += 1;
let mut trailing = bytes.clone();
trailing.push(0);
check_case("scene-trailing", &e, trailing, None);
for metadata in [json!(null), json!("bad")] {
e = entry.clone();
e.metadata = metadata;
check_case("scene-metadata", &e, bytes.clone(), None);
}
}
#[test]
fn scene_tile_chunk_edges_flags_and_reserved_bytes() {
for count in [1_usize, 1023, 1024, 1025, 2049] {
let (mut entry, mut bytes, mut expected) = empty_scene();
bytes[12] = 0xff; // Only the active flag is meaningful.
bytes[13..17].copy_from_slice(&(-7_i32).to_le_bytes());
bytes[17] = 32;
bytes[18..20].copy_from_slice(&[0xff; 2]); // Reserved fields are ignored.
bytes[20..24].copy_from_slice(&(-0.5_f32).to_le_bytes());
bytes[24..28].copy_from_slice(&2.0_f32.to_le_bytes());
bytes[28..32].copy_from_slice(&(count as u32).to_le_bytes());
bytes[32..36].copy_from_slice(&1_u32.to_le_bytes());
bytes[36..40].copy_from_slice(&(count as u32).to_le_bytes());
bytes.splice(44..44, std::iter::repeat_n([0xff, 255, 0x34, 0x12], count).flatten());
entry.size = bytes.len() as u64;
entry.decoded_size += (count * size_of::<Tile>()) as u64;
expected[0] = json!([
true,
-7,
32,
-0.5,
2.0,
count,
1,
vec![(true, 0x1234, 255, true, true); count]
]);
check_case("scene-chunks", &entry, bytes, Some(expected));
}
}
#[test]
fn pcm_metadata_and_source_range_corpus() {
for metadata in [
json!(null),
json!({}),
json!({"sample_rate":-1,"channels":1}),
json!({"sample_rate":44100,"channels":"bad"}),
] {
let mut e = pcm_entry(2);
e.metadata = metadata;
check_case("pcm-metadata", &e, vec![0x34, 0x12], None);
}
// These fields are currently descriptive, not extra PCM layout constraints.
let mut e = pcm_entry(2);
e.metadata = json!({"sample_rate":0,"channels":0});
e.decoded_size = 999;
check_case(
"pcm-permissive-metadata",
&e,
vec![0x34, 0x12],
Some(json!([[[0x1234], 0, null, null]])),
);
for offset in [1, u64::MAX] {
let mut e = pcm_entry(2);
e.offset = offset;
check_case("source-range", &e, vec![0x34, 0x12], None);
}
let mut e = pcm_entry(2);
e.size = u64::MAX;
check_case("source-size-overflow", &e, vec![0x34, 0x12], None);
}
pub(in crate::asset) fn glyph_snapshot(bank: &GlyphBank) -> Value {
json!([
bank.tile_size as u8,
bank.width,
bank.height,
bank.pixel_indices,
bank.palettes
.iter()
.map(|p| p.iter().map(|c| c.raw()).collect::<Vec<_>>())
.collect::<Vec<_>>()
])
}
pub(in crate::asset) fn sound_snapshot(bank: &SoundBank) -> Value {
json!(
bank.samples
.iter()
.map(|s| (&s.data, s.sample_rate, s.loop_start, s.loop_end))
.collect::<Vec<_>>()
)
}
pub(in crate::asset) fn scene_snapshot(bank: &SceneBank) -> Value {
json!(
bank.layers
.iter()
.map(|l| json!([
l.active,
l.glyph_asset_id,
l.tile_size as u8,
l.parallax_factor.x,
l.parallax_factor.y,
l.tilemap.width,
l.tilemap.height,
l.tilemap
.tiles
.iter()
.map(|t| (t.active, t.glyph.glyph_id, t.glyph.palette_id, t.flip_x, t.flip_y))
.collect::<Vec<_>>()
]))
.collect::<Vec<_>>()
)
}
#[test]
fn shared_scene_arithmetic_rejects_overflow_before_allocating() {
let (_, bytes, _) = empty_scene();
let mut header = bytes[12..44].to_vec();
header[16..20].copy_from_slice(&1_u32.to_le_bytes());
header[20..24].copy_from_slice(&1_u32.to_le_bytes());
header[24..28].copy_from_slice(&1_u32.to_le_bytes());
assert!(scene_layer(&header, usize::MAX, u64::MAX, &mut 0).is_err());
let mut decoded = usize::MAX;
assert!(scene_layer(&header, 44, 48, &mut decoded).is_err());
assert!(scene_layer(&header, 44, 47, &mut 0).is_err());
assert!(scene_layer(&header[..31], 44, 48, &mut 0).is_err());
}
#[test]
fn physically_truncated_glyph_palette_is_rejected_by_both_modes() {
let fixture = FileFixture::new("truncated-glyph", &[0; 63]);
let entry = AssetEntry {
asset_id: 3,
asset_name: "empty-glyph".into(),
bank_type: BankType::GLYPH,
offset: 0,
size: 64,
decoded_size: 64,
codec: AssetCodec::None,
metadata: json!({"tile_size":8,"width":0,"height":0,"palette_count":1,"palette_authored":1}),
};
let source = fixture.source(64);
assert!(AssetManager::perform_load_glyph_bank(&entry, source.clone()).is_err());
assert!(resumable::conformance_result(&entry, &source).is_err());
std::fs::write(&fixture.0, [0; 64]).unwrap();
let expected = json!([8, 0, 0, [], vec![vec![0; 16]]]);
assert_eq!(
glyph_snapshot(&AssetManager::perform_load_glyph_bank(&entry, source.clone()).unwrap()),
expected
);
assert_eq!(resumable::conformance_result(&entry, &source).unwrap(), expected);
}
@@ -246,7 +246,7 @@ impl Decoder {
}
}
BankType::SOUNDS => {
let sample_rate = entry.metadata_as_sound_bank()?.sample_rate;
let sample_rate = decode::sound_rate(entry)?;
State::Sound {
sample_rate,
data: Vec::with_capacity(entry.size as usize / 2),
@@ -254,15 +254,9 @@ impl Decoder {
}
}
BankType::SCENE => {
entry.metadata_as_scene_bank()?;
decode::scene_metadata(entry)?;
State::Scene {
layers: std::array::from_fn(|_| SceneLayer {
active: false,
glyph_asset_id: 0,
tile_size: TileSize::Size8,
parallax_factor: ParallaxFactor { x: 1.0, y: 1.0 },
tilemap: TileMap { width: 0, height: 0, tiles: Vec::new() },
}),
layers: decode::empty_layers(),
layer: 0,
tiles_left: 0,
header: false,
@@ -279,25 +273,18 @@ impl Decoder {
if *position < *packed {
let n = (*packed - *position).min(bytes.len());
self.reader.read_exact(&mut bytes[..n]).map_err(|e| e.to_string())?;
for &v in &bytes[..n] {
if bank.pixel_indices.len() < bank.width * bank.height {
bank.pixel_indices.push(v >> 4)
}
if bank.pixel_indices.len() < bank.width * bank.height {
bank.pixel_indices.push(v & 15)
}
}
decode::append_pixels(
&mut bank.pixel_indices,
&bytes[..n],
bank.width * bank.height,
);
*position += n;
return Ok(None);
}
if *palette_position < *palette_bytes {
let n = (*palette_bytes - *palette_position).min(bytes.len());
self.reader.read_exact(&mut bytes[..n]).map_err(|e| e.to_string())?;
for (i, rgba) in bytes[..n].as_chunks::<4>().0.iter().enumerate() {
let at = *palette_position / 4 + i;
bank.palettes[at / 16][at % 16] =
Color::rgba(rgba[0], rgba[1], rgba[2], rgba[3]);
}
decode::write_palettes(&mut bank.palettes, *palette_position, &bytes[..n]);
*palette_position += n;
return Ok(None);
}
@@ -311,9 +298,7 @@ impl Decoder {
if *remaining > 0 {
let n = (*remaining).min(bytes.len());
self.reader.read_exact(&mut bytes[..n]).map_err(|e| e.to_string())?;
for pair in bytes[..n].as_chunks::<2>().0 {
data.push(i16::from_le_bytes([pair[0], pair[1]]));
}
decode::append_pcm(data, &bytes[..n]);
*remaining -= n;
return Ok(None);
}
@@ -327,12 +312,7 @@ impl Decoder {
self.reader
.read_exact(&mut bytes[..SCENE_HEADER_BYTES_V1])
.map_err(|e| e.to_string())?;
if bytes[..4] != SCENE_PAYLOAD_MAGIC_V1
|| u16::from_le_bytes([bytes[4], bytes[5]]) != SCENE_PAYLOAD_VERSION_V1
|| u16::from_le_bytes([bytes[6], bytes[7]]) as usize != SCENE_LAYER_COUNT_V1
{
return Err("Invalid scene header".into());
}
decode::scene_header(&bytes[..SCENE_HEADER_BYTES_V1])?;
self.position += SCENE_HEADER_BYTES_V1;
*header = true;
return Ok(None);
@@ -341,17 +321,7 @@ impl Decoder {
let n = (*tiles_left).min(bytes.len() / SCENE_TILE_RECORD_BYTES_V1);
let count = n * SCENE_TILE_RECORD_BYTES_V1;
self.reader.read_exact(&mut bytes[..count]).map_err(|e| e.to_string())?;
for b in bytes[..count].as_chunks::<SCENE_TILE_RECORD_BYTES_V1>().0 {
layers[*layer - 1].tilemap.tiles.push(Tile {
active: b[0] & 1 != 0,
glyph: Glyph {
glyph_id: u16::from_le_bytes([b[2], b[3]]),
palette_id: b[1],
},
flip_x: b[0] & 2 != 0,
flip_y: b[0] & 4 != 0,
});
}
decode::append_tiles(&mut layers[*layer - 1].tilemap.tiles, &bytes[..count]);
*tiles_left -= n;
self.position += count;
return Ok(None);
@@ -361,50 +331,18 @@ impl Decoder {
.read_exact(&mut bytes[..SCENE_LAYER_HEADER_BYTES_V1])
.map_err(|e| e.to_string())?;
self.position += SCENE_LAYER_HEADER_BYTES_V1;
let u32at = |at| u32::from_le_bytes(bytes[at..at + 4].try_into().unwrap());
let width = u32at(16) as usize;
let height = u32at(20) as usize;
let count = u32at(24) as usize;
let tile_size = match bytes[5] {
8 => TileSize::Size8,
16 => TileSize::Size16,
32 => TileSize::Size32,
_ => return Err("Invalid scene tile size".into()),
};
let x = f32::from_le_bytes(bytes[8..12].try_into().unwrap());
let y = f32::from_le_bytes(bytes[12..16].try_into().unwrap());
if !x.is_finite()
|| !y.is_finite()
|| width.checked_mul(height) != Some(count)
|| count
.checked_mul(SCENE_TILE_RECORD_BYTES_V1)
.and_then(|n| n.checked_add(self.position))
.is_none_or(|n| n as u64 > self.entry.size)
{
return Err("Invalid scene layer range".into());
}
*decoded = decoded
.checked_add(SCENE_DECODED_LAYER_OVERHEAD_BYTES_V1)
.and_then(|n| {
count.checked_mul(size_of::<Tile>()).and_then(|b| n.checked_add(b))
})
.ok_or("Scene size overflow")?;
layers[*layer] = SceneLayer {
active: bytes[0] & 1 != 0,
glyph_asset_id: i32::from_le_bytes(bytes[1..5].try_into().unwrap()),
tile_size,
parallax_factor: ParallaxFactor { x, y },
tilemap: TileMap { width, height, tiles: Vec::with_capacity(count) },
};
layers[*layer] = decode::scene_layer(
&bytes[..SCENE_LAYER_HEADER_BYTES_V1],
self.position,
self.entry.size,
decoded,
)?;
let count = layers[*layer].tilemap.width * layers[*layer].tilemap.height;
*layer += 1;
*tiles_left = count;
return Ok(None);
}
if self.position as u64 != self.entry.size
|| *decoded as u64 != self.entry.decoded_size
{
return Err("Invalid scene payload size".into());
}
decode::scene_finish(&self.entry, self.position, *decoded)?;
let State::Scene { layers, .. } = std::mem::replace(&mut self.state, State::Done)
else {
unreachable!()
@@ -416,6 +354,30 @@ impl Decoder {
}
}
// Test-only access to the production state machine; no alternate decode rules.
#[cfg(test)]
pub(super) fn conformance_result(
entry: &AssetEntry,
source: &Arc<RwLock<AssetsPayloadSource>>,
) -> Result<serde_json::Value, String> {
use std::io::Seek;
let mut decoder = Decoder::new(entry, source)?;
for _ in 0..100 {
let before = decoder.reader.stream_position().unwrap();
let result = decoder.step();
let after = decoder.reader.stream_position().unwrap();
assert!(after >= before && after - before <= 4096, "unbounded or rewound decode step");
if let Some(bank) = result? {
return Ok(match bank {
Decoded::Glyph(bank) => super::decode::tests::glyph_snapshot(&bank),
Decoded::Sound(bank) => super::decode::tests::sound_snapshot(&bank),
Decoded::Scene(bank) => super::decode::tests::scene_snapshot(&bank),
});
}
}
panic!("conformance fixture exceeded its step budget")
}
#[cfg(test)]
mod tests {
use super::*;
@@ -556,4 +518,55 @@ mod tests {
assert_eq!(manager.preloads_ready(), Ok(true));
assert_eq!(manager.slot_info(SlotRef::audio(0)).asset_id, Some(1));
}
#[test]
fn canceled_and_superseded_partial_decoders_never_publish() {
use crate::memory_banks::SoundBankPoolAccess;
let bytes = vec![0x12; 16384];
let entry = AssetEntry {
asset_id: 1,
asset_name: "pcm".into(),
bank_type: BankType::SOUNDS,
offset: 0,
size: bytes.len() as u64,
decoded_size: bytes.len() as u64,
codec: AssetCodec::None,
metadata: serde_json::json!({"sample_rate":44100,"channels":1}),
};
let mut newer = entry.clone();
newer.asset_id = 2;
let banks = Arc::new(crate::memory_banks::MemoryBanks::new());
let mut manager = AssetManager::new(
vec![entry, newer],
AssetsPayloadSource::from_bytes(bytes),
banks.clone(),
banks.clone(),
banks.clone(),
);
let lane = Arc::new(ManualLane::default());
assert!(manager.bind_background_work(lane.clone()));
let handle = manager.load(1, 0).unwrap();
assert!(lane.step());
assert!(lane.step());
assert_eq!(manager.status(handle), LoadStatus::LOADING);
assert_eq!(manager.cancel(handle), AssetOpStatus::Ok);
while lane.step() {}
assert_eq!(manager.status(handle), LoadStatus::CANCELED);
assert!(manager.sound_policy.staging.read().unwrap().is_empty());
manager.apply_commits();
assert!(banks.sound_bank_slot(0).is_none());
let old = manager.load(1, 0).unwrap();
assert!(lane.step());
assert!(lane.step());
let new = manager.load(2, 0).unwrap();
assert_eq!(old, new); // Stable slot handle, different request generation.
assert!(lane.step()); // Discard the in-flight old decoder.
assert!(manager.sound_policy.staging.read().unwrap().is_empty());
while lane.step() {}
assert_eq!(manager.status(new), LoadStatus::READY);
assert!(banks.sound_bank_slot(0).is_none());
assert_eq!(manager.commit(new), AssetOpStatus::Ok);
manager.apply_commits();
assert_eq!(manager.slot_info(SlotRef::audio(0)).asset_id, Some(2));
assert_eq!(banks.sound_bank_slot(0).unwrap().samples[0].data, vec![0x1212; 8192]);
}
}
+3 -2
View File
@@ -203,7 +203,8 @@ impl AudioBridge for Audio {
fn begin_frame(&mut self) { /* Staging may already contain debugger steps. */
}
fn finish_frame(&mut self) {
self.transport.publish(std::mem::take(&mut self.commands), false);
let commands = std::mem::replace(&mut self.commands, self.transport.take_buffer());
self.transport.publish(commands, false);
self.frame_commands_admitted = 0;
}
fn abort_frame(&mut self) {
@@ -221,7 +222,7 @@ impl AudioBridge for Audio {
if success {
// Publish the earlier frame prefix first so a callback stop cannot
// be undone by an old play on resume. Do not replenish frame quota.
let mut commands = std::mem::take(&mut self.commands);
let mut commands = std::mem::replace(&mut self.commands, self.transport.take_buffer());
commands.append(&mut self.lifecycle_commands);
self.transport.publish(commands, true);
} else {
@@ -14,6 +14,7 @@ struct Mailbox {
paused: bool,
frame: Option<Vec<AudioCommand>>,
lifecycle: Option<Vec<AudioCommand>>,
recycled: [Option<Vec<AudioCommand>>; 4],
}
pub struct AudioDelivery {
@@ -36,7 +37,31 @@ impl AudioTransport {
}
pub fn consume_silently(&self) {
let _ = self.try_take();
if let Some(delivery) = self.try_take() {
for batch in [delivery.frame, delivery.lifecycle].into_iter().flatten() {
self.recycle_commands(batch);
}
}
}
/// Main/service lane only: drop command owners and retain bounded empty storage.
pub fn recycle_commands(&self, mut commands: Vec<AudioCommand>) {
commands.clear();
let mut state = self.0.lock().expect("audio mailbox");
if let Some(slot) = state.recycled.iter_mut().find(|slot| slot.is_none()) {
*slot = Some(commands);
}
// Excess empty buffers are destroyed here, never in the data callback.
}
pub(super) fn take_buffer(&self) -> Vec<AudioCommand> {
self.0
.lock()
.expect("audio mailbox")
.recycled
.iter_mut()
.find_map(Option::take)
.unwrap_or_default()
}
pub fn ready(&self) -> bool {
@@ -84,3 +109,36 @@ mod tests {
assert!(transport.try_take().is_some());
}
}
#[cfg(test)]
mod reuse_tests {
use super::*;
#[test]
fn service_recycles_storage_and_releases_sample_owners() {
use prometeu_hal::{LoopMode, sample::Sample};
let transport = AudioTransport::default();
let sample = Arc::new(Sample::new(48000, vec![1]));
let weak = Arc::downgrade(&sample);
let mut batch = Vec::with_capacity(1024);
let allocation = batch.as_ptr();
batch.push(AudioCommand::Play {
sample,
voice_id: 0,
volume: 255,
pan: 0,
pitch: 1.0,
priority: 0,
loop_mode: LoopMode::Off,
});
transport.recycle_commands(batch);
assert!(weak.upgrade().is_none());
let reused = transport.take_buffer();
assert_eq!(reused.as_ptr(), allocation);
assert_eq!(reused.capacity(), 1024);
assert!(reused.is_empty());
for _ in 0..8 {
transport.recycle_commands(Vec::with_capacity(16));
}
assert_eq!(transport.0.lock().unwrap().recycled.iter().flatten().count(), 4);
}
}
@@ -65,6 +65,8 @@ struct Request {
control: Arc<PublicationControl>,
receiver: mpsc::Receiver<Completion>,
admitted: Instant,
watchdog_canceled_at: Option<Instant>,
worker_error: Option<Error>,
result: RequestResult,
job: AsyncWorkJobId,
released: bool,
@@ -523,6 +525,8 @@ impl Session {
control,
receiver,
admitted,
watchdog_canceled_at: None,
worker_error: None,
result: RequestResult { id, state: RequestState::Pending, reason: RequestReason::None },
job,
released: !saves_payload,
@@ -664,6 +668,8 @@ impl Session {
control,
receiver,
admitted: now,
watchdog_canceled_at: None,
worker_error: None,
result: RequestResult { id, state: RequestState::Pending, reason: RequestReason::None },
job,
released: false,
@@ -686,27 +692,29 @@ impl Session {
let Some(r) = &mut self.request else { return };
match r.receiver.try_recv() {
Ok(completion) => {
r.worker_error = completion.result.error;
let current =
self.mode.is_some() && r.session == self.context && r.epoch == self.epoch;
if current {
self.slots[r.local].protected = false;
if r.result.reason == RequestReason::None {
if r.result.state == RequestState::Pending {
r.result.state = match completion.result.durable {
DurableResult::Success => RequestState::Success,
DurableResult::Failure => RequestState::Failure,
DurableResult::Indeterminate => RequestState::Indeterminate,
};
r.result.reason = completion
.result
.error
.map_or(RequestReason::None, RequestReason::Storage);
r.result.reason = if r.watchdog_canceled_at.is_some() {
RequestReason::Timeout
} else {
r.worker_error.map_or(RequestReason::None, RequestReason::Storage)
};
if completion.result.durable == DurableResult::Success && r.saves_payload {
let mut s = r.source.write().unwrap();
s.descriptor =
completion.card.current().slots[self.logical[r.local] as usize];
s.dirty = false;
}
} // A timeout result is immutable; returned card requires explicit reconciliation.
} // A restart escalation is immutable; returned card requires explicit reconciliation.
if self.provisional && completion.result.durable != DurableResult::Success {
self.boot_error =
Some(completion.result.error.unwrap_or(Error::InvalidState));
@@ -755,7 +763,10 @@ impl Session {
return;
}
let elapsed = now.saturating_duration_since(r.admitted);
if elapsed >= Duration::from_millis(500) {
if elapsed >= Duration::from_millis(500) && !r.control.canceled() {
// Main-lane cancellation cause is independent of the worker's durable
// outcome. Do not relabel an earlier removal or termination cancel.
r.watchdog_canceled_at = Some(now);
r.control.cancel();
}
if elapsed >= Duration::from_secs(1) {
@@ -788,6 +799,12 @@ impl Session {
.map(|r| r.result)
.ok_or(Error::InvalidRange)
}
/// Original worker error for native diagnostics, independent of guest deadline reason.
/// Pending and RAM-only requests have no worker error. Retention follows query/release.
pub fn request_storage_error(&self, context: u64, id: u64) -> Result<Option<Error>> {
self.query(context, id)?;
Ok(self.request.as_ref().filter(|r| r.id == id).and_then(|r| r.worker_error))
}
pub fn release_request(&mut self, context: u64, id: u64) -> Result<()> {
self.query(context, id)?;
if self.deferred.is_some() {
@@ -280,8 +280,157 @@ fn async_commit_protects_only_source_and_retains_result_without_occupying_lane()
controller.stop().unwrap();
}
// The sentinel executes after the save on the single lane. Wall time only bounds
// a broken fixture; all watchdog observations use explicit simulated instants.
fn finish_queued_save(lane: Arc<crate::AsyncWorkLane>) {
let (tx, rx) = std::sync::mpsc::channel();
lane.submit_asset_load(move |_| {
tx.send(()).unwrap();
crate::services::async_work::AsyncWorkJobOutcome::Completed
})
.unwrap();
let mut controller =
crate::AsyncWorkLaneController::start(crate::AsyncWorkLaneConfig::default(), lane);
rx.recv_timeout(std::time::Duration::from_secs(5)).unwrap();
controller.stop().unwrap();
}
fn assert_guest_query(
session: &mut super::super::session::Session,
id: u64,
state: i64,
reason: i64,
) {
use prometeu_bytecode::Value;
use prometeu_hal::{
HostReturn, host_return::GuestStringAllocator, syscalls::Syscall, vm_fault::VmFault,
};
struct NoStrings;
impl GuestStringAllocator for NoStrings {
fn byte_limit(&self) -> u64 {
0
}
fn build(
&mut self,
_: usize,
_: &mut dyn FnMut(&mut dyn std::fmt::Write) -> std::fmt::Result,
) -> std::result::Result<Arc<str>, VmFault> {
panic!("MemQuery must not allocate strings")
}
}
let mut service = super::super::MemcardService::new();
std::mem::swap(session, &mut service.session);
let args = [Value::Int64(service.session.context() as i64), Value::Int64(id as i64)];
let mut values = Vec::with_capacity(Syscall::MemQuery.results_count());
let mut strings = NoStrings;
let mut ret = HostReturn::new(&mut values, Syscall::MemQuery.results_count(), &mut strings);
service.guest(Syscall::MemQuery, &args, &mut ret, 4, 0, true).unwrap();
assert!(!ret.overflowed());
assert!(ret.failure().is_none());
assert_eq!(values, vec![Value::Int64(0), Value::Int64(state), Value::Int64(reason)]);
std::mem::swap(session, &mut service.session);
}
#[test]
fn blocked_host_timeout_release_keeps_source_and_prevents_replacement_work() {
fn queued_memcard_watchdog_preserves_timeout_and_guest_reason() {
use super::super::session::{RequestReason, RequestState, Session};
use std::time::{Duration, Instant};
for bootstrap in [false, true] {
let (mut card, disk) = formatted();
add(&mut card, 4, 1);
let before = card.current().slots[0];
let mut s = Session::default();
let c = s.load_card(card, 4, 1, Session::required_bytes(1).unwrap()).unwrap();
s.write(c, 0, 0, b"pending").unwrap();
let lane = Arc::new(crate::AsyncWorkLane::default());
let now = Instant::now();
if bootstrap {
s.begin_boot();
}
let id = s.commit(c, 0, 7, now, &lane).unwrap();
if bootstrap {
s.finish_boot(true, &lane, now).unwrap();
}
s.poll(now + Duration::from_millis(499));
assert_guest_query(&mut s, id, 0, 0);
for _ in 0..2 {
s.poll(now + Duration::from_millis(500));
}
assert_guest_query(&mut s, id, 0, 0);
assert_eq!(s.release_request(c, id), Err(Error::Busy));
assert_eq!(s.clear(c, 0), Err(Error::Busy));
finish_queued_save(lane);
s.poll(now + Duration::from_millis(501));
let result = s.query(c, id).unwrap();
assert_eq!(result.state, RequestState::Failure);
assert_eq!(result.reason, RequestReason::Timeout);
assert_eq!(s.request_storage_error(c, id), Ok(Some(Error::Canceled)));
assert_guest_query(&mut s, id, 2, 15);
assert!(s.unsaved());
assert_eq!(reboot(&disk).current().slots[0], before);
assert_eq!(s.query(c + 1, id), Err(Error::AccessDenied));
s.release_request(c, id).unwrap();
assert_eq!(s.query(c, id), Err(Error::InvalidRange));
s.clear(c, 0).unwrap();
}
}
#[test]
fn lifecycle_cancellation_is_not_a_watchdog_timeout() {
use super::super::session::{RequestReason, RequestState, Session};
use std::time::{Duration, Instant};
for end in [false, true] {
let (mut card, _) = formatted();
add(&mut card, 4, 1);
let mut s = Session::default();
let c = s.load_card(card, 4, 1, Session::required_bytes(1).unwrap()).unwrap();
let lane = Arc::new(crate::AsyncWorkLane::default());
let now = Instant::now();
let id = s.commit(c, 0, 7, now, &lane).unwrap();
if end {
s.end_session();
} else {
s.notify_removal();
}
s.poll(now + Duration::from_millis(500));
finish_queued_save(lane);
s.poll(now + Duration::from_millis(501));
if end {
assert_eq!(s.query(c, id), Err(Error::AccessDenied));
let card = s.take_idle_card().unwrap().unwrap();
let new = s.load_card(card, 4, 1, Session::required_bytes(1).unwrap()).unwrap();
assert_ne!(new, c);
assert_eq!(s.query(new, id), Err(Error::InvalidRange));
} else {
assert_eq!(s.query(c, id).unwrap().state, RequestState::Failure);
assert_eq!(s.query(c, id).unwrap().reason, RequestReason::Storage(Error::Canceled));
assert_guest_query(&mut s, id, 2, 8);
}
}
}
#[test]
fn deferred_boot_memcard_deadline_stays_timeout() {
use super::super::session::{RequestReason, Session};
use std::time::{Duration, Instant};
let (mut card, _) = formatted();
add(&mut card, 4, 1);
let mut s = Session::default();
let c = s.load_card(card, 4, 1, Session::required_bytes(1).unwrap()).unwrap();
s.begin_boot();
let lane = crate::AsyncWorkLane::default();
let now = Instant::now();
let id = s.commit(c, 0, 7, now, &lane).unwrap();
s.poll(now + Duration::from_millis(500));
assert_eq!(s.query(c, id).unwrap().reason, RequestReason::Timeout);
assert_guest_query(&mut s, id, 2, 15);
assert_eq!(lane.pending_depth(), 0);
s.clear(c, 0).unwrap();
s.release_request(c, id).unwrap();
}
#[test]
fn blocked_host_memcard_watchdog_preserves_outcome_and_escalation() {
use crate::services::async_work::{
AsyncWorkLane, AsyncWorkLaneConfig, AsyncWorkLaneController,
};
@@ -290,6 +439,7 @@ fn blocked_host_timeout_release_keeps_source_and_prevents_replacement_work() {
struct Blocked {
store: Box<dyn MemcardStorage>,
publication: bool,
fail_publication: bool,
entered: std::sync::mpsc::Sender<()>,
resume: std::sync::mpsc::Receiver<()>,
}
@@ -310,12 +460,24 @@ fn blocked_host_timeout_release_keeps_source_and_prevents_replacement_work() {
if if self.publication { b.starts_with(b"PMMAP1\0\0") } else { o >= DATA_OFFSET } {
self.entered.send(()).unwrap();
self.resume.recv().unwrap();
if self.fail_publication {
return Err(MemcardIoError::IoError);
}
}
self.store.write_at(o, b)
}
}
for publication in [false, true] {
let (mut card, _) = formatted();
for (publication, fail_publication, escalate, release_early) in [
(false, false, false, false),
(true, false, false, false),
(true, true, false, false),
(false, false, true, false),
(true, false, true, false),
(true, true, true, false),
(false, false, true, true),
(true, false, true, true),
] {
let (mut card, disk) = formatted();
add(&mut card, 4, 1);
let (et, er) = std::sync::mpsc::channel();
let (rt, rr) = std::sync::mpsc::channel();
@@ -324,6 +486,7 @@ fn blocked_host_timeout_release_keeps_source_and_prevents_replacement_work() {
entered: et,
resume: rr,
publication,
fail_publication,
}))
.unwrap();
let mut s = Session::default();
@@ -336,22 +499,84 @@ fn blocked_host_timeout_release_keeps_source_and_prevents_replacement_work() {
let id = s.commit(c, 0, 7, admitted, &lane).unwrap();
er.recv_timeout(Duration::from_secs(1)).unwrap();
s.poll(admitted + Duration::from_millis(500));
s.poll(admitted + Duration::from_secs(1));
let result = s.query(c, id).unwrap();
assert_eq!(
result.state,
if publication { RequestState::Indeterminate } else { RequestState::Failure }
);
assert_eq!(result.reason, RequestReason::RestartRequired);
assert!(s.host_faulted());
s.release_request(c, id).unwrap();
assert_guest_query(&mut s, id, 0, 0);
let terminal = if escalate {
s.poll(admitted + Duration::from_secs(1));
let result = s.query(c, id).unwrap();
assert_eq!(
result.state,
if publication { RequestState::Indeterminate } else { RequestState::Failure }
);
assert_eq!(result.reason, RequestReason::RestartRequired);
assert!(s.host_faulted());
if release_early {
s.release_request(c, id).unwrap();
}
Some(result)
} else {
None
};
assert_eq!(s.write(c, 0, 0, b"unsafe"), Err(Error::Busy));
assert_eq!(s.commit(c, 0, 7, Instant::now(), &lane), Err(Error::Busy));
assert_eq!(s.commit(c, 0, 7, admitted, &lane), Err(Error::Busy));
rt.send(()).unwrap();
controller.stop().unwrap();
s.poll(Instant::now());
assert!(s.unsaved());
assert!(s.host_faulted());
s.poll(admitted + Duration::from_millis(if escalate { 1001 } else { 501 }));
if let Some(terminal) = terminal {
if release_early {
assert_eq!(s.query(c, id), Err(Error::InvalidRange));
} else {
assert_eq!(s.query(c, id), Ok(terminal));
assert_guest_query(&mut s, id, if publication { 3 } else { 2 }, 15);
s.release_request(c, id).unwrap();
}
assert!(s.unsaved());
assert!(s.host_faulted());
assert_eq!(s.commit(c, 0, 7, admitted, &lane), Err(Error::ReadOnly));
} else {
let expected = if fail_publication {
RequestState::Indeterminate
} else if publication {
RequestState::Success
} else {
RequestState::Failure
};
assert_eq!(s.query(c, id).unwrap().state, expected);
assert_eq!(s.query(c, id).unwrap().reason, RequestReason::Timeout);
assert_guest_query(
&mut s,
id,
if fail_publication {
3
} else if publication {
1
} else {
2
},
15,
);
assert_eq!(
s.request_storage_error(c, id),
Ok(if fail_publication {
Some(Error::Io(MemcardIoError::IoError))
} else if publication {
None
} else {
Some(Error::Canceled)
})
);
assert_eq!(s.unsaved(), expected != RequestState::Success);
assert_eq!(s.removal_pending(), fail_publication);
assert!(!s.host_faulted());
s.release_request(c, id).unwrap();
}
let mut durable = reboot(&disk);
let mut bytes = [0; 7];
let n = durable.read_slot(0, &mut bytes).unwrap();
if publication && !fail_publication {
assert_eq!(&bytes[..n], b"pending");
} else {
assert_eq!(n, 0);
}
}
}
@@ -660,3 +885,232 @@ fn ending_a_session_releases_a_completed_unconsumed_request() {
assert!(s.take_idle_card().unwrap().is_some());
controller.stop().unwrap();
}
#[test]
fn persistent_save_preempts_real_multichunk_asset_decoders() {
use crate::memcard::session::{RequestState, Session};
use crate::{AsyncWorkLane, AsyncWorkLaneConfig, AsyncWorkLaneController};
use prometeu_drivers::{
AssetManager, GlyphBankPoolAccess, MemoryBanks, SceneBankPoolAccess, SoundBankPoolAccess,
};
use prometeu_hal::{
AssetBridge,
asset::{AssetCodec, AssetEntry, AssetOpStatus, BankType, LoadStatus},
background_work::{AssetWork, BackgroundWork, WorkUnavailable},
cartridge::AssetsPayloadSource,
};
use std::{
sync::mpsc,
thread,
time::{Duration, Instant},
};
type Events = Arc<Mutex<Vec<(&'static str, thread::ThreadId)>>>;
struct ObservedLane {
lane: Arc<AsyncWorkLane>,
events: Events,
paused: mpsc::Sender<()>,
resume: Mutex<Option<mpsc::Receiver<()>>>,
}
impl BackgroundWork for ObservedLane {
fn submit_asset(&self, mut work: AssetWork) -> std::result::Result<(), WorkUnavailable> {
let events = self.events.clone();
let paused = self.paused.clone();
let resume = self.resume.lock().unwrap().take().unwrap();
let mut step = 0;
self.lane.submit_asset(Box::new(move |canceled| {
let result = work(canceled);
step += 1;
events.lock().unwrap().push(("asset", thread::current().id()));
if step == 4 {
paused.send(()).unwrap();
resume.recv_timeout(Duration::from_secs(10)).unwrap();
}
result
}))
}
}
struct ObservedStore {
inner: Store,
events: Events,
}
impl MemcardStorage for ObservedStore {
fn len(&self) -> std::result::Result<u64, MemcardIoError> {
self.inner.len()
}
fn read_only(&self) -> bool {
false
}
fn read_at(
&mut self,
offset: u64,
out: &mut [u8],
) -> std::result::Result<(), MemcardIoError> {
self.inner.read_at(offset, out)
}
fn write_at(
&mut self,
offset: u64,
bytes: &[u8],
) -> std::result::Result<(), MemcardIoError> {
self.events.lock().unwrap().push(("save write", thread::current().id()));
self.inner.write_at(offset, bytes)
}
fn sync(&mut self) -> std::result::Result<(), MemcardIoError> {
self.events.lock().unwrap().push(("save sync", thread::current().id()));
self.inner.sync()
}
}
for kind in [BankType::GLYPH, BankType::SOUNDS, BankType::SCENE] {
let (card, disk) = formatted();
drop(card);
let events: Events = Arc::default();
let mut card = Card::mount(Box::new(ObservedStore {
inner: Store(disk.clone()),
events: events.clone(),
}))
.unwrap();
add(&mut card, 1, 1);
let mut session = Session::default();
let context = session.load_card(card, 1, 1, 1024 * 1024).unwrap();
session.write(context, 0, 0, b"persistent under asset pressure").unwrap();
events.lock().unwrap().clear();
let (bytes, decoded_size, metadata) = match kind {
BankType::GLYPH => {
let mut bytes = vec![0x12; 16384];
bytes.extend([255; 64]);
(
bytes,
32768 + 64,
serde_json::json!({"tile_size":8,"width":256,"height":128,"palette_count":1,"palette_authored":1}),
)
}
BankType::SOUNDS => {
(vec![0x12; 16385], 16385, serde_json::json!({"sample_rate":44100,"channels":1}))
}
BankType::SCENE => {
let mut bytes = vec![0; 140];
bytes[..8].copy_from_slice(&[b'S', b'C', b'N', b'E', 1, 0, 4, 0]);
for start in [12, 44, 76, 108] {
bytes[start + 5] = 8;
}
bytes[12] = 1;
bytes[28..32].copy_from_slice(&4097_u32.to_le_bytes());
bytes[32..36].copy_from_slice(&1_u32.to_le_bytes());
bytes[36..40].copy_from_slice(&4097_u32.to_le_bytes());
bytes.splice(44..44, std::iter::repeat_n([7, 3, 0x34, 0x12], 4097).flatten());
(
bytes,
64 + 4097 * size_of::<prometeu_hal::tile::Tile>() as u64,
serde_json::json!({}),
)
}
};
let entry = AssetEntry {
asset_id: 1,
asset_name: "pressure".into(),
bank_type: kind,
offset: 0,
size: bytes.len() as u64,
decoded_size,
codec: AssetCodec::None,
metadata,
};
let banks = Arc::new(MemoryBanks::new());
let mut manager = AssetManager::new(
vec![entry],
AssetsPayloadSource::from_bytes(bytes),
banks.clone(),
banks.clone(),
banks.clone(),
);
let lane = Arc::new(AsyncWorkLane::default());
let (paused_tx, paused_rx) = mpsc::channel();
let (resume_tx, resume_rx) = mpsc::channel();
let observed = Arc::new(ObservedLane {
lane: lane.clone(),
events: events.clone(),
paused: paused_tx,
resume: Mutex::new(Some(resume_rx)),
});
assert!(manager.bind_background_work(observed));
let mut controller =
AsyncWorkLaneController::start(AsyncWorkLaneConfig::default(), lane.clone());
let handle = manager.load(1, 0).unwrap();
paused_rx.recv_timeout(Duration::from_secs(10)).unwrap();
assert_eq!(manager.status(handle), LoadStatus::LOADING);
assert!(
banks.glyph_bank_slot(0).is_none()
&& banks.sound_bank_slot(0).is_none()
&& banks.scene_bank_slot(0).is_none()
);
let request = session.commit(context, 0, 7, Instant::now(), &lane).unwrap();
resume_tx.send(()).unwrap();
let deadline = Instant::now() + Duration::from_secs(10);
loop {
session.poll(Instant::now());
if session.query(context, request).unwrap().state != RequestState::Pending
&& manager.status(handle) == LoadStatus::READY
{
break;
}
assert!(Instant::now() < deadline, "save/asset did not complete for {kind:?}");
thread::yield_now();
}
assert_eq!(session.query(context, request).unwrap().state, RequestState::Success);
assert!(
banks.glyph_bank_slot(0).is_none()
&& banks.sound_bank_slot(0).is_none()
&& banks.scene_bank_slot(0).is_none()
);
assert_eq!(manager.commit(handle), AssetOpStatus::Ok);
manager.apply_commits();
match kind {
BankType::GLYPH => {
let bank = banks.glyph_bank_slot(0).unwrap();
assert_eq!((bank.width, bank.height), (256, 128));
assert_eq!(
bank.pixel_indices,
(0..32768).map(|i| 1 + (i % 2) as u8).collect::<Vec<_>>()
);
assert!(bank.palettes[0].iter().all(|c| c.raw() == 0xffffffff));
}
BankType::SOUNDS => {
let bank = banks.sound_bank_slot(0).unwrap();
assert_eq!(bank.samples[0].sample_rate, 44100);
assert_eq!(bank.samples[0].data, vec![0x1212; 8192]);
}
BankType::SCENE => {
let bank = banks.scene_bank_slot(0).unwrap();
assert_eq!(
(bank.layers[0].tilemap.width, bank.layers[0].tilemap.height),
(4097, 1)
);
assert!(bank.layers[0].active);
assert_eq!(bank.layers[0].tilemap.tiles.len(), 4097);
assert!(bank.layers[0].tilemap.tiles.iter().all(|t| t.active
&& t.flip_x
&& t.flip_y
&& t.glyph.glyph_id == 0x1234
&& t.glyph.palette_id == 3));
assert!(bank.layers[1..].iter().all(|l| l.tilemap.tiles.is_empty()));
}
}
controller.stop().unwrap();
let observed = events.lock().unwrap();
assert_eq!(observed[..4].iter().map(|e| e.0).collect::<Vec<_>>(), vec!["asset"; 4]);
assert_eq!(observed[4].0, "save write");
let resumed = observed[4..].iter().position(|e| e.0 == "asset").unwrap() + 4;
assert_eq!(observed[resumed - 1].0, "save sync");
assert!(observed[resumed..].iter().all(|e| e.0 == "asset"));
let worker = observed[0].1;
assert_ne!(worker, thread::current().id());
assert!(observed.iter().all(|e| e.1 == worker));
drop(observed);
drop(session);
let mut recovered = Session::default();
let c = recovered.load_card(reboot(&disk), 1, 1, 1024 * 1024).unwrap();
let mut saved = [0; 64];
let n = recovered.read(c, 0, 0, &mut saved).unwrap();
assert_eq!(&saved[..n], b"persistent under asset pressure");
}
}
+4
View File
@@ -18,3 +18,7 @@ fpmath = { version = "=0.1.1", features = ["soft-float"] }
[dev-dependencies]
prometeu-test-support = { path = "../../dev/prometeu-test-support" }
[[example]]
name = "string_provenance_benchmark"
required-features = ["resource-diagnostics"]
@@ -0,0 +1,83 @@
//! Run in release with --features resource-diagnostics; query setup is excluded.
#[allow(dead_code)]
#[path = "../tests/support/callables.rs"]
mod callables;
use prometeu_bytecode::model::ConstantPoolEntry;
use prometeu_vm::string_provenance_qualification::Fixture;
use std::{
alloc::{GlobalAlloc, Layout, System},
sync::atomic::{AtomicUsize, Ordering},
time::Instant,
};
struct Counting;
static ALLOC: AtomicUsize = AtomicUsize::new(0);
static REALLOC: AtomicUsize = AtomicUsize::new(0);
static FREE: AtomicUsize = AtomicUsize::new(0);
unsafe impl GlobalAlloc for Counting {
unsafe fn alloc(&self, l: Layout) -> *mut u8 {
ALLOC.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(l) }
}
unsafe fn dealloc(&self, p: *mut u8, l: Layout) {
FREE.fetch_add(1, Ordering::Relaxed);
unsafe { System.dealloc(p, l) }
}
unsafe fn realloc(&self, p: *mut u8, l: Layout, n: usize) -> *mut u8 {
REALLOC.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(p, l, n) }
}
}
#[global_allocator]
static GLOBAL: Counting = Counting;
fn counts() -> [usize; 3] {
[ALLOC.load(Ordering::Relaxed), REALLOC.load(Ordering::Relaxed), FREE.load(Ordering::Relaxed)]
}
fn main() {
for n in [1024, 16384, 65536] {
let constants = vec![ConstantPoolEntry::String("same".into()); n];
let start = Instant::now();
let image = callables::image_constants(
&[("RET", callables::sig(&[], &[]), vec![])],
vec![],
&constants,
)
.unwrap();
let image_ns = start.elapsed().as_nanos();
let payload = image.payload_bytes();
let before = counts();
let start = Instant::now();
let mut fixture = Fixture::new(image, n);
let creation_ns = start.elapsed().as_nanos();
let after = counts();
println!(
"setup n={n} image_load_ns={image_ns} image_payload_bytes={payload} session_ns={creation_ns} session_allocs={} auxiliary_bytes={} work={:?}",
after[0] - before[0],
fixture.auxiliary_bytes(),
fixture.work()
);
for name in [
"image_hit",
"image_miss",
"session_hit",
"session_miss",
"import_hit",
"snapshot_mixed",
"import_miss",
] {
let before = counts();
let start = Instant::now();
for _ in 0..1000 {
fixture.run(name);
}
let ns = start.elapsed().as_nanos();
let after = counts();
let delta = std::array::from_fn::<_, 3, _>(|i| after[i] - before[i]);
if name.ends_with("hit") || name == "image_miss" || name == "session_miss" {
assert_eq!(delta, [0; 3]);
}
println!(
"query n={n} name={name} iterations=1000 ns={ns} alloc_realloc_free={delta:?}"
);
}
}
}
+12 -12
View File
@@ -1263,6 +1263,18 @@ pub(crate) fn metered_lookup_intrinsic(
Ok(None)
}
/// Runtime-owned catalogue, including input and scalar math.
pub fn intrinsic_catalog() -> impl Iterator<Item = &'static IntrinsicMeta> {
INTRINSICS
.iter()
.chain(crate::input_intrinsics::INPUT_INTRINSICS.iter())
.chain(crate::math_intrinsics::MATH_INTRINSICS.iter())
.chain(crate::random_intrinsics::RANDOM_INTRINSICS.iter())
}
pub fn builtin_constant_catalog() -> impl Iterator<Item = &'static BuiltinConstMeta> {
BUILTIN_CONSTS.iter()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1432,15 +1444,3 @@ mod tests {
assert_eq!(pixel.field("color").map(|field| field.start_slot), Some(2));
}
}
/// Runtime-owned catalogue, including input and scalar math.
pub fn intrinsic_catalog() -> impl Iterator<Item = &'static IntrinsicMeta> {
INTRINSICS
.iter()
.chain(crate::input_intrinsics::INPUT_INTRINSICS.iter())
.chain(crate::math_intrinsics::MATH_INTRINSICS.iter())
.chain(crate::random_intrinsics::RANDOM_INTRINSICS.iter())
}
pub fn builtin_constant_catalog() -> impl Iterator<Item = &'static BuiltinConstMeta> {
BUILTIN_CONSTS.iter()
}
+4
View File
@@ -25,6 +25,7 @@ mod program_image;
mod registry_verifier;
pub use program_image::{ImageStringLease, ValidatedProgramImage};
// Image/session creation and native provenance consume this in step 19.
mod string_identity_index;
#[allow(dead_code)]
mod string_ledger;
// Registry-backed consumers are integrated with executable images in PLN-0176.
@@ -52,3 +53,6 @@ pub use virtual_machine::{BudgetReport, LogicalFrameEndingReason, VirtualMachine
pub use vm_init_error::{LoaderPatchError, VmInitError};
pub use verifier::VerifierError;
#[cfg(feature = "resource-diagnostics")]
pub use virtual_machine::string_provenance_qualification;
@@ -1,5 +1,6 @@
//! PBX v1 publication: immutable metadata and shared payload, after bindings
//! and bounded whole-program verification. No public unchecked constructor.
use crate::string_identity_index::Identity;
use crate::{LoaderPatchError, VmInitError};
use prometeu_bytecode::isa::core::CoreOpCode as Op;
use prometeu_bytecode::load_limits::{
@@ -107,8 +108,30 @@ impl ValidatedProgramImage {
Some(ImageStringLease { content: content.clone(), domain: self.strings.clone() })
}
pub fn retains_string(&self, content: &Arc<str>) -> bool {
self.lookup_string(content, || {})
}
fn lookup_string(&self, content: &Arc<str>, mut compare: impl FnMut()) -> bool {
let strings = self.strings.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
strings.records.iter().any(|(weak, _)| std::ptr::eq(weak.as_ptr(), Arc::as_ptr(content)))
let target = Identity::of(content);
let mut low = 0;
let mut high = strings.records.len();
while low < high {
let mid = low + (high - low) / 2;
compare();
match record_identity(&strings.records[mid]).cmp(&target) {
std::cmp::Ordering::Less => low = mid + 1,
std::cmp::Ordering::Greater => high = mid,
std::cmp::Ordering::Equal => return true,
}
}
false
}
#[cfg(any(test, feature = "resource-diagnostics"))]
pub(crate) fn lookup_work(&self, content: &Arc<str>) -> (bool, usize) {
let mut work = 0;
let found = self.lookup_string(content, || work += 1);
(found, work)
}
pub fn constant_content(&self) -> (u64, usize) {
let mut strings = self.strings.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
@@ -205,6 +228,7 @@ impl ValidatedProgramImage {
};
values.push(value);
}
sort_string_records(&mut records, b)?;
b.refund(Q::TemporaryBytes, constants_capacity);
// Binding no longer needs source declarations. They were validated,
// used exactly, and can be released before final payload publication.
@@ -256,6 +280,48 @@ impl ValidatedProgramImage {
}
}
fn record_identity(record: &(Weak<str>, usize)) -> Identity {
Identity { address: record.0.as_ptr() as *const u8 as usize, length: record.1 }
}
// In-place heapsort: every record comparison (four field reads + comparison)
// and swap (two record reads + writes) is precharged. No scratch allocation.
fn sort_string_records(records: &mut [(Weak<str>, usize)], b: &mut LoadBudget) -> Result<(), E> {
fn sift(
records: &mut [(Weak<str>, usize)],
mut root: usize,
end: usize,
b: &mut LoadBudget,
) -> Result<(), E> {
while root < end / 2 {
let mut child = root * 2 + 1;
if child + 1 < end {
b.work(5)?;
if record_identity(&records[child]) < record_identity(&records[child + 1]) {
child += 1;
}
}
b.work(5)?;
if record_identity(&records[root]) >= record_identity(&records[child]) {
break;
}
b.work(4)?;
records.swap(root, child);
root = child;
}
Ok(())
}
for root in (0..records.len() / 2).rev() {
sift(records, root, records.len(), b)?;
}
for end in (1..records.len()).rev() {
b.work(4)?;
records.swap(0, end);
sift(records, 0, end, b)?;
}
Ok(())
}
pub(crate) fn scratch<T>(count: usize, b: &mut LoadBudget) -> Result<Vec<T>, E> {
let bytes = count
.checked_mul(size_of::<T>())
@@ -309,3 +309,65 @@ fn sealed_image_load_probe() {
elapsed
);
}
#[test]
fn string_identity_binary_search_preserves_constant_order_and_lease_indices() {
let constants = (0..1024).map(|i| K::String(format!("owner-{i}"))).collect::<Vec<_>>();
let bytes = fixture(&[1, 0], 0, &constants, None, &[]);
let image = ValidatedProgramImage::load(&bytes, &mut budget(), caps::NONE).unwrap();
for (i, value) in image.constants().iter().enumerate() {
let Value::String(text) = value else { panic!() };
assert_eq!(&**text, format!("owner-{i}"));
let (found, work) = image.lookup_work(text);
assert!(found && work <= 11, "binary work={work}");
assert!(Arc::ptr_eq(text, image.lease_constant(i).unwrap().content()));
assert!(!image.lookup_work(&Arc::from(text.as_ref())).0);
}
let lease = image.lease_constant(1023).unwrap();
drop(image);
assert_eq!(lease.retained_content(), (10, 1));
let domain = lease.domain.lock().unwrap();
assert_eq!(record_identity(&domain.records[0]), Identity::of(lease.content()));
}
#[test]
fn string_heap_sort_precharges_each_comparison_and_swap_without_scratch() {
let owners = (0..16).map(|_| Arc::<str>::from("same")).collect::<Vec<_>>();
let source = owners.iter().rev().map(|v| (Arc::downgrade(v), v.len())).collect::<Vec<_>>();
let mut full = budget();
let mut records = source.clone();
sort_string_records(&mut records, &mut full).unwrap();
assert!(records.windows(2).all(|v| record_identity(&v[0]) < record_identity(&v[1])));
assert_eq!(full.used(Q::TemporaryBytes), 0);
for limit in 0..full.used(Q::Work) {
let mut b = LoadBudget::new(LoadLimits { work: limit, ..Default::default() });
let mut records = source.clone();
let error = sort_string_records(&mut records, &mut b).unwrap_err();
assert_eq!(error.resource, Some(Q::Work));
assert!(b.used(Q::Work) <= limit);
assert_eq!(b.used(Q::TemporaryBytes), 0);
// Failed sort still owns exactly the original Weak identities.
records.sort_unstable_by_key(record_identity);
let mut expected = source.clone();
expected.sort_unstable_by_key(record_identity);
assert!(records.iter().zip(&expected).all(|(a, b)| Weak::ptr_eq(&a.0, &b.0)));
}
}
#[test]
fn string_sort_exhaustion_rolls_back_sealed_load_domains() {
let constants = vec![K::String("same".into()); 2];
let bytes = fixture(&[1, 0], 0, &constants, None, &[]);
let mut full = budget();
ValidatedProgramImage::load(&bytes, &mut full, caps::NONE).unwrap();
// Post-sort sharing costs 2 + 2 + 1 work units. Sorting two records
// costs 9 or 13 units, depending on their addresses. Removing another
// five leaves at most eight: failure is inside sorting for either order.
let mut b =
LoadBudget::new(LoadLimits { work: full.used(Q::Work) - 5 - 5, ..Default::default() });
let error = ValidatedProgramImage::load(&bytes, &mut b, caps::NONE).unwrap_err();
assert!(matches!(error,VmInitError::RegistryLoadFailed(e) if e.resource==Some(Q::Work)));
for resource in DOMAINS {
assert_eq!(b.used(resource), 0, "{resource:?}");
}
}
@@ -385,6 +385,10 @@ impl<T> BudgetVec<T> {
let bytes = b.bytes::<T>(capacity, q)?;
let mut extra = b.reserve(q, bytes - self.charge.amount)?;
let temporary = b.reserve(RuntimeResource::TemporaryBytes, self.charge.amount)?;
#[cfg(test)]
if FAIL_NEXT_VEC_ALLOCATION.with(|fail| fail.replace(false)) {
return Err(b.failure(q, ResourceReason::AllocationFailed, bytes));
}
let mut candidate = Vec::new();
candidate
.try_reserve_exact(capacity)
@@ -405,6 +409,21 @@ impl<T> BudgetVec<T> {
self.charge.absorb(growth.extra);
}
}
/// Rebuild a table with all fallible capacity/coexistence admission completed
/// before moving any old entry. The relocation callback must not allocate.
pub(crate) fn rebuild_with(
&mut self,
len: usize,
vacant: impl FnMut() -> T,
relocate: impl FnOnce(&mut [T], &mut [T]),
) -> Result<(), ResourceFailure> {
let mut growth = self.prepare_capacity(len.max(self.data.capacity()))?;
growth.candidate.resize_with(len, vacant);
relocate(&mut self.data, &mut growth.candidate);
self.data = growth.candidate;
self.charge.absorb(growth.extra);
Ok(())
}
pub fn reserve(&mut self, additional: usize) -> Result<(), ResourceFailure> {
let growth = self.prepare_growth(additional)?;
self.commit_growth(growth);
@@ -445,9 +464,6 @@ impl<T> BudgetVec<T> {
pub fn clear(&mut self) {
self.data.clear();
}
pub(crate) fn retain(&mut self, keep: impl FnMut(&T) -> bool) {
self.data.retain(keep);
}
pub(crate) fn extend_from_slice(&mut self, values: &[T]) -> Result<(), ResourceFailure>
where
T: Clone,
@@ -509,3 +525,8 @@ pub(crate) struct VecGrowth<T> {
_temporary: Reservation,
old_capacity: usize,
}
#[cfg(test)]
thread_local! {
pub(crate) static FAIL_NEXT_VEC_ALLOCATION: std::cell::Cell<bool> = const { std::cell::Cell::new(false) };
}
@@ -0,0 +1,123 @@
//! Private allocation identity index. Values retain Weak ownership, never content.
use crate::resource_budget::{
BudgetVec, ResourceFailure, ResourceReason, RuntimeBudget, RuntimeResource as Q,
};
use std::sync::Arc;
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub(crate) struct Identity {
pub(crate) address: usize,
pub(crate) length: usize,
}
impl Identity {
pub(crate) fn of(content: &Arc<str>) -> Self {
Self { address: Arc::as_ptr(content) as *const u8 as usize, length: content.len() }
}
fn hash(self) -> usize {
// Mix aligned addresses and lengths; collisions still compare both fields.
let mut n = (self.address as u64) ^ (self.length as u64).wrapping_mul(0x9e3779b97f4a7c15);
n = (n ^ (n >> 30)).wrapping_mul(0xbf58476d1ce4e5b9);
n = (n ^ (n >> 27)).wrapping_mul(0x94d049bb133111eb);
(n ^ (n >> 31)) as usize
}
}
#[derive(Debug)]
enum Bucket<T> {
Empty,
Tombstone,
Occupied(Identity, T),
}
#[derive(Debug)]
pub(crate) struct IdentityIndex<T> {
buckets: BudgetVec<Bucket<T>>,
live: usize,
tombstones: usize,
}
impl<T> IdentityIndex<T> {
pub(crate) fn new(budget: &RuntimeBudget) -> Self {
Self { buckets: BudgetVec::new(budget, Q::AuxiliaryBytes), live: 0, tombstones: 0 }
}
pub(crate) fn len(&self) -> usize {
self.live
}
pub(crate) fn contains(&self, key: Identity, mut probe: impl FnMut()) -> bool {
let size = self.buckets.len();
if size == 0 {
return false;
}
let mut at = key.hash() & (size - 1);
loop {
probe();
match &self.buckets[at] {
Bucket::Empty => return false,
Bucket::Occupied(found, _) if *found == key => return true,
_ => at = (at + 1) & (size - 1),
}
}
}
pub(crate) fn prepare(&mut self, budget: &RuntimeBudget) -> Result<(), ResourceFailure> {
let size = self.buckets.len();
let next = self.live + 1;
let capacity = if size == 0 {
8
} else if next > size / 2 {
size.checked_mul(2).ok_or_else(|| {
budget.failure(Q::AuxiliaryBytes, ResourceReason::SizeOverflow, u64::MAX)
})?
} else if next + self.tombstones > size / 2 {
size
} else {
return Ok(());
};
self.buckets.rebuild_with(
capacity,
|| Bucket::Empty,
|old, new| {
for slot in old {
if let Bucket::Occupied(key, value) = std::mem::replace(slot, Bucket::Empty) {
Self::place(new, key, value);
}
}
},
)?;
self.tombstones = 0;
Ok(())
}
fn place(buckets: &mut [Bucket<T>], key: Identity, value: T) -> bool {
let mut at = key.hash() & (buckets.len() - 1);
loop {
match &buckets[at] {
Bucket::Empty | Bucket::Tombstone => {
let reused = matches!(buckets[at], Bucket::Tombstone);
buckets[at] = Bucket::Occupied(key, value);
return reused;
}
Bucket::Occupied(found, _) => {
assert_ne!(*found, key, "duplicate allocation identity")
}
}
at = (at + 1) & (buckets.len() - 1);
}
}
pub(crate) fn insert(&mut self, key: Identity, value: T) {
assert!(self.live < self.buckets.len() / 2, "prepare must precede publication");
if Self::place(&mut self.buckets, key, value) {
self.tombstones -= 1;
}
self.live += 1;
}
pub(crate) fn retain(&mut self, mut keep: impl FnMut(&T) -> bool) {
for slot in &mut *self.buckets {
if matches!(slot, Bucket::Occupied(_, value) if !keep(value)) {
// Erase the key before releasing the Weak/reservations. Its
// allocation address cannot be recycled while Weak survives.
let removed = std::mem::replace(slot, Bucket::Tombstone);
self.live -= 1;
self.tombstones += 1;
drop(removed);
}
}
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,72 @@
use super::*;
#[test]
fn collision_chains_distinguish_full_identity_and_survive_tombstones() {
let b = RuntimeBudget::default();
let mut table = IdentityIndex::new(&b);
let keys: Vec<_> = (0..1000)
.map(|address| Identity { address, length: 4 })
.filter(|k| k.hash() & 7 == 0)
.take(4)
.collect();
for (i, key) in keys.iter().enumerate() {
table.prepare(&b).unwrap();
table.insert(*key, i);
}
for key in &keys {
assert!(table.contains(*key, || {}));
}
table.retain(|i| *i != 1);
assert!(!table.contains(keys[1], || {}));
assert!(table.contains(keys[3], || {}));
assert!(!table.contains(Identity { length: 5, ..keys[3] }, || {}));
table.prepare(&b).unwrap(); // Rebuild removes tombstones at half occupancy.
table.insert(keys[1], 1);
assert_eq!(table.len(), 4);
assert_eq!(table.tombstones, 0);
}
#[test]
fn tombstone_reuse_keeps_collision_successors_and_length_is_part_of_key() {
let b = RuntimeBudget::default();
let mut t = IdentityIndex::new(&b);
let key = Identity { address: 16, length: 0 };
t.prepare(&b).unwrap();
t.insert(key, 1);
t.prepare(&b).unwrap();
t.insert(Identity { length: 1, ..key }, 2);
t.retain(|v| *v != 1);
assert_eq!(t.tombstones, 1);
t.prepare(&b).unwrap();
t.insert(key, 3);
assert_eq!(t.tombstones, 0);
assert!(t.contains(key, || {}));
assert!(t.contains(Identity { length: 1, ..key }, || {}));
}
#[test]
fn controlled_probe_counts_and_half_load_survive_growth_and_rebuild() {
let b = RuntimeBudget::default();
let mut table = IdentityIndex::new(&b);
for n in 0..1024 {
table.prepare(&b).unwrap();
table.insert(Identity { address: n * 16, length: n % 7 }, n);
assert!(table.live * 2 <= table.buckets.len());
}
let mut max_probes = 0;
for n in 0..2048 {
let mut probes = 0;
assert_eq!(
table.contains(Identity { address: n * 16, length: n % 7 }, || probes += 1),
n < 1024
);
max_probes = max_probes.max(probes);
}
// A reproducible synthetic-key bound for this fixture, not a hash-table
// worst-case claim or a constraint on allocator-selected real addresses.
assert!(max_probes <= 32, "fixture probes={max_probes}");
table.retain(|n| n % 2 == 0);
table.prepare(&b).unwrap();
for n in 0..1024 {
assert_eq!(table.contains(Identity { address: n * 16, length: n % 7 }, || {}), n % 2 == 0);
}
}
+23 -11
View File
@@ -4,6 +4,7 @@
use crate::resource_budget::{
BudgetVec, Reservation, ResourceFailure, ResourceReason, RuntimeBudget, RuntimeResource as Q,
};
use crate::string_identity_index::{Identity, IdentityIndex};
use prometeu_bytecode::Value;
use std::sync::{Arc, Mutex, MutexGuard, Weak};
@@ -17,7 +18,7 @@ struct Entry {
#[derive(Debug)]
struct Domain {
budget: RuntimeBudget,
entries: BudgetVec<Entry>,
entries: IdentityIndex<Entry>,
_state: Reservation,
}
@@ -49,7 +50,7 @@ impl StringLedger {
Q::AuxiliaryBytes,
budget.bytes::<Mutex<Domain>>(1, Q::AuxiliaryBytes)? + 2 * size_of::<usize>() as u64,
)?;
let entries = BudgetVec::new(&budget, Q::AuxiliaryBytes);
let entries = IdentityIndex::new(&budget);
Ok(Self(Arc::new(Mutex::new(Domain { budget, entries, _state: state }))))
}
fn domain(&self) -> MutexGuard<'_, Domain> {
@@ -57,7 +58,7 @@ impl StringLedger {
}
/// Safepoint reconciliation is bounded by the accepted allocation count.
/// Retain compacts in place; unused capacity remains charged until drop.
/// Dead buckets become tombstones; retained capacity stays charged until drop.
pub(crate) fn reconcile(&self) -> usize {
self.domain().reconcile()
}
@@ -65,9 +66,19 @@ impl StringLedger {
self.domain().entries.len()
}
pub(crate) fn retains(&self, content: &Arc<str>) -> bool {
let address = Arc::as_ptr(content);
self.domain().entries.iter().any(|entry| std::ptr::eq(entry.content.as_ptr(), address))
self.lookup(content, || {})
}
fn lookup(&self, content: &Arc<str>, mut probe: impl FnMut()) -> bool {
self.domain().entries.contains(Identity::of(content), &mut probe)
}
#[cfg(any(test, feature = "resource-diagnostics"))]
pub(crate) fn lookup_work(&self, content: &Arc<str>) -> (bool, usize) {
let mut work = 0;
let found = self.lookup(content, || work += 1);
(found, work)
}
pub(crate) fn lease(&self, content: &Arc<str>) -> Option<StringLease> {
self.retains(content)
.then(|| StringLease { content: content.clone(), domain: self.clone() })
@@ -239,23 +250,24 @@ impl Domain {
}
Err(failure) => return Err(failure),
};
if let Err(failure) = self.entries.reserve(1) {
if let Err(failure) = self.entries.prepare(&self.budget) {
if failure.reason != ResourceReason::Limit {
return Err(failure);
}
self.reconcile();
self.entries.reserve(1)?;
self.entries.prepare(&self.budget)?;
}
Ok(charges)
}
fn publish(&mut self, content: &Arc<str>, charges: Charges) {
self.entries
.push(Entry {
self.entries.insert(
Identity::of(content),
Entry {
content: Arc::downgrade(content),
_bytes: charges.bytes,
_allocation: charges.allocation,
})
.expect("prepare reserved one ledger record before materialization");
},
);
}
}
@@ -259,3 +259,95 @@ fn formatting_refuses_result_and_temporary_limits_with_no_content_publication()
assert_eq!(budget.used(Q::StringAllocations), 0);
assert_eq!(ledger.tracked_allocations(), 0);
}
#[test]
fn index_growth_refuses_auxiliary_temporary_and_allocator_before_materialization() {
for failure in [0, 1, 2] {
let (budget, ledger) = fixture(&[]);
let kept = (0..4).map(|_| ledger.import("kept").unwrap()).collect::<Vec<_>>();
let auxiliary = budget.used(Q::AuxiliaryBytes);
match failure {
0 => budget.test_lower_limit(Q::AuxiliaryBytes, auxiliary),
1 => budget.test_lower_limit(Q::TemporaryBytes, 0),
_ => crate::resource_budget::FAIL_NEXT_VEC_ALLOCATION.with(|fail| fail.set(true)),
}
let mut called = false;
let result = ledger.build(4, &mut |w| {
called = true;
w.write_str("next")
});
let Err(MaterializeError::Resource(error)) = result else {
panic!("expected prepublication refusal")
};
assert!(!called);
assert_eq!(
error.resource,
if failure == 1 { Q::TemporaryBytes } else { Q::AuxiliaryBytes }
);
assert_eq!(
error.reason,
if failure == 2 { ResourceReason::AllocationFailed } else { ResourceReason::Limit }
);
assert_eq!(ledger.tracked_allocations(), 4);
assert_eq!(budget.used(Q::StringBytes), 16);
assert_eq!(budget.used(Q::StringAllocations), 4);
assert_eq!(budget.used(Q::AuxiliaryBytes), auxiliary);
assert_eq!(budget.used(Q::TemporaryBytes), 0);
for value in &kept {
assert!(ledger.retains(value));
}
}
}
#[test]
fn index_initial_allocation_failure_and_rebuild_failure_refund_reservations() {
let (budget, ledger) = fixture(&[]);
let initial = budget.used(Q::AuxiliaryBytes);
crate::resource_budget::FAIL_NEXT_VEC_ALLOCATION.with(|fail| fail.set(true));
assert_eq!(ledger.import("x").unwrap_err().reason, ResourceReason::AllocationFailed);
assert_eq!(ledger.tracked_allocations(), 0);
assert_eq!(budget.used(Q::AuxiliaryBytes), initial);
assert_eq!(budget.used(Q::StringBytes), 0);
assert_eq!(budget.used(Q::StringAllocations), 0);
let mut kept = (0..4).map(|_| ledger.import("same").unwrap()).collect::<Vec<_>>();
let old = Arc::downgrade(&kept.pop().unwrap());
assert_eq!(ledger.reconcile(), 1);
let auxiliary = budget.used(Q::AuxiliaryBytes);
budget.test_lower_limit(Q::TemporaryBytes, 0);
assert_eq!(ledger.import("new").unwrap_err().resource, Q::TemporaryBytes);
assert_eq!(ledger.tracked_allocations(), 3);
assert_eq!(budget.used(Q::AuxiliaryBytes), auxiliary);
assert_eq!(budget.used(Q::TemporaryBytes), 0);
assert_eq!(budget.used(Q::StringBytes), 12);
for value in &kept {
assert_eq!(Arc::strong_count(value), 1);
assert!(ledger.retains(value));
}
assert!(old.upgrade().is_none());
// Rebuild under a restored test budget; no production ceiling changes.
budget.test_lower_limit(Q::TemporaryBytes, 64 * 1024 * 1024);
let newer = ledger.import("same").unwrap();
assert!(ledger.retains(&newer));
assert!(!ledger.retains(&Arc::from("same")));
assert!(ledger.lookup_work(&newer).0);
}
#[test]
fn tombstone_rebuild_allocator_refusal_preserves_live_keys_and_charges() {
let (budget, ledger) = fixture(&[]);
let mut owners = (0..4).map(|_| ledger.import("same").unwrap()).collect::<Vec<_>>();
owners.pop();
assert_eq!(ledger.reconcile(), 1);
let auxiliary = budget.used(Q::AuxiliaryBytes);
crate::resource_budget::FAIL_NEXT_VEC_ALLOCATION.with(|fail| fail.set(true));
assert_eq!(ledger.import("new").unwrap_err().reason, ResourceReason::AllocationFailed);
assert_eq!(ledger.tracked_allocations(), 3);
assert_eq!(budget.used(Q::StringBytes), 12);
assert_eq!(budget.used(Q::StringAllocations), 3);
assert_eq!(budget.used(Q::AuxiliaryBytes), auxiliary);
assert_eq!(budget.used(Q::TemporaryBytes), 0);
assert!(owners.iter().all(|v| ledger.retains(v)));
let new = ledger.import("new").unwrap();
assert!(ledger.retains(&new));
assert_eq!(budget.used(Q::AuxiliaryBytes), auxiliary);
}
@@ -3890,3 +3890,6 @@ mod tests {
#[cfg(test)]
#[path = "virtual_machine/callable_tests.rs"]
mod callable_tests;
#[cfg(feature = "resource-diagnostics")]
pub mod string_provenance_qualification;
@@ -331,3 +331,45 @@ fn recursive_fn_calls_use_bounded_ordinary_frames() {
);
assert_eq!(v.call_stack.len(), 4);
}
#[test]
fn root_return_scratch_refusal_keeps_transferred_roots_without_partial_sweep() {
let mut v = vm(&[
("MAKE_CLOSURE fn=1,type=4\nPOP\nMAKE_CLOSURE fn=1,type=4\nRET", sig(&[], &[4]), vec![]),
("PUSH_I32 55\nRET", sig(&[], &[0]), vec![]),
]);
v.gc_alloc_threshold = 1;
for _ in 0..3 {
v.step(&mut Native, &mut HostContext::new(None)).unwrap();
}
let Value::HeapRef(owner) = *v.operand_stack.last().unwrap() else { panic!() };
let live = v.heap.len();
v.budget.test_lower_limit(RuntimeResource::TemporaryBytes, 0);
assert!(matches!(v.step(&mut Native, &mut HostContext::new(None)),
Err(LogicalFrameEndingReason::Trap(t)) if t.code == TRAP_RESOURCE_LIMIT));
assert!(v.call_stack.is_empty());
assert_eq!(v.operand_stack.as_slice(), &[Value::HeapRef(owner)]);
assert!(v.heap.is_valid(owner));
assert_eq!(v.heap.len(), live);
v.budget.test_lower_limit(RuntimeResource::TemporaryBytes, 1024 * 1024);
v.handle_safepoint().unwrap();
assert_eq!(v.heap.len(), 1);
assert!(v.heap.is_valid(owner));
}
#[test]
fn bad_return_height_preserves_frame_and_owners() {
let mut v = vm(&[
("MAKE_CLOSURE fn=1,type=4\nRET", sig(&[], &[4]), vec![]),
("PUSH_I32 55\nRET", sig(&[], &[0]), vec![]),
]);
v.step(&mut Native, &mut HostContext::new(None)).unwrap();
v.operand_stack.push(Value::Int32(9)).unwrap();
let before = v.operand_stack.as_slice().to_vec();
assert!(matches!(v.step(&mut Native, &mut HostContext::new(None)),
Err(LogicalFrameEndingReason::Trap(t)) if t.code == prometeu_bytecode::TRAP_BAD_RET_SLOTS));
assert_eq!(v.call_stack.len(), 1);
assert_eq!(v.operand_stack.as_slice(), before);
let Value::HeapRef(owner) = before[before.len() - 2] else { panic!() };
assert!(v.heap.is_valid(owner));
}
@@ -46,21 +46,13 @@ impl VirtualMachine {
)
.map_err(|(_, error)| self.value_trap(error, opcode, pc))?;
// Copy return values (preserving order: pop return_slots values, then reverse to push back)
let mut return_vals = BudgetVec::new(&self.budget, RuntimeResource::TemporaryBytes);
return_vals.reserve(return_slots).map_err(|e| self.resource_trap(e, opcode, pc))?;
for _ in 0..return_slots {
return_vals
.push(self.pop().map_err(LogicalFrameEndingReason::Panic)?)
.map_err(|e| self.resource_trap(e, opcode, pc))?;
// Validation is complete. Move the ordered result suffix ahead of this
// frame's parameters/locals without cloning owners or reserving scratch.
if return_slots != 0 {
self.operand_stack[frame.stack_base..].rotate_right(return_slots);
}
return_vals.reverse();
self.operand_stack.truncate(frame.stack_base + return_slots);
self.call_stack.pop();
self.operand_stack.truncate(frame.stack_base);
for val in return_vals.drain(..) {
self.push_trap(val, opcode, pc)?;
}
self.pc = frame.return_pc as usize;
if self.call_stack.is_empty() {
self.handle_safepoint().map_err(|e| self.resource_trap(e, opcode, pc))?;
@@ -62,7 +62,7 @@ impl VirtualMachine {
if self.temporary_slots() == 0 {
return Err("Stack underflow".into());
}
self.operand_stack.pop().ok_or("Stack underflow".into())
self.operand_stack.pop().ok_or_else(|| "Stack underflow".into())
}
pub fn pop_number(&mut self) -> Result<f64, String> {
@@ -82,7 +82,7 @@ impl VirtualMachine {
if self.temporary_slots() == 0 {
return Err("Stack underflow".into());
}
self.operand_stack.last().ok_or("Stack underflow".into())
self.operand_stack.last().ok_or_else(|| "Stack underflow".into())
}
pub(super) fn pop_trap<S: Into<String>>(
@@ -112,8 +112,22 @@ impl VirtualMachine {
where
F: FnOnce(Value, Value) -> Result<Value, OpError>,
{
let b = self.pop_trap(opcode, start_pc, format!("{:?} requires two operands", opcode))?;
let a = self.pop_trap(opcode, start_pc, format!("{:?} requires two operands", opcode))?;
let b = self.pop().map_err(|_| {
self.trap(
TRAP_STACK_UNDERFLOW,
opcode as u16,
format!("{:?} requires two operands", opcode),
start_pc,
)
})?;
let a = self.pop().map_err(|_| {
self.trap(
TRAP_STACK_UNDERFLOW,
opcode as u16,
format!("{:?} requires two operands", opcode),
start_pc,
)
})?;
match f(a, b) {
Ok(res) => {
self.push_trap(res, opcode, start_pc)?;
@@ -0,0 +1,81 @@
//! Opt-in diagnostics for the actual native provenance boundaries.
use super::*;
use crate::ValidatedProgramImage;
use std::{hint::black_box, sync::Arc};
pub struct Fixture {
vm: VirtualMachine,
owners: Vec<Arc<str>>,
image_hit: Arc<str>,
miss: Arc<str>,
}
impl Fixture {
pub fn new(image: Arc<ValidatedProgramImage>, count: usize) -> Self {
let Value::String(image_hit) = image.constants().last().unwrap() else { panic!() };
let image_hit = image_hit.clone();
let mut vm = VirtualMachine::from_image(image, Default::default());
let ledger = crate::string_ledger::StringLedger::new(vm.budget.clone()).unwrap();
let owners = (0..count).map(|_| ledger.import("same").unwrap()).collect::<Vec<_>>();
vm.strings = Some(ledger);
vm.operand_stack.push(Value::String(image_hit.clone())).unwrap();
vm.operand_stack.push(Value::String(owners.last().unwrap().clone())).unwrap();
Self { vm, owners, image_hit, miss: Arc::from("same") }
}
pub fn auxiliary_bytes(&self) -> u64 {
self.vm.budget.used(RuntimeResource::AuxiliaryBytes)
}
pub fn work(&self) -> [usize; 4] {
let image = self.vm.validated_image.as_ref().unwrap();
let session = self.vm.strings.as_ref().unwrap();
let (ih, iw) = image.lookup_work(&self.image_hit);
let (im, mw) = image.lookup_work(&self.miss);
let (sh, sw) = session.lookup_work(self.owners.last().unwrap());
let (sm, tw) = session.lookup_work(&self.miss);
assert!(ih && sh && !im && !sm);
[iw, mw, sw, tw]
}
pub fn run(&mut self, workload: &str) {
match workload {
"image_hit" => {
black_box(
self.vm
.validated_image
.as_ref()
.unwrap()
.retains_string(black_box(&self.image_hit)),
);
}
"image_miss" => {
black_box(
self.vm.validated_image.as_ref().unwrap().retains_string(black_box(&self.miss)),
);
}
"session_hit" => {
black_box(
self.vm
.strings
.as_ref()
.unwrap()
.retains(black_box(self.owners.last().unwrap())),
);
}
"session_miss" => {
black_box(self.vm.strings.as_ref().unwrap().retains(black_box(&self.miss)));
}
"import_hit" => {
black_box(
self.vm
.import_string_value(&Value::String(self.owners.last().unwrap().clone()))
.unwrap(),
);
}
"import_miss" => {
black_box(self.vm.import_string_value(&Value::String(self.miss.clone())).unwrap());
}
"snapshot_mixed" => {
black_box(self.vm.operand_stack_top(2));
}
_ => panic!("unknown workload"),
}
}
}
@@ -699,28 +699,28 @@ fn ordered_multi_group_return_preserves_native_owners_and_receiver_identity() {
}
#[test]
fn unknown_return_and_late_return_resource_failures_keep_current_frame_and_operands() {
fn unknown_return_refusal_keeps_frame_but_valid_return_needs_no_scratch() {
let mut code = vec![];
arg(&mut code, OpCode::GetGlobal, 0);
op(&mut code, OpCode::Ret);
let image = image(&code, vec![function(0, code.len(), &[], &[0], 0, 1)]);
for (value, limits, expected) in [
(Value::Boolean(true), RuntimeLimits::default(), TRAP_TYPE),
(
Value::Int32(3),
RuntimeLimits::default().lowered(RuntimeResource::TemporaryBytes, 0).unwrap(),
TRAP_RESOURCE_LIMIT,
),
] {
for value in [Value::Boolean(true), Value::Int32(3)] {
let limits = RuntimeLimits::default().lowered(RuntimeResource::TemporaryBytes, 0).unwrap();
let mut vm = VirtualMachine::from_image(image.clone(), limits);
vm.prepare_boot_call().unwrap();
vm.globals.push(value.clone()).unwrap();
let result = run(&mut vm);
assert!(
matches!(result,LogicalFrameEndingReason::Trap(info) if info.code==expected && info.pc==6)
);
assert_eq!(vm.call_stack.len(), 1);
if matches!(value, Value::Boolean(_)) {
assert!(
matches!(result, LogicalFrameEndingReason::Trap(info) if info.code == TRAP_TYPE && info.pc == 6)
);
assert_eq!(vm.call_stack.len(), 1);
} else {
assert_eq!(result, LogicalFrameEndingReason::EndOfRom);
assert!(vm.call_stack.is_empty());
}
assert_eq!(vm.operand_stack.as_slice(), &[value]);
assert_eq!(vm.budget.used(RuntimeResource::TemporaryBytes), 0);
}
}
@@ -0,0 +1,136 @@
//! Count only this thread's warmed instruction region, including reallocations.
use prometeu_bytecode::{assemble, load_limits::LoadBudget, registry::*};
use prometeu_hal::{HostContext, syscalls::caps};
use prometeu_test_support::program::{TestProgram, int64_signature};
use prometeu_vm::{ValidatedProgramImage, VirtualMachine};
use std::alloc::{GlobalAlloc, Layout, System};
use std::cell::Cell;
struct Native;
impl prometeu_hal::NativeInterface for Native {
fn syscall(
&mut self,
_: u32,
_: &[prometeu_bytecode::Value],
_: &mut prometeu_hal::HostReturn,
_: &mut HostContext,
) -> Result<(), prometeu_hal::vm_fault::VmFault> {
panic!("no syscalls in allocation fixtures")
}
}
thread_local! {
static COUNTS: Cell<Option<(usize, usize)>> = const { Cell::new(None) };
}
struct Counter;
fn count(alloc: usize, free: usize) {
let _ = COUNTS.try_with(|c| {
if let Some((a, d)) = c.get() {
c.set(Some((a + alloc, d + free)));
}
});
}
unsafe impl GlobalAlloc for Counter {
unsafe fn alloc(&self, l: Layout) -> *mut u8 {
count(1, 0);
unsafe { System.alloc(l) }
}
unsafe fn alloc_zeroed(&self, l: Layout) -> *mut u8 {
count(1, 0);
unsafe { System.alloc_zeroed(l) }
}
unsafe fn dealloc(&self, p: *mut u8, l: Layout) {
count(0, 1);
unsafe { System.dealloc(p, l) }
}
unsafe fn realloc(&self, p: *mut u8, l: Layout, n: usize) -> *mut u8 {
count(1, 1);
unsafe { System.realloc(p, l, n) }
}
}
#[global_allocator]
static ALLOCATOR: Counter = Counter;
#[test]
fn warmed_instruction_regions_do_not_allocate() {
let mut observations = vec![];
for (name, caller, callee, results, steps) in [
("scalar", "CALL 1\nPOP\nJMP 0", "PUSH_I64 42\nRET", 1, 5),
("void", "CALL 1\nJMP 0", "RET", 0, 3),
("multi", "CALL 1\nPOP\nPOP\nJMP 0", "PUSH_I64 11\nPUSH_I64 22\nRET", 2, 7),
("arithmetic_peek", "PUSH_I64 3\nDUP\nADD\nPOP\nJMP 0", "RET", 0, 5),
] {
let mut code = assemble(caller).unwrap();
let offset = code.len();
let body = assemble(callee).unwrap();
let mut program = TestProgram::single(code.clone(), int64_signature(0, 0), 4, 8);
program.functions.push(FunctionDecl {
signature: int64_signature(0, results),
code_offset: offset as u32,
code_len: body.len() as u32,
local_slots: 4,
max_stack_slots: 8,
..Default::default()
});
code.extend(body);
program.code = code;
let image = ValidatedProgramImage::load(
&program.serialize(),
&mut LoadBudget::new(Default::default()),
caps::NONE,
)
.unwrap();
let mut vm = VirtualMachine::from_image(image, Default::default());
vm.prepare_boot_call().unwrap();
let mut host = HostContext::new(None);
for _ in 0..steps * 10 {
vm.step(&mut Native, &mut host).unwrap();
}
COUNTS.with(|c| c.set(Some((0, 0))));
for _ in 0..steps * 1000 {
vm.step(&mut Native, &mut host).unwrap();
}
let counts = COUNTS.with(|c| c.replace(None).unwrap());
println!("{name}: 1000 iterations, {counts:?} allocations/deallocations");
observations.push((name, counts));
}
for (name, counts) in observations {
assert_eq!(counts, (0, 0), "{name}");
}
}
#[cfg(feature = "resource-diagnostics")]
#[allow(dead_code)]
#[path = "support/callables.rs"]
mod provenance_callables;
#[cfg(feature = "resource-diagnostics")]
#[test]
fn provenance_queries_allocate_nothing_at_all_qualification_sizes() {
use prometeu_bytecode::model::ConstantPoolEntry;
use prometeu_vm::string_provenance_qualification::Fixture;
for count in [1024_usize, 16384, 65536] {
let constants = vec![ConstantPoolEntry::String("same".into()); count];
let image = provenance_callables::image_constants(
&[("RET", provenance_callables::sig(&[], &[]), vec![])],
vec![],
&constants,
)
.unwrap();
let mut fixture = Fixture::new(image, count);
let work = fixture.work();
let bound = count.ilog2() as usize + 1;
assert!(work[0] <= bound && work[1] <= bound);
for workload in ["image_hit", "image_miss", "session_hit", "session_miss", "import_hit"] {
COUNTS.with(|c| c.set(Some((0, 0))));
for _ in 0..100 {
fixture.run(workload);
}
let counts = COUNTS.with(|c| c.replace(None)).unwrap();
assert_eq!(counts, (0, 0), "{workload}, owners={count}");
}
COUNTS.with(|c| c.set(Some((0, 0))));
fixture.run("snapshot_mixed");
let counts = COUNTS.with(|c| c.replace(None)).unwrap();
assert_eq!(counts, (1, 1), "only snapshot Value storage allocates");
}
}
@@ -216,3 +216,47 @@ fn integer_division_and_remainder_overflow_wrap_without_host_panic() {
assert_eq!(machine.operand_stack_top(1), vec![Value::Int64(expected)]);
}
}
#[test]
fn maximum_width_mixed_return_preserves_order_and_caller_locals() {
// TestProgram catalog: Int64, Vec2, Float = 0, 6, 2.
let mut caller = String::from("PUSH_I64 73\nSET_LOCAL 0\nCALL 1\n");
caller.push_str(&"POP\n".repeat(1024));
caller.push_str("GET_LOCAL 0\nRET");
let mut callee = String::new();
let mut results = vec![];
let mut expected = vec![];
for i in 0..256 {
callee.push_str(&format!("PUSH_I64 {i}\nPUSH_F64 f64:0x3ff0000000000000\nPUSH_F64 f64:0x4000000000000000\nPUSH_F64 f64:0x4008000000000000\n"));
results.extend([TypeId(0), TypeId(6), TypeId(2)]);
expected.extend([Value::Int64(i), Value::Float(1.), Value::Float(2.), Value::Float(3.)]);
}
callee.push_str("RET");
let caller_code = assemble(&caller).unwrap();
let body = assemble(&callee).unwrap();
let mut program = TestProgram::single(caller_code.clone(), int64_signature(0, 1), 1, 1024);
program.functions.push(FunctionDecl {
signature: Signature { parameters: vec![], results },
code_offset: caller_code.len() as u32,
code_len: body.len() as u32,
local_slots: 3,
max_stack_slots: 1024,
..Default::default()
});
program.code.extend(body);
let image = ValidatedProgramImage::load(
&program.serialize(),
&mut LoadBudget::new(Default::default()),
caps::NONE,
)
.unwrap();
let mut machine = VirtualMachine::from_image(image, Default::default());
machine.prepare_boot_call().unwrap();
for _ in 0..3 + 1024 + 1 {
machine.step(&mut Native, &mut HostContext::new(None)).unwrap();
}
expected.reverse();
assert_eq!(machine.operand_stack_top(1024), expected);
assert_eq!(run(&mut machine, 10000).reason, LogicalFrameEndingReason::EndOfRom);
assert_eq!(machine.operand_stack_top(1), vec![Value::Int64(73)]);
}
@@ -61,7 +61,9 @@ fn integer_limits_are_exact_and_abs_does_not_wrap() {
] {
assert_eq!(call(base, &[b.clone(), a.clone()]).unwrap(), a);
assert_eq!(call(base + 1, &[a.clone(), b.clone()]).unwrap(), b);
assert!(matches!(call(base+3,&[a.clone()]),Err(E::Numeric(e)) if e.code==RANGE));
assert!(
matches!(call(base+3,std::slice::from_ref(&a)),Err(E::Numeric(e)) if e.code==RANGE)
);
assert!(matches!(call(base+2,&[b.clone(),b,a]),Err(E::Numeric(e)) if e.code==DOMAIN));
}
assert_eq!(
@@ -45,7 +45,7 @@ fn independent_words_signed_and_float_bits_for_all_seed_patterns() {
});
}
code.push_str("RET");
let mut v = vm(&[(&code, sig(&[], &vec![if floating { 2 } else { 0 }; 8]), vec![], 1)]);
let mut v = vm(&[(&code, sig(&[], &[if floating { 2 } else { 0 }; 8]), vec![], 1)]);
let r = run(&mut v, 10000);
assert_eq!(r.reason, End::EndOfRom);
let expected: Vec<_> = vectors(seed)
@@ -114,6 +114,111 @@ fn repository_sources_have_exactly_one_coverage_owner() {
);
}
fn repository_policy(script: &str) -> Output {
Command::new("python3")
.arg("-B")
.arg("-c")
.arg(format!(
r#"
import importlib.util
import json
import sys
from pathlib import Path
root = Path(sys.argv[1])
spec = importlib.util.spec_from_file_location("coverage_auditor", root / "scripts/audit-coverage-domains.py")
auditor = importlib.util.module_from_spec(spec)
spec.loader.exec_module(auditor)
config = json.loads((root / "files/config/runtime-edge-coverage-domains.json").read_text())
domains = config["domains"]
trees = [
("crates/console/prometeu-system/src/services/memcard", "system/fs", "system/core"),
("crates/console/prometeu-drivers/src/asset", "asset/bank", "drivers/core"),
]
{script}
"#
))
.arg(repository())
.output()
.unwrap()
}
#[test]
fn specialized_entry_files_and_all_descendants_have_the_expected_owner() {
success(repository_policy(
r#"
for stem, expected, residual in trees:
names = [stem + ".rs"]
descendants = sorted((root / stem).rglob("*.rs"))
assert descendants, stem
names += [path.relative_to(root).as_posix() for path in descendants]
names += [stem + "/future.rs", stem + "/future/nested.rs"]
for name in names:
actual = auditor.owners(name, domains)
assert actual == [expected], (name, expected, actual)
# Exclusive residual fallback would pass the generic audit but is wrong.
import copy
fallback = copy.deepcopy(domains)
fallback[expected]["paths"].remove(stem + "/")
fallback[residual]["exclude_paths"].remove(stem + "/")
name = names[-1]
assert auditor.owners(name, fallback) == [residual]
assert auditor.owners(name, fallback) != [expected]
overlap = copy.deepcopy(domains)
overlap[residual]["exclude_paths"].remove(stem + "/")
assert set(auditor.owners(name, overlap)) == {expected, residual}
"#,
));
}
#[test]
fn specialized_ownership_survives_absent_and_zero_instrumentation() {
success(repository_policy(
r#"
import tempfile
with tempfile.TemporaryDirectory(prefix="prometeu-domain-trees-") as directory:
fixture = Path(directory)
records = []
# Use the real map and source layout, with independent synthetic evidence.
for source in (root / "crates").glob("**/src/**/*.rs"):
name = source.relative_to(root).as_posix()
path = fixture / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("// ownership fixture\n")
records.append({"filename": name, "summary": {"lines": {"count": 1, "covered": 1}}})
for stem, expected, residual in trees:
for suffix in ["future/absent.rs", "future/zero.rs"]:
path = fixture / stem / suffix
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("// no executable lines\n")
records.append({"filename": stem + "/future/zero.rs", "summary": {"lines": {"count": 0, "covered": 0}}})
config_path = fixture / "config.json"
config_path.write_text(json.dumps(config))
report_path = fixture / "report.json"
report_path.write_text(json.dumps({"data": [{"files": records}]}))
before = auditor.audit(root, root / "files/config/runtime-edge-coverage-domains.json")
result = auditor.audit(fixture, config_path, report_path)
assert len(result["domains"]) == 15
assert all(domain["gate_passed"] for domain in result["domains"].values())
for stem, expected, residual in trees:
domain = result["domains"][expected]
assert domain["source_files"] == before["domains"][expected]["source_files"] + 2
assert stem + "/future/zero.rs" in domain["files"]
assert stem + "/future/absent.rs" not in domain["files"]
assert result["domains"][residual]["source_files"] == before["domains"][residual]["source_files"]
# Missing residual exclusions must fail even for sources absent from reports.
for stem, expected, residual in trees:
domains[residual]["exclude_paths"].remove(stem + "/")
config_path.write_text(json.dumps(config))
try:
auditor.audit(fixture, config_path, report_path)
except auditor.AuditError as error:
assert "expected exactly one domain" in str(error), error
else:
raise AssertionError("overlapping module trees passed ownership audit")
"#,
));
}
#[test]
fn exclusions_and_absolute_relative_report_names_have_exclusive_ownership() {
let fixture = Fixture::new();
@@ -1,3 +1,7 @@
mod reclamation;
pub use reclamation::AudioReclaimer;
use reclamation::{CallbackLane, Packet};
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use prometeu_drivers::{AudioCommand, Channel, MAX_CHANNELS, OUTPUT_SAMPLE_RATE};
use prometeu_drivers::{AudioDelivery, AudioTransport};
@@ -10,9 +14,15 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant};
trait AudioStreamHandle {}
trait AudioStreamHandle {
fn stop(&self) {}
}
impl AudioStreamHandle for cpal::Stream {}
impl AudioStreamHandle for cpal::Stream {
fn stop(&self) {
let _ = self.pause();
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum HostAudioError {
@@ -97,6 +107,7 @@ pub struct HostAudio {
last_progress: Instant,
pub perf_consumer: Option<ringbuf::wrap::CachingCons<Arc<HeapRb<u64>>>>,
_stream: Option<Box<dyn AudioStreamHandle>>,
reclaimer: Option<AudioReclaimer>,
}
impl Default for HostAudio {
@@ -115,6 +126,7 @@ impl HostAudio {
last_progress: Instant::now(),
perf_consumer: None,
_stream: None,
reclaimer: None,
}
}
@@ -129,7 +141,7 @@ impl HostAudio {
self.last_heartbeat = 0;
self.last_progress = Instant::now();
self.perf_consumer = None;
self._stream = None;
self.stop_stream();
let device = runtime.default_output_device().ok_or(HostAudioError::NoOutputDevice)?;
@@ -143,7 +155,7 @@ impl HostAudio {
let failed = self.failed.clone();
let heartbeat = self.heartbeat.clone();
let mut mixer = AudioMixer::new();
let (mut callback, reclaimer) = AudioCallback::new();
// To pass performance data from the audio thread to the main thread
let audio_perf_rb = HeapRb::<u64>::new(64);
@@ -154,13 +166,9 @@ impl HostAudio {
&config,
move |data: &mut [f32]| {
heartbeat.fetch_add(1, Ordering::Relaxed);
if let Some(delivery) = transport.try_take() {
mixer.apply_delivery(delivery);
}
// Mixes audio
mixer.fill_buffer(data);
callback.render(&transport, data);
// Sends processing time in microseconds
let _ = perf_prod.try_push(mixer.last_processing_time.as_micros() as u64);
let _ = perf_prod.try_push(callback.mixer.last_processing_time.as_micros() as u64);
},
move |err| {
failed.store(true, Ordering::Relaxed);
@@ -169,6 +177,7 @@ impl HostAudio {
)?;
runtime.play_stream(&stream)?;
self.reclaimer = Some(reclaimer);
self.perf_consumer = Some(perf_cons);
self._stream = Some(Box::new(stream));
Ok(())
@@ -177,6 +186,9 @@ impl HostAudio {
/// Called even while guest execution is backpressured. A failed or stalled
/// device cannot retain the only publication slot indefinitely.
pub fn service(&mut self) {
if let Some(reclaimer) = &mut self.reclaimer {
reclaimer.service(&self.transport);
}
let heartbeat = self.heartbeat.load(Ordering::Relaxed);
if heartbeat != self.last_heartbeat {
self.last_heartbeat = heartbeat;
@@ -187,7 +199,7 @@ impl HostAudio {
|| self.last_progress.elapsed() >= Duration::from_secs(2))
{
eprintln!("[HostAudio] Output unavailable/stalled; consuming audio silently.");
self._stream = None;
self.stop_stream();
self.perf_consumer = None;
}
if self._stream.is_none() {
@@ -195,6 +207,16 @@ impl HostAudio {
}
}
fn stop_stream(&mut self) {
if let Some(stream) = self._stream.take() {
stream.stop();
drop(stream); // Stop/drop the callback before any silent consumer takes over.
}
if let Some(mut reclaimer) = self.reclaimer.take() {
reclaimer.service(&self.transport);
}
}
pub fn update_stats(&mut self, stats: &mut crate::stats::HostStats) {
if let Some(cons) = &mut self.perf_consumer {
while let Some(us) = cons.try_pop() {
@@ -204,6 +226,40 @@ impl HostAudio {
}
}
impl Drop for HostAudio {
fn drop(&mut self) {
self.stop_stream();
self.transport.consume_silently();
}
}
/// The same bounded runtime-owned data path used by CPAL and allocator tests.
pub struct AudioCallback {
mixer: AudioMixer,
lane: CallbackLane,
}
impl AudioCallback {
pub fn new() -> (Self, AudioReclaimer) {
let (lane, reclaimer) = reclamation::pool();
(Self { mixer: AudioMixer::new(), lane }, reclaimer)
}
pub fn render(&mut self, transport: &AudioTransport, data: &mut [f32]) {
let mut packet = self.lane.take();
if let Some(packet) = &mut packet {
// Reserve reclamation before acquiring any publication ownership.
self.mixer.retire_inactive(packet);
if let Some(delivery) = transport.try_take() {
self.mixer.apply_delivery_into(delivery, packet);
}
}
self.mixer.mix(data);
if let Some(mut packet) = packet {
self.mixer.retire_inactive(&mut packet);
self.lane.give_back(packet);
}
}
}
#[cfg(test)]
#[allow(clippy::items_after_test_module)]
mod tests {
@@ -347,27 +403,59 @@ impl AudioMixer {
}
}
pub fn apply_delivery(&mut self, delivery: AudioDelivery) {
fn apply_delivery_into(&mut self, delivery: AudioDelivery, packet: &mut Packet) {
if delivery.epoch < self.epoch {
packet.batches = [delivery.frame, delivery.lifecycle];
return;
}
if delivery.epoch != self.epoch {
self.voices = Default::default();
for voice in &mut self.voices {
packet.retire(voice.sample.take());
*voice = Channel::default();
}
self.epoch = delivery.epoch;
}
// A frame pending before a lifecycle hook is always applied first.
for batch in [delivery.frame, delivery.lifecycle].into_iter().flatten() {
for command in batch {
self.process_command(command);
for (slot, batch) in packet.batches.iter_mut().zip([delivery.frame, delivery.lifecycle]) {
*slot = batch;
}
// Move commands out of slots, keeping each vector allocation in its packet.
for index in 0..2 {
if let Some(mut batch) = packet.batches[index].take() {
for command in &mut batch {
let command =
std::mem::replace(command, AudioCommand::Stop { voice_id: MAX_CHANNELS });
self.process_command(command, packet);
}
packet.batches[index] = Some(batch);
}
}
self.paused = delivery.paused;
}
pub fn process_command(&mut self, cmd: AudioCommand) {
fn retire_inactive(&mut self, packet: &mut Packet) {
for voice in &mut self.voices {
if !voice.active {
packet.retire(voice.sample.take());
}
}
}
// Signal-only tests use synchronous reclamation outside a data callback.
#[cfg(test)]
fn apply_delivery(&mut self, delivery: AudioDelivery) {
self.apply_delivery_into(delivery, &mut Packet::new());
}
#[cfg(test)]
fn fill_buffer(&mut self, buffer: &mut [f32]) {
self.mix(buffer);
self.retire_inactive(&mut Packet::new());
}
fn process_command(&mut self, cmd: AudioCommand, packet: &mut Packet) {
match cmd {
AudioCommand::Play { sample, voice_id, volume, pan, pitch, priority, loop_mode } => {
if voice_id < MAX_CHANNELS {
packet.retire(self.voices[voice_id].sample.take());
self.voices[voice_id] = Channel {
sample: Some(sample),
active: true,
@@ -378,12 +466,14 @@ impl AudioMixer {
loop_mode,
priority,
};
} else {
packet.retire(Some(sample));
}
}
AudioCommand::Stop { voice_id } => {
if voice_id < MAX_CHANNELS {
self.voices[voice_id].active = false;
self.voices[voice_id].sample = None;
packet.retire(self.voices[voice_id].sample.take());
}
}
AudioCommand::SetVolume { voice_id, volume } => {
@@ -404,7 +494,7 @@ impl AudioMixer {
}
}
pub fn fill_buffer(&mut self, buffer: &mut [f32]) {
fn mix(&mut self, buffer: &mut [f32]) {
let start = std::time::Instant::now();
// Zeroes the buffer (stereo)
for sample in buffer.iter_mut() {
@@ -417,17 +507,19 @@ impl AudioMixer {
}
for voice in self.voices.iter_mut() {
if !voice.active {
continue;
}
let Some(sample) = voice.sample.as_ref() else {
continue;
};
let len = sample.data.len();
if len == 0 {
voice.active = false;
voice.sample = None;
continue;
}
let (loop_start, end) = match (sample.loop_start, sample.loop_end) {
(Some(start), Some(end)) if start < end && end as usize <= len => {
let (loop_start, end) = match (voice.loop_mode, sample.loop_start, sample.loop_end) {
(LoopMode::On, Some(start), Some(end)) if start < end && end as usize <= len => {
(start as f64, end as f64)
}
_ => (0.0, len as f64),
@@ -441,7 +533,6 @@ impl AudioMixer {
let pos = voice.pos as usize;
if pos >= len {
voice.active = false;
voice.sample = None;
break;
}
let s1 = sample.data[pos] as f32 / 32768.0;
@@ -460,7 +551,6 @@ impl AudioMixer {
if voice.pitch >= (end - voice.pos) / ratio {
if voice.loop_mode == LoopMode::Off {
voice.active = false;
voice.sample = None;
break;
}
let length = end - loop_start;
@@ -512,6 +602,168 @@ mod control_tests {
);
}
#[test]
fn one_shot_ignores_markers_and_loop_preserves_intro() {
for looping in [false, true] {
let (mut audio, mut mixer) = pair();
let sample =
Sample::new(48_000, vec![1024, 2048, 4096, 8192, 12288, 16384, 20480, 24576])
.with_loop(2, 4);
play(&mut audio, Arc::new(sample), 1.0, looping);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [0.0; 20];
mixer.fill_buffer(&mut pcm);
let expected = if looping {
[1024, 2048, 4096, 8192, 4096, 8192, 4096, 8192, 4096, 8192]
} else {
[1024, 2048, 4096, 8192, 12288, 16384, 20480, 24576, 0, 0]
};
for (frame, value) in pcm.as_chunks::<2>().0.iter().zip(expected) {
assert_eq!(*frame, [value as f32 / 32768.0, 0.0]);
}
assert_eq!(mixer.voices[0].active, looping);
assert_eq!(mixer.voices[0].sample.is_some(), looping);
}
}
#[test]
fn absent_partial_and_invalid_markers_fall_back_to_full_sample() {
for (start, end) in [
(None, None),
(Some(1), None),
(None, Some(2)),
(Some(2), Some(2)),
(Some(3), Some(1)),
(Some(0), Some(5)),
(Some(u32::MAX), Some(u32::MAX)),
(Some(0), Some(0)),
] {
for looping in [false, true] {
let (mut audio, mut mixer) = pair();
let mut sample = Sample::new(48_000, vec![4096, 8192, 12288, 16384]);
sample.loop_start = start;
sample.loop_end = end;
play(&mut audio, Arc::new(sample), 1.0, looping);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [0.0; 16];
mixer.fill_buffer(&mut pcm);
for (i, frame) in pcm.as_chunks::<2>().0.iter().enumerate() {
let value = if looping || i < 4 { (i % 4 + 1) as f32 / 8.0 } else { 0.0 };
assert_eq!(*frame, [value, 0.0], "markers {start:?}/{end:?}, loop {looping}");
}
assert_eq!(mixer.voices[0].active, looping);
}
}
}
#[test]
fn fractional_interpolation_uses_loop_seam_or_actual_tail() {
// Both rate conversion and fractional pitch produce a half-frame step.
for (rate, pitch) in [(48_000, 0.5), (24_000, 1.0), (96_000, 0.25)] {
for looping in [false, true] {
let (mut audio, mut mixer) = pair();
play(
&mut audio,
Arc::new(Sample::new(rate, vec![4096, 8192, 16384, 24576]).with_loop(1, 3)),
pitch,
looping,
);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [0.0; 18];
mixer.fill_buffer(&mut pcm);
let expected = if looping {
[0.125, 0.1875, 0.25, 0.375, 0.5, 0.375, 0.25, 0.375, 0.5]
} else {
[0.125, 0.1875, 0.25, 0.375, 0.5, 0.625, 0.75, 0.375, 0.0]
};
for (frame, value) in pcm.as_chunks::<2>().0.iter().zip(expected) {
assert!((frame[0] - value).abs() <= 1e-6);
assert_eq!(frame[1], 0.0);
}
assert_eq!(mixer.voices[0].active, looping);
}
}
}
#[test]
fn empty_samples_terminate_without_indexing() {
for looping in [false, true] {
for marked in [false, true] {
let (mut audio, mut mixer) = pair();
let sample = Sample::new(48_000, vec![]);
let sample = if marked { sample.with_loop(0, 1) } else { sample };
play(&mut audio, Arc::new(sample), f64::MAX, looping);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [1.0; 8];
mixer.fill_buffer(&mut pcm);
assert_eq!(pcm, [0.0; 8]);
assert!(!mixer.voices[0].active);
assert!(mixer.voices[0].sample.is_none());
}
}
}
#[test]
fn multiple_wrap_pitch_preserves_selected_interval_and_signal() {
for (rate, pitch) in [(48_000, 5.0), (96_000, 2.5)] {
let (mut audio, mut mixer) = pair();
play(
&mut audio,
Arc::new(Sample::new(rate, vec![4096, 8192, 16384, 24576]).with_loop(1, 3)),
pitch,
true,
);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [0.0; 10];
mixer.fill_buffer(&mut pcm);
assert_eq!(pcm, [0.125, 0.0, 0.25, 0.0, 0.5, 0.0, 0.25, 0.0, 0.5, 0.0]);
assert_eq!(mixer.voices[0].pos, 1.0);
}
}
#[test]
fn extreme_pitch_and_source_rates_keep_positions_finite() {
for rate in [0, 1, 44_100, 48_000, 96_000, u32::MAX] {
for looping in [false, true] {
let (mut audio, mut mixer) = pair();
play(
&mut audio,
Arc::new(Sample::new(rate, vec![4096, 8192, 16384, 24576]).with_loop(1, 3)),
f64::MAX,
looping,
);
audio.finish_frame();
apply(&audio, &mut mixer);
let mut pcm = [0.0; 128];
mixer.fill_buffer(&mut pcm);
assert!(pcm.iter().all(|value| value.is_finite()));
assert!(mixer.voices[0].pos.is_finite());
if rate == 0 {
assert_eq!(mixer.voices[0].pos, 0.0);
assert!(pcm.as_chunks::<2>().0.iter().all(|frame| *frame == [0.125, 0.0]));
} else if looping {
assert!((1.0..3.0).contains(&mixer.voices[0].pos));
assert!(
pcm.as_chunks::<2>()
.0
.iter()
.skip(1)
.all(|frame| (0.25..=0.5).contains(&frame[0]))
);
} else {
assert_eq!(&pcm[..2], &[0.125, 0.0]);
assert!(pcm[2..].iter().all(|value| *value == 0.0));
}
assert_eq!(mixer.voices[0].active, looping || rate == 0);
}
}
}
#[test]
fn pcm_controls_preserve_position_and_stop_is_idempotent() {
let (mut audio, mut mixer) = pair();
@@ -632,7 +884,7 @@ mod control_tests {
let mut pcm = [0.0; 128];
mixer.fill_buffer(&mut pcm);
assert!(pcm.iter().all(|v| v.is_finite()));
assert!(pcm.chunks_exact(2).all(|frame| frame == [0.5, 0.0]));
assert!(pcm.as_chunks::<2>().0.iter().all(|frame| *frame == [0.5, 0.0]));
assert!(mixer.voices[0].active);
assert!(mixer.voices[0].pos >= 1.0 && mixer.voices[0].pos < 3.0);
}
@@ -705,6 +957,7 @@ mod control_tests {
audio.finish_frame();
let deadline = Instant::now() + Duration::from_secs(2);
while !audio.ready() && Instant::now() < deadline {
host.service();
std::thread::sleep(Duration::from_millis(10));
}
assert!(audio.ready());
@@ -713,12 +966,119 @@ mod control_tests {
audio.set_pan(0, 255);
audio.set_pitch(0, 1.5);
audio.finish_frame();
std::thread::sleep(Duration::from_millis(100));
for _ in 0..10 {
host.service();
std::thread::sleep(Duration::from_millis(10));
}
assert!(audio.ready());
audio.stop(0);
audio.finish_frame();
std::thread::sleep(Duration::from_millis(100));
for _ in 0..10 {
host.service();
std::thread::sleep(Duration::from_millis(10));
}
assert!(audio.ready());
assert!(!host.failed.load(Ordering::Relaxed));
}
}
#[cfg(test)]
mod reclamation_tests {
use super::*;
use prometeu_drivers::{Audio, MemoryBanks};
use prometeu_hal::{AudioBridge, sample::Sample};
#[test]
fn stale_delivery_moves_whole_vectors_to_service_without_changing_playback() {
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let sample = Arc::new(Sample::new(48000, vec![1000; 16]));
let weak = Arc::downgrade(&sample);
audio.play_sample(sample, 0, 255, 0, 1.0, 0, LoopMode::On);
audio.finish_frame();
let stale = audio.transport().try_take().unwrap();
let allocation = stale.frame.as_ref().unwrap().as_ptr();
audio.reset();
let mut mixer = AudioMixer::new();
mixer.apply_delivery(audio.transport().try_take().unwrap());
let mut packet = Packet::new();
mixer.apply_delivery_into(stale, &mut packet);
assert_eq!(packet.batches[0].as_ref().unwrap().as_ptr(), allocation);
assert!(weak.upgrade().is_some());
mixer.mix(&mut [0.0; 32]);
assert!(mixer.voices.iter().all(|voice| !voice.active));
drop(packet); // Main/service thread, including stale command owners.
assert!(weak.upgrade().is_none());
}
type Callback = Box<dyn FnMut(&mut [f32]) + Send>;
struct ManualStream(Arc<std::sync::Mutex<Option<Callback>>>);
impl AudioStreamHandle for ManualStream {}
impl Drop for ManualStream {
fn drop(&mut self) {
self.0.lock().unwrap().take();
}
}
struct ManualRuntime(Arc<std::sync::Mutex<Option<Callback>>>);
impl AudioRuntime for ManualRuntime {
type Device = ();
type Stream = ManualStream;
fn default_output_device(&self) -> Option<()> {
Some(())
}
fn build_output_stream<D, E>(
&self,
_: &(),
_: &cpal::StreamConfig,
data: D,
_: E,
) -> Result<ManualStream, HostAudioError>
where
D: FnMut(&mut [f32]) + Send + 'static,
E: FnMut(String) + Send + 'static,
{
*self.0.lock().unwrap() = Some(Box::new(data));
Ok(ManualStream(self.0.clone()))
}
fn play_stream(&self, _: &ManualStream) -> Result<(), HostAudioError> {
Ok(())
}
}
#[test]
fn failure_stall_and_teardown_release_callback_and_pending_owners() {
for mode in 0..3 {
let callbacks = Arc::new(std::sync::Mutex::new(None));
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let mut host = HostAudio::new();
host.transport = audio.transport();
host.init_with_runtime(&ManualRuntime(callbacks.clone())).unwrap();
let mut owners = vec![];
for _ in 0..2 {
let sample = Arc::new(Sample::new(48000, vec![1000; 16]));
owners.push(Arc::downgrade(&sample));
audio.play_sample(sample, 0, 255, 0, 1.0, 0, LoopMode::On);
audio.finish_frame();
callbacks.lock().unwrap().as_mut().unwrap()(&mut []);
}
let sample = Arc::new(Sample::new(48000, vec![1000; 16]));
owners.push(Arc::downgrade(&sample));
audio.play_sample(sample, 0, 255, 0, 1.0, 0, LoopMode::On);
audio.finish_frame();
assert!(!audio.ready());
if mode == 0 {
host.failed.store(true, Ordering::Relaxed);
}
if mode == 1 {
host.last_progress = Instant::now() - Duration::from_secs(3);
host.last_heartbeat = host.heartbeat.load(Ordering::Relaxed);
}
if mode < 2 {
host.service();
assert!(host._stream.is_none());
}
drop(host);
assert!(audio.ready());
assert!(callbacks.lock().unwrap().is_none());
assert!(owners.iter().all(|owner| owner.upgrade().is_none()));
}
}
}
@@ -0,0 +1,100 @@
//! Two packets circulate free -> callback -> returned -> service -> free.
//! While the callback owns a packet, at most one packet can occupy the return
//! lane, so its capacity-two enqueue cannot fail. Neither lane grows.
use prometeu_drivers::{AudioCommand, AudioTransport, MAX_CHANNELS, MAX_COMMANDS_PER_FRAME};
use prometeu_hal::sample::Sample;
use ringbuf::{
HeapRb,
traits::{Consumer, Producer, Split},
wrap::{CachingCons, CachingProd},
};
use std::sync::Arc;
pub(super) const RETIRED_CAPACITY: usize = 3 * MAX_COMMANDS_PER_FRAME + MAX_CHANNELS;
pub(super) struct Packet {
retired: Vec<Arc<Sample>>,
pub batches: [Option<Vec<AudioCommand>>; 2],
}
impl Packet {
pub(super) fn new() -> Self {
Self { retired: Vec::with_capacity(RETIRED_CAPACITY), batches: [None, None] }
}
pub(super) fn retire(&mut self, sample: Option<Arc<Sample>>) {
if let Some(sample) = sample {
// Every owner comes from at most 3072 Play commands or 16 slots.
assert!(self.retired.len() < RETIRED_CAPACITY, "audio retirement bound");
self.retired.push(sample);
}
}
fn recycle(&mut self, transport: &AudioTransport) {
self.retired.clear();
for batch in &mut self.batches {
if let Some(batch) = batch.take() {
transport.recycle_commands(batch);
}
}
}
}
type Prod = CachingProd<Arc<HeapRb<Packet>>>;
type Cons = CachingCons<Arc<HeapRb<Packet>>>;
pub(super) struct CallbackLane {
free: Cons,
returned: Prod,
}
/// Main-thread endpoint. Service independently of guest progress or pause.
pub struct AudioReclaimer {
free: Prod,
returned: Cons,
}
pub(super) fn pool() -> (CallbackLane, AudioReclaimer) {
let (mut free, free_cons) = HeapRb::new(2).split();
let (returned, returned_cons) = HeapRb::new(2).split();
for _ in 0..2 {
assert!(free.try_push(Packet::new()).is_ok());
}
(CallbackLane { free: free_cons, returned }, AudioReclaimer { free, returned: returned_cons })
}
impl CallbackLane {
pub(super) fn take(&mut self) -> Option<Packet> {
self.free.try_pop()
}
pub(super) fn give_back(&mut self, packet: Packet) {
// Conservation proves capacity; there is no overflow/drop fallback.
assert!(self.returned.try_push(packet).is_ok(), "audio packet conservation");
}
}
impl AudioReclaimer {
pub fn service(&mut self, transport: &AudioTransport) {
while let Some(mut packet) = self.returned.try_pop() {
packet.recycle(transport);
assert!(self.free.try_push(packet).is_ok(), "audio packet conservation");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn two_packets_conserve_capacity_across_service_starvation() {
let (mut callback, mut service) = pool();
let first = callback.take().unwrap();
let second = callback.take().unwrap();
assert!(callback.take().is_none());
callback.give_back(first);
callback.give_back(second);
assert!(callback.take().is_none());
let transport = AudioTransport::default();
for _ in 0..1000 {
service.service(&transport);
for _ in 0..2 {
let packet = callback.take().unwrap();
assert_eq!(packet.retired.capacity(), RETIRED_CAPACITY);
assert!(packet.retired.is_empty());
assert!(packet.batches.iter().all(Option::is_none));
callback.give_back(packet);
}
assert!(callback.take().is_none());
}
}
}
@@ -433,6 +433,9 @@ impl ApplicationHandler for HostRunner {
// Consume time in exact 60Hz slices.
while self.accumulator >= self.frame_target_dt {
// Reclaim returned audio owners before another guest attempt, even
// when the previous attempt paused or stopped at backpressure.
self.audio.service();
// Unless the debugger is waiting for a 'start' command, advance the system.
if !self.debugger.waiting_for_start {
self.firmware.tick(&self.input.signals, &mut self.hardware);
@@ -834,7 +837,7 @@ mod tests {
// real socket bind/connect coordination. Deterministic debugger state changes
// are covered below the host layer in `debugger.rs`.
#[test]
#[ignore = "requires localhost TCP bind/connect; run via `cargo test -p prometeu-host-desktop-winit --lib -- --ignored`"]
#[ignore = "requires localhost TCP bind/connect; run via `make test-debugger-socket`"]
fn test_debug_port_opens() {
let mut runner = HostRunner::new(None, None);
let port = 9999;
@@ -894,7 +897,7 @@ mod tests {
}
#[test]
#[ignore = "requires localhost TCP bind/connect; run via `cargo test -p prometeu-host-desktop-winit --lib -- --ignored`"]
#[ignore = "requires localhost TCP bind/connect; run via `make test-debugger-socket`"]
fn test_debug_reconnection() {
let mut runner = HostRunner::new(None, None);
let port = 9998;
@@ -937,7 +940,7 @@ mod tests {
}
#[test]
#[ignore = "requires localhost TCP bind/connect; run via `cargo test -p prometeu-host-desktop-winit --lib -- --ignored`"]
#[ignore = "requires localhost TCP bind/connect; run via `make test-debugger-socket`"]
fn test_debug_refuse_second_connection() {
let mut runner = HostRunner::new(None, None);
let port = 9997;
@@ -973,7 +976,7 @@ mod tests {
}
#[test]
#[ignore = "requires localhost TCP bind/connect; run via `cargo test -p prometeu-host-desktop-winit --lib -- --ignored`"]
#[ignore = "requires localhost TCP bind/connect; run via `make test-debugger-socket`"]
fn test_get_state_returns_response() {
let mut runner = HostRunner::new(None, None);
let port = 9996;
@@ -1018,7 +1021,7 @@ mod tests {
}
#[test]
#[ignore = "requires localhost TCP bind/connect; run via `cargo test -p prometeu-host-desktop-winit --lib -- --ignored`"]
#[ignore = "requires localhost TCP bind/connect; run via `make test-debugger-socket`"]
fn test_debug_resume_on_disconnect() {
let mut runner = HostRunner::new(None, None);
let port = 9995;
@@ -0,0 +1,347 @@
use prometeu_drivers::{Audio, MemoryBanks};
use prometeu_hal::{AudioBridge, LoopMode, sample::Sample};
use prometeu_host_desktop_winit::audio::AudioCallback;
use std::alloc::{GlobalAlloc, Layout, System};
use std::cell::Cell;
use std::sync::Arc;
thread_local! {
static COUNTS: Cell<Option<(usize, usize)>> = const { Cell::new(None) };
}
struct Counter;
fn count(alloc: usize, free: usize) {
let _ = COUNTS.try_with(|c| {
if let Some((a, d)) = c.get() {
c.set(Some((a + alloc, d + free)));
}
});
}
unsafe impl GlobalAlloc for Counter {
unsafe fn alloc(&self, l: Layout) -> *mut u8 {
count(1, 0);
unsafe { System.alloc(l) }
}
unsafe fn alloc_zeroed(&self, l: Layout) -> *mut u8 {
count(1, 0);
unsafe { System.alloc_zeroed(l) }
}
unsafe fn dealloc(&self, p: *mut u8, l: Layout) {
count(0, 1);
unsafe { System.dealloc(p, l) }
}
unsafe fn realloc(&self, p: *mut u8, l: Layout, n: usize) -> *mut u8 {
count(1, 1);
unsafe { System.realloc(p, l, n) }
}
}
#[global_allocator]
static ALLOCATOR: Counter = Counter;
fn measured(f: impl FnOnce()) -> (usize, usize) {
COUNTS.with(|c| c.set(Some((0, 0))));
f();
COUNTS.with(|c| c.replace(None).unwrap())
}
fn play(audio: &mut Audio, voice: usize, looping: bool) -> std::sync::Weak<Sample> {
let sample = Arc::new(Sample::new(48000, vec![1000; 4]));
let weak = Arc::downgrade(&sample);
assert_eq!(
audio.play_sample(
sample,
voice,
255,
0,
1.0,
0,
if looping { LoopMode::On } else { LoopMode::Off }
),
prometeu_hal::AudioOpStatus::Ok
);
weak
}
fn render(callback: &mut AudioCallback, audio: &Audio, pcm: &mut [f32], label: &str) {
let counts = measured(|| callback.render(&audio.transport(), pcm));
println!("{label}: {counts:?}");
assert_eq!(counts, (0, 0), "{label}");
}
#[test]
fn callback_owners_do_not_use_allocator() {
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let (mut callback, mut reclaimer) = AudioCallback::new();
let mut pcm = [0.0; 32];
let weak = play(&mut audio, 0, false);
audio.finish_frame();
render(&mut callback, &audio, &mut pcm, "natural_completion_and_batch");
assert!(weak.upgrade().is_some(), "owner must await service");
reclaimer.service(&audio.transport());
assert!(weak.upgrade().is_none());
for _ in 0..8 {
play(&mut audio, 0, true);
audio.stop(0);
let replacement = play(&mut audio, 0, true);
audio.finish_frame();
render(&mut callback, &audio, &mut pcm, "replacement_stop_fifo");
assert!(pcm.iter().step_by(2).all(|&v| v == 1000.0 / 32768.0));
reclaimer.service(&audio.transport());
audio.set_paused(true);
render(&mut callback, &audio, &mut pcm, "pause");
assert!(pcm.iter().all(|&v| v == 0.0));
reclaimer.service(&audio.transport());
audio.reset();
render(&mut callback, &audio, &mut pcm, "epoch_reset");
assert!(replacement.upgrade().is_some());
reclaimer.service(&audio.transport());
assert!(replacement.upgrade().is_none());
}
audio.stop(0);
audio.finish_frame();
render(&mut callback, &audio, &mut pcm, "control_batch");
}
#[test]
fn stalled_service_bounds_retention_and_recovers_pending_delivery() {
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let (mut callback, mut reclaimer) = AudioCallback::new();
let owners: Vec<_> = (0..16).map(|voice| play(&mut audio, voice, false)).collect();
audio.finish_frame();
// Keep all voices playing until both packets have been returned.
render(&mut callback, &audio, &mut [], "start_16");
render(&mut callback, &audio, &mut [], "exhaust_pool");
let mut pcm = [0.0; 32];
render(&mut callback, &audio, &mut pcm, "finish_16_without_packet");
assert!(owners.iter().all(|owner| owner.upgrade().is_some()));
let next = play(&mut audio, 0, true);
audio.finish_frame();
for _ in 0..10 {
render(&mut callback, &audio, &mut pcm, "starved_silent_pending");
assert!(!audio.ready());
assert!(pcm.iter().all(|&v| v == 0.0));
}
reclaimer.service(&audio.transport());
render(&mut callback, &audio, &mut pcm, "recover");
assert!(audio.ready());
reclaimer.service(&audio.transport());
assert!(owners.iter().all(|owner| owner.upgrade().is_none()));
assert!(next.upgrade().is_some());
drop(callback); // Teardown belongs to the service/main thread.
drop(reclaimer);
assert!(next.upgrade().is_none());
}
#[test]
fn full_frame_and_combined_lifecycle_fit_one_packet() {
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let (mut callback, mut reclaimer) = AudioCallback::new();
let mut owners = Vec::new();
for voice in 0..16 {
owners.push(play(&mut audio, voice, false));
}
audio.finish_frame();
render(&mut callback, &audio, &mut [], "initial_16");
reclaimer.service(&audio.transport());
// The published frame, interrupted next-frame prefix and lifecycle each
// admit 1024 commands. Every command replaces an owner on the same voice.
for _ in 0..1024 {
owners.push(play(&mut audio, 0, false));
}
audio.finish_frame();
for _ in 0..1024 {
owners.push(play(&mut audio, 0, false));
}
audio.begin_lifecycle();
for _ in 0..1024 {
owners.push(play(&mut audio, 0, false));
}
audio.finish_lifecycle(true);
let mut pcm = [0.0; 32];
render(&mut callback, &audio, &mut pcm, "3072_commands_plus_completion");
reclaimer.service(&audio.transport());
assert!(owners.iter().skip(16).all(|owner| owner.upgrade().is_none()));
audio.reset();
render(&mut callback, &audio, &mut pcm, "reset_remaining_voices");
reclaimer.service(&audio.transport());
assert!(owners.iter().all(|owner| owner.upgrade().is_none()));
}
/// Physical observations, deliberately excluded from ordinary CI. The pressure
/// fixture reads the checked-in asset pack and writes a temporary host save;
/// it does not certify asset decoding or guest memcard scheduling.
#[test]
#[ignore = "requires physical audio output; 16 quiet voices and temporary file I/O"]
fn physical_callback_duration_16_voices_with_io_pressure() {
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use prometeu_host_desktop_winit::fs_backend::HostDirBackend;
use prometeu_system::fs::FsBackend;
use ringbuf::{
HeapRb,
traits::{Consumer, Producer, Split},
};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::{Duration, Instant};
let device = cpal::default_host().default_output_device().expect("physical output");
let device_name = device.name().unwrap();
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let (mut callback, mut reclaimer) = AudioCallback::new();
let mut owners = vec![];
for voice in 0..16 {
let sample = Arc::new(Sample::new(48000, vec![32; 4800]));
owners.push(Arc::downgrade(&sample));
audio.play_sample(sample, voice, 8, 127, 1.0, 0, LoopMode::On);
}
audio.finish_frame();
let config = cpal::StreamConfig {
channels: 2,
sample_rate: cpal::SampleRate(48000),
buffer_size: cpal::BufferSize::Fixed(256),
};
let transport = audio.transport();
let (mut timings, mut observations) = HeapRb::<(u64, usize)>::new(4096).split();
let allocator_calls = Arc::new(AtomicUsize::new(0));
let failed = Arc::new(AtomicBool::new(false));
let counter = allocator_calls.clone();
let error = failed.clone();
let stream = device
.build_output_stream(
&config,
move |data: &mut [f32], _: &cpal::OutputCallbackInfo| {
let start = Instant::now();
let (alloc, free) = measured(|| callback.render(&transport, data));
counter.fetch_add(alloc + free, Ordering::Relaxed);
let _ = timings.try_push((start.elapsed().as_nanos() as u64, data.len() / 2));
},
move |_| {
error.store(true, Ordering::Relaxed);
},
None,
)
.unwrap();
let root =
std::env::temp_dir().join(format!("prometeu-audio-qualification-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let mut filesystem = HostDirBackend::new(root.clone());
let asset = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../../test-cartridges/stress-console/assets.pa");
let save = vec![0x5a; 1024 * 1024];
stream.play().unwrap();
let deadline = Instant::now() + Duration::from_secs(2);
let mut io_cycles = 0;
while Instant::now() < deadline {
let payload = std::fs::read(&asset).unwrap();
assert!(!payload.is_empty());
filesystem.write_file("/user/audio-pressure.bin", &save).unwrap();
reclaimer.service(&audio.transport());
io_cycles += 1;
std::thread::sleep(Duration::from_millis(2));
}
stream.pause().unwrap();
let live_before = owners.iter().filter(|owner| owner.upgrade().is_some()).count();
drop(stream);
reclaimer.service(&audio.transport());
let live_after = owners.iter().filter(|owner| owner.upgrade().is_some()).count();
let mut samples = vec![];
let mut frames = vec![];
while let Some((time, count)) = observations.try_pop() {
samples.push(time);
frames.push(count);
}
filesystem.delete("/user/audio-pressure.bin").unwrap();
std::fs::remove_dir(root.join("user")).unwrap();
std::fs::remove_dir(root).unwrap();
samples.sort_unstable();
assert!(!samples.is_empty(), "physical stream delivered no callbacks");
println!(
"device={device_name:?}, callbacks={}, io_cycles={io_cycles}, frames_min={}, frames_max={}, p50_ns={}, p95_ns={}, p99_ns={}, max_ns={}, allocator_calls={}, owners_before={live_before}, owners_after={live_after}",
samples.len(),
frames.iter().min().unwrap(),
frames.iter().max().unwrap(),
samples[samples.len() / 2],
samples[samples.len() * 95 / 100],
samples[samples.len() * 99 / 100],
samples.last().unwrap(),
allocator_calls.load(Ordering::Relaxed)
);
assert!(!failed.load(Ordering::Relaxed));
assert_eq!(allocator_calls.load(Ordering::Relaxed), 0);
assert_eq!((live_before, live_after), (16, 0));
}
#[test]
#[ignore = "release timing observation with temporary asset-read/save-write pressure"]
fn callback_duration_16_voices_with_io_pressure() {
use prometeu_host_desktop_winit::fs_backend::HostDirBackend;
use prometeu_system::fs::FsBackend;
use std::time::Instant;
let mut audio = Audio::new(Arc::new(MemoryBanks::new()));
let (mut callback, mut reclaimer) = AudioCallback::new();
let owners: Vec<_> = (0..16).map(|voice| play(&mut audio, voice, true)).collect();
audio.finish_frame();
let mut pcm = [0.0; 512]; // 256 stereo frames: nominal 5.333 ms at 48 kHz.
callback.render(&audio.transport(), &mut pcm);
reclaimer.service(&audio.transport());
let root = std::env::temp_dir().join(format!("prometeu-audio-timing-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let mut filesystem = HostDirBackend::new(root.clone());
let asset = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../../../test-cartridges/stress-console/assets.pa");
let save = vec![0x5a; 1024 * 1024];
let mut samples = Vec::with_capacity(1000);
let mut counts = (0, 0);
for _ in 0..1000 {
assert!(!std::fs::read(&asset).unwrap().is_empty());
filesystem.write_file("/user/audio-pressure.bin", &save).unwrap();
let start = Instant::now();
let measured = measured(|| callback.render(&audio.transport(), &mut pcm));
samples.push(start.elapsed().as_nanos());
counts.0 += measured.0;
counts.1 += measured.1;
reclaimer.service(&audio.transport());
}
samples.sort_unstable();
let before = owners.iter().filter(|owner| owner.upgrade().is_some()).count();
drop(callback);
drop(reclaimer);
let after = owners.iter().filter(|owner| owner.upgrade().is_some()).count();
filesystem.delete("/user/audio-pressure.bin").unwrap();
std::fs::remove_dir(root.join("user")).unwrap();
std::fs::remove_dir(root).unwrap();
println!(
"synthetic callbacks=1000, frames=256, nominal_period_ns=5333333, io_cycles=1000, p50_ns={}, p95_ns={}, p99_ns={}, max_ns={}, alloc_free={counts:?}, owners_before={before}, owners_after={after}",
samples[500], samples[950], samples[990], samples[999]
);
assert_eq!(counts, (0, 0));
assert_eq!((before, after), (16, 0));
}
#[test]
fn unloaded_banks_empty_samples_and_controls_reclaim_on_service() {
use prometeu_drivers::SoundBankPoolInstaller;
use prometeu_hal::sound_bank::SoundBank;
let banks = Arc::new(MemoryBanks::new());
let sample = Arc::new(Sample::new(48000, vec![1000; 16]));
let weak = Arc::downgrade(&sample);
banks.install_sound_bank(0, Arc::new(SoundBank::new(vec![sample])));
let mut audio = Audio::new(banks.clone());
let (mut callback, mut reclaimer) = AudioCallback::new();
audio.play(0, 0, 0, 255, 0, 1.0, 0, LoopMode::On);
audio.finish_frame();
banks.clear_sound_banks();
let mut pcm = [0.0; 16];
render(&mut callback, &audio, &mut pcm, "unloaded_bank");
assert!(weak.upgrade().is_some());
reclaimer.service(&audio.transport());
audio.set_volume(0, 127);
audio.set_pan(0, 255);
audio.set_pitch(0, 0.5);
audio.stop(0);
let empty = Arc::new(Sample::new(48000, vec![]));
let weak_empty = Arc::downgrade(&empty);
audio.play_sample(empty, 1, 255, 0, 1.0, 0, LoopMode::Off);
audio.finish_frame();
render(&mut callback, &audio, &mut pcm, "controls_stop_empty");
assert!(pcm.iter().all(|&sample| sample == 0.0));
assert!(weak.upgrade().is_some() && weak_empty.upgrade().is_some());
reclaimer.service(&audio.transport());
assert!(weak.upgrade().is_none() && weak_empty.upgrade().is_none());
}
@@ -58,8 +58,33 @@ or symlink aliases while producing plausible percentages.
**Trade-offs:**
New crates require explicit ownership. Reports with missing domains, duplicate
sources or zero instrumented lines are rejected before threshold comparison.
The current source map assigns 232 files to 15 exclusive owners without
changing the original eight domain baselines.
The initial ownership extension assigned 232 files to 15 exclusive owners
without changing the original eight domain baselines. Source counts grow with
the repository; the current evidence records the measured layout.
### Exclusive ownership must also be the intended ownership
**What:**
Module entry files and their sibling directories belong to the same specialized
domain. Memcard descendants belong to system/fs; asset-driver descendants
belong to asset/bank. Both residual owners exclude the entry file and the whole
module tree. Regression tests call the production auditor's ownership function
for current and future paths, and audit independent reports containing absent
and zero-line descendants.
**Why:**
A prefix ending in `.rs` covers the entry file but not the directory created
when a module is split. A residual owner can absorb every new descendant and
still pass a generic exactly-one-owner audit. The resulting percentages look
valid but measure the wrong responsibility. Expected-owner assertions detect
this failure before instrumentation or percentage comparison.
**Trade-offs:**
Correcting ownership moves the denominator and may reduce a specialized
domain's margin. Preserve reviewed floors and investigate fresh failures with
behavioral evidence. Historical measurements remain useful only when their
original source map and fingerprints remain intact; they should not be
silently rewritten to match the repaired map.
## Patterns and Algorithms
@@ -80,6 +105,8 @@ changing the original eight domain baselines.
- Allowing an absent or zero-line report to pass a zero baseline.
- Assigning the whole VM to the system integration domain, or counting one
source in both a specialized owner and its enclosing residual crate owner.
- Treating exactly one owner as proof that a module retained its intended
owner after its implementation moved into sibling source files.
- Treating local host measurements as evidence for unexecuted platforms.
## Published Extension and Evidence
@@ -97,6 +124,18 @@ Mathematics and its portable MPFR-generated fixture now exercise vm/core;
bytecode has its own gate. The standalone MPFR qualification remains separate.
This evidence does not claim Linux or Intel Mac coverage execution.
The module-tree ownership correction derives from accepted DEC-0059. Its
[current measured evidence](../../../docs/vm-arch/COVERAGE_DOMAINS_CURRENT.json)
preserves the earlier evidence records and restores system/core to 87.71%
without changing the other 14 floors. The initial repaired map had 257 source files
and 15 exclusive domains. Its domain gates passed: system/fs measured 78.73%,
system/core 93.69%, asset/bank 90.52% and drivers/core 89.17%.
Ten policy tests cover both generic validity and intended specialized owners.
That initial repository target passed 1,021 tests with 11 ignored while strict
workspace Clippy still reported VM diagnostics. The final strict gate subsequently
passed on the completed source, as recorded in the [runtime quality lesson](../DSC-0064-runtime-branch-quality-review/LSN-0070-runtime-quality-needs-independent-oracles-and-owned-evidence.md).
The current evidence retains the initial ownership comparison separately.
## Takeaways
- Runtime-edge coverage is safer when CI enforces domain ownership instead of trusting a workspace aggregate.
@@ -104,3 +143,40 @@ This evidence does not claim Linux or Intel Mac coverage execution.
- Coverage gates that live in repository helpers are easier to evolve, audit, and keep consistent than CI-local logic.
- Audit source ownership and evidence validity before comparing percentages.
A passing percentage is useful only when its denominator has a clear owner.
## A Shared Gate Must Share Failure Semantics Too
The strict-gate correction derives from DEC-0063. A common coverage helper did
not establish common CI policy while one pipeline only ran permissive lint/tests
and another produced domain reports. Both CI consumers now provision the declared
tools and invoke the same Makefile-owned `ci-domains` target. Strict Clippy covers
workspace targets and features with warnings denied; concrete lint fixes replace
warning suppression. The existing coverage map and floors remain unchanged.
The Makefile owns the ordered steps. A serial runner performs format, strict lint,
all-target tests, the five named debugger socket cases, fresh coverage/export and
all domain audits/gates. Serial child Make invocations avoid races between clean,
Cargo execution and report generation. A second independent gate in the same
checkout still requires operator coordination.
Failure publication is part of the contract. Before a run, old reports are
removed; every executed step receives a log, exit status, revision and source/config
identity. A later failure retains current partial reports. An early failure cannot
archive old HTML as current success. Both CI providers publish available evidence
on failure and preserve the gate's nonzero status. Regression fixtures exercise
failed execution, success and failure after partial report generation.
Ignored tests are not one qualification category. Five localhost socket tests
belong in automation, but physical audio, endurance, packaged installation and
manual timing/layout probes need their own named execution. Restricting the socket
filter prevents accidentally turning a device requirement into a CI prerequisite.
The local macOS/aarch64 strict gate passed 1,039 ordinary tests, five socket tests,
three runner regressions and all fifteen coverage gates; fresh coverage separately
replayed 1,039 tests. The ordinary phases reported thirteen ignored tests; five
of these ran in the separate socket step, while eight remained unselected. The initial sandbox
socket refusal is distinguished from the successful authorized run. Pipeline
configuration was inspected, but no remote or Linux gate was executed. Platform
equivalence concerns required commands, not identical measured coverage. See the
[published operational contract and evidence](../../../docs/vm-arch/COVERAGE_DOMAINS.md)
for prerequisites, manual commands and practical limits.
@@ -86,6 +86,35 @@ surfaces unless the caller violated the structural ABI.
Do not reopen FS public API scope while wiring internal async-lane consumers.
The lane can support FS-style work without deciding request/poll semantics.
## Share format rules without sharing execution strategy
The glyph, sound and scene decoders now consume one private driver validation
module. Complete-buffer decoding can materialize a whole payload, while the
shared worker retains state and reads at most 4096 bytes per decode step. Sharing
validation does not require replacing cooperative IO with a full-buffer read.
A differential test alone is insufficient: both paths can agree on a mistake.
Use independent expected banks and rejection fixtures, then compare each entry
point against those expectations. Include physical file carriers: an in-memory
slice rejects an out-of-range request before a decoder runs, which can conceal
a short-file discrepancy in a reader that accepts EOF. The initial corpus found
that legacy PCM and scene loads accepted missing declared bytes. After an
explicit compatibility decision, both modes reject those files while still
ignoring a physically present odd PCM tail.
The production pressure regression pauses the real decoder after a safe step,
queues a persistent memcard save, and observes save writes and synchronization
before the next asset step. It checks all three multi-chunk asset kinds, the
single worker identity, complete output, deferred frame publication and saved
data after remount. This establishes priority and retained progress through
observable ordering, rather than a host-specific latency threshold. Separate
regressions cover cancellation, superseding and preload publication.
This refinement follows DEC-0064, summarized in the [runtime quality lesson](../DSC-0064-runtime-branch-quality-review/LSN-0070-runtime-quality-needs-independent-oracles-and-owned-evidence.md).
The published contract is [Asset Management](../../../docs/specs/runtime/15-asset-management.md#43-decoding-validation-and-physical-payload-completeness).
The ordering fixture uses the persistent transaction protocol with a deterministic
storage backend; file-host latency measurements remain separate evidence.
## Takeaways
- A runtime async lane is an ownership boundary, not just an implementation
@@ -67,3 +67,28 @@ String concatenation still creates a new string when the language semantics requ
- The right performance question is often "where is the repeated cost?" rather than "where is any cost at all?"
- Immutable payload sharing is a strong default when public semantics must remain stable and hot-path copies are the real issue.
- Internal telemetry should help engineering decisions without silently becoming a product contract.
## Successful execution can allocate without materializing guest values
The later stack/return correction (DEC-0061) demonstrated another source of
recurring native allocations: eager diagnostic Strings and disposable return
storage. A warmed scalar call/return loop allocated and freed 3,000 times per
1,000 iterations; lazy diagnostics and in-place return transfer reduced both
counts to zero in debug and release. The original string materialization evidence
above remains a separate observation.
Validate before moving owners. Rotating only the returning frame's suffix and
then truncating locals preserves ordered logical groups, Arc identity and heap
aliases without cloning or scratch storage. Remove the frame after transfer so
the existing last-root safepoint sees the results. GC scratch refusal is a later
failure boundary: transferred roots must remain intact and sweeping must not
start after failed marking admission.
Account for storage that actually exists. Eliminating temporary result storage
can make a return succeed under a zero temporary budget without changing any
quota ceiling. A triggered collector still needs its own admitted scratch.
The [execution allocation qualification](../../../docs/vm-arch/execution-allocation-qualification.md)
records instruction regions, before/after counts and limits. The
[memory contract](../../../docs/specs/runtime/03-memory-stack-heap-and-allocation.md)
remains normative. These measurements say nothing about gameplay throughput.
@@ -46,6 +46,18 @@ The mixer also reduces large pitch steps before multiplication when wrapping loo
## Evidence and Limits
### Sample Metadata Does Not Select Playback Mode
Loop markers describe an optional interval within a sample; the playback command decides whether to use it. Applying a loop end to one-shot playback silently discards the tail, even though the data remains available. The mixer now selects the full sample boundary when looping is off. When looping is on, a complete interval satisfying `start < end <= length` preserves the intro and wraps at the selected end; absent, partial or invalid markers fall back to the full sample. Empty samples terminate before indexing.
Interpolation follows the same selected boundary. A one-shot interpolates into the actual tail and finally toward zero, while a loop seam interpolates toward the loop-start frame. Distinct intro, loop and tail values make these behaviors observable: an eight-frame sample with markers `(2, 4)` emits all eight frames once with looping off, and emits frame indices `0, 1, 2, 3, 2, 3, 2, 3` with looping on. Constant PCM alone cannot detect this defect.
The correction derives from DEC-0060 in DSC-0064. Synthetic regressions cover exact PCM, fractional seams and tails, incomplete and invalid markers, empty samples, rate conversion, multiple wraps and extreme finite pitch. These tests exercise the existing delivery and control paths without establishing new device timing or reclamation guarantees. The [sample contract](../../../docs/specs/runtime/05-audio-peripheral.md#6-samples-interpolation-and-ownership) remains normative.
Local macOS/aarch64 qualification on 2026-10-14 passed 19 host audio tests (one physical-device test ignored), nine driver audio tests and ten runtime audio tests. The existing `make ci-domains` target passed with 1,027 tests passed, zero failed and 11 ignored, including all 15 domain gates. Strict all-target/all-feature Clippy passed for the desktop host. That initial run preceded repair of existing VM lint errors. The later strict shared gate passed on the completed source; see the [runtime quality lesson](../DSC-0064-runtime-branch-quality-review/LSN-0070-runtime-quality-needs-independent-oracles-and-owned-evidence.md) for final acceptance and complementary production decode/save pressure evidence.
### Original Voice-Control Qualification
Validation was run locally on macOS/aarch64 on 2026-10-14:
- `cargo test --workspace --all-targets --all-features --no-fail-fast`: **1,016 passed**, zero failed, eleven ignored across 55 reported suites.
@@ -65,3 +77,37 @@ The API is a foreground PCM control surface. It does not establish musical seque
- Keep shutdown, pause and session invalidation independent of guest saturation.
- Place completion-sensitive state with the component that actually observes completion.
- Test both explicit and implicit runtime boundaries, including debugger transitions.
## Completion and Reclamation Have Different Owners
The callback reclamation correction derives from DEC-0062. A nonblocking mailbox
alone did not make the callback allocator-free: dropping a consumed Vec freed its
storage, and finishing a voice could free the final Sample and PCM allocation.
The useful boundary is ownership destruction, not just explicit allocation calls.
Two preallocated packets now move discarded references and vector allocations to
main-thread service. The bound follows from admitted input: 1,024 frame commands,
2,048 combined lifecycle commands and sixteen initial voices imply at most 3,088
retired owners and two vectors per packet. Conservation, rather than an overflow
fallback, guarantees room to return a packet. When both await service, publications
remain pending and naturally finished voices become silent while retaining at most
sixteen references in their existing slots. A stalled reclaimer cannot create an
unbounded queue or resurrect a finished voice.
The original signal observations above remain valid, but physical release now
happens during service. Service runs before guest attempts even during pause or
backpressure. It clears retired owners and reuses bounded empty command storage.
Stream shutdown precedes silent consumption, preserving a single transport consumer.
Tests distinguish logical silence, delayed retention and eventual destruction with
weak references; owner release is no longer inferred merely from silent PCM.
The [callback qualification](../../../docs/vm-arch/audio-callback-qualification.md)
records zero runtime allocator calls in debug/release paths, the capacity-saturation
case and local device timing. A physical release run observed 375 callbacks on
MacBook Pro Speakers with 256-frame buffers and a 30.458 µs maximum runtime duration;
all sixteen owners released at teardown. Its asset-pack reads and temporary 1 MiB
host-file writes are an explicitly limited pressure workload, not guest save or
asset-pipeline certification. Earlier device-open/stall failures are retained in
the evidence rather than erased by a successful retry. The
[audio specification](../../../docs/specs/runtime/05-audio-peripheral.md) defines
the bounded reclamation contract; timing measurements do not define device deadlines.
@@ -109,6 +109,36 @@ AGD-0055's future plan. It does not approve that agenda's proposal, weaken the
final PBX contract or label transferred code as executed. Other agendas retain
their own collection/string/stress/game obligations.
## Index ownership identity without owning content
Provenance is an allocation-identity question, not an equality-of-text question.
The image index sorts the existing Weak records by address and length during
sealed loading. The session index uses a half-full open-addressed table. Both
preserve Arc values and explicit native leases, so ordinary string reads remain
direct and accounting does not keep content alive strongly.
The allocation lifetime matters as much as the key. A Weak prevents the allocator
from recycling an address while its record remains indexed. Remove the key before
dropping that Weak. Tombstones preserve collision chains, and a metered rebuild
removes them without losing live identities. Admit the new capacity and its
coexistence with old storage before moving entries or constructing new strings;
failed admission must leave live owners recognizable and unpublished charges
refunded.
The [release qualification](../../../docs/vm-arch/string-provenance-qualification.md)
measured the existing linear boundary queries before changing them. At 65,536
owners the indexed image took 16–17 comparisons and the session fixtures took
1–2 probes, with no lookup allocation. The tradeoff is explicit: session metadata
rose from about 4 MiB to 10 MiB and setup became more expensive under the same
quotas. This is boundary-work evidence, not a claim of faster game frames.
Synthetic collision keys and work-limit tests complement real-address timings;
allocator-selected address order is not a stable test oracle across loads.
This implementation follows DEC-0065, summarized in the [runtime quality lesson](../DSC-0064-runtime-branch-quality-review/LSN-0070-runtime-quality-needs-independent-oracles-and-owned-evidence.md).
The [published ownership contract](../../../docs/specs/runtime/02c-program-registry-and-pbx-v1.md#memory-ownership-and-quotas)
remains normative; native import, snapshot retention and typed storage tests
verify the same owner lifetimes through the new indices.
## Takeaways
- Keep each plan atomic to its originating agenda and identify integration ownership before execution.
@@ -40,12 +40,33 @@ already completed result, which exposed an ownership leak during integration.
Cancellation and entry into publication share an atomic control state. At the
deadline, a canceled attempt that never entered publication can report failure.
Once publication may have started, the result remains indeterminate until
reconciliation. A blocked host requires restart escalation even when publication
Once publication may have started, cancellation alone cannot determine the outcome.
A worker that returns a known success before escalation still reports success;
an uncertain publication requires reconciliation. A blocked host requires restart escalation even when publication
was provably prevented: storage and source ownership still have to remain valid.
The runtime neither kills the thread nor starts a replacement worker over its
resources. The watchdog runs outside that worker.
The watchdog cause is a third observation, independent of durable outcome and
resource lifetime. Keeping it only in the terminal result loses information:
a worker's generic cancellation error can overwrite the reason the cancellation
was requested. Conversely, setting a terminal timeout before the worker returns
can accidentally suppress its eventual success. The request now records the
first watchdog cancellation separately, leaves the result Pending, and preserves
the original worker error for native diagnostics. Completion before escalation
combines the actual durable state with guest reason 15, including Success/15.
Removal or termination cancellation that happened first retains its own reason.
This correction follows accepted decision DEC-0058. Deterministic regression
fixtures hold queued work until after the simulated 500ms poll and deliver it
at 501ms. Channel barriers hold IO before or inside publication, exercising
Failure/15, Success/15, Indeterminate/15 and immutable restart results after 1s.
The tests invoke the actual guest query handler and assert all three ABI slots;
wall-clock receive limits only detect a hung fixture. They also verify deferred
and composite bootstrap saves, dirty snapshots, durable bytes after reopening,
result release and teardown isolation. These checks establish behavior under
controlled scheduling, not real-device latency.
## A checksum does not make an old reference safe
Two valid map pages can reference different generations of payload and public
@@ -0,0 +1,122 @@
---
id: LSN-0070
ticket: runtime-branch-quality-review
title: Runtime Quality Needs Independent Oracles and Owned Evidence
created: 2026-10-14
tags: [quality, runtime, ownership, validation]
---
## Context
A branch review found eight gaps spanning deadline reasons, coverage ownership,
one-shot audio, recurring VM allocations, callback reclamation, CI enforcement,
asset decoder parity and string provenance lookup. Accepted decisions DEC-0058
through DEC-0065 were executed by PLN-0189 through PLN-0196. This lesson preserves
the resulting reasoning after removal of those workflow artifacts; the linked
published contracts remain normative.
The common failure mode was evidence that proved a nearby property while leaving
the intended property unchecked. A callback can avoid locks while still freeing
memory. Two decoders can agree on output while accepting the same malformed file.
A coverage percentage can rise because ownership moved, without any new test.
A fast lookup can hide expensive construction and uncharged metadata.
## Key Decisions
### Observe the boundary where the guarantee becomes true
The [memcard contract](../../../docs/specs/runtime/08-save-memory-and-memcard.md)
keeps the first deadline escalation reason stable while distinguishing durable
completion from pending or failed work. Fake-clock boundary tests and explicit
publication ordering make those distinctions observable without sleeping.
The [audio contract](../../../docs/specs/runtime/05-audio-peripheral.md) distinguishes
one-shot completion from looping and logical inactivity from physical ownership.
A finished voice may become silent before its sample can be reclaimed. Two bounded
return packets move sample and vector owners to host service; pool exhaustion
retains bounded ownership and defers delivery. Count both allocations and frees
inside the callback, then independently prove eventual owner release outside it.
### Shared mechanics still need independent expected results
The [asset contract](../../../docs/specs/runtime/15-asset-management.md) uses common
metadata validation and decoding mechanics for synchronous and incremental paths.
Conformance checks include independently specified pixels, palette bytes and PCM,
plus malformed metadata and physically truncated payloads. Agreement between two
entry points is useful, but cannot by itself establish correctness.
The real production decoder/save ordering fixture pauses at a safe decode step,
queues persistent work and checks writes and synchronization before decode resumes.
That deterministic ordering oracle is stronger evidence of priority than a latency
threshold on one host. Host latency remains a separate observation.
### Move cost deliberately and account for the destination
Recurring VM stack/return work reuses capacity after warmup; release allocator
checks measure the execution region separately from setup. String provenance uses
image and session indices under existing ownership quotas. Index metadata and
construction cost are charged and measured, including collision and refusal paths.
See the [Registry contract](../../../docs/specs/runtime/02c-program-registry-and-pbx-v1.md)
and [provenance qualification](../../../docs/vm-arch/string-provenance-qualification.md).
Lookup improvements do not establish faster game frames or cheaper construction.
### The repository owns acceptance and measurement identity
The [coverage policy](../../../docs/vm-arch/COVERAGE_DOMAINS.md) assigns each source
to exactly one domain, includes module descendants, and keeps acceptance floors
independent of observed percentages. Ownership repair and behavioral improvement
must be evaluated separately against the same report before comparing fresh runs.
The shared strict gate records source/config identity, preserves command failures,
removes stale reports and publishes available evidence even when a stage fails.
Local execution and CI call the same target. A successful focused test does not
replace the full gate; a successful local gate does not certify an unexecuted OS.
## Published Evidence and Its Limits
The final source commit is `3f9d2355462d3b5675f5edc6e488ab325eb024d8`, with Rust
fingerprint `309a069f8b0560d1e9d1f9f4e011b709c89729383aeacd100c8c2ff5c3cff28d`.
On macOS/aarch64 with rustc 1.98.1, all twelve `make ci-domains` stages passed:
1,060 ordinary tests, five socket tests, three runner tests and all fifteen fixed
coverage floors. Thirteen ordinary tests were ignored. The fresh coverage replay
also passed 1,060 tests; those are repeated tests, not additional unique cases.
[Current coverage evidence](../../../docs/vm-arch/COVERAGE_DOMAINS_CURRENT.json)
retains the earlier ownership-repair comparison and identifies the final corpus.
The ten focused coverage tests and two release/all-feature VM allocation tests
passed again during closure without source changes.
[Audio qualification](../../../docs/vm-arch/audio-callback-qualification.md) retains
the named physical-device smoke and the initial device failures. Complementary
release measurement ran 1,000 sixteen-voice callbacks while a separate process
repeated the real production decoder/FileMemcard fixture fourteen times. Callback
p50/p95/p99/max were 14.542/97.667/399.041/1,070.708 microseconds against a nominal
5,333.333-microsecond period. Callback allocation/free counts were both zero;
sample owners went from sixteen to zero after teardown. The
[measurement record](../../../docs/vm-arch/audio-production-pressure-qualification.json)
records overlapping process intervals, environment and individual pressure rounds.
This is host-concurrent pressure evidence, not same-process console integration
or a worst-case latency guarantee. Each pressure round alternates twenty production
loads and durable 64-KiB commits; the separate deterministic fixture establishes
save preemption during decoding. Physical audio, synthetic timing, allocator
counts and save ordering answer different questions and retain separate scopes.
No Linux/remote CI, human listening assessment, long-duration soak or gameplay
speedup is claimed by this closure.
## Related Domain Lessons
- [Save publication and resource ownership](../DSC-0061-memcard-fixed-slots-v1/LSN-0067-save-publication-and-resource-ownership-are-separate.md).
- [Repository-owned domain gates](../DSC-0002-runtime-edge-test-plan/LSN-0037-domain-gates-must-be-owned-by-the-repository.md).
- [Audio delivery and lifecycle](../DSC-0055-guest-audio-voice-control/LSN-0069-audio-admission-needs-a-delivery-boundary-and-independent-lifecycle-control.md).
- [Materialization and recurring VM cost](../DSC-0014-perf-vm-allocation-and-copy-pressure/LSN-0035-first-materialization-is-not-the-same-as-hot-path-copy-pressure.md).
- [Serial async lanes](../DSC-0009-perf-async-background-work-lanes-for-assets-and-fs/LSN-0050-serial-async-lanes-bound-backlog-complexity.md).
- [Registry boundaries and owned values](../DSC-0058-program-registry-type-and-function-metadata/LSN-0059-registry-boundaries-and-owned-values.md).
## Pitfalls and Takeaways
- Test the promised boundary with an independent oracle, including failure paths.
- Separate admission, execution, publication and reclamation when reasoning about cost.
- Preserve owner counts and quota accounting when replacing scans or moving work.
- Bind acceptance evidence to source/config fingerprints, not merely a nearby commit.
- Close deferred gates on the final source, while keeping environmental limits explicit.
- Preserve reusable reasoning and durable measurements; discard completed workflow scaffolding.
@@ -234,6 +234,23 @@ Image constants and session-created strings are separate accounting domains; cha
Each domain tracks UTF-8 bytes and allocation count using a bounded Weak<str> ledger without a strong accounting owner. Reads remain direct Arc accesses, not handle-table lookups. Reconcile dead entries at safepoints and before quota refusal, release Weak plus charge then; Weak may retain native allocation after last strong drop, so no immediate-release promise. Keep accounting domains alive while tracked allocations survive, including host/tool retention. Import untracked native strings as charged new allocations; explicit session transfer copies into the destination domain, while shared-image constants remain shared.
Provenance keys include both allocation data address and UTF-8 length. Image
Weak records are sorted once during sealed loading by a bounded in-place heap
sort, with record comparisons/reads/writes charged before work through LoadBudget;
binary search serves native boundary identity queries. This does not reorder
constants, exports or lease indices. Record capacities remain part of the
existing image payload accounting.
Session provenance uses a precharged open-addressed BudgetVec table with at most
50% live occupancy, geometric growth and explicit tombstones. Charge retained
capacity through AuxiliaryBytes and old/new capacity coexistence through
TemporaryBytes before rehashing. Reserve content and bucket growth before
materialization; refusal must not publish content or an entry and must refund
provisional reservations. Remove stale keys before releasing their Weak ownership;
reuse tombstones and rebuild under the same capacity accounting. Queries allocate
nothing, ordinary reads do not query the index, and resource ceilings and
reconciliation safepoints are unchanged.
Check result sizes/overflow and reserve before materializing; roll back reservations and initialized state on failure. Charge transient buffer/result coexistence. No hidden unmetered concatenation/native result paths. Do not introduce reentrant GC or scheduling at arbitrary allocation points to satisfy quotas; dead guest objects stay charged until safe sweep. Layout headers, metadata, payload capacities, native ownership, auxiliary buffers and temporary storage must have explicit nonduplicated domains. Limits are not RSS or upfront reservations.
LoadLimits uses the same defaults on hosts; cartridges cannot raise them. Test configurations may lower limits. MiB=1,048,576 bytes.
+46 -2
View File
@@ -84,12 +84,56 @@ Reset, termination, crash or cartridge replacement ends the audio session, stops
## 6 Samples, interpolation and ownership
Samples are immutable PCM16 mono, with a source sample rate and optional loop bounds. Linear interpolation uses fractional playback position. Playback rate combines source/output sample-rate ratio and pitch. Looping uses the sample's valid loop interval, or the full sample when no valid interval is present. Loop steps may cross multiple repetitions; finite positive pitch must not accidentally terminate a short loop or produce an infinite position.
Samples are immutable PCM16 mono, with a source sample rate and optional loop bounds. With looping off, playback ignores all loop markers and ends at the full sample length. With looping on, both markers must be present and satisfy `start < end <= sample length`; otherwise the loop interval is the full sample `[0, length)`. Playback starts at position zero, preserving the intro before the selected loop start, and wraps at the exclusive loop end. Empty samples terminate without reading PCM data.
Unloading or replacing a bank does not interrupt a sample already retained by an admitted command or mixer voice. New plays resolve the current bank. Memory can remain retained until the command is cancelled or the playback ends, stops, is replaced, or its session ends. The driver must not retain a second sample reference indefinitely after the mixer observes natural completion.
Linear interpolation uses fractional playback position. With looping off, the next value is the next actual sample frame, or zero beyond the full sample length. With looping on, interpolation across the selected loop end uses the loop-start frame. Playback rate combines source/output sample-rate ratio and pitch. Loop steps may cross multiple repetitions; finite positive pitch, including `f64::MAX`, must not accidentally terminate a short loop or produce an infinite position.
Unloading or replacing a bank does not interrupt a sample already retained by an admitted command or mixer voice. New plays resolve the current bank. Memory may remain retained after the command is cancelled or playback ends, stops, is replaced, or its session ends, until the host service reclaims the retired owner. Logical inactivity and physical reclamation are separate. The driver must not maintain a second playback mirror that cannot observe completion.
## 7 Host responsibilities and evidence
The desktop host consumes batches, mixes PCM and schedules the physical output. If no output device is available, it consumes committed batches silently. A failed or stalled device is diagnosed and detached so pending publication cannot block the guest indefinitely. The desktop implementation treats two seconds without callback progress as a stalled output.
Headless execution consumes committed batches silently and preserves incomplete frame/callback staging. It validates admission and runtime behavior, without proving audible output. Synthetic mixer tests establish signal behavior; a real device smoke establishes stream initialization and callback consumption, while listening remains a separate observation. Bit-identical PCM across physical devices is not required.
### Bounded desktop reclamation
The successful desktop data callback MUST perform no allocation or deallocation
of runtime-owned memory, including command-vector storage and the last Sample/PCM
owner. This excludes CPAL/device internals and asynchronous driver error reporting.
It is not an absolute device-latency guarantee.
Initialization allocates exactly two retirement packets outside the callback.
Each has room for two returned command-vector owners and 3,088 retired Sample
references: one 1,024-command frame, one combined 2,048-command lifecycle delivery,
and sixteen existing voices. Fixed-capacity SPSC free and return lanes each hold
two packets and MUST NOT grow. A checked-out packet implies a free return slot;
returning it never waits or frees overflow ownership.
The callback MUST obtain a packet before taking a publication. It moves commands
out of vector slots, retaining the vector allocation in the packet, and moves
replaced/stopped/reset voice owners into retirement storage. Stale deliveries are
returned whole. Natural completion marks a voice inactive. Without an available
packet, no publication is taken, accepted commands remain pending under existing
backpressure, and at most sixteen inactive voice slots retain their Sample owners.
Inactive slots produce silence and do not advance. They are existing mixer slots,
not another playback mirror. Pending pause/session transitions apply when the
mixer next obtains a packet and processes control state; already emitted device
buffers cannot be recalled.
`HostAudio::service` MUST drain and recycle packets independently of guest progress,
including paused/backpressured execution, before the next guest attempt. It frees
retired references and clears returned commands outside the callback. Empty vector
storage can be reused through a bounded four-buffer transport cache; excess empty
storage is released on the service lane. These buffers retain no Sample owners.
There is no change to guest quotas, FIFO/prefix order, session epochs, sixteen-voice
capacity, or pause position.
Failed/stalled output is stopped and dropped before silent consumption takes over,
so only one consumer drains the publication transport. Teardown releases callback,
packet and pending-publication ownership outside the data callback. Headless/test
hosts continue consuming silently through the same transport, without constructing
an audio callback or requiring a device.
See [callback qualification](../../vm-arch/audio-callback-qualification.md) for
executed allocation, signal, ownership and device observations.
@@ -236,6 +236,32 @@ not proof of rollback. If cessation is unconfirmed, isolate mount/resources,
block new work and retain source/scratch. Late completion belongs to original
reconciliation. Never kill threads unsafely, free buffers in use or create a
replacement worker. Report restart requirement when host recovery is impossible.
The first watchdog cancellation is recorded independently of the request result,
before signaling cancellation. Repeated polls preserve that cause. At 500ms the
request remains Pending with no terminal reason until the worker returns or the
1s escalation occurs. Removal or session termination that canceled first does
not become a watchdog timeout merely because a later poll crosses 500ms.
On completion before escalation, the worker's durable outcome determines
Success, Failure or Indeterminate. A recorded watchdog cancellation supplies
Timeout (guest reason 15) independently of that state; native diagnostics retain
the original worker storage error. For example:
- A save canceled while queued at 500ms, then completed at 501ms without entering
publication, reports Failure/15. The durable slot is unchanged.
- A save already inside publication at cancellation can finish successfully and
report Success/15. Its saved snapshot becomes clean; cancellation is not rollback.
- An uncertain publication reports Indeterminate/15 and requires reconciliation.
- At 1s without confirmed cessation, RestartRequired reports Failure/15 before
publication entry or Indeterminate/15 after entry. Late completion returns card
ownership for reconciliation but cannot rewrite this terminal guest result or
mark the snapshot clean. Releasing the result does not release in-use resources.
A deferred bootstrap save that has not reached the worker expires as Failure/15
at 500ms. Memory-only commits remain MemoryOnly with reason 0 and perform no IO.
The status/state/reason result slots and request identities do not change.
Fast-profile claims require measurements under asset load, including tail latency.
## 8. Player choice and lifecycle
+26
View File
@@ -242,6 +242,32 @@ If a scene glyph dependency cannot be resolved:
- draw/composition MUST fail as a fatal machine error with clear logging;
- runtime MUST NOT continue scene composition after the missing dependency is detected.
### 4.3 Decoding validation and physical payload completeness
Complete-buffer/reader decoding and resumable decoding MUST use the same driver
format authority for glyph layout, sound metadata and PCM conversion, and scene
headers, layers, tile ranges and decoded-size validation. Execution mode MUST NOT
change acceptance or semantic bank output for the same declared asset slice.
Every byte of the declared slice MUST be physically available. A short file read
MUST fail; it must not publish a partial sound or accept a physically complete
scene whose byte length is shorter than the declared asset size. This applies
even when the missing sound byte would have been an ignored odd PCM tail.
A physically present final unpaired PCM byte remains accepted and ignored.
The shared worker retains resumable state and reads at most 4096 payload bytes
per decode step. Scene header and layer steps read their fixed headers; tile
steps use complete tile records. PCM steps end on sample boundaries except for
the final odd tail. Save work takes priority at the next safe step. Cancellation
and target generation are checked before continuing and before staging a
complete bank; frame-boundary publication remains separate from decoding.
The conformance corpus checks explicit outputs and refusal outcomes across
buffer, reader, legacy load and resumable entry points. It covers empty supported
forms, chunk edges, palettes, truncated and trailing data, metadata, declared
sizes, scene ranges and nonfinite parallax. Ignored reserved fields remain
ignored; this validation sharing does not introduce new codecs or payload versions.
## 5 Banks and Slots
The current runtime exposes bank types:
+176 -24
View File
@@ -28,10 +28,10 @@ LLVM lines still need an owner.
| --- | --- |
| system/os | OS facades, process and task services |
| system/runtime | System-side VM lifecycle, scheduling and render integration |
| system/fs | Runtime filesystem and memcard service |
| system/fs | Runtime filesystem, memcard.rs and the complete memcard/ module tree |
| system/programs | Resident system programs |
| system/core | Remaining system sources, excluding the four specialized owners |
| asset/bank | Existing asset, cartridge and asset-driver prefixes |
| asset/bank | HAL asset/cartridge prefixes, driver asset.rs and the complete driver asset/ module tree |
| firmware | Firmware state and transitions |
| host-dependent | Desktop host |
| host/headless | Headless host and automation implementation |
@@ -43,6 +43,10 @@ LLVM lines still need an owner.
| dev/test-support | Shared test-program construction and development infrastructure source stubs |
Existing specialized paths retain their original ownership.
Both module entry files and sibling module directories need explicit prefixes.
Memcard and asset-driver descendants, including future nested modules, retain
their specialized owner even when absent from the coverage report or reporting
zero executable lines. The corresponding residual exclusions cover both forms.
Residual domains use broad crate source prefixes with explicit exclusions,
so a new module inherits the appropriate owner. A new crate or conflicting
prefix must be assigned before the ownership audit can pass.
@@ -60,6 +64,10 @@ Quality tests remain in prometeu-quality-checks/tests. They enforce the map
and drive the auditor/helper with independent source/report fixtures,
including omitted/overlapping owners, exclusions, absolute/relative filenames,
absent/empty evidence, invalid statistics and percentage regression.
Expected-owner tests use the auditor's ownership function for entry files,
every existing descendant and future nested paths. Synthetic reports also
check absent and zero-line descendants. Exclusive residual fallback is not
sufficient: it can silently transfer specialized code into the wrong gate.
The empty quality/layer library stubs inherit infrastructure ownership;
they do not create separate gates with zero executable lines.
@@ -93,13 +101,16 @@ this percentage is at least baseline_percent. Counts for functions and
regions remain supporting evidence. Scenario/contract assertions continue
to govern behavior; a percentage does not replace them.
Measured coverage and the acceptance floor are separate values. This explicitly
reviewed revision gives every domain at least 5.00 percentage points of headroom
from the recorded local measurement. Its rule is
Measured coverage and the acceptance floor are separate values. The historical
margin revision gave every domain at least 5.00 percentage points of headroom
from that recorded local measurement. Its rule was
`new_floor = min(original_floor, measured_percent - 5.00)`, using the same
downward truncation. Floors already providing that margin remain unchanged.
Five percentage points means an 88.56% measurement permits an 83.56% floor;
it is not a relative five-percent reduction.
That historical margin is not a guarantee about later measurements or corrected
ownership. The current map restores the reviewed system/core floor of 87.71%
and preserves every other floor.
Floors remain fixed during CI. A failing run does not automatically lower them.
An intentional scope or floor revision requires explicit review with fresh
@@ -108,30 +119,107 @@ fresh measurement.
## Commands and CI
List domains, inspect source ownership and validate coverage evidence:
`make ci-domains` is the authoritative repository quality gate. Both
[GitHub CI](../../.github/workflows/ci.yml) and
[Jenkins](../../files/config/Jenkinsfile) provision dependencies and invoke that
same target. `make ci` is an alias. Invoke one gate per checkout at a time.
make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
python3 scripts/audit-coverage-domains.py --report target/llvm-cov/report.json --json
The Makefile supplies this ordered sequence to the serial evidence runner:
Produce a fresh report and run all gates:
1. Clean LLVM coverage instrumentation from the previous run.
2. Check formatting with `cargo fmt --all -- --check`.
3. Run `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
4. Test gate failure/report handling with the Python regression suite.
5. Run `cargo test --workspace --all-targets --all-features --no-fail-fast`.
6. Run the five ignored debugger socket tests through `make test-debugger-socket`.
7. Generate fresh workspace/all-feature LLVM coverage, HTML, full JSON,
Cobertura XML and summary JSON.
8. Audit source ownership/evidence and check every fixed coverage domain floor.
make ci-domains
The socket command is deliberately restricted:
The existing Jenkins configuration invokes this repository target. Its
coverage-domain-gate step also performs the ownership/evidence audit.
GitHub's workspace-test step discovers the quality regression tests; it
does not separately produce a domain coverage report.
```sh
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
```
Inspect a domain:
It selects `test_debug_port_opens`, `test_debug_reconnection`,
`test_debug_refuse_second_connection`, `test_get_state_returns_response` and
`test_debug_resume_on_disconnect`. It does not select the audio-device test.
Local sandbox TCP permissions must permit bind/connect; a permission denial is
not evidence of a debugger behavior regression.
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core
scripts/coverage-domain-evidence.sh --check-all
The runner executes child Make targets with `-j1` even if the outer invocation
uses `-j`. CI invokes `make ci-domains` serially. Do not run a separate Cargo,
coverage-clean or report process concurrently in the same checkout: serialized
steps inside one gate cannot protect against an independent process.
Dependencies are the existing Rust/LLVM coverage tooling, Bash, Git, jq and rg,
plus Python 3 for the standard-library ownership auditor.
### Prerequisites and reports
Both pipelines use `bash scripts/provision-quality-gate.sh`. A Debian/Ubuntu
agent needs root or sudo for apt provisioning, and an existing Rust/rustup
installation. GitHub installs stable Rust first; Jenkins must supply a supported
Rust toolchain and rustup in CARGO_HOME/bin. The script installs rustfmt, Clippy,
llvm-tools-preview and pinned cargo-llvm-cov 0.8.5 (`--locked`).
Linux packages include build-essential, pkg-config, libasound2-dev, libudev-dev,
libxkbcommon-dev, libwayland-dev, libx11-dev, libxrandr-dev, libxi-dev, Python 3,
jq, ripgrep, Git, curl and CA certificates. ALSA development files satisfy CPAL's
Linux build dependency; a running audio device is not needed. Non-Linux local
hosts must already provide native compiler/SDK tools, Bash, Make, pkg-config,
Python 3, jq, rg and Git. The provisioning script verifies these commands.
`target/quality-gate/metadata.json` records revision, dirty working tree, host,
Rust/coverage-tool versions, configuration/source fingerprints, ordered step
exit codes and report hashes. Each step has a separate combined-output log.
The runner stops on the first failed Make target and returns its nonzero exit
status; the log preserves underlying Cargo diagnostics. Three regression cases
check failure logs/identity, fresh success evidence and partial-report retention
when a later gate fails.
Both pipelines discard old generated report directories before provisioning and
publish current artifacts even after gate failure. An early failed gate has
metadata/logs but no old coverage report. Failure after generation preserves the
partial current reports. Provisioning failure can leave no report at all and
still fails CI. Reports are archived from `target/quality-gate/**` and
`target/llvm-cov/**`; Jenkins also publishes current HTML when present. Artifact
publication does not turn a failed gate into success.
List domains or inspect ownership and existing evidence:
```sh
make coverage-domain-list
python3 scripts/audit-coverage-domains.py
make coverage-domain-audit
make coverage-domain-evidence DOMAIN=vm/core
scripts/coverage-domain-evidence.sh --check --verbose hal/core
```
### Explicit manual qualification
Ordinary all-target tests and coverage leave ignored device/measurement tests
ignored. The shared target only opts in to the five socket tests above. It does
not certify the following manual workloads:
```sh
# Physical desktop audio, exact named test.
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
# Physical callback duration under host I/O pressure.
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
# Eight-hour active release soak.
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
# Packaged installation; set PROMETEU_PROTOCOL_TEST_INSTALL to an extracted distribution first.
cargo test -p prometeu-cli --test automation_process packaged_installation_smoke -- --ignored --exact --nocapture
# File-host commit latency with the production asset decoder.
cargo test --release -p prometeu-system --all-features qualify_file_commits_with_production_asset_decoder -- --ignored --nocapture
# Explicit layout observation.
cargo test --release -p prometeu-bytecode --test registry_v1 reference_map_probe -- --ignored --exact --nocapture
```
Other ignored timing probes remain opt-in by their individual test names.
All-target execution can print ordinary benchmark observations; this does not
establish completion of ignored manual qualification. For scope and prior measured
results see [resource stability](resource-stability-qualification.md) and
[audio callback qualification](audio-callback-qualification.md).
## Historical local baseline
@@ -153,14 +241,14 @@ on macOS/aarch64:
These are historical starting floors; current acceptance floors appear below.
## Current local margins
## Historical margin revision
The [margin evidence](COVERAGE_DOMAINS_MARGIN.json) records fresh measurements,
original and revised floors, toolchain, fingerprints and executed validation.
Only domains with less than 5.00 percentage points of headroom were adjusted.
The historical evidence above remains unchanged.
| Domain | Measured lines | Original floor | Current floor | Margin (points) |
| Domain | Measured lines | Original floor | Revised floor | Margin (points) |
| --- | ---: | ---: | ---: | ---: |
| system/os | 82.24% | 77.24% | 77.24% | 5.00 |
| system/runtime | 81.19% | 76.50% | 76.19% | 5.00 |
@@ -183,6 +271,70 @@ The full local `make ci-domains` validation passed with 939 tests passed and
in verification; its final floor uses the lower measurement. All gates were
rechecked against the verification report after that final adjustment.
## Current coverage measurement
The [current evidence](COVERAGE_DOMAINS_CURRENT.json) identifies the committed
source by HEAD and content fingerprints, records the strict shared gate and
retains the earlier ownership-repair measurement as historical evidence.
Both original historical JSON records above remain unchanged. The latest
measurement uses the repaired module-tree map with all 15 fixed floors.
| Domain | Covered / total lines | Measured lines | Fixed floor | Margin (points) |
| --- | ---: | ---: | ---: | ---: |
| system/os | 1410 / 1711 | 82.40% | 77.24% | 5.16 |
| system/runtime | 2757 / 3309 | 83.31% | 76.19% | 7.12 |
| system/fs | 1946 / 2469 | 78.81% | 76.08% | 2.73 |
| system/programs | 556 / 567 | 98.05% | 93.05% | 5.00 |
| asset/bank | 3176 / 3434 | 92.48% | 86.76% | 5.72 |
| firmware | 1990 / 2314 | 85.99% | 83.20% | 2.79 |
| host-dependent | 1531 / 2393 | 63.97% | 40.05% | 23.92 |
| host/headless | 739 / 855 | 86.43% | 83.39% | 3.04 |
| vm/core | 8522 / 9526 | 89.46% | 84.29% | 5.17 |
| bytecode | 3085 / 3303 | 93.39% | 88.26% | 5.13 |
| hal/core | 2093 / 2369 | 88.34% | 83.56% | 4.78 |
| drivers/core | 2775 / 3107 | 89.31% | 79.51% | 9.80 |
| system/core | 1353 / 1444 | 93.69% | 87.71% | 5.98 |
| tooling | 396 / 612 | 64.70% | 59.75% | 4.95 |
| dev/test-support | 99 / 115 | 86.08% | 80.96% | 5.12 |
The fresh macOS/aarch64 `make ci-domains` run passed all 15 domain gates,
1,060 ordinary tests (13 ignored), five explicit socket tests and three
runner regression tests. The fresh coverage replay also passed 1,060 tests;
it is not an additional set of unique behavioral tests. All ten focused
ownership/evidence tests passed again on the committed source. No floor
was lowered. The retained ownership-repair measurement used the earlier
non-strict gate; the current measurement passes strict workspace Clippy.
Platform-specific compilation and host scenarios can change report totals.
Local evidence does not certify an unexecuted Linux or Intel Mac run; evaluate
those runs against the configured floors and investigate differences.
## Strict shared-gate qualification
On 2026-10-14, macOS/aarch64 with rustc 1.98.1, the shared target passed strict
workspace/all-target/all-feature Clippy, all 1,060 ordinary tests (13 ignored),
the five explicitly selected socket tests, three gate-runner regression tests,
and all 15 fixed coverage domain gates. The separate fresh coverage execution
also passed 1,060 tests with 13 ignored. Those repeated runs are not distinct
behavioral tests and must not be added into a unique-test count.
The baseline strict invocation failed on the concrete VM lint sources above;
the repaired sources pass without new lint allowances. Socket tests initially
failed because the sandbox denied localhost TCP operations; the authorized
outside-sandbox run passed all five. Gate failure fixtures preserve nonzero
Make status, logs and commit/config identity, discard stale coverage, and retain
fresh partial reports when a later gate fails. Interrupted runners record failure
rather than success.
GitHub's YAML was parsed locally, and both pipeline definitions were inspected
for shared provisioning, `make ci-domains` and unconditional failure-artifact
publication. No remote GitHub or Jenkins build was launched. No Linux gate ran:
the local Docker API was denied by the sandbox and the unrestricted daemon query
did not return a usable result before interruption. Local macOS coverage does not
certify Linux compilation or domain percentages. Such failures must be investigated
on that platform without lowering the existing floors.
No physical audio, eight-hour soak, packaged-installation or ignored manual
latency/layout qualification was selected by this gate. Prior manual observations
remain separate evidence. Current executable metadata and reports are emitted
under `target/quality-gate/` and `target/llvm-cov/` by every invocation.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,121 @@
# Desktop audio callback qualification
The desktop callback moves discarded Sample references and command-vector owners
into two preallocated return packets. Host service reclaims them outside the data
callback. Logical playback completion does not imply immediate physical release.
The [audio peripheral contract](../specs/runtime/05-audio-peripheral.md) is normative.
## Capacity and ownership proof
One publication can contain 1,024 frame commands and a combined 2,048-command
lifecycle batch. Each command introduces at most one Sample reference, and the
mixer starts with at most sixteen. Therefore at most 3,088 Sample owners can retire
in one callback, even if every command replaces a voice and every resulting voice
finishes immediately. Each packet reserves that reference capacity plus two vector
slots before starting the stream. Stale command references remain in the returned
vectors instead of moving into the retired-reference vector.
Exactly two packets circulate through capacity-two SPSC lanes. While the callback
owns a packet, no more than one other packet can occupy the return lane. Enqueue
therefore needs no waiting, resizing or overflow disposal. Symmetrically, service
can always return a reclaimed packet to the free lane. If both packets await service,
the callback leaves publications untouched. Up to sixteen naturally completed
voices retain owners in their existing slots, becoming silent until reclamation
capacity returns. PCM content can be large; this is a bound on references and fixed
metadata, not an additional PCM byte quota.
Service clears retired references, clears commands and caches at most four empty
command vectors for producer reuse. It runs before guest attempts and during pause
or backpressure. Stream stop/drop precedes silent-consumer takeover. No additional
reclamation worker or playback mirror is introduced.
## Allocation and semantic evidence
`tests/audio_callback_allocations.rs` installs a thread-local allocator counter.
Only runtime callback execution is counted; fixture construction, I/O, reporting,
service reclamation and teardown run outside that region. It counts alloc,
alloc_zeroed, realloc and dealloc (realloc contributes to both counters).
The original last-owner natural completion made zero allocations and two
deallocations; applying a control batch freed one vector. After the change,
debug and release counters report zero allocations/deallocations for natural
completion, all controls, repeated play/stop/replacement, pause, epoch reset,
empty samples, bank unload, 16-voice completion without a packet, ten starved
callbacks and recovery. The saturation case combines 3,072 Play commands with
sixteen existing voices and completion, exercising the full 3,088-owner bound.
Weak-reference assertions prove retention until service and final release.
Unit tests additionally check stale-vector identity and deferred release, 1,000
pool recycling cycles, vector-allocation reuse, bounded empty-buffer caching,
failed/stalled stream takeover and teardown. Existing exact PCM tests retain
one-shot tails, loop seams, fractional interpolation, FIFO/prefix order and pause
position. Runtime audio controls and headless tests retain their logical admission
behavior; the Rust silent-consumer interface remains unchanged.
Reproduce allocation qualification with:
```sh
cargo test -p prometeu-host-desktop-winit --test audio_callback_allocations -- --nocapture
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations -- --nocapture
```
## Timing observations
Executed on 2026-10-14, macOS/aarch64, rustc 1.98.1, release profile. Both workloads
used sixteen voices and 256 stereo frames at 48 kHz (nominal period 5.333 ms).
The explicitly named pressure workload reads the checked-in stress-console asset
pack and writes a temporary 1 MiB file through HostDirBackend. It does not execute
guest asset decoding or the memcard save scheduler; it is not a substitute for a
separate full integration workload.
| Workload | Callbacks | I/O cycles | p50 | p95 | p99 | Maximum |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Synthetic callback invocation, interleaved I/O | 1,000 | 1,000 | 13.791 µs | 30.458 µs | 54.292 µs | 72.166 µs |
| MacBook Pro Speakers, concurrent main-thread I/O | 375 | 273 | 11.458 µs | 12.875 µs | 15.916 µs | 30.458 µs |
Both observed zero runtime allocator calls and sixteen live Sample owners before
teardown, zero afterward. Physical buffers were observed at exactly 256 frames.
Timing includes the runtime render call and counter overhead, excludes I/O and
CPAL/device internals, and does not measure device presentation latency. No timing
threshold is asserted. These are local observations, not cross-device certification.
Explicit commands (both timing tests are ignored by ordinary CI):
```sh
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
cargo test --release -p prometeu-host-desktop-winit --test audio_callback_allocations physical_callback_duration_16_voices_with_io_pressure -- --ignored --exact --nocapture
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
```
The ordinary physical smoke passed on retry. The first sandboxed attempt could
not open the device; the first unrestricted attempt received no callbacks and
triggered the existing two-second watchdog. Those failures remain part of the
qualification evidence and demonstrate why successful local timing is not an
availability guarantee. No human listening assessment is claimed.
Strict host Clippy passes. Repository-wide strict-gate status remains separate
from these behavioral and device observations.
## Production decode and save pressure
A complementary release observation on 2026-10-14 used the same 1,000-callback,
16-voice synthetic harness while a separate process repeatedly ran
`qualify_file_commits_with_production_asset_decoder` from `prometeu-system`.
Each pressure round performs twenty production glyph loads and durable 64-KiB
FileMemcard commits. Fourteen rounds completed successfully during the audio
process interval. The two test binaries were built before measurement; a worker
started the first pressure process before audio and repeated it until audio
exited, then joined the final invocation. Monotonic process intervals confirm
concurrency in the [measurement record](audio-production-pressure-qualification.json).
Callback p50/p95/p99/max were 14.542/97.667/399.041/1,070.708 microseconds for the
nominal 5,333.333-microsecond period. Allocation and deallocation counts remained
zero; sixteen live sample owners became zero after teardown. All test processes
exited successfully. The record identifies the committed source, toolchain,
per-round save observations and the additional policy tests running on the host.
This measures host contention between processes, not an integrated console
session or a physical callback under this exact workload. The previously named
physical-device observation remains separate. The pressure fixture alternates
loads and saves; deterministic production-decoder tests establish save preemption
during decoding. None of these host-specific maxima is a real-time guarantee.
@@ -0,0 +1,255 @@
{
"scope": "Concurrent host processes: real production decode/FileMemcard fixture and 16-voice callback; not a same-process console integration",
"audio": {
"start_ns": 1213231353849333,
"end_ns": 1213235637563291,
"exit_code": 0,
"callbacks": 1000,
"voices": 16,
"frames": 256,
"nominal_period_ns": 5333333,
"p50_ns": 14542,
"p95_ns": 97667,
"p99_ns": 399041,
"max_ns": 1070708,
"allocations": 0,
"deallocations": 0,
"owners_before": 16,
"owners_after": 0
},
"pressure": [
{
"start_ns": 1213231351766125,
"end_ns": 1213232142082875,
"exit_code": 0,
"first_io_us": {
"p50": 8,
"p95": 43,
"max": 220
},
"commit_us": {
"p50": 14187,
"p95": 29711,
"max": 30902
},
"samples": 20
},
{
"start_ns": 1213232142089250,
"end_ns": 1213232647182500,
"exit_code": 0,
"first_io_us": {
"p50": 6,
"p95": 15,
"max": 29
},
"commit_us": {
"p50": 16130,
"p95": 19710,
"max": 23276
},
"samples": 20
},
{
"start_ns": 1213232647184250,
"end_ns": 1213232911448208,
"exit_code": 0,
"first_io_us": {
"p50": 8,
"p95": 38,
"max": 59
},
"commit_us": {
"p50": 8550,
"p95": 10179,
"max": 10965
},
"samples": 20
},
{
"start_ns": 1213232911451958,
"end_ns": 1213233156142708,
"exit_code": 0,
"first_io_us": {
"p50": 9,
"p95": 15,
"max": 26
},
"commit_us": {
"p50": 8981,
"p95": 10341,
"max": 11438
},
"samples": 20
},
{
"start_ns": 1213233156146791,
"end_ns": 1213233422857166,
"exit_code": 0,
"first_io_us": {
"p50": 8,
"p95": 18,
"max": 19
},
"commit_us": {
"p50": 10074,
"p95": 12108,
"max": 12709
},
"samples": 20
},
{
"start_ns": 1213233422860875,
"end_ns": 1213233691177833,
"exit_code": 0,
"first_io_us": {
"p50": 7,
"p95": 23,
"max": 89
},
"commit_us": {
"p50": 10453,
"p95": 12621,
"max": 12941
},
"samples": 20
},
{
"start_ns": 1213233691180500,
"end_ns": 1213233907892666,
"exit_code": 0,
"first_io_us": {
"p50": 6,
"p95": 17,
"max": 48
},
"commit_us": {
"p50": 7962,
"p95": 10914,
"max": 11195
},
"samples": 20
},
{
"start_ns": 1213233907895666,
"end_ns": 1213234115777250,
"exit_code": 0,
"first_io_us": {
"p50": 15,
"p95": 111,
"max": 245
},
"commit_us": {
"p50": 8040,
"p95": 12131,
"max": 14538
},
"samples": 20
},
{
"start_ns": 1213234115783333,
"end_ns": 1213234378236041,
"exit_code": 0,
"first_io_us": {
"p50": 18,
"p95": 77,
"max": 143
},
"commit_us": {
"p50": 9393,
"p95": 11115,
"max": 11919
},
"samples": 20
},
{
"start_ns": 1213234378243041,
"end_ns": 1213234604680916,
"exit_code": 0,
"first_io_us": {
"p50": 10,
"p95": 94,
"max": 1120
},
"commit_us": {
"p50": 8773,
"p95": 10268,
"max": 11906
},
"samples": 20
},
{
"start_ns": 1213234604683875,
"end_ns": 1213234822563625,
"exit_code": 0,
"first_io_us": {
"p50": 9,
"p95": 104,
"max": 188
},
"commit_us": {
"p50": 8485,
"p95": 10250,
"max": 11311
},
"samples": 20
},
{
"start_ns": 1213234822567208,
"end_ns": 1213235110141666,
"exit_code": 0,
"first_io_us": {
"p50": 17,
"p95": 55,
"max": 122
},
"commit_us": {
"p50": 8975,
"p95": 16481,
"max": 16947
},
"samples": 20
},
{
"start_ns": 1213235110144333,
"end_ns": 1213235379186208,
"exit_code": 0,
"first_io_us": {
"p50": 13,
"p95": 26,
"max": 49
},
"commit_us": {
"p50": 9699,
"p95": 13929,
"max": 15228
},
"samples": 20
},
{
"start_ns": 1213235379189083,
"end_ns": 1213235676903791,
"exit_code": 0,
"first_io_us": {
"p50": 23,
"p95": 1085,
"max": 1339
},
"commit_us": {
"p50": 9024,
"p95": 20041,
"max": 21674
},
"samples": 20
}
],
"date": "2026-10-14",
"git_head": "3f9d2355462d3b5675f5edc6e488ab325eb024d8",
"source_sha256": "309a069f8b0560d1e9d1f9f4e011b709c89729383aeacd100c8c2ff5c3cff28d",
"host": "macOS-27.0.1-arm64-arm-64bit-Mach-O",
"rustc": "1.98.1",
"profile": "release",
"pressure_test": "prometeu-system: qualify_file_commits_with_production_asset_decoder",
"audio_test": "prometeu-host-desktop-winit: callback_duration_16_voices_with_io_pressure",
"orchestration": "Build both release test binaries first. Start a worker that repeatedly invokes the production pressure test with --ignored --nocapture. After the first pressure process starts, run the exact audio test with --ignored --exact --nocapture. Stop repetition when audio exits and join the final pressure invocation. Record monotonic start/end times and require all exits zero. Focused coverage-policy tests also ran during part of this host observation."
}
@@ -0,0 +1,62 @@
# VM execution allocation qualification
Successful stack reads construct diagnostics only on failure. Return completion
validates the exact frame height and ordered signature groups before rotating the
current frame suffix right by its result width and truncating parameters and
locals. Void returns truncate directly. This preserves result order and owners
without a temporary result vector or Value clones.
Only the returning frame is removed. Caller locals and continuation remain intact.
Direct RET and branches to function end share completion while retaining their
existing instruction charges. The last-frame safepoint still runs after results
are rooted in the operand stack. If GC scratch admission fails, those roots remain
published and no sweep occurs.
## Measurement
On 2026-10-14, aarch64 macOS with rustc 1.98.1, the dedicated
`crates/console/prometeu-vm/tests/execution_allocations.rs` integration binary
measured the following native allocation/deallocation calls. Both debug and
release produced the same counts:
| Instruction loop | Iterations | Before alloc/free | After alloc/free |
| --- | ---: | ---: | ---: |
| CALL / PUSH_I64 / RET / POP / JMP | 1,000 | 3,000 / 3,000 | 0 / 0 |
| CALL / RET / JMP (void) | 1,000 | 0 / 0 | 0 / 0 |
| CALL / two PUSH_I64 / RET / two POP / JMP | 1,000 | 5,000 / 5,000 | 0 / 0 |
| PUSH_I64 / DUP / ADD / POP / JMP | 1,000 | 6,000 / 6,000 | 0 / 0 |
Run `cargo test -p prometeu-vm --test execution_allocations -- --nocapture`
and the same command with `--release` before `-p`.
The allocator wrapper enables counting only on the executing test thread after
ten warm-up iterations. It includes alloc, alloc_zeroed, dealloc and realloc
(realloc contributes to both counters). Fixture construction, image validation,
reporting and destruction occur outside the measured region. The fixtures do not
create GC pressure. Error diagnostics, deliberate guest allocations, collection
and dead-string reclamation are outside this zero-allocation claim. This is an
instruction-region qualification, not a gameplay throughput measurement.
## Semantic and resource evidence
VM regressions exercise direct, closure and trait calls, taken and untaken
function-end branches, captured strings, ordered mixed result groups, Arc pointer
identity and heap receiver identity. A 1,024-slot mixed return checks the published
result-width ceiling and caller-local preservation. Internal tests check height
and type refusal before mutation, last-root closure survival, GC scratch refusal
without partial sweep, and successful collection after scratch admission returns.
Lifecycle tests continue to check saved continuation and completion boundaries.
Actual storage remains quota-accounted. Removing the disposable return vector
removes its TemporaryBytes reservation; a valid scalar return can therefore
succeed with zero temporary capacity when no collection is needed. This does not
raise a ceiling, introduce an unmetered cache, or bypass the temporary memory
required by a triggered collector. Operand and call stack capacity accounting,
GC thresholds, verifier contracts and syscall ABI are unchanged.
The focused VM and transitive-root layer tests qualify these behaviors. Workspace
quality-gate status is tracked separately; these measurements do not establish
that all repository diagnostics or platform checks pass.
See the [memory and allocation contract](../specs/runtime/03-memory-stack-heap-and-allocation.md)
and the [materialization and copy-pressure lesson](../../discussion/lessons/DSC-0014-perf-vm-allocation-and-copy-pressure/LSN-0035-first-materialization-is-not-the-same-as-hot-path-copy-pressure.md).
@@ -29,11 +29,17 @@ cargo test --workspace --features prometeu-host-headless/resource-diagnostics
cargo test --release --workspace --features prometeu-host-headless/resource-diagnostics
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --frames 216000
cargo run --release -p prometeu-vm --features resource-diagnostics --example resource_qualification -- --seconds 28800
cargo test --release -p prometeu-vm --features resource-diagnostics --lib -- --ignored --nocapture
cargo test --release -p prometeu-vm --features resource-diagnostics --lib sealed_image_load_probe -- --ignored --nocapture
cargo test --release -p prometeu-vm --features resource-diagnostics --lib storage_and_gc_probe -- --ignored --nocapture
cargo test --release -p prometeu-bytecode reference_map_probe -- --ignored --nocapture
cargo test -p prometeu-host-desktop-winit --lib -- --ignored
cargo test -p prometeu-host-desktop-winit --lib runner::tests -- --ignored
cargo test -p prometeu-host-desktop-winit --lib audio::control_tests::desktop_audio_device_smoke -- --ignored --exact --nocapture
```
The reproduction commands name manual probes individually. The original recorded
desktop qualification grouped ignored tests; current automation selects only the
five debugger socket cases and leaves device audio explicit.
The `resource-diagnostics` feature adds engineering observations through Rust
only. It adds no guest syscall. The eight-hour example is a manual command;
the repository CI, release workflow and Jenkins pipeline do not invoke it. Normal builds omit timing histograms, additional
@@ -0,0 +1,126 @@
# String provenance qualification
Measured on 2026-10-14, macOS arm64, Rust 1.98.1, release profile. The workload
uses 1,024, 16,384 and 65,536 independently allocated equal-text owners in each
image/session domain. Setup is outside query timing. Each query row executes
1,000 operations. The native import and mixed snapshot cases call the actual VM
boundary methods; a separate diagnostics feature exposes only the fixture.
Reproduce with:
```sh
cargo run --release -p prometeu-vm --features resource-diagnostics --example string_provenance_benchmark
cargo test -p prometeu-vm --test execution_allocations --all-features
```
## Structure and ownership
Image Weak records are sorted once by allocation data address and UTF-8 length.
An in-place heap sort precharges LoadBudget work before each record comparison
(four field reads plus comparison) and swap (two reads plus two writes). The
existing scratch/retained payload accounting includes the record capacity;
sorting adds no auxiliary allocation. Constant-pool and export order, including
lease indices, are unchanged. Binary search performs at most floor(log2(N))+1
record comparisons for a nonempty table. In-place dead-record retention preserves
sorted order.
Session records use an open-addressed BudgetVec table with no strong content
owner, at most 50% live occupancy, geometric growth and tombstones. Full identity
is checked after hashing; equal text in another allocation is a miss. Growth and
same-capacity rebuild reserve the retained capacity delta in AuxiliaryBytes and
the old allocation in TemporaryBytes before allocating or moving records.
Publication reserves content and table space before constructing an Arc. Lookup
neither reserves nor resizes. Reconciliation removes a key before releasing its
Weak and content charges, preventing recycled addresses from matching stale keys.
Dead content is still reclaimed at the existing safepoints/pressure pass.
## Query work and allocator calls
| Owners per domain | Image hit comparisons before → after | Image miss comparisons before → after | Session hit probes before → after | Session miss probes before → after |
| ---: | ---: | ---: | ---: | ---: |
| 1,024 | 1024 → 10 | 1024 → 10 | 1024 → 1 | 1024 → 1 |
| 16,384 | 16384 → 14 | 16384 → 15 | 16384 → 1 | 16384 → 1 |
| 65,536 | 65536 → 16 | 65536 → 17 | 65536 → 2 | 65536 → 1 |
The hit is the last-created owner, which was last in the original linear ledger.
The miss has equal text in a distinct allocation. Hash probe observations depend
on allocator-selected addresses and are not worst-case constant-time guarantees.
A separate fixed synthetic-key collision fixture verifies chains, full keys and
tombstones. Integration tests bound image comparisons and assert zero allocations,
reallocations and deallocations for image/session hits/misses and tracked imports
at all three sizes. Ordinary string reads still access Arc directly.
All direct queries and tracked imports reported `[0, 0, 0]` allocator calls
before and after. Mixed two-value snapshots reported `[1000, 0, 1000]`, for their
Value-vector storage. Untracked imports reported `[1001, 0, 1]`: 1,000 new Arcs
plus one table growth; dead content remains Weak-tracked until reconciliation.
The benchmark asserts zero calls in the lookup regions; setup, reporting and
fixture teardown are excluded. The CI allocator oracle is thread-local.
## Supporting timings
Elapsed milliseconds for 1,000 operations; a single local run per implementation,
not a percentile estimate or a CI threshold. Warmup, allocator layout, scheduling
and CPU frequency can affect these values.
| Owners | Workload | Linear (ms) | Indexed (ms) |
| ---: | --- | ---: | ---: |
| 1,024 | `image_hit` | 1.602 | 0.058 |
| 1,024 | `image_miss` | 1.511 | 0.046 |
| 1,024 | `session_hit` | 1.160 | 0.030 |
| 1,024 | `session_miss` | 0.684 | 0.025 |
| 1,024 | `import_hit` | 1.513 | 0.101 |
| 1,024 | `snapshot_mixed` | 2.031 | 0.216 |
| 1,024 | `import_miss` | 2.233 | 0.376 |
| 16,384 | `image_hit` | 10.658 | 0.026 |
| 16,384 | `image_miss` | 11.847 | 0.023 |
| 16,384 | `session_hit` | 10.890 | 0.009 |
| 16,384 | `session_miss` | 11.412 | 0.009 |
| 16,384 | `import_hit` | 21.726 | 0.035 |
| 16,384 | `snapshot_mixed` | 33.012 | 0.083 |
| 16,384 | `import_miss` | 22.832 | 0.887 |
| 65,536 | `image_hit` | 44.191 | 0.029 |
| 65,536 | `image_miss` | 43.959 | 0.025 |
| 65,536 | `session_hit` | 47.441 | 0.010 |
| 65,536 | `session_miss` | 45.249 | 0.009 |
| 65,536 | `import_hit` | 88.518 | 0.038 |
| 65,536 | `snapshot_mixed` | 132.199 | 0.094 |
| 65,536 | `import_miss` | 90.935 | 3.137 |
## Creation and memory cost
The `image_load_ns` diagnostic includes fixture assembly/serialization and sealed
loading, not just the sort. Session setup includes the VM, ledger and retained
owners. Auxiliary measurements include that fixture's small VM/domain state;
image payload measurements include the complete immutable payload, not only its
index. These are charged native capacities, not process RSS.
| Owners | Image fixture before → after (ms) | Session setup before → after (ms) | Session AuxiliaryBytes before → after | Image payload bytes (unchanged) | Session setup allocations before → after |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 1,024 | 0.600 → 1.128 | 0.236 → 0.348 | 65,696 → 164,016 | 74,072 | 1046 → 1044 |
| 16,384 | 1.702 → 4.067 | 1.298 → 2.889 | 1,048,736 → 2,621,616 | 1,179,992 | 16410 → 16408 |
| 65,536 | 6.248 → 14.130 | 5.230 → 9.885 | 4,194,464 → 10,485,936 | 4,718,936 | 65564 → 65562 |
At 65,536 session owners the charged auxiliary footprint rises from roughly 4 MiB
to 10 MiB. The 50% occupancy policy costs capacity and increases setup work, while
removing whole-ledger traversals at native boundaries. Existing quotas are
unchanged; this metadata can reduce available headroom for other session owners.
No extra strong Arc owner, guest handle, content interning or new GC schedule was
introduced.
## Refusal and scope of evidence
Tests force initial allocation, growth and tombstone rebuild failure, auxiliary
and coexistence limits, and image sort work exhaustion. They verify no partial
entry/content publication, retained live identities and refunded provisional
charges. Allocator failures are injected at the fallible allocation seam; stable
Arc allocation can still abort on physical allocator exhaustion, as before.
Cross-session copies, native lease/snapshot survival and typed storage/root tests
remain part of the regression suite. Address ordering can change between loads;
the sort refusal fixture is bounded for either order, not tied to a prior address
layout.
These synthetic boundary measurements establish lookup work and ownership
behavior. They do not establish gameplay/frame speedups, long-run memory stability,
other-platform timings or hardware-independent latency limits. The normative
ownership contract is in [Registry and PBX](../specs/runtime/02c-program-registry-and-pbx-v1.md#memory-ownership-and-quotas).
+34 -40
View File
@@ -4,55 +4,49 @@ pipeline {
environment {
CARGO_HOME = '/var/jenkins_home/.cargo'
CARGO_TARGET_DIR = 'target'
CARGO_TERM_COLOR = 'always'
}
stages {
stage('Build') {
stage('Prepare reports') {
steps {
dir('target/quality-gate') { deleteDir() }
dir('target/llvm-cov') { deleteDir() }
}
}
stage('Provision') {
steps {
sh '''
set -eux
set -eu
export PATH="$CARGO_HOME/bin:$PATH"
bash scripts/provision-quality-gate.sh
'''
}
}
stage('Quality gate') {
steps {
sh '''
set -eu
export PATH="$CARGO_HOME/bin:$PATH"
make ci-domains
'''
// withChecks(name: 'Test', includeStage: true) {
// sh '''
// set -eux
// make ci-domains
// '''
// recordCoverage(
// tools: [[parser: 'COBERTURA', pattern: 'target/llvm-cov/cobertura.xml']],
// sourceCodeRetention: 'LAST_BUILD',
// enabledForFailure: true,
// failOnError: false,
// checksAnnotationScope: 'MODIFIED_LINES',
// id: 'rust-coverage',
// name: 'Rust Coverage',
// checksName: 'Rust Coverage',
// qualityGates: [
// [metric: 'LINE', baseline: 'MODIFIED_LINES', threshold: 0.0],
// [metric: 'LINE', baseline: 'PROJECT', threshold: 0.0],
// [metric: 'BRANCH', baseline: 'PROJECT', threshold: 0.0]
// ]
// )
// }
}
} // Test
}
}
stage('Reports') {
steps {
publishHTML(target: [
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: 'target/llvm-cov/html',
reportFiles: 'index.html',
reportName: 'Rust Coverage HTML',
reportTitles: 'Coverage Report'
])
archiveArtifacts artifacts: 'target/llvm-cov/**', fingerprint: true
}
} // Reports
post {
always {
archiveArtifacts artifacts: 'target/quality-gate/**,target/llvm-cov/**',
fingerprint: true, allowEmptyArchive: true
publishHTML(target: [
allowMissing: true,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: 'target/llvm-cov/html',
reportFiles: 'index.html',
reportName: 'Rust Coverage HTML',
reportTitles: 'Coverage Report'
])
}
}
}
@@ -19,7 +19,8 @@
"baseline_percent": 76.08,
"paths": [
"crates/console/prometeu-system/src/services/fs/",
"crates/console/prometeu-system/src/services/memcard.rs"
"crates/console/prometeu-system/src/services/memcard.rs",
"crates/console/prometeu-system/src/services/memcard/"
]
},
"system/programs": {
@@ -36,7 +37,8 @@
"crates/console/prometeu-hal/src/asset_bridge.rs",
"crates/console/prometeu-hal/src/cartridge.rs",
"crates/console/prometeu-hal/src/cartridge_loader.rs",
"crates/console/prometeu-drivers/src/asset.rs"
"crates/console/prometeu-drivers/src/asset.rs",
"crates/console/prometeu-drivers/src/asset/"
]
},
"firmware": {
@@ -88,11 +90,12 @@
"crates/console/prometeu-drivers/src/"
],
"exclude_paths": [
"crates/console/prometeu-drivers/src/asset.rs"
"crates/console/prometeu-drivers/src/asset.rs",
"crates/console/prometeu-drivers/src/asset/"
]
},
"system/core": {
"baseline_percent": 80.0,
"baseline_percent": 87.71,
"paths": [
"crates/console/prometeu-system/src/"
],
@@ -103,6 +106,7 @@
"crates/console/prometeu-system/src/services/vm_runtime",
"crates/console/prometeu-system/src/services/fs/",
"crates/console/prometeu-system/src/services/memcard.rs",
"crates/console/prometeu-system/src/services/memcard/",
"crates/console/prometeu-system/src/programs/"
]
},
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Shared CI prerequisites. Run on a disposable Debian/Ubuntu agent with sudo
# (or root), or on an already provisioned macOS development machine.
set -euo pipefail
if [[ "$(uname -s)" == Linux ]]; then
admin=()
if [[ "$EUID" -ne 0 ]]; then admin=(sudo); fi
"${admin[@]}" apt-get update
"${admin[@]}" apt-get install -y --no-install-recommends \
build-essential pkg-config libasound2-dev libudev-dev libxkbcommon-dev \
libwayland-dev libx11-dev libxrandr-dev libxi-dev \
python3 jq ripgrep git curl ca-certificates
fi
for command in rustup cargo python3 jq rg git make pkg-config; do
command -v "$command" >/dev/null || { echo "Missing quality-gate prerequisite: $command" >&2; exit 1; }
done
rustup component add rustfmt clippy llvm-tools-preview
if [[ "$(cargo llvm-cov --version 2>/dev/null || true)" != 'cargo-llvm-cov 0.8.5' ]]; then
cargo install cargo-llvm-cov --version 0.8.5 --locked
fi
if [[ "$(uname -s)" == Linux ]]; then pkg-config --exists alsa; fi
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""Run Makefile-owned gate steps serially and retain evidence on failure."""
import hashlib
import json
import os
from pathlib import Path
import platform
import shutil
import subprocess
import sys
from datetime import datetime, timezone
def capture(*command):
try:
result = subprocess.run(command, text=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, check=False)
return {"exit_code": result.returncode, "output": result.stdout.strip()}
except OSError as error:
return {"exit_code": 127, "output": str(error)}
def sha(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def main():
root = Path(__file__).resolve().parent.parent
os.chdir(root)
output = root / "target/quality-gate"
# These two directories contain generated reports only. Never archive a
# previous run's coverage as evidence for an early lint/test failure.
for directory in (output, root / "target/llvm-cov"):
if directory.exists():
shutil.rmtree(directory)
output.mkdir(parents=True)
inputs = [Path("Makefile"), Path("Cargo.lock"), Path(".github/workflows/ci.yml"),
Path("files/config/Jenkinsfile"), Path("files/config/runtime-edge-coverage-domains.json")]
inputs.extend(sorted(Path("scripts").rglob("*.py")))
inputs.extend(sorted(Path("scripts").glob("*.sh")))
fingerprints = {str(path): sha(path) for path in inputs if path.is_file()}
sources = "".join(f"{path}\0{sha(path)}\n" for path in sorted(Path("crates").rglob("*.rs")))
evidence = {
"started_utc": datetime.now(timezone.utc).isoformat(),
"host": platform.platform(), "revision": capture("git", "rev-parse", "HEAD"),
"working_tree": capture("git", "status", "--porcelain"),
"rustc": capture("rustc", "-vV"), "cargo_llvm_cov": capture("cargo", "llvm-cov", "--version"),
"config_sha256": fingerprints,
"rust_source_sha256": hashlib.sha256(sources.encode()).hexdigest(),
"source_hash_algorithm": "SHA-256 of sorted path + NUL + file SHA-256 hex + newline records",
"steps": [], "status": "running",
}
metadata = output / "metadata.json"
def persist():
metadata.write_text(json.dumps(evidence, indent=2) + "\n")
persist()
exit_code = 0
try:
if not sys.argv[1:]:
raise ValueError("Makefile must supply the required gate steps")
for target in sys.argv[1:]:
if not target.replace("-", "").isalnum():
raise ValueError(f"invalid make target: {target!r}")
command = ["make", "--no-print-directory", "-j1", target]
log = output / f"{target}.log"
print(f"Quality gate: {target}", flush=True)
with log.open("w") as stream:
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, errors="replace")
for line in process.stdout:
stream.write(line)
print(line, end="", flush=True)
result = process.wait()
exit_code = result if result >= 0 else 128 - result
evidence["steps"].append({"target": target, "command": command,
"exit_code": exit_code, "log_sha256": sha(log)})
persist()
if exit_code:
break
except KeyboardInterrupt:
evidence["error"] = "Interrupted"
exit_code = 130
except Exception as error:
evidence["error"] = str(error)
exit_code = 1
finally:
evidence["status"] = "passed" if exit_code == 0 else "failed"
evidence["exit_code"] = exit_code
evidence["finished_utc"] = datetime.now(timezone.utc).isoformat()
evidence["reports_sha256"] = {
str(path.relative_to(root)): sha(path)
for path in sorted((root / "target/llvm-cov").rglob("*")) if path.is_file()
}
persist()
return exit_code
if __name__ == "__main__":
sys.exit(main())
+55
View File
@@ -0,0 +1,55 @@
"""Exercise real make failures without running Cargo or modifying the repository."""
import json
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
class QualityGateTests(unittest.TestCase):
def run_gate(self, makefile, targets):
temporary = tempfile.TemporaryDirectory()
self.addCleanup(temporary.cleanup)
root = Path(temporary.name)
(root / "scripts").mkdir()
shutil.copy(Path(__file__).resolve().parents[1] / "run-quality-gate.py", root / "scripts")
(root / "Makefile").write_text(makefile)
(root / "target/llvm-cov").mkdir(parents=True)
(root / "target/llvm-cov/stale.json").write_text("old evidence")
result = subprocess.run([sys.executable, str(root / "scripts/run-quality-gate.py"), *targets],
capture_output=True, text=True)
evidence = json.loads((root / "target/quality-gate/metadata.json").read_text())
self.assertFalse((root / "target/llvm-cov/stale.json").exists())
return root, result, evidence
def test_failure_preserves_logs_identity_exit_and_stops_later_steps(self):
root, result, evidence = self.run_gate(
'fail:\n\t@echo diagnostic\n\t@exit 7\nlater:\n\t@touch should-not-exist\n',
['fail', 'later'])
self.assertNotEqual(result.returncode, 0)
self.assertEqual(evidence['exit_code'], result.returncode)
self.assertEqual(evidence['status'], 'failed')
self.assertEqual([step['target'] for step in evidence['steps']], ['fail'])
self.assertIn('diagnostic', (root / 'target/quality-gate/fail.log').read_text())
self.assertIn('Makefile', evidence['config_sha256'])
self.assertIn('revision', evidence)
self.assertEqual(evidence['reports_sha256'], {})
self.assertFalse((root / 'should-not-exist').exists())
def test_success_fingerprints_only_current_reports(self):
_, result, evidence = self.run_gate(
'report:\n\t@mkdir -p target/llvm-cov\n\t@echo fresh > target/llvm-cov/report.json\n',
['report'])
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertEqual(evidence['status'], 'passed')
self.assertEqual(list(evidence['reports_sha256']), ['target/llvm-cov/report.json'])
def test_partial_report_is_retained_when_a_later_gate_fails(self):
_, result, evidence = self.run_gate(
'report:\n\t@mkdir -p target/llvm-cov\n\t@echo fresh > target/llvm-cov/report.json\nfail:\n\t@exit 1\n',
['report', 'fail'])
self.assertNotEqual(result.returncode, 0)
self.assertEqual(evidence['status'], 'failed')
self.assertIn('target/llvm-cov/report.json', evidence['reports_sha256'])