[PERF] Async Background Work Lanes for Assets and FS
Intrepid/Prometeu/Runtime/pipeline/head There was a failure building this commit
Intrepid/Prometeu/Runtime/pipeline/pr-master This commit looks good

This commit is contained in:
2026-06-28 09:35:08 +01:00
parent bd147938a0
commit 8f37d971cd
10 changed files with 1591 additions and 458 deletions
@@ -0,0 +1,118 @@
---
id: PLN-0123
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Work Lane Specification Propagation
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, fs, async, scheduler, runtime, spec]
---
# PLN-0123 - Async Work Lane Specification Propagation
## Briefing
`DEC-0034` establishes a third serial async work lane, asset backlog semantics,
stable bank-slot handles, superseding, progress, telemetry, and FS/memcard lane
boundaries. The canonical specs must be updated before code work relies on the
new contract.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Publish the accepted async work lane and asset backlog contract in the canonical
runtime specs.
## Escopo
Included:
- Update asset management spec.
- Update syscall ABI spec if public backlog APIs are named there.
- Update syscall policy/status language for operational asset states.
- Update events/concurrency spec with the async work lane as an implementation
lane that does not introduce guest callbacks.
- Add a cross-reference from FS/app-home agenda or notes to the async IO lane.
Fora de Escopo:
- Implementing runtime code.
- Finalizing public FS API semantics.
- Changing render worker contracts.
## Plano de Execucao
### Step 1 - Update Asset Management
**What:** Add the third async lane, backlog, handle, superseding, progress, and
telemetry contract.
**How:** Extend `docs/specs/runtime/15-asset-management.md` around the existing
`asset.load/status/commit/cancel` section. Preserve the existing
`asset.load(asset_id, slot)` identity contract.
**Files:** `docs/specs/runtime/15-asset-management.md`.
### Step 2 - Update Host ABI
**What:** Add the backlog API direction if the ABI spec is the canonical place
for syscall shapes.
**How:** Extend the asset surface section with the planned backlog operations,
marking exact numeric ids as implementation-plan work if ids are not assigned
yet.
**Files:** `docs/specs/runtime/16-host-abi-and-syscalls.md`.
### Step 3 - Update Syscall Policies
**What:** Clarify that `superseded`, `canceled`, `backend_unavailable`, and
decode failures are operational statuses, not faults.
**How:** Add asset-domain examples under status-first policy.
**Files:** `docs/specs/runtime/16a-syscall-policies.md`.
### Step 4 - Update Events and Concurrency
**What:** Document that the async work lane is an implementation lane.
**How:** Add a section next to render worker concurrency stating that asset/IO
lane work does not introduce guest callbacks or hidden guest execution.
**Files:** `docs/specs/runtime/09-events-and-concurrency.md`.
### Step 5 - Cross-Reference FS Work
**What:** Record that FS may consume the async IO lane, while public FS API
semantics remain owned by the app-home filesystem discussion.
**How:** Add a note in `AGD-0006` or its future decision input, without deciding
FS request/poll shape in this plan.
**Files:** `discussion/workflow/agendas/AGD-0006-app-home-filesystem-surface-and-semantics.md`.
## Criterios de Aceite
- [ ] Specs state that the async work lane is serial and separate from render.
- [ ] Asset spec states backlog keying by `bank_type/slot`.
- [ ] Asset spec states stable slot handles with `slot_state` and `request_state`.
- [ ] Asset spec states superseding behavior.
- [ ] Asset spec states main-lane commit/install.
- [ ] FS is cross-referenced as a permitted lane consumer without public API decisions.
## Tests / Validacao
- Run `discussion validate`.
- Review spec links for broken relative paths.
- No cargo tests are required for this documentation-only plan.
## Riscos
- Public ABI names may be refined later; the spec must avoid assigning ids too
early if implementation work has not reserved them.
- FS scope can leak into this work; keep FS public API decisions in `AGD-0006`.
@@ -0,0 +1,119 @@
---
id: PLN-0124
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Work Lane Runtime Infrastructure
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, fs, async, scheduler, runtime]
---
# PLN-0124 - Async Work Lane Runtime Infrastructure
## Briefing
`DEC-0034` requires one runtime-owned async work lane, separate from VM/main
execution and separate from render worker execution. This plan introduces the
lane infrastructure before migrating asset loading.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Create a serial async work lane primitive that can execute one active job at a
time, expose observable state, and support deterministic shutdown.
## Escopo
Included:
- Runtime-owned async work lane type.
- Single active job execution.
- Ordered pending job storage.
- Cooperative cancellation hook.
- Shutdown and join behavior.
- Basic lane telemetry.
Fora de Escopo:
- Public asset backlog syscalls.
- Asset handle model migration.
- FS public API.
- Render worker changes.
## Plano de Execucao
### Step 1 - Locate Runtime Service Boundary
**What:** Choose the crate/module boundary for the async lane.
**How:** Place generic lane infrastructure near VM runtime services, not inside
desktop host code and not inside render worker modules.
**Files:** likely `crates/console/prometeu-system/src/services/` or
`crates/console/prometeu-system/src/services/vm_runtime/`.
### Step 2 - Define Job and Lane Types
**What:** Add a lane type with pending queue, active job metadata, generation,
state, progress, and cancellation flag.
**How:** Use typed job enums or traits that do not require guest callbacks.
Start with internal asset job support but keep the lane generic enough for
memcard/FS later.
**Files:** new module under `prometeu-system` service layer.
### Step 3 - Implement Serial Worker
**What:** Implement one worker loop that takes one pending job and runs it to
completion or cooperative cancellation.
**How:** Use a fixed worker thread on host builds. Preserve a logical-lane API
so future hardware can map it to a dedicated core.
**Files:** async lane module; lifecycle wiring in VM runtime service setup.
### Step 4 - Implement Shutdown
**What:** Add explicit stop/shutdown behavior.
**How:** Wake the lane, stop accepting new jobs, report pending jobs as canceled
or shutdown-discarded, and join the worker with bounded behavior.
**Files:** async lane module and VM runtime lifecycle modules.
### Step 5 - Expose Internal Telemetry
**What:** Add counters for pending depth, active job, submitted, completed,
failed, canceled, and superseded.
**How:** Use cheap atomics or lane-owned state snapshots. Do not update
expensive percentile data inside inner decode loops.
**Files:** async lane module; telemetry structs where runtime telemetry lives.
## Criterios de Aceite
- [ ] Lane has one active job maximum.
- [ ] Lane is separate from render worker code.
- [ ] Lane can be started and stopped deterministically.
- [ ] Pending jobs are observable.
- [ ] Cooperative cancellation can be signaled.
- [ ] No guest callback path is introduced.
## Tests / Validacao
- Unit tests for enqueue, single active execution, ordering, cancellation signal,
and shutdown.
- Concurrency tests must synchronize on real state transitions, not sleeps.
- Run targeted `cargo test` for the owning crate.
## Riscos
- Accidentally creating a generic pool instead of a serial lane.
- Coupling the lane to desktop host thread details.
- Reusing render worker handoff semantics where ordered backlog is required.
@@ -0,0 +1,127 @@
---
id: PLN-0125
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Asset Backlog and Stable Slot Handles
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, runtime]
---
# PLN-0125 - Asset Backlog and Stable Slot Handles
## Briefing
`DEC-0034` changes asset loading from request-owned thread spawning to a serial
backlog keyed by `bank_type/slot`. Handles become stable bank-slot observers
with separate resident slot state and request state.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Refactor `AssetManager` so asset requests are tracked by target bank slot,
deduplicated/superseded by target, and observable through stable handles.
## Escopo
Included:
- Stable handle structure with `bank_type`, `slot`, `slot_state`, and
`request_state`.
- Backlog keyed by `bank_type/slot`.
- Already-resident fast path.
- Superseding for queued and active requests.
- Main-lane commit/install boundary.
Fora de Escopo:
- Public backlog syscall wiring.
- Percentile telemetry.
- FS public API.
## Plano de Execucao
### Step 1 - Model Slot Handles
**What:** Replace transient load-operation assumptions with stable bank-slot
handle state.
**How:** Introduce structures equivalent to:
```text
handle: bank_type, slot
slot_state: loaded_asset_id, resident_state, slot_generation
request_state: requested_asset_id, request_generation, state, position, progress
```
**Files:** `crates/console/prometeu-drivers/src/asset.rs` and HAL asset types if
public type definitions belong there.
### Step 2 - Key Backlog by Target
**What:** Ensure at most one current request per `bank_type/slot`.
**How:** Add a target index from `(bank_type, slot)` to current request
generation/handle state. Enqueue only when the target does not already have the
same resident asset and no newer request supersedes it.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 3 - Implement Already-Resident Fast Path
**What:** Return a ready handle immediately when the requested asset already
resides in the target slot.
**How:** Compare requested `asset_id` with the target `slot_state.loaded_asset_id`
and resident validity before creating a queued job.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 4 - Implement Superseding
**What:** New request for the same target supersedes previous queued or active
request.
**How:** Remove queued older request immediately. For active work, update target
generation so the old result is discarded if it cannot stop cooperatively.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`; async lane module
from `PLN-0124`.
### Step 5 - Preserve Main-Lane Commit
**What:** Keep install/commit into resident banks on the main runtime lane.
**How:** Async lane produces staged materialized result. `asset.commit(handle)`
installs only when the request generation is current and ready.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`; VM runtime dispatch if
commit behavior changes.
## Criterios de Aceite
- [ ] No asset load spawns an OS thread per request.
- [ ] At most one request exists per `bank_type/slot`.
- [ ] Same-target newer request supersedes older queued request.
- [ ] Same-target newer request prevents active stale result from installing.
- [ ] Already-resident target returns ready handle without backlog entry.
- [ ] Handles remain queryable for empty, queued, active, ready, committed,
canceled, superseded, and error states.
## Tests / Validacao
- Unit tests for handle state transitions.
- Unit tests for target-keyed superseding.
- Unit tests for already-resident fast path.
- Integration test for stale active result discard.
- Existing asset load/commit tests must continue passing.
## Riscos
- Confusing handle identity with request generation.
- Installing a stale async result after superseding.
- Losing compatibility with existing `asset.status/commit/cancel` behavior.
@@ -0,0 +1,124 @@
---
id: PLN-0126
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Asset Backlog Public API and Status Surface
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, syscall, runtime]
---
# PLN-0126 - Asset Backlog Public API and Status Surface
## Briefing
`DEC-0034` keeps `asset.load/status/commit/cancel` and adds a small status-first
backlog inspection and ordering surface. This plan wires that surface through
HAL syscall metadata and runtime dispatch.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Expose asset backlog inspection and reordering while preserving status-first
semantics and existing asset load identity.
## Escopo
Included:
- Syscall registry additions for backlog APIs.
- Runtime dispatch handlers.
- Status enum additions or normalization.
- ABI/spec alignment with `PLN-0123`.
Fora de Escopo:
- Async lane internals.
- Asset decode refactor.
- FS API.
## Plano de Execucao
### Step 1 - Finalize Syscall Shapes
**What:** Choose final names, arity, and return slots for backlog operations.
**How:** Start from `DEC-0034`:
```text
asset.backlog_info()
asset.backlog_position(handle)
asset.backlog_move(handle, new_position)
asset.backlog_promote(handle)
asset.backlog_demote(handle)
asset.target_status(bank_type, slot)
```
**Files:** `crates/console/prometeu-hal/src/syscalls/domains/asset.rs`,
`docs/specs/runtime/16-host-abi-and-syscalls.md`.
### Step 2 - Add Status Values
**What:** Represent `empty`, `invalid`, `queued`, `active`, `ready`,
`committed`, `canceled`, `superseded`, `error`, and `backend_unavailable`.
**How:** Extend or map existing `LoadStatus` and `AssetOpStatus` without
turning operational states into traps.
**Files:** HAL asset status types and `crates/console/prometeu-drivers/src/asset.rs`.
### Step 3 - Add Dispatch
**What:** Wire new syscalls through VM runtime dispatch.
**How:** Add match arms near existing `AssetLoad`, `AssetStatus`,
`AssetCommit`, and `AssetCancel` arms. Return status-first values.
**Files:** `crates/console/prometeu-system/src/services/vm_runtime/dispatch.rs`.
### Step 4 - Implement API Methods
**What:** Add AssetManager methods backing each public operation.
**How:** Methods must read stable handle/target state and reorder only queued
requests. `promote` and `demote` are shortcuts over move semantics.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 5 - Validate ABI Metadata
**What:** Ensure registry ids, args, returns, capability, cost, and
non-deterministic flags are correct.
**How:** Update syscall metadata tests and any declared PBS/runtime syscall
tables.
**Files:** `crates/console/prometeu-hal/src/syscalls/tests.rs` and related
registry files.
## Criterios de Aceite
- [ ] Public backlog operations are registered with stable metadata.
- [ ] All backlog operations are status-first.
- [ ] `promote` is equivalent to moving to first pending position.
- [ ] `demote` moves to the end of pending backlog.
- [ ] Reordering a non-queued request returns operational status.
- [ ] Structural ABI misuse remains Trap.
## Tests / Validacao
- HAL syscall metadata tests.
- VM runtime dispatch tests for each new syscall.
- AssetManager unit tests for move/promote/demote.
- Tests for `target_status` on empty, queued, active, ready, and superseded
targets.
## Riscos
- Adding too many public ABI details before spec propagation is accepted.
- Returning payload values that are ambiguous when status is not success.
- Treating known but non-mutable states as traps instead of statuses.
@@ -0,0 +1,120 @@
---
id: PLN-0127
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Lane Memcard and FS Integration Boundaries
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, fs, memcard, async, runtime]
---
# PLN-0127 - Async Lane Memcard and FS Integration Boundaries
## Briefing
`DEC-0034` allows memcard and FS work to consume the async IO lane but keeps
public FS API semantics out of scope. This plan establishes integration
boundaries without deciding app-home FS API shape.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Define and implement safe internal boundaries for memcard and FS use of the
async work lane, preserving lane priority and keeping FS public API decisions in
`AGD-0006`.
## Escopo
Included:
- Memcard as a priority lane consumer candidate.
- FS as a permitted internal lane consumer.
- Priority ordering in the lane.
- Updates to `AGD-0006` to acknowledge the lane.
Fora de Escopo:
- Public FS request/poll design.
- App-home path/status contract.
- Full memcard async conversion if it requires public ABI changes.
## Plano de Execucao
### Step 1 - Update FS Agenda
**What:** Record that an async IO lane exists and may be used by FS.
**How:** Add a concise section to `AGD-0006` stating that FS public API remains
owned by that agenda, but implementation may use the async lane from `DEC-0034`.
**Files:** `discussion/workflow/agendas/AGD-0006-app-home-filesystem-surface-and-semantics.md`.
### Step 2 - Identify Memcard Operations
**What:** List memcard operations that can benefit from async IO.
**How:** Inspect `crates/console/prometeu-system/src/services/memcard.rs` and
VM runtime dispatch for `mem.*` operations. Classify read, write, commit, and
clear behavior.
**Files:** memcard service and VM runtime dispatch modules.
### Step 3 - Define Lane Priority
**What:** Encode the initial priority order from `DEC-0034`.
**How:** Add priority classes:
```text
memcard commit/write
FS write/config
asset load
non-critical list/read
```
**Files:** async lane module from `PLN-0124`.
### Step 4 - Add Internal Consumer Hooks
**What:** Provide internal APIs for memcard/FS to submit lane jobs later.
**How:** Add types/interfaces without changing public FS syscall behavior.
No public FS request/poll operations are introduced here.
**Files:** async lane module; FS/memcard service modules as needed.
### Step 5 - Preserve Current Public Behavior
**What:** Ensure existing memcard and FS syscalls keep their current observable
surface unless a later plan explicitly changes it.
**How:** Do not alter return shapes in this plan. Add tests proving existing
status-first behavior remains intact.
**Files:** VM runtime FS/memcard tests.
## Criterios de Aceite
- [ ] `AGD-0006` acknowledges the async IO lane and keeps FS public API in its
own scope.
- [ ] Async lane supports priority classes required by `DEC-0034`.
- [ ] Memcard and FS can be represented as internal lane consumers.
- [ ] Existing public memcard/FS syscall shapes are unchanged.
- [ ] No FS public request/poll API is introduced by this plan.
## Tests / Validacao
- Existing FS/memcard tests continue passing.
- Unit tests for lane priority ordering.
- Tests proving asset jobs do not overlap with higher-priority memcard jobs
once both are submitted through the lane.
## Riscos
- Accidentally deciding FS public API in the wrong discussion.
- Making memcard async in a way that changes status-first behavior.
- Starving asset loads if priority rules do not include fairness.
@@ -0,0 +1,126 @@
---
id: PLN-0128
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Asset Pipeline Contract Tests and Telemetry
status: open
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, telemetry, tests, runtime]
---
# PLN-0128 - Async Asset Pipeline Contract Tests and Telemetry
## Briefing
`DEC-0034` requires observable progress, backlog state, superseding, and cheap
telemetry. This plan adds the validation and telemetry evidence needed to prove
the new async asset pipeline is correct.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Provide contract tests and telemetry for the async asset lane so future
implementations can be changed without regressing lane semantics.
## Escopo
Included:
- Contract tests for lane and asset backlog behavior.
- Progress reporting tests.
- Superseding and stale result discard tests.
- Telemetry counters and percentile updates.
- No-sleep deterministic test helpers where needed.
Fora de Escopo:
- Implementing public API wiring if `PLN-0126` has not run.
- Full benchmarking infrastructure.
- Host UI display of telemetry.
## Plano de Execucao
### Step 1 - Build Deterministic Test Hooks
**What:** Add test hooks for blocking a lane job at read/decode/stage.
**How:** Use channels/barriers or explicit fake job steps. Do not rely on
`thread::sleep` for correctness.
**Files:** async lane tests; asset test support modules.
### Step 2 - Test Backlog Ordering
**What:** Prove queued jobs run serially and in backlog order.
**How:** Submit multiple target slots and assert active job, pending positions,
promote, demote, and move behavior.
**Files:** asset manager tests and lane tests.
### Step 3 - Test Superseding
**What:** Prove newer request for the same `bank_type/slot` supersedes older
requests.
**How:** Cover queued superseding and active superseding. For active work, prove
stale completion cannot install.
**Files:** asset manager tests.
### Step 4 - Test Stable Handle Observability
**What:** Prove handles are queryable for empty, queued, active, ready,
committed, canceled, superseded, and error states.
**How:** Use `target_status` and handle status APIs to inspect both `slot_state`
and `request_state`.
**Files:** asset manager tests; VM dispatch tests if public API is available.
### Step 5 - Test Progress
**What:** Prove progress uses `0..10000` and phase boundaries.
**How:** Fake read/decode/stage phases with known completion points. Validate
non-linear phases do not report false internal progress.
**Files:** asset manager tests; telemetry tests.
### Step 6 - Add Telemetry
**What:** Add counters and percentile aggregation.
**How:** Update submitted/completed/failed/canceled/superseded counters when job
state closes. Update percentiles by `bank_type` and lightweight `asset_id`
window only at job completion.
**Files:** asset manager telemetry, async lane telemetry, existing runtime
telemetry types if shared.
## Criterios de Aceite
- [ ] Tests prove one active job at a time.
- [ ] Tests prove backlog order and reordering.
- [ ] Tests prove queued and active superseding.
- [ ] Tests prove stale active results are discarded.
- [ ] Tests prove handles remain queryable across all required states.
- [ ] Tests prove progress phase behavior.
- [ ] Telemetry is updated at job closure, not inside inner decode loops.
## Tests / Validacao
- Run targeted cargo tests for asset manager, async lane, and VM runtime
dispatch.
- Run existing asset-bank tests to prove current behavior remains compatible.
- Run `discussion validate` after plan state changes.
## Riscos
- Tests that depend on sleeps will be flaky.
- Telemetry may become too expensive if updated inside decode loops.
- Public API tests must wait for `PLN-0126` if syscall wiring is not present.