Intrepid/prometeu-runtime
Intrepid/prometeu-runtime
4
0
Fork 0
Code Pull Requests Activity

[PERF] Async Background Work Lanes for Assets and FS
All checks were successful
Intrepid/Prometeu/Runtime/pipeline/pr-master This commit looks good
Details

Browse Source
This commit is contained in:
bQUARKz 2026-07-01 10:09:33 +01:00
parent b84946a3a0
commit 5d3db4c22a
Signed by: bquarkz
SSH Key Fingerprint: SHA256:Z7dgqoglWwoK6j6u4QC87OveEq74WOhFN+gitsxtkf8
10 changed files with 102 additions and 1629 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
discussion/index.ndjson
View File

@ -1,4 +1,4 @@
{"type":"meta","next_id":{"DSC":43,"AGD":44,"DEC":35,"PLN":129,"LSN":50,"CLSN":1}}
{"type":"meta","next_id":{"DSC":43,"AGD":44,"DEC":35,"PLN":129,"LSN":51,"CLSN":1}}
{"type":"discussion","id":"DSC-0039","status":"abandoned","ticket":"render-pipeline-family-and-future-3d","title":"Render Pipeline Family and Future 3D","created_at":"2026-06-04","updated_at":"2026-06-04","tags":["gfx","renderer","runtime","architecture","pipeline"],"agendas":[{"id":"AGD-0039","file":"AGD-0039-render-pipeline-family-and-future-3d.md","status":"abandoned","created_at":"2026-06-04","updated_at":"2026-06-04","_override_reason":"User explicitly chose to close this agenda without a new decision because DSC-0038 already established enough architecture for future extension, and 3D is intentionally deferred."}],"decisions":[],"plans":[],"lessons":[],"_override_reason":"User explicitly chose to close this agenda without a new decision because DSC-0038 already established enough architecture for future extension, and 3D is intentionally deferred."}
{"type":"discussion","id":"DSC-0040","status":"done","ticket":"vm-render-parallel-execution-boundary","title":"VM and Render Parallel Execution Boundary","created_at":"2026-06-04","updated_at":"2026-06-06","tags":["runtime","renderer","vm","concurrency","architecture","perf"],"agendas":[],"decisions":[],"plans":[],"lessons":[{"id":"LSN-0048","file":"discussion/lessons/DSC-0040-vm-render-parallel-execution-boundary/LSN-0048-render-workers-need-a-closed-packet-contract.md","status":"done","created_at":"2026-06-06","updated_at":"2026-06-06"}]}
{"type":"discussion","id":"DSC-0041","status":"open","ticket":"foreground-stack-game-pause-shell-vm-backed","title":"Foreground Stack, Game Pause, and VM-Backed Shell Coexistence","created_at":"2026-06-05","updated_at":"2026-06-05","tags":["runtime","os","lifecycle","shell","game","vm","foreground","architecture"],"agendas":[{"id":"AGD-0041","file":"AGD-0041-foreground-stack-game-pause-shell-vm-backed.md","status":"open","created_at":"2026-06-05","updated_at":"2026-06-05"}],"decisions":[],"plans":[],"lessons":[]}
@ -18,7 +18,7 @@
{"type":"discussion","id":"DSC-0006","status":"open","ticket":"vm-owned-random-service","title":"Agenda - VM-Owned Random Service","created_at":"2026-03-27","updated_at":"2026-03-27","tags":[],"agendas":[{"id":"AGD-0005","file":"AGD-0005-vm-owned-random-service.md","status":"open","created_at":"2026-03-27","updated_at":"2026-03-27"}],"decisions":[],"plans":[],"lessons":[]}
{"type":"discussion","id":"DSC-0007","status":"open","ticket":"app-home-filesystem-surface-and-semantics","title":"Agenda - App Home Filesystem Surface and Semantics","created_at":"2026-03-27","updated_at":"2026-03-27","tags":[],"agendas":[{"id":"AGD-0006","file":"AGD-0006-app-home-filesystem-surface-and-semantics.md","status":"open","created_at":"2026-03-27","updated_at":"2026-03-27"}],"decisions":[],"plans":[],"lessons":[]}
{"type":"discussion","id":"DSC-0008","status":"done","ticket":"perf-runtime-telemetry-hot-path","title":"Agenda - [PERF] Runtime Telemetry Hot Path","created_at":"2026-03-27","updated_at":"2026-06-04","tags":[],"agendas":[],"decisions":[],"plans":[],"lessons":[{"id":"LSN-0026","file":"discussion/lessons/DSC-0008-perf-runtime-telemetry-hot-path/LSN-0026-push-based-telemetry-model.md","status":"done","created_at":"2026-04-10","updated_at":"2026-04-10"}]}
{"type":"discussion","id":"DSC-0009","status":"in_progress","ticket":"perf-async-background-work-lanes-for-assets-and-fs","title":"Agenda - [PERF] Async Background Work Lanes for Assets and FS","created_at":"2026-03-27","updated_at":"2026-06-28","tags":["perf","asset","fs","async","scheduler","runtime"],"agendas":[{"id":"AGD-0008","file":"AGD-0008-perf-async-background-work-lanes-for-assets-and-fs.md","status":"accepted","created_at":"2026-03-27","updated_at":"2026-06-28"}],"decisions":[{"id":"DEC-0034","file":"DEC-0034-async-work-lane-and-asset-backlog-contract.md","status":"accepted","created_at":"2026-06-28","updated_at":"2026-06-28","ref_agenda":"AGD-0008"}],"plans":[{"id":"PLN-0123","file":"PLN-0123-async-work-lane-specification-propagation.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]},{"id":"PLN-0124","file":"PLN-0124-async-work-lane-runtime-infrastructure.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]},{"id":"PLN-0125","file":"PLN-0125-asset-backlog-and-stable-slot-handles.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]},{"id":"PLN-0126","file":"PLN-0126-asset-backlog-public-api-and-status-surface.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]},{"id":"PLN-0127","file":"PLN-0127-async-lane-memcard-and-fs-integration-boundaries.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]},{"id":"PLN-0128","file":"PLN-0128-async-asset-pipeline-contract-tests-and-telemetry.md","status":"done","created_at":"2026-06-28","updated_at":"2026-06-28","ref_decisions":["DEC-0034"]}],"lessons":[]}
{"type":"discussion","id":"DSC-0009","status":"done","ticket":"perf-async-background-work-lanes-for-assets-and-fs","title":"Agenda - [PERF] Async Background Work Lanes for Assets and FS","created_at":"2026-03-27","updated_at":"2026-07-01","tags":["perf","asset","fs","async","scheduler","runtime"],"agendas":[],"decisions":[],"plans":[],"lessons":[{"id":"LSN-0050","file":"discussion/lessons/DSC-0009-perf-async-background-work-lanes-for-assets-and-fs/LSN-0050-serial-async-lanes-bound-backlog-complexity.md","status":"done","created_at":"2026-07-01","updated_at":"2026-07-01"}]}
{"type":"discussion","id":"DSC-0010","status":"done","ticket":"perf-host-desktop-frame-pacing-and-presentation","title":"Agenda - [PERF] Host Desktop Frame Pacing and Presentation","created_at":"2026-03-27","updated_at":"2026-04-20","tags":[],"agendas":[],"decisions":[],"plans":[],"lessons":[{"id":"LSN-0036","file":"discussion/lessons/DSC-0010-perf-host-desktop-frame-pacing-and-presentation/LSN-0036-frame-publication-and-host-invalidation-must-be-separate.md","status":"done","created_at":"2026-04-20","updated_at":"2026-04-20"}]}
{"type":"discussion","id":"DSC-0011","status":"abandoned","ticket":"perf-gfx-render-pipeline-and-dirty-regions","title":"Agenda - [PERF] GFX Render Pipeline and Dirty Regions","created_at":"2026-03-27","updated_at":"2026-06-28","tags":[],"agendas":[{"id":"AGD-0010","file":"AGD-0010-perf-gfx-render-pipeline-and-dirty-regions.md","status":"abandoned","created_at":"2026-03-27","updated_at":"2026-06-28","_override_reason":"User explicitly chose to remove this agenda because it no longer applies to the current render scenario; future optimizations can be discussed when needed."}],"decisions":[],"plans":[],"lessons":[],"_override_reason":"User explicitly chose to abandon this discussion because it no longer applies to the current render scenario; future optimizations can be discussed when needed."}
{"type":"discussion","id":"DSC-0012","status":"done","ticket":"perf-runtime-introspection-syscalls","title":"Agenda - [PERF] Runtime Introspection Syscalls","created_at":"2026-03-27","updated_at":"2026-04-19","tags":["perf","runtime","syscall","telemetry","debug","asset"],"agendas":[],"decisions":[],"plans":[],"lessons":[{"id":"LSN-0034","file":"discussion/lessons/DSC-0012-perf-runtime-introspection-syscalls/LSN-0034-host-owned-debug-boundaries.md","status":"done","created_at":"2026-04-19","updated_at":"2026-04-19"}]}

100
discussion/lessons/DSC-0009-perf-async-background-work-lanes-for-assets-and-fs/LSN-0050-serial-async-lanes-bound-backlog-complexity.md Normal file
View File

@ -0,0 +1,100 @@
---
id: LSN-0050
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Serial Async Lanes Bound Backlog Complexity
created: 2026-07-01
tags: [runtime, asset, async, scheduler, telemetry]
---
## Context
The async asset work moved from request-owned worker creation toward a
runtime-owned serial lane. The important shift is not just "use a worker
thread"; it is the introduction of a third logical execution lane with explicit
ownership, priority, cancellation, progress, and telemetry.
This work followed `DEC-0034` and plans `PLN-0123` through `PLN-0128`. The
published model separates:
- main runtime execution, where guest-visible state is committed;
- render worker execution, where closed render packets are consumed;
- async IO/decode/persistence work, where one background job is active at a
time.
## Key Decisions
### Async Work Lane and Asset Backlog Contract
**What:** Asset IO/decode and compatible persistence work use a runtime-owned
serial async lane. Asset requests are queued by target bank slot, not by
unbounded transient request identity.
**Why:** A serial lane keeps the runtime observable and portable. It preserves
the intended hardware mental model while avoiding a desktop-biased thread pool
or one OS thread per asset request.
**Trade-offs:** The lane intentionally sacrifices parallel decode throughput in
exchange for bounded state, deterministic ownership, simpler telemetry, and
clear priority rules. If future implementations add more physical parallelism,
they still need to preserve the logical serial contract where the spec requires
it.
## Patterns and Algorithms
The asset backlog is bounded by target identity. A request targets a concrete
bank type and slot. A newer request for the same target supersedes the older
pending request, and an older active result is discarded when its generation is
no longer current.
Stable handles observe slots. A handle should not be treated as a worker job or
thread token. The durable identity is the bank slot plus request generation:
slot state says what is resident; request state says what is queued, active,
ready, canceled, superseded, or failed.
Commit remains a main-lane operation. The background lane may read and decode,
but publication into resident runtime state happens at predictable ownership
points. This keeps VM execution, render handoff, and frame observation from
racing with background mutation.
Progress and telemetry are closure-oriented. Use integer progress and update
expensive aggregate telemetry when jobs close, not inside decode loops. Tests
should synchronize on explicit state transitions rather than sleeps.
Priority is part of the lane contract. Memcard commit/write work can outrank
ordinary asset loads, FS write/config work can be represented internally, and
non-critical read/list work belongs below asset loads. Public FS semantics
remain owned by the filesystem discussion; sharing the lane is not permission
to decide the FS API here.
## Pitfalls
Do not hide concurrency behind per-request `thread::spawn`. That appears simple
locally but loses boundedness, priority, cancellation, and telemetry.
Do not let a handle mean "the current background job." Handles must survive
empty slots, completed work, cancellation, superseding, errors, and already
resident fast paths.
Do not install background results directly from the worker. Decode completion is
not the same as publication. Use request generation checks before committing a
result.
Do not turn operational states into traps. Queued, active, canceled,
superseded, error, and backend-unavailable outcomes belong in status-first
surfaces unless the caller violated the structural ABI.
Do not reopen FS public API scope while wiring internal async-lane consumers.
The lane can support FS-style work without deciding request/poll semantics.
## Takeaways
- A runtime async lane is an ownership boundary, not just an implementation
thread.
- Backlog keying by target slot bounds complexity better than queue length
limits exposed to the guest.
- Stable slot handles plus request generations prevent stale async results from
mutating newer state.
- Main-lane commit keeps background IO/decode compatible with deterministic
runtime publication.
- Telemetry belongs at state transitions and job closure, not inside hot decode
loops.

557
discussion/workflow/agendas/AGD-0008-perf-async-background-work-lanes-for-assets-and-fs.md
View File

@ -1,557 +0,0 @@
---
id: AGD-0008
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Agenda - [PERF] Async Background Work Lanes for Assets and FS
status: accepted
created: 2026-03-27
resolved: 2026-06-28
decision: DEC-0034
tags: [perf, asset, fs, async, scheduler, runtime]
---
# Agenda - [PERF] Async Background Work Lanes for Assets and FS
## Contexto
Depois das decisoes de render worker, a concorrencia de render saiu do escopo
desta agenda. O problema restante e a lane de trabalho assincrono para IO,
decode e persistencia.
O modelo de execucao que queremos discutir agora assume uma terceira core/lane
dedicada a trabalhos assincronos de runtime:
```text
Core/lane 1: VM, firmware, SystemOS e frame logico
Core/lane 2: render worker / rasterizacao assincrona
Core/lane 3: async work lane para asset IO/decode e possivelmente FS
```
Essa terceira lane nao deve virar um pool grande e solto. A intencao e ter uma
fila previsivel, com capacidade limitada, telemetria e regras de saturacao.
Para assets, a necessidade e concreta: `asset.load()` ainda cria uma thread do
SO por requisicao. Para FS, a motivacao provavel era evitar que operacoes de
leitura/escrita/listagem/delete, hoje chamadas de forma sincrona pelo dispatch,
roubem tempo do frame quando o backend real envolver IO. Precisamos verificar
se FS deve usar a mesma lane agora ou apenas reservar compatibilidade.
## Problema
`asset.load()` hoje dispara `thread::spawn` por requisicao de asset nao
residente. Isso escala mal, cria jitter e deixa o runtime sem uma politica
unica de backpressure.
O dominio `fs` tambem tem operacoes potencialmente bloqueantes (`read`,
`write`, `list_dir`, `delete`, memcard commit/read/write) e a tabela de
syscalls ja as marca como nao deterministicas/custosas. Ainda assim, a
implementacao atual executa essas chamadas diretamente no caminho de dispatch.
Sem um contrato explicito, cada dominio pode inventar sua propria concorrencia:
asset criando threads, FS bloqueando o frame, e memcard herdando comportamento
acidental do backend.
## Pontos Criticos
- `thread::spawn` por request escala mal e cria jitter.
- A terceira core/lane deve ser um recurso de plataforma controlado, nao uma
permissao geral para criar threads.
- Asset loading precisa de um pool/fila limitada para multiplos loads
pendentes.
- FS pode justificar a mesma lane quando envolver IO real, mas nao devemos
assinar uma API assincrona ampla sem entender a semantica de retorno.
- Commit/install em bancos residentes pode ter restricoes de determinismo e
ownership; decode pode ser assincrono, mas publicacao/instalacao talvez tenha
que cruzar de volta para a lane principal.
- A estrategia precisa funcionar em hardware alvo com poucos cores e tambem no
host desktop sem transformar desktop em referencia de custo errada.
- Render worker ja tem seu proprio contrato e nao deve compartilhar essa lane.
## Hotspots Atuais
- [asset.rs](/Users/niltonconstantino/personal/workspace.personal/intrepid/prometeu/runtime/crates/console/prometeu-drivers/src/asset.rs#L438) - `AssetManager::load`.
- [asset.rs](/Users/niltonconstantino/personal/workspace.personal/intrepid/prometeu/runtime/crates/console/prometeu-drivers/src/asset.rs#L507) - `thread::spawn` por load.
- [fs.rs](/Users/niltonconstantino/personal/workspace.personal/intrepid/prometeu/runtime/crates/console/prometeu-hal/src/syscalls/domains/fs.rs#L4) - `fs` e `mem` marcados como superficies custosas/nao deterministicas.
- [dispatch.rs](/Users/niltonconstantino/personal/workspace.personal/intrepid/prometeu/runtime/crates/console/prometeu-system/src/services/vm_runtime/dispatch.rs#L549) - chamadas `fs.*` executam no dispatch.
- [virtual_fs.rs](/Users/niltonconstantino/personal/workspace.personal/intrepid/prometeu/runtime/crates/console/prometeu-system/src/services/fs/virtual_fs.rs#L69) - VFS normaliza path e chama backend diretamente.
## Alvo da Discussao
Fechar um modelo de terceira core/lane para trabalho assincrono de runtime,
comecando por asset loading e avaliando se FS deve entrar na mesma lane no v1.
O alvo nao e criar multitarefa geral, callbacks para guest, ou um scheduler
paralelo da VM. A lane deve executar trabalho de IO/decode/sync e publicar
resultados observaveis por status/polling em pontos previsiveis.
## O Que Precisa Ser Definido
1. Topologia da terceira lane.
Definir:
- uma async work lane dedicada, separada da VM e do render worker;
- se a lane e uma thread fixa no host e uma core dedicada no hardware alvo;
- fallback quando nao houver core fisico dedicado;
- proibicao explicita de `thread::spawn` por request no caminho normal.
2. Pool/fila de asset loading.
Definir:
- backlog de pedidos sem limite operacional fixo;
- exatamente um job ativo por vez na lane;
- o "pool" significa backlog e estados de job, nao multiplos workers reais;
- politica de substituicao/cancelamento para pedidos que miram o mesmo
bank/slot;
- status para pending/backlog, ativo, progresso, cancelamento, erro de
decode e sucesso.
3. Fases do asset load.
Separar:
- request e alocacao de handle;
- leitura de bytes;
- decode/materializacao;
- staging de resultado;
- install/commit em banco residente.
Ponto a fechar: quais fases rodam na terceira lane e quais voltam para a
lane principal por determinismo/ownership.
4. Inclusao de FS na mesma lane.
Verificar se `fs` deve usar a terceira lane no v1 para:
- leituras grandes;
- escritas;
- listagem;
- delete;
- memcard commit/read/write.
Ponto a fechar: FS sync publico continua bloqueante/status-first, vira
request/poll, ou ganha apenas uma implementacao interna que drena trabalho em
pontos de frame.
5. Separacao por dominio.
Decidir se `asset` e `fs` compartilham uma fila com classes de prioridade ou
se a terceira lane contem subfilas dedicadas.
6. Politica de prioridade.
Definir:
- prioridade de loads visuais vs audio;
- prioridade de save/config;
- prioridade de memcard commit;
- starvation prevention;
- limite de trabalho de instalacao/publicacao por frame.
7. Modelo de retorno e telemetria.
Fechar como guest/runtime observa:
- `pending`;
- `loading`;
- `ready`;
- `canceled`;
- `backend_unavailable`;
- backlog;
- posicao de um asset no backlog;
- progresso percentual do asset ativo;
- drops/retries quando aplicavel.
8. Orcamento para hardware barato.
Definir o baseline:
- VM principal;
- render worker;
- terceira async work lane;
- limites de memoria para jobs pendentes e buffers temporarios.
## Opcoes
### Opcao A - Terceira lane unica com subfilas e pool limitado de assets
- **Abordagem:** criar uma async work lane dedicada. Assets usam um pool/fila
de jobs nessa lane; FS tambem pode submeter jobs na mesma lane, com
prioridade menor ou classe separada.
- **Pro:** modelo simples de budget; evita explosao de threads; encaixa no alvo
de terceira core; permite compartilhar backpressure e telemetria.
- **Contra:** assets e FS podem competir por IO; precisa politica clara para
evitar que save/config fique atras de loads longos.
- **Manutenibilidade:** boa se a lane tiver tipos de job e estados explicitos.
### Opcao B - Lane de assets agora, FS apenas reservado
- **Abordagem:** fechar v1 somente para asset loading, removendo `spawn` por
request. FS continua sincrono por enquanto, mas o contrato da lane reserva
compatibilidade para FS no futuro.
- **Pro:** menor escopo; resolve a dor concreta de assets rapidamente; evita
alterar semantica de FS antes da agenda de app home filesystem.
- **Contra:** nao resolve stalls reais de FS/memcard; pode exigir refatorar a
lane depois para aceitar FS.
- **Manutenibilidade:** boa no curto prazo, com risco de segunda migracao.
### Opcao C - Lanes separadas para asset e FS
- **Abordagem:** criar uma lane/pool para assets e outra para FS.
- **Pro:** isolamento forte; FS/save nao compete diretamente com loads de
textura/audio/cena.
- **Contra:** consome mais threads/cores; contraria a premissa de terceira core
unica; pior para hardware simples.
- **Manutenibilidade:** media; isolamento ajuda, mas a topologia fica pesada.
### Opcao D - Pool global pequeno sem nocao de terceira lane
- **Abordagem:** trocar `spawn` por request por um pool global fixo.
- **Pro:** simples de implementar em host desktop.
- **Contra:** perde a arquitetura de lanes por core; pode mascarar custo em
desktop; fica menos claro como mapear para hardware alvo.
- **Manutenibilidade:** baixa para o modelo de console se virar dependencia
acidental de scheduler do SO.
## Sugestao / Recomendacao
Recomendo seguir com a Opcao A como direcao de arquitetura, mas fechar a
decisao em duas camadas:
1. Contrato obrigatorio agora:
- terceira async work lane como recurso runtime;
- assets deixam de usar `thread::spawn` por requisicao;
- asset loading passa por backlog consultavel/reordenavel;
- status, progresso e telemetria de backlog ficam explicitos.
2. Contrato condicionado para FS:
- FS deve ser analisado como consumidor legitimo da mesma lane porque suas
operacoes podem envolver IO real e ja sao custosas/nao deterministicas;
- a decisao precisa escolher entre incluir FS no v1 ou reservar a lane para
FS sem mudar ainda a superficie publica;
- memcard precisa ser considerado junto com FS, porque usa a mesma
capacidade e backend de persistencia.
O ponto importante: a terceira core/lane deve ser a unidade de concorrencia.
O pool de assets deve existir dentro desse modelo, nao como criacao livre de
threads por load.
## Respostas Consolidadas Ate Aqui
1. O pool de assets deve ser uma lane serial.
- Ha exatamente um job ativo na async work lane.
- O pool representa backlog, estados e controle de ordem, nao multiplos
workers executando em paralelo.
2. O backlog de pedidos deve ser semanticamente ilimitado.
- Nao existe `queue_full` como status normal do dominio.
- Como cada pedido mira um `bank_type/slot` e pedidos novos para o mesmo
alvo substituem os anteriores, o backlog efetivo fica limitado pela soma
dos slots dos bancos.
- O contrato nao precisa expor limite de fila, mas a implementacao deve
garantir que metadados de handles cancelados/superseded nao crescam sem
coleta.
3. Asset loading precisa expor consulta e reordenacao do backlog.
- Deve existir API publica/runtime para consultar onde um asset esta no
backlog.
- Deve existir capacidade de adiantar ou atrasar um pedido de asset na fila.
- O asset ativo deve expor progresso percentual.
- Requests novos para o mesmo `bank_type/slot` devem cancelar pedidos
anteriores pendentes para aquele alvo, evitando carregar asset que ja foi
substituido antes de executar.
- Se o alvo ja estiver carregado com o mesmo `asset_id`, o novo request nao
entra no backlog; ele retorna imediatamente pronto com handle disponivel.
- Se um novo request substituir o alvo atualmente ativo, a lane deve cancelar
em ponto cooperativo quando possivel; se o trabalho nao puder ser
interrompido, o resultado antigo deve ser descartado ao final por geracao
superseded.
- `superseded` significa que um request foi substituido por outro request
mais novo para o mesmo `bank_type/slot`. Nao e erro e nao e cancelamento
manual; e obsolescencia operacional do pedido antigo.
- Handles representam o slot do bank como alvo estavel. Dentro do estado do
handle deve existir uma secao de request/backlog para aquele slot,
informando se ha pedido pendente/ativo, qual asset foi solicitado,
progresso, posicao e geracao.
- Handles devem ser capazes de consultar o estado do alvo `bank_type/slot`
a qualquer momento. Se nao houver asset valido carregado e tambem nao
houver request pendente para aquele alvo, a consulta deve retornar estado
explicito de vazio/invalido, nao falhar por ausencia de request ativo.
4. Decode/materializacao fria rodam na terceira lane.
- Leitura e decode de asset podem acontecer fora da lane principal.
- O resultado deve voltar como material pronto/staged.
5. Install/commit em bancos residentes continua na lane principal.
- A async lane prepara trabalho.
- A lane principal publica/instala em pontos previsiveis por ownership,
determinismo e integracao com bancos/render/audio.
6. FS e memcard entram como consumidores possiveis da mesma lane, mas precisam
ser melhor amarrados com a agenda de app home filesystem.
- A `DSC-0007` / `AGD-0006` deve ser ajustada para estabelecer melhor a
superficie publica e o modelo async/sync de FS.
- Esta agenda deve afirmar a existencia da lane para trabalhos async de IO
desse tipo e permitir que FS a consuma.
- Questoes de API publica de FS, request/poll e semantica de app home devem
ficar a cargo da agenda 6.
- FS, memcard e assets nao devem operar sobrepostos quando isso gerar
concorrencia real de backend; se houver competicao, a fila unica resolve a
ordem.
7. Prioridade quando houver competicao:
- memcard commit/write;
- FS write/config;
- asset visual/audio load;
- list/read nao critico.
8. Status-first continua valendo.
- `backend_unavailable`, `canceled`, erro de decode e estados de job sao
status operacionais.
- Cancelamento por superseding de pedido no mesmo bank/slot e resultado
operacional normal, nao fault.
- Nao ha `queue_full` no contrato esperado.
- Trap permanece para violacao estrutural, como argumento invalido,
capability ausente ou handle estruturalmente invalido.
9. Telemetria minima:
- backlog atual;
- posicao por asset;
- progresso percentual do asset ativo;
- jobs submetidos, concluidos, falhos e cancelados;
- tempo por job;
- percentis por asset enquanto estiver carregando na lane.
10. Escopo da agenda.
- O foco desta agenda e a lane, assets e memcard/game persistence.
- FS entra como consumidor permitido da lane de IO, mas as decisoes de API
publica de FS pertencem a `AGD-0006`.
11. API de backlog.
- A API inicial sugerida cobre os casos esperados.
- `asset.backlog_promote(handle)` deve ser mantido como atalho oficial para
`asset.backlog_move(handle, 1)`.
- `asset.backlog_demote(handle)` deve ser mantido como atalho oficial para
mover o pedido para o fim dos pendentes.
- Os atalhos nao criam semantica nova; apenas evitam que o caller calcule
posicoes comuns.
- O handle deve permitir consulta estavel do slot do bank mesmo quando nao ha
asset valido carregado.
- O estado do handle deve separar:
- estado do slot/residencia atual;
- estado do request/backlog atual para aquele slot.
- Operacoes que mudam estado, como commit/cancel/reorder, atuam sobre a
secao de request/backlog do handle e precisam validar geracao/request atual
para nao agir sobre um pedido substituido.
12. Cancelamento por substituicao.
- Quando um request novo chega para o mesmo `bank_type/slot`, o request
anterior e marcado imediatamente como `superseded`.
- Se o request anterior ainda estiver no backlog, ele sai da fila.
- Se o request anterior estiver ativo, a lane tenta cancelar
cooperativamente quando a fase atual permitir.
- Se a fase ativa nao puder ser interrompida com baixo custo, ela termina e
o resultado e descartado ao final porque a geracao do alvo ja mudou.
- Como o handle pertence ao slot do bank, ele continua consultavel; sua
secao de request deve indicar que a geracao anterior foi `superseded` ou
que nao ha request ativo para aquele alvo.
13. Fases de progresso.
- `read/decode/stage` e suficiente como modelo minimo para `GLYPH`, `SOUND`
e `SCENE`.
- Subfases futuras podem detalhar progresso dentro de `decode` sem mudar o
contrato geral.
14. Percentis.
- Percentis devem existir por `bank_type` como telemetria principal.
- Percentis por `asset_id` tambem sao uteis, mas podem usar janela pequena
ou agregador leve.
- Percentis sao atualizados no fechamento do job, nao durante o hot path.
15. Ajuste esperado na agenda de FS.
- `AGD-0006` deve registrar que existe uma async IO lane compartilhavel por
FS/memcard/assets.
- Esta agenda so decide que FS pode consumir a lane.
- A API publica de FS, incluindo request/poll ou sync aparente, pertence a
`AGD-0006`.
## API Inicial Sugerida Para Assets
Esta agenda ainda nao fecha nomes finais de ABI, mas a direcao inicial sugerida
e manter `asset.load/status/commit/cancel` e adicionar uma superficie pequena
para backlog:
1. `asset.backlog_info() -> (status, pending_count, active_handle, active_asset_id, active_bank_type, active_slot, active_progress)`
- Consulta o estado geral da lane de assets.
- `active_progress` usa escala inteira, preferencialmente `0..10000` para
permitir duas casas decimais sem float.
2. `asset.backlog_position(handle) -> (status, state, position, progress)`
- Consulta um pedido especifico.
- `position = 0` significa job ativo.
- `position > 0` significa posicao no backlog.
- `position = -1` pode representar nao enfileirado/pronto/cancelado,
dependendo do `state`.
3. `asset.backlog_move(handle, new_position) -> status`
- Move um pedido pendente para uma posicao absoluta.
- `new_position = 1` significa primeiro pendente depois do ativo.
- Nao interrompe diretamente o job ativo; substituicao do mesmo
`bank_type/slot` usa a regra de superseding.
4. `asset.backlog_promote(handle) -> status`
- Atalho para mover o pedido para a frente do backlog pendente.
5. `asset.backlog_demote(handle) -> status`
- Atalho para mover o pedido para o fim do backlog pendente.
6. `asset.target_status(bank_type, slot) -> (status, asset_id, handle, state, position, progress)`
- Consulta por alvo operacional, que e o que realmente controla superseding.
- Ajuda quando o caller quer saber "o que esta programado para este slot".
Estados sugeridos para `state`:
- `empty`;
- `invalid`;
- `queued`;
- `active`;
- `ready`;
- `committed`;
- `canceled`;
- `superseded`;
- `error`;
- `backend_unavailable`.
Regra de design: APIs de backlog sao operacionais e status-first. Reordenar um
handle inexistente, ja concluido ou ja superseded retorna status operacional
apropriado quando o handle e conhecido; violações estruturais continuam Trap.
Handles sao observaveis e representam slots de banks. Um handle conhecido pode
ser consultado mesmo quando o slot alvo nao contem asset valido carregado; nesse
caso a consulta deve retornar estado como `empty` ou `invalid`.
O estado do handle deve conter pelo menos duas partes:
```text
slot_state:
loaded_asset_id
resident_state
request_state:
requested_asset_id
generation
queued/active/ready/canceled/superseded/error
backlog_position
progress
```
Mutacoes seguem regras mais estritas e devem falhar operacionalmente quando a
secao de request/backlog do handle nao representa a geracao atual.
## Progresso e Percentis
### Progresso percentual
Sugestao:
- usar progresso em base inteira `0..10000`;
- quando o tamanho total de bytes for conhecido, calcular progresso por fases
ponderadas;
- quando o decode nao for linear ou nao houver tamanho confiavel, expor
progresso por fase com marcos conservadores.
Modelo inicial de fases:
```text
queued -> 0
read -> 0..4000
decode -> 4000..9000
stage -> 9000..10000
ready -> 10000
```
Se uma fase nao consegue reportar progresso interno, ela deve manter o ultimo
marco e saltar para o proximo marco ao concluir. Isso evita progresso falso
sem perder visibilidade de fase.
### Percentis por asset
Sugestao:
- registrar duracoes por asset no fim de cada job, nao por tick/frame;
- manter agregadores pequenos por `asset_id` e por `bank_type`;
- calcular percentis a partir de janela/ring buffer pequeno ou histograma
barato por bucket de duracao;
- nunca atualizar percentis no loop quente de decode por item/pixel;
- o job ativo pode expor tempo decorrido e fase atual; percentis sao atualizados
apenas no fechamento do job.
## Perguntas em Aberto
- [x] A agenda esta madura para virar decisao normativa ou ainda falta algum
caso de asset/memcard/lane?
- [x] A estrutura minima de `slot_state` e `request_state` no handle esta
completa ou falta algum campo para commit/cancel/reorder seguro?
## Resolucao
A agenda esta madura para decisao normativa.
A decisao deve formalizar:
- a terceira async work lane serial;
- um job ativo por vez;
- backlog efetivamente limitado pela unicidade de `bank_type/slot`;
- handles como observadores estaveis de slots de bank;
- `slot_state` e `request_state` dentro do handle;
- cancelamento/superseding por novo request para o mesmo alvo;
- load pronto imediato quando o asset ja esta residente no alvo;
- decode/materializacao na lane async;
- commit/install na lane principal;
- memcard como consumidor prioritario da mesma lane;
- FS como consumidor permitido, deixando API publica de FS para `AGD-0006`;
- progresso por fases e percentis atualizados no fechamento do job.
## Dependencias
- `docs/specs/runtime/09-events-and-concurrency.md`
- `docs/specs/runtime/15-asset-management.md`
- `docs/specs/runtime/16a-syscall-policies.md`
- `discussion/workflow/agendas/AGD-0006-app-home-filesystem-surface-and-semantics.md`
## Criterio para Encerrar
Pode virar PR quando houver decisao escrita sobre:
- existencia e papel da terceira async work lane;
- relacao dessa lane com a terceira core no hardware alvo e fallback no host;
- desenho do backlog serial de asset loading;
- regra de cancelamento/substituicao para requests no mesmo bank/slot;
- proibicao de `thread::spawn` por load no caminho normal;
- fases de asset load que rodam na lane vs lane principal;
- inclusao condicionada de FS/memcard nessa mesma lane;
- API de consulta e reordenacao do backlog de assets;
- comportamento de request imediato quando o asset ja esta residente no alvo;
- regra de bounded backlog por unicidade de `bank_type/slot`;
- politica de prioridade entre asset, FS e memcard;
- status e telemetria para backlog, progresso, percentis, cancelamento e falha.
## Discussao
- 2026-06-28: Agenda atualizada para remover render worker do escopo, assumir
uma terceira core/lane para trabalhos assincronos, exigir pool/fila limitada
para asset loading e reavaliar o papel de FS/memcard como consumidores da
mesma lane.
- 2026-06-28: Discussao consolidou que a async work lane deve ser serial, com
um job ativo, backlog semanticamente ilimitado, consulta/reordenacao publica
de backlog de assets, progresso percentual do asset ativo, decode na lane
async, install/commit na lane principal, e FS/memcard como consumidores a
alinhar com a agenda de app home filesystem.
- 2026-06-28: Acrescentada regra de asset pipe: requests para o mesmo bank/slot
devem cancelar pedidos anteriores pendentes para aquele alvo, tratando o
pedido mais novo como substituto operacional.
- 2026-06-28: Discussao refinou que o alvo canonico e `bank_type/slot`; um
asset ja residente deve retornar handle pronto sem entrar no backlog; o
backlog efetivo e limitado pela soma dos slots dos bancos; FS fica como
consumidor permitido da lane, enquanto API publica de FS permanece na agenda
6; progresso deve usar fases conservadoras e percentis devem ser atualizados
no fechamento do job, nao no hot path.
- 2026-06-28: Open questions restantes foram consolidadas: API de backlog
aceita com `promote/demote` como atalhos; superseding marca requests
substituidos por outro request para o mesmo `bank_type/slot`; cancelamento do
ativo e cooperativo quando barato e descarte por geracao quando nao for;
`read/decode/stage` cobre GLYPH/SOUND/SCENE; percentis ficam por `bank_type`
e por `asset_id`; API publica de FS permanece em `AGD-0006`.
- 2026-06-28: Adicionada a expectativa de que handles possam consultar o alvo
`bank_type/slot` a qualquer momento, inclusive quando nao houver asset valido
carregado; mutacoes devem continuar protegidas por request/geracao para evitar
commit/cancel acidental de pedidos substituidos.
- 2026-06-28: Refinado o modelo de handle: o handle pertence ao slot do bank e
contem uma secao de estado de request/backlog para aquele slot. Consulta e
residencia ficam estaveis por alvo; mutacoes atuam sobre a geracao atual do
request dentro do handle.

336
discussion/workflow/decisions/DEC-0034-async-work-lane-and-asset-backlog-contract.md
View File

@ -1,336 +0,0 @@
---
id: DEC-0034
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Work Lane and Asset Backlog Contract
status: accepted
created: 2026-06-28
accepted: 2026-06-28
ref_agenda: AGD-0008
plans: [PLN-0123, PLN-0124, PLN-0125, PLN-0126, PLN-0127, PLN-0128]
tags: [perf, asset, fs, async, scheduler, runtime]
---
# DEC-0034 - Async Work Lane and Asset Backlog Contract
## Status
Accepted.
## Contexto
`asset.load()` currently creates an OS thread per non-resident asset request.
That behavior does not match the target runtime model: the machine should have
explicit execution lanes, bounded ownership, observable backpressure, and no
uncontrolled thread creation from guest-visible operations.
Render worker concurrency is already handled by separate render decisions. This
decision covers the third runtime lane for asynchronous IO/decode/persistence
work:
```text
Lane 1: VM, firmware, SystemOS, logical frame
Lane 2: render worker / asynchronous rasterization
Lane 3: async work lane for asset IO/decode and game persistence work
```
The concrete first consumer is asset loading. Game persistence through memcard
may also consume this lane. FS is allowed to consume this lane, but public FS API
shape belongs to the app-home filesystem agenda.
## Decisao
Prometeu MUST introduce a third async work lane as the runtime-owned execution
place for asset IO/decode work and compatible persistence IO work.
The async work lane MUST be serial: it has exactly one active job at a time.
The runtime MUST NOT create one OS thread per `asset.load` request in the normal
path.
Asset loading MUST use a backlog keyed by target `bank_type/slot`. A target can
have at most one current request. A newer request for the same `bank_type/slot`
MUST supersede the previous request for that target.
Asset handles MUST represent stable bank slots, not transient worker threads.
A handle MUST remain queryable even when the slot has no valid loaded asset and
even when there is no active request for that slot. The handle state MUST
separate resident slot state from request/backlog state.
Asset IO/read and decode/materialization MUST run on the async work lane.
Asset install/commit into resident banks MUST happen on the main runtime lane at
predictable ownership points.
Memcard work MAY consume the same async work lane and has higher priority than
ordinary asset loads when contention exists. FS MAY consume the same lane for
IO-style work, but this decision does not define the public FS API.
## Rationale
A serial lane maps cleanly to the intended third-core mental model and avoids a
desktop-biased generic thread pool. It gives the runtime one place to observe
IO/decode backlog, progress, priority, cancellation, and telemetry.
Keying requests by `bank_type/slot` makes the backlog naturally bounded by the
sum of bank slots. There is no need for a guest-visible `queue_full` state:
there cannot be unbounded distinct pending targets if each target has at most
one current request.
Keeping commit/install on the main lane preserves deterministic publication
semantics and avoids letting the async lane mutate resident graphics/audio/scene
state while the VM, render handoff, or frame boundary is observing it.
Separating `slot_state` and `request_state` inside the handle avoids ambiguity:
callers can inspect the slot at any time, while mutating operations still remain
protected by request generation.
## Invariantes / Contrato
### Async Work Lane
- The async work lane MUST be runtime-owned.
- The async work lane MUST be separate from the render worker.
- The async work lane MUST execute at most one active job at a time.
- The runtime MUST NOT use `thread::spawn` per asset request in the normal
asset loading path.
- The host may implement the lane as a fixed worker thread. Hardware targets may
map it to a dedicated core when available.
- If a physical third core is unavailable, the implementation MUST preserve the
same logical lane contract.
### Asset Backlog
- Asset requests MUST target a concrete `bank_type/slot`.
- The backlog is ordered and serial.
- Each `bank_type/slot` MUST have at most one current request.
- A newer request for the same `bank_type/slot` MUST supersede any earlier
pending request for that target.
- If the target already contains the requested `asset_id` as a valid resident
asset, `asset.load` MUST return a handle in ready state without adding a job
to the backlog.
- The effective backlog size is bounded by the sum of targetable bank slots,
even though the contract does not expose a fixed queue limit.
- Implementations MUST collect stale metadata for canceled or superseded
requests so the handle/request history does not grow without bound.
### Handles
An asset handle MUST represent a stable bank slot target. The handle state MUST
include at least:
```text
handle:
bank_type
slot
slot_state:
loaded_asset_id
resident_state
slot_generation
request_state:
requested_asset_id
request_generation
state
backlog_position
progress
```
The handle MUST be queryable when:
- the slot is empty;
- the slot contains a valid resident asset;
- a request is queued;
- a request is active;
- a request is ready for commit;
- a request has been canceled;
- a request has been superseded;
- a request ended in error.
Mutating operations such as commit, cancel, promote, demote, or move MUST act on
the current request generation. They MUST NOT accidentally mutate a newer
request through an older handle view.
### Superseding
`superseded` means a request was replaced by a newer request for the same
`bank_type/slot`.
- Superseding is an operational state, not a fault.
- If the old request is queued, it MUST be removed from the backlog.
- If the old request is active, the lane SHOULD cancel cooperatively when the
current phase supports cheap cancellation.
- If active work cannot be interrupted cheaply, it MAY finish, but its result
MUST be discarded when its generation no longer matches the target's current
request generation.
### Asset Backlog API Direction
The existing asset operations remain the base surface:
```text
asset.load(asset_id, slot) -> (status, handle)
asset.status(handle) -> status
asset.commit(handle) -> status
asset.cancel(handle) -> status
```
The asset backlog surface SHOULD add a small status-first API:
```text
asset.backlog_info()
-> (status, pending_count, active_handle, active_asset_id,
active_bank_type, active_slot, active_progress)
asset.backlog_position(handle)
-> (status, state, position, progress)
asset.backlog_move(handle, new_position)
-> status
asset.backlog_promote(handle)
-> status
asset.backlog_demote(handle)
-> status
asset.target_status(bank_type, slot)
-> (status, asset_id, handle, state, position, progress)
```
`asset.backlog_promote(handle)` is an official shortcut for moving a queued
request to position `1`, the first pending position after the active job.
`asset.backlog_demote(handle)` is an official shortcut for moving a queued
request to the end of the pending backlog.
The final ABI names and exact return shapes may be refined during planning, but
the implementation MUST preserve the capabilities above.
### Progress
Progress MUST be represented without floating point. The recommended scale is
`0..10000`.
The initial phase model is:
```text
queued -> 0
read -> 0..4000
decode -> 4000..9000
stage -> 9000..10000
ready -> 10000
```
If a phase cannot report internal progress, it MUST keep the previous progress
mark and advance at phase completion. Implementations MUST NOT invent false
precision for non-linear decode phases.
### Telemetry
Telemetry MUST be cheap and must not add hot-path per-item decode cost.
Minimum telemetry:
- current backlog depth;
- target/request position;
- active job progress;
- jobs submitted;
- jobs completed;
- jobs failed;
- jobs canceled;
- jobs superseded;
- job duration;
- percentiles by `bank_type`;
- lightweight percentiles or small-window samples by `asset_id`.
Percentiles MUST be updated when a job closes, not inside the inner decode loop.
### Priority
When async lane consumers contend, the initial priority order is:
1. memcard commit/write;
2. FS write/config work;
3. asset visual/audio/scene load;
4. non-critical list/read work.
This priority order is a lane arbitration rule. It does not define public FS
syscall semantics.
### FS Boundary
FS MAY consume the async work lane for IO-style work.
This decision MUST NOT define the public FS API, app-home FS semantics, or
whether FS is exposed as request/poll or sync-appearing operations. Those
questions belong to `AGD-0006`.
`AGD-0006` MUST be updated or interpreted with the existence of this async IO
lane in mind.
## Impactos
### Spec
- `docs/specs/runtime/15-asset-management.md` must absorb the async lane,
backlog, handle, superseding, progress, and telemetry contract.
- `docs/specs/runtime/16-host-abi-and-syscalls.md` must absorb any final public
asset backlog syscall names and return shapes.
- `docs/specs/runtime/16a-syscall-policies.md` may need status catalog updates
for superseded/canceled/backend unavailable states if not already covered by
asset domain status.
- `docs/specs/runtime/09-events-and-concurrency.md` should mention the async
work lane as an implementation-side lane that does not introduce guest
callbacks.
- `AGD-0006` / future FS spec work must account for FS as a possible consumer of
the async IO lane.
### Runtime
- Asset loading must stop spawning a thread per request.
- AssetManager needs a serial backlog keyed by `bank_type/slot`.
- Asset handles need stable slot identity plus separated `slot_state` and
`request_state`.
- Commit/install remains a main-lane operation.
- Superseding and generation checks become required correctness mechanisms.
### Host
- Desktop host may implement the async work lane with one fixed worker thread.
- Host presentation/render worker remains separate and unaffected.
### Firmware
- No direct firmware behavior is required for asset backlog mechanics.
- Firmware flows that rely on preload or memcard may later use lane telemetry or
priority policy if needed.
### Tooling
- Test tools may need support for inspecting backlog state and forcing request
ordering.
- Asset fixtures should cover queued, active, ready, canceled, superseded, and
already-resident paths.
## Referencias
- `AGD-0008` - Async Background Work Lanes for Assets and FS.
- `docs/specs/runtime/15-asset-management.md`
- `docs/specs/runtime/16-host-abi-and-syscalls.md`
- `docs/specs/runtime/16a-syscall-policies.md`
- `docs/specs/runtime/09-events-and-concurrency.md`
- `docs/specs/runtime/08-save-memory-and-memcard.md`
- `AGD-0006` - App Home Filesystem Surface and Semantics.
## Propagacao Necessaria
1. Update asset management specs with the lane/backlog/handle contract.
2. Update syscall specs if backlog APIs become public ABI.
3. Update FS agenda/spec work to acknowledge the async IO lane without deciding
FS public API in this decision.
4. Plan implementation of a serial async work lane and asset backlog.
5. Plan tests for superseding, stable slot handles, immediate-ready resident
assets, progress, telemetry, and main-lane commit.
## Revision Log
- 2026-06-28: Initial draft from `AGD-0008`.

118
discussion/workflow/plans/PLN-0123-async-work-lane-specification-propagation.md
View File

@ -1,118 +0,0 @@
---
id: PLN-0123
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Work Lane Specification Propagation
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, fs, async, scheduler, runtime, spec]
---
# PLN-0123 - Async Work Lane Specification Propagation
## Briefing
`DEC-0034` establishes a third serial async work lane, asset backlog semantics,
stable bank-slot handles, superseding, progress, telemetry, and FS/memcard lane
boundaries. The canonical specs must be updated before code work relies on the
new contract.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Publish the accepted async work lane and asset backlog contract in the canonical
runtime specs.
## Escopo
Included:
- Update asset management spec.
- Update syscall ABI spec if public backlog APIs are named there.
- Update syscall policy/status language for operational asset states.
- Update events/concurrency spec with the async work lane as an implementation
lane that does not introduce guest callbacks.
- Add a cross-reference from FS/app-home agenda or notes to the async IO lane.
Fora de Escopo:
- Implementing runtime code.
- Finalizing public FS API semantics.
- Changing render worker contracts.
## Plano de Execucao
### Step 1 - Update Asset Management
**What:** Add the third async lane, backlog, handle, superseding, progress, and
telemetry contract.
**How:** Extend `docs/specs/runtime/15-asset-management.md` around the existing
`asset.load/status/commit/cancel` section. Preserve the existing
`asset.load(asset_id, slot)` identity contract.
**Files:** `docs/specs/runtime/15-asset-management.md`.
### Step 2 - Update Host ABI
**What:** Add the backlog API direction if the ABI spec is the canonical place
for syscall shapes.
**How:** Extend the asset surface section with the planned backlog operations,
marking exact numeric ids as implementation-plan work if ids are not assigned
yet.
**Files:** `docs/specs/runtime/16-host-abi-and-syscalls.md`.
### Step 3 - Update Syscall Policies
**What:** Clarify that `superseded`, `canceled`, `backend_unavailable`, and
decode failures are operational statuses, not faults.
**How:** Add asset-domain examples under status-first policy.
**Files:** `docs/specs/runtime/16a-syscall-policies.md`.
### Step 4 - Update Events and Concurrency
**What:** Document that the async work lane is an implementation lane.
**How:** Add a section next to render worker concurrency stating that asset/IO
lane work does not introduce guest callbacks or hidden guest execution.
**Files:** `docs/specs/runtime/09-events-and-concurrency.md`.
### Step 5 - Cross-Reference FS Work
**What:** Record that FS may consume the async IO lane, while public FS API
semantics remain owned by the app-home filesystem discussion.
**How:** Add a note in `AGD-0006` or its future decision input, without deciding
FS request/poll shape in this plan.
**Files:** `discussion/workflow/agendas/AGD-0006-app-home-filesystem-surface-and-semantics.md`.
## Criterios de Aceite
- [ ] Specs state that the async work lane is serial and separate from render.
- [ ] Asset spec states backlog keying by `bank_type/slot`.
- [ ] Asset spec states stable slot handles with `slot_state` and `request_state`.
- [ ] Asset spec states superseding behavior.
- [ ] Asset spec states main-lane commit/install.
- [ ] FS is cross-referenced as a permitted lane consumer without public API decisions.
## Tests / Validacao
- Run `discussion validate`.
- Review spec links for broken relative paths.
- No cargo tests are required for this documentation-only plan.
## Riscos
- Public ABI names may be refined later; the spec must avoid assigning ids too
early if implementation work has not reserved them.
- FS scope can leak into this work; keep FS public API decisions in `AGD-0006`.

119
discussion/workflow/plans/PLN-0124-async-work-lane-runtime-infrastructure.md
View File

@ -1,119 +0,0 @@
---
id: PLN-0124
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Work Lane Runtime Infrastructure
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, fs, async, scheduler, runtime]
---
# PLN-0124 - Async Work Lane Runtime Infrastructure
## Briefing
`DEC-0034` requires one runtime-owned async work lane, separate from VM/main
execution and separate from render worker execution. This plan introduces the
lane infrastructure before migrating asset loading.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Create a serial async work lane primitive that can execute one active job at a
time, expose observable state, and support deterministic shutdown.
## Escopo
Included:
- Runtime-owned async work lane type.
- Single active job execution.
- Ordered pending job storage.
- Cooperative cancellation hook.
- Shutdown and join behavior.
- Basic lane telemetry.
Fora de Escopo:
- Public asset backlog syscalls.
- Asset handle model migration.
- FS public API.
- Render worker changes.
## Plano de Execucao
### Step 1 - Locate Runtime Service Boundary
**What:** Choose the crate/module boundary for the async lane.
**How:** Place generic lane infrastructure near VM runtime services, not inside
desktop host code and not inside render worker modules.
**Files:** likely `crates/console/prometeu-system/src/services/` or
`crates/console/prometeu-system/src/services/vm_runtime/`.
### Step 2 - Define Job and Lane Types
**What:** Add a lane type with pending queue, active job metadata, generation,
state, progress, and cancellation flag.
**How:** Use typed job enums or traits that do not require guest callbacks.
Start with internal asset job support but keep the lane generic enough for
memcard/FS later.
**Files:** new module under `prometeu-system` service layer.
### Step 3 - Implement Serial Worker
**What:** Implement one worker loop that takes one pending job and runs it to
completion or cooperative cancellation.
**How:** Use a fixed worker thread on host builds. Preserve a logical-lane API
so future hardware can map it to a dedicated core.
**Files:** async lane module; lifecycle wiring in VM runtime service setup.
### Step 4 - Implement Shutdown
**What:** Add explicit stop/shutdown behavior.
**How:** Wake the lane, stop accepting new jobs, report pending jobs as canceled
or shutdown-discarded, and join the worker with bounded behavior.
**Files:** async lane module and VM runtime lifecycle modules.
### Step 5 - Expose Internal Telemetry
**What:** Add counters for pending depth, active job, submitted, completed,
failed, canceled, and superseded.
**How:** Use cheap atomics or lane-owned state snapshots. Do not update
expensive percentile data inside inner decode loops.
**Files:** async lane module; telemetry structs where runtime telemetry lives.
## Criterios de Aceite
- [ ] Lane has one active job maximum.
- [ ] Lane is separate from render worker code.
- [ ] Lane can be started and stopped deterministically.
- [ ] Pending jobs are observable.
- [ ] Cooperative cancellation can be signaled.
- [ ] No guest callback path is introduced.
## Tests / Validacao
- Unit tests for enqueue, single active execution, ordering, cancellation signal,
and shutdown.
- Concurrency tests must synchronize on real state transitions, not sleeps.
- Run targeted `cargo test` for the owning crate.
## Riscos
- Accidentally creating a generic pool instead of a serial lane.
- Coupling the lane to desktop host thread details.
- Reusing render worker handoff semantics where ordered backlog is required.

127
discussion/workflow/plans/PLN-0125-asset-backlog-and-stable-slot-handles.md
View File

@ -1,127 +0,0 @@
---
id: PLN-0125
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Asset Backlog and Stable Slot Handles
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, runtime]
---
# PLN-0125 - Asset Backlog and Stable Slot Handles
## Briefing
`DEC-0034` changes asset loading from request-owned thread spawning to a serial
backlog keyed by `bank_type/slot`. Handles become stable bank-slot observers
with separate resident slot state and request state.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Refactor `AssetManager` so asset requests are tracked by target bank slot,
deduplicated/superseded by target, and observable through stable handles.
## Escopo
Included:
- Stable handle structure with `bank_type`, `slot`, `slot_state`, and
`request_state`.
- Backlog keyed by `bank_type/slot`.
- Already-resident fast path.
- Superseding for queued and active requests.
- Main-lane commit/install boundary.
Fora de Escopo:
- Public backlog syscall wiring.
- Percentile telemetry.
- FS public API.
## Plano de Execucao
### Step 1 - Model Slot Handles
**What:** Replace transient load-operation assumptions with stable bank-slot
handle state.
**How:** Introduce structures equivalent to:
```text
handle: bank_type, slot
slot_state: loaded_asset_id, resident_state, slot_generation
request_state: requested_asset_id, request_generation, state, position, progress
```
**Files:** `crates/console/prometeu-drivers/src/asset.rs` and HAL asset types if
public type definitions belong there.
### Step 2 - Key Backlog by Target
**What:** Ensure at most one current request per `bank_type/slot`.
**How:** Add a target index from `(bank_type, slot)` to current request
generation/handle state. Enqueue only when the target does not already have the
same resident asset and no newer request supersedes it.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 3 - Implement Already-Resident Fast Path
**What:** Return a ready handle immediately when the requested asset already
resides in the target slot.
**How:** Compare requested `asset_id` with the target `slot_state.loaded_asset_id`
and resident validity before creating a queued job.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 4 - Implement Superseding
**What:** New request for the same target supersedes previous queued or active
request.
**How:** Remove queued older request immediately. For active work, update target
generation so the old result is discarded if it cannot stop cooperatively.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`; async lane module
from `PLN-0124`.
### Step 5 - Preserve Main-Lane Commit
**What:** Keep install/commit into resident banks on the main runtime lane.
**How:** Async lane produces staged materialized result. `asset.commit(handle)`
installs only when the request generation is current and ready.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`; VM runtime dispatch if
commit behavior changes.
## Criterios de Aceite
- [ ] No asset load spawns an OS thread per request.
- [ ] At most one request exists per `bank_type/slot`.
- [ ] Same-target newer request supersedes older queued request.
- [ ] Same-target newer request prevents active stale result from installing.
- [ ] Already-resident target returns ready handle without backlog entry.
- [ ] Handles remain queryable for empty, queued, active, ready, committed,
canceled, superseded, and error states.
## Tests / Validacao
- Unit tests for handle state transitions.
- Unit tests for target-keyed superseding.
- Unit tests for already-resident fast path.
- Integration test for stale active result discard.
- Existing asset load/commit tests must continue passing.
## Riscos
- Confusing handle identity with request generation.
- Installing a stale async result after superseding.
- Losing compatibility with existing `asset.status/commit/cancel` behavior.

124
discussion/workflow/plans/PLN-0126-asset-backlog-public-api-and-status-surface.md
View File

@ -1,124 +0,0 @@
---
id: PLN-0126
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Asset Backlog Public API and Status Surface
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, syscall, runtime]
---
# PLN-0126 - Asset Backlog Public API and Status Surface
## Briefing
`DEC-0034` keeps `asset.load/status/commit/cancel` and adds a small status-first
backlog inspection and ordering surface. This plan wires that surface through
HAL syscall metadata and runtime dispatch.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Expose asset backlog inspection and reordering while preserving status-first
semantics and existing asset load identity.
## Escopo
Included:
- Syscall registry additions for backlog APIs.
- Runtime dispatch handlers.
- Status enum additions or normalization.
- ABI/spec alignment with `PLN-0123`.
Fora de Escopo:
- Async lane internals.
- Asset decode refactor.
- FS API.
## Plano de Execucao
### Step 1 - Finalize Syscall Shapes
**What:** Choose final names, arity, and return slots for backlog operations.
**How:** Start from `DEC-0034`:
```text
asset.backlog_info()
asset.backlog_position(handle)
asset.backlog_move(handle, new_position)
asset.backlog_promote(handle)
asset.backlog_demote(handle)
asset.target_status(bank_type, slot)
```
**Files:** `crates/console/prometeu-hal/src/syscalls/domains/asset.rs`,
`docs/specs/runtime/16-host-abi-and-syscalls.md`.
### Step 2 - Add Status Values
**What:** Represent `empty`, `invalid`, `queued`, `active`, `ready`,
`committed`, `canceled`, `superseded`, `error`, and `backend_unavailable`.
**How:** Extend or map existing `LoadStatus` and `AssetOpStatus` without
turning operational states into traps.
**Files:** HAL asset status types and `crates/console/prometeu-drivers/src/asset.rs`.
### Step 3 - Add Dispatch
**What:** Wire new syscalls through VM runtime dispatch.
**How:** Add match arms near existing `AssetLoad`, `AssetStatus`,
`AssetCommit`, and `AssetCancel` arms. Return status-first values.
**Files:** `crates/console/prometeu-system/src/services/vm_runtime/dispatch.rs`.
### Step 4 - Implement API Methods
**What:** Add AssetManager methods backing each public operation.
**How:** Methods must read stable handle/target state and reorder only queued
requests. `promote` and `demote` are shortcuts over move semantics.
**Files:** `crates/console/prometeu-drivers/src/asset.rs`.
### Step 5 - Validate ABI Metadata
**What:** Ensure registry ids, args, returns, capability, cost, and
non-deterministic flags are correct.
**How:** Update syscall metadata tests and any declared PBS/runtime syscall
tables.
**Files:** `crates/console/prometeu-hal/src/syscalls/tests.rs` and related
registry files.
## Criterios de Aceite
- [ ] Public backlog operations are registered with stable metadata.
- [ ] All backlog operations are status-first.
- [ ] `promote` is equivalent to moving to first pending position.
- [ ] `demote` moves to the end of pending backlog.
- [ ] Reordering a non-queued request returns operational status.
- [ ] Structural ABI misuse remains Trap.
## Tests / Validacao
- HAL syscall metadata tests.
- VM runtime dispatch tests for each new syscall.
- AssetManager unit tests for move/promote/demote.
- Tests for `target_status` on empty, queued, active, ready, and superseded
targets.
## Riscos
- Adding too many public ABI details before spec propagation is accepted.
- Returning payload values that are ambiguous when status is not success.
- Treating known but non-mutable states as traps instead of statuses.

120
discussion/workflow/plans/PLN-0127-async-lane-memcard-and-fs-integration-boundaries.md
View File

@ -1,120 +0,0 @@
---
id: PLN-0127
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Lane Memcard and FS Integration Boundaries
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, fs, memcard, async, runtime]
---
# PLN-0127 - Async Lane Memcard and FS Integration Boundaries
## Briefing
`DEC-0034` allows memcard and FS work to consume the async IO lane but keeps
public FS API semantics out of scope. This plan establishes integration
boundaries without deciding app-home FS API shape.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Define and implement safe internal boundaries for memcard and FS use of the
async work lane, preserving lane priority and keeping FS public API decisions in
`AGD-0006`.
## Escopo
Included:
- Memcard as a priority lane consumer candidate.
- FS as a permitted internal lane consumer.
- Priority ordering in the lane.
- Updates to `AGD-0006` to acknowledge the lane.
Fora de Escopo:
- Public FS request/poll design.
- App-home path/status contract.
- Full memcard async conversion if it requires public ABI changes.
## Plano de Execucao
### Step 1 - Update FS Agenda
**What:** Record that an async IO lane exists and may be used by FS.
**How:** Add a concise section to `AGD-0006` stating that FS public API remains
owned by that agenda, but implementation may use the async lane from `DEC-0034`.
**Files:** `discussion/workflow/agendas/AGD-0006-app-home-filesystem-surface-and-semantics.md`.
### Step 2 - Identify Memcard Operations
**What:** List memcard operations that can benefit from async IO.
**How:** Inspect `crates/console/prometeu-system/src/services/memcard.rs` and
VM runtime dispatch for `mem.*` operations. Classify read, write, commit, and
clear behavior.
**Files:** memcard service and VM runtime dispatch modules.
### Step 3 - Define Lane Priority
**What:** Encode the initial priority order from `DEC-0034`.
**How:** Add priority classes:
```text
memcard commit/write
FS write/config
asset load
non-critical list/read
```
**Files:** async lane module from `PLN-0124`.
### Step 4 - Add Internal Consumer Hooks
**What:** Provide internal APIs for memcard/FS to submit lane jobs later.
**How:** Add types/interfaces without changing public FS syscall behavior.
No public FS request/poll operations are introduced here.
**Files:** async lane module; FS/memcard service modules as needed.
### Step 5 - Preserve Current Public Behavior
**What:** Ensure existing memcard and FS syscalls keep their current observable
surface unless a later plan explicitly changes it.
**How:** Do not alter return shapes in this plan. Add tests proving existing
status-first behavior remains intact.
**Files:** VM runtime FS/memcard tests.
## Criterios de Aceite
- [ ] `AGD-0006` acknowledges the async IO lane and keeps FS public API in its
own scope.
- [ ] Async lane supports priority classes required by `DEC-0034`.
- [ ] Memcard and FS can be represented as internal lane consumers.
- [ ] Existing public memcard/FS syscall shapes are unchanged.
- [ ] No FS public request/poll API is introduced by this plan.
## Tests / Validacao
- Existing FS/memcard tests continue passing.
- Unit tests for lane priority ordering.
- Tests proving asset jobs do not overlap with higher-priority memcard jobs
once both are submitted through the lane.
## Riscos
- Accidentally deciding FS public API in the wrong discussion.
- Making memcard async in a way that changes status-first behavior.
- Starving asset loads if priority rules do not include fairness.

126
discussion/workflow/plans/PLN-0128-async-asset-pipeline-contract-tests-and-telemetry.md
View File

@ -1,126 +0,0 @@
---
id: PLN-0128
ticket: perf-async-background-work-lanes-for-assets-and-fs
title: Async Asset Pipeline Contract Tests and Telemetry
status: done
created: 2026-06-28
completed:
ref_decisions: [DEC-0034]
tags: [perf, asset, async, telemetry, tests, runtime]
---
# PLN-0128 - Async Asset Pipeline Contract Tests and Telemetry
## Briefing
`DEC-0034` requires observable progress, backlog state, superseding, and cheap
telemetry. This plan adds the validation and telemetry evidence needed to prove
the new async asset pipeline is correct.
## Decisions de Origem
- `DEC-0034` - Async Work Lane and Asset Backlog Contract.
## Alvo
Provide contract tests and telemetry for the async asset lane so future
implementations can be changed without regressing lane semantics.
## Escopo
Included:
- Contract tests for lane and asset backlog behavior.
- Progress reporting tests.
- Superseding and stale result discard tests.
- Telemetry counters and percentile updates.
- No-sleep deterministic test helpers where needed.
Fora de Escopo:
- Implementing public API wiring if `PLN-0126` has not run.
- Full benchmarking infrastructure.
- Host UI display of telemetry.
## Plano de Execucao
### Step 1 - Build Deterministic Test Hooks
**What:** Add test hooks for blocking a lane job at read/decode/stage.
**How:** Use channels/barriers or explicit fake job steps. Do not rely on
`thread::sleep` for correctness.
**Files:** async lane tests; asset test support modules.
### Step 2 - Test Backlog Ordering
**What:** Prove queued jobs run serially and in backlog order.
**How:** Submit multiple target slots and assert active job, pending positions,
promote, demote, and move behavior.
**Files:** asset manager tests and lane tests.
### Step 3 - Test Superseding
**What:** Prove newer request for the same `bank_type/slot` supersedes older
requests.
**How:** Cover queued superseding and active superseding. For active work, prove
stale completion cannot install.
**Files:** asset manager tests.
### Step 4 - Test Stable Handle Observability
**What:** Prove handles are queryable for empty, queued, active, ready,
committed, canceled, superseded, and error states.
**How:** Use `target_status` and handle status APIs to inspect both `slot_state`
and `request_state`.
**Files:** asset manager tests; VM dispatch tests if public API is available.
### Step 5 - Test Progress
**What:** Prove progress uses `0..10000` and phase boundaries.
**How:** Fake read/decode/stage phases with known completion points. Validate
non-linear phases do not report false internal progress.
**Files:** asset manager tests; telemetry tests.
### Step 6 - Add Telemetry
**What:** Add counters and percentile aggregation.
**How:** Update submitted/completed/failed/canceled/superseded counters when job
state closes. Update percentiles by `bank_type` and lightweight `asset_id`
window only at job completion.
**Files:** asset manager telemetry, async lane telemetry, existing runtime
telemetry types if shared.
## Criterios de Aceite
- [ ] Tests prove one active job at a time.
- [ ] Tests prove backlog order and reordering.
- [ ] Tests prove queued and active superseding.
- [ ] Tests prove stale active results are discarded.
- [ ] Tests prove handles remain queryable across all required states.
- [ ] Tests prove progress phase behavior.
- [ ] Telemetry is updated at job closure, not inside inner decode loops.
## Tests / Validacao
- Run targeted cargo tests for asset manager, async lane, and VM runtime
dispatch.
- Run existing asset-bank tests to prove current behavior remains compatible.
- Run `discussion validate` after plan state changes.
## Riscos
- Tests that depend on sleeps will be flaky.
- Telemetry may become too expensive if updated inside decode loops.
- Public API tests must wait for `PLN-0126` if syscall wiring is not present.