fix App, clean up, add metadata convergence
This commit is contained in:
parent
ee6b1b54c9
commit
f34c5a7691
SSH Key Fingerprint:
SHA256:Z7dgqoglWwoK6j6u4QC87OveEq74WOhFN+gitsxtkf8
@ -1,557 +0,0 @@
|
||||
# Prometeu Packer (prometeu-packer) — Specification (Draft)
|
||||
|
||||
> **Status:** Draft / Community-facing
|
||||
>
|
||||
> This document specifies the **Prometeu Packer**, the tooling responsible for **asset management** and producing two build artifacts:
|
||||
>
|
||||
> * `build/assets.pa` — the ROM payload containing packed asset bytes
|
||||
> * `build/asset_table.json` — a machine-readable descriptor of the packed assets
|
||||
>
|
||||
> The Packer is deliberately **agnostic of cartridge shipping**. A separate **Cartridge Shipper** (outside the Packer) consumes `build/asset_table.json` to generate the final `cartridge/manifest.json`, copy `assets.pa` to the cartridge directory, and zip the cartridge into a distributable format.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals and Non-Goals
|
||||
|
||||
### 1.1 Goals
|
||||
|
||||
1. **Be the guardian of sanity** in a constantly mutating `assets/` workspace.
|
||||
|
||||
* Users may be disorganized.
|
||||
* The directory may contain WIP, junk, unused files, duplicates, outdated exports.
|
||||
* The Packer must help users identify and fix mistakes.
|
||||
|
||||
2. Provide a robust, deterministic, **tooling-grade** asset pipeline.
|
||||
|
||||
* Stable asset identity.
|
||||
* Deterministic packing order.
|
||||
* Reproducible output bytes.
|
||||
|
||||
3. Support both **raw (direct) assets** and **virtual assets**.
|
||||
|
||||
* Raw assets: the payload in ROM is exactly the source bytes.
|
||||
* Virtual assets: the payload is derived from multiple inputs via a build pipeline (e.g., PNG + palettes → `TILES` payload).
|
||||
|
||||
4. Produce an output descriptor (`build/asset_table.json`) suitable for:
|
||||
|
||||
* a Cartridge Shipper to generate the runtime manifest
|
||||
* CI checks
|
||||
* a future IDE / GUI tooling
|
||||
|
||||
5. Provide an extensive **diagnostics chain** (doctor) with structured error codes and suggested fixes.
|
||||
|
||||
### 1.2 Non-Goals
|
||||
|
||||
* The Packer **does not**:
|
||||
|
||||
* generate `cartridge/manifest.json`
|
||||
* decide preload slots
|
||||
* copy files into `cartridge/`
|
||||
* compile PBS bytecode
|
||||
* zip cartridges
|
||||
|
||||
These responsibilities belong to a separate **Cartridge Shipper**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Repository / Project Layout (Golden Pipeline)
|
||||
|
||||
The Prometeu project uses the following canonical structure:
|
||||
|
||||
* `src/` — PBS scripts
|
||||
* `assets/` — mutable asset workspace (WIP allowed)
|
||||
* `build/` — generated artifacts and caches
|
||||
* `cartridge/` — final cartridge directory (produced by Cartridge Shipper)
|
||||
* `prometeu.json` — project description (dependencies, version, etc.)
|
||||
* `sdk/` — SDK/tooling and libraries
|
||||
|
||||
The Packer owns the asset workspace metadata under:
|
||||
|
||||
* `assets/.prometeu/` — Packer control directory (registry, cache, quarantine)
|
||||
|
||||
---
|
||||
|
||||
## 3. Crate Topology
|
||||
|
||||
### 3.1 Crates
|
||||
|
||||
* **`prometeu-packer`**
|
||||
|
||||
* **lib**: `prometeu_packer_core`
|
||||
* **bin**: `prometeu-packer`
|
||||
|
||||
* a thin CLI wrapper delegating to `prometeu_packer_core::run()`
|
||||
|
||||
* **`prometeu` dispatcher**
|
||||
|
||||
* provides a wrapper command **`prometeup`** (or integrated subcommand)
|
||||
* delegates to `prometeu-packer` for packer operations
|
||||
|
||||
### 3.2 Design Principle
|
||||
|
||||
Treat the Packer like the compiler: core library + CLI wrapper.
|
||||
|
||||
* The core library enables future integrations (IDE, GUI, watch mode) without shelling out.
|
||||
* CLI is a stable interface for users and CI.
|
||||
|
||||
---
|
||||
|
||||
## 4. Mental Model: A “Git-like” Asset Workspace
|
||||
|
||||
The Packer treats `assets/` like a **dirty working tree**.
|
||||
|
||||
* `assets/` can contain *anything*.
|
||||
* Only the assets registered in the Packer registry are considered part of the build.
|
||||
|
||||
This is analogous to Git:
|
||||
|
||||
* working tree (chaos) vs index (truth)
|
||||
|
||||
The **source of truth** for “what counts” is the registry:
|
||||
|
||||
* `assets/.prometeu/index.json`
|
||||
|
||||
---
|
||||
|
||||
## 5. Core Concepts
|
||||
|
||||
### 5.1 Managed Asset
|
||||
|
||||
A **managed asset** is an entry in `assets/.prometeu/index.json` pointing to an **asset root directory** that contains an anchor file:
|
||||
|
||||
* `asset.json` (the asset specification)
|
||||
|
||||
Everything else is free-form.
|
||||
|
||||
### 5.2 Asset Identity
|
||||
|
||||
Each asset has stable identity:
|
||||
|
||||
* `asset_id: u32` — stable within the project (used by runtime/tooling)
|
||||
* `asset_uuid: string` — globally unique stable identifier (useful for IDE and future migrations)
|
||||
|
||||
Names and paths may change, but identity remains.
|
||||
|
||||
### 5.3 Asset Types (Bank Targets)
|
||||
|
||||
Assets ultimately target a **bank type** in the runtime:
|
||||
|
||||
* `TILES`
|
||||
* `SOUNDS`
|
||||
* (future) `SPRITESHEET`, `MAP`, `FONT`, `RAW_BLOB`, etc.
|
||||
|
||||
The Packer does **not** define bank memory semantics. It defines the *ROM payload* and its metadata.
|
||||
|
||||
### 5.4 Raw vs Virtual Assets
|
||||
|
||||
* **Raw assets**: ROM payload equals the source bytes.
|
||||
* **Virtual assets**: ROM payload is derived from input(s) via deterministic build steps.
|
||||
|
||||
Examples:
|
||||
|
||||
* PNG + palette files → `TILES` payload (indexed pixels + packed palettes)
|
||||
* WAV → PCM16LE payload
|
||||
* Multiple PNGs → atlas spritesheet
|
||||
|
||||
---
|
||||
|
||||
## 6. Directory Structure and Control Files
|
||||
|
||||
### 6.1 Workspace
|
||||
|
||||
`assets/` is a mutable workspace:
|
||||
|
||||
* users may create nested organization trees
|
||||
* junk files are allowed
|
||||
|
||||
### 6.2 Control Directory
|
||||
|
||||
The Packer stores its truth + tools state in:
|
||||
|
||||
```
|
||||
assets/
|
||||
.prometeu/
|
||||
index.json
|
||||
cache/
|
||||
fingerprints.json
|
||||
build-cache.json
|
||||
quarantine/
|
||||
...
|
||||
```
|
||||
|
||||
* `index.json` — registry of managed assets
|
||||
* `cache/` — fingerprints and incremental build cache
|
||||
* `quarantine/` — optional area to move detected junk (only by explicit user action)
|
||||
|
||||
---
|
||||
|
||||
## 7. Asset Specification: `asset.json`
|
||||
|
||||
`asset.json` describes:
|
||||
|
||||
1. **the output ROM payload** expected by runtime
|
||||
2. **the build pipeline** (for virtual assets)
|
||||
3. **metadata** needed by runtime/shipper
|
||||
|
||||
This spec is modular: **each asset format** (e.g. `TILES/indexed_v1`) has its own dedicated specification document.
|
||||
|
||||
### 7.1 Common Fields (All Assets)
|
||||
|
||||
* `schema_version`
|
||||
* `name`
|
||||
* `type` (bank type)
|
||||
* `codec` (e.g. `RAW`; future: compression)
|
||||
* `inputs` (for virtual assets)
|
||||
* `output` (format + required metadata)
|
||||
* `build` (optional pipeline configuration)
|
||||
|
||||
### 7.2 Virtual Asset Pipeline Declaration
|
||||
|
||||
Virtual assets must be declared in a way that is:
|
||||
|
||||
* deterministic
|
||||
* fully materialized (no silent inference)
|
||||
* explicit about defaults (defaults may exist, but must be written into `asset.json` or build outputs)
|
||||
|
||||
---
|
||||
|
||||
## 8. Build Artifacts Produced by the Packer
|
||||
|
||||
### 8.1 `build/assets.pa`
|
||||
|
||||
**`assets.pa`** is the ROM asset payload used by the runtime.
|
||||
|
||||
**Definition:** a contiguous binary blob where each managed asset contributes a payload region.
|
||||
|
||||
#### Key Properties
|
||||
|
||||
* Deterministic asset order (by `asset_id`)
|
||||
* Offsets are recorded in `build/asset_table.json`
|
||||
* Alignment rules (configurable by packer, default: no alignment unless required by a format)
|
||||
|
||||
**Note:** `assets.pa` is intentionally simple.
|
||||
|
||||
* No internal header is required.
|
||||
* The authoritative structure comes from the descriptor (`asset_table.json`).
|
||||
|
||||
Future versions may introduce chunk tables, but v1 keeps ROM simple.
|
||||
|
||||
### 8.2 `build/asset_table.json`
|
||||
|
||||
**`asset_table.json`** is the canonical descriptor output of the Packer.
|
||||
|
||||
It contains:
|
||||
|
||||
* `assets_pa` file info (size, hash)
|
||||
* `asset_table[]` entries describing each payload slice
|
||||
* optional diagnostics/warnings
|
||||
|
||||
#### Asset Table Entry
|
||||
|
||||
An entry describes a ROM slice and its runtime meaning:
|
||||
|
||||
* `asset_id` — stable u32
|
||||
* `asset_uuid` — stable UUID string
|
||||
* `asset_name` — stable user-facing name
|
||||
* `bank_type` — e.g. `TILES`, `SOUNDS`
|
||||
* `offset` — byte offset in `assets.pa`
|
||||
* `size` — bytes stored in ROM
|
||||
* `decoded_size` — bytes after decode (equal to `size` when `codec=RAW`)
|
||||
* `codec` — `RAW` (future: compression)
|
||||
* `metadata` — format-specific metadata needed by runtime/shipper
|
||||
|
||||
Additional tooling fields:
|
||||
|
||||
* `source_root` — path to asset dir
|
||||
* `inputs` — resolved input paths
|
||||
* `source_hashes` — stable fingerprints of inputs
|
||||
|
||||
`asset_table.json` is machine-readable and designed for:
|
||||
|
||||
* cartridge shipper consumption
|
||||
* IDE visualization
|
||||
* debugging
|
||||
|
||||
---
|
||||
|
||||
## 9. Determinism Rules
|
||||
|
||||
1. Asset packing order MUST be deterministic.
|
||||
|
||||
* Default: increasing `asset_id`
|
||||
|
||||
2. All derived outputs MUST be deterministic.
|
||||
|
||||
* No random seeds unless explicitly declared
|
||||
* Any seed must be written to output metadata
|
||||
|
||||
3. Default values MUST be materialized.
|
||||
|
||||
* If the packer infers something, it must be written into `asset.json` (via `--fix`) or recorded in build outputs.
|
||||
|
||||
---
|
||||
|
||||
## 10. Diagnostics and the “Sanity Guardian” Chain
|
||||
|
||||
The Packer provides structured diagnostics:
|
||||
|
||||
* `code` — stable diagnostic code
|
||||
* `severity` — `error | warning | info`
|
||||
* `path` — affected file
|
||||
* `message` — human friendly
|
||||
* `help` — extended context
|
||||
* `fixes[]` — suggested automated or manual fixes
|
||||
|
||||
### 10.1 Diagnostic Classes
|
||||
|
||||
1. **Registered Errors** (break build)
|
||||
|
||||
* registry entry missing anchor file
|
||||
* `asset.json` invalid
|
||||
* missing inputs
|
||||
* format/metadata mismatch
|
||||
|
||||
2. **Workspace Warnings** (does not break build)
|
||||
|
||||
* orphaned `asset.json` (not registered)
|
||||
* unused large files
|
||||
* duplicate inputs by hash
|
||||
|
||||
3. **Policy Hints** (optional)
|
||||
|
||||
* naming conventions
|
||||
* missing preview
|
||||
|
||||
### 10.2 `doctor` Modes
|
||||
|
||||
* `doctor` (default) — validate registry only (fast)
|
||||
* `doctor --workspace` — deep scan workspace (slow)
|
||||
|
||||
---
|
||||
|
||||
## 11. Incremental Build, Cache, and Fingerprints
|
||||
|
||||
The Packer maintains fingerprints of inputs:
|
||||
|
||||
* size
|
||||
* mtime
|
||||
* strong hash (sha256)
|
||||
|
||||
Stored in:
|
||||
|
||||
* `assets/.prometeu/cache/fingerprints.json`
|
||||
|
||||
This enables:
|
||||
|
||||
* detecting changes
|
||||
* rebuild only what changed
|
||||
* producing stable reports
|
||||
|
||||
The cache must never compromise determinism.
|
||||
|
||||
---
|
||||
|
||||
## 12. Quarantine and Garbage Collection
|
||||
|
||||
### 12.1 Quarantine
|
||||
|
||||
The Packer can optionally move suspected junk to:
|
||||
|
||||
* `assets/.prometeu/quarantine/`
|
||||
|
||||
Rules:
|
||||
|
||||
* Quarantine is **never automatic** without user consent.
|
||||
* Packer must explain exactly what will be moved.
|
||||
|
||||
### 12.2 Garbage Collection (`gc`)
|
||||
|
||||
The Packer can report unused files:
|
||||
|
||||
* files not referenced by any registered asset
|
||||
* orphaned asset dirs
|
||||
|
||||
Actions:
|
||||
|
||||
* list candidates
|
||||
* optionally move to quarantine
|
||||
* never delete without explicit user request
|
||||
|
||||
---
|
||||
|
||||
## 13. CLI Commands (Comprehensive)
|
||||
|
||||
> The CLI is a stable interface; all commands are implemented by calling `prometeu_packer_core`.
|
||||
|
||||
### 13.1 `prometeu packer init`
|
||||
|
||||
Creates the control directory and initial registry:
|
||||
|
||||
* creates `assets/.prometeu/index.json`
|
||||
* creates caches directory
|
||||
|
||||
### 13.2 `prometeu packer add <path> [--name <name>] [--type <TILES|SOUNDS|...>]`
|
||||
|
||||
Registers a new managed asset.
|
||||
|
||||
* does not require moving files
|
||||
* can create an asset root directory if desired
|
||||
* generates `asset.json` with explicit defaults
|
||||
* allocates `asset_id` and `asset_uuid`
|
||||
|
||||
Variants:
|
||||
|
||||
* `add --dir` creates a dedicated asset root dir
|
||||
* `add --in-place` anchors next to the file
|
||||
|
||||
### 13.3 `prometeu packer forget <name|id|uuid>`
|
||||
|
||||
Removes an asset from the registry without deleting files.
|
||||
|
||||
Useful for WIP and cleanup.
|
||||
|
||||
### 13.4 `prometeu packer rm <name|id|uuid> [--delete]`
|
||||
|
||||
Removes the asset from the registry.
|
||||
|
||||
* default: no deletion
|
||||
* `--delete` can remove the asset root dir (dangerous; must confirm in UI tooling, or require a force flag in CLI)
|
||||
|
||||
### 13.5 `prometeu packer list`
|
||||
|
||||
Lists managed assets:
|
||||
|
||||
* id, uuid, name
|
||||
* type
|
||||
* status (ok/error)
|
||||
|
||||
### 13.7 `prometeu packer show <name|id|uuid>`
|
||||
|
||||
Shows detailed information:
|
||||
|
||||
* resolved inputs
|
||||
* metadata
|
||||
* fingerprints
|
||||
* last build summary
|
||||
|
||||
### 13.8 `prometeu packer doctor [--workspace] [--strict] [--fix]`
|
||||
|
||||
Runs diagnostics:
|
||||
|
||||
* `--workspace` deep scan
|
||||
* `--strict` warnings become errors
|
||||
* `--fix` applies safe automatic fixes (materialize defaults, normalize paths)
|
||||
|
||||
### 13.9 `prometeu packer build [--out build/assets.pa] [--table build/asset_table.json]`
|
||||
|
||||
Builds:
|
||||
|
||||
* `build/assets.pa`
|
||||
* `build/asset_table.json`
|
||||
|
||||
Key behaviors:
|
||||
|
||||
* validates registry before packing
|
||||
* packs assets deterministically
|
||||
* for virtual assets, runs build pipelines
|
||||
* records all offsets and metadata
|
||||
|
||||
### 13.10 `prometeu packer watch`
|
||||
|
||||
Watches registered inputs and registry changes.
|
||||
|
||||
* emits events (future)
|
||||
* rebuilds incrementally
|
||||
|
||||
`watch` is optional in v0 but recommended.
|
||||
|
||||
### 13.11 `prometeu packer gc [--workspace] [--quarantine]`
|
||||
|
||||
Reports unused files.
|
||||
|
||||
* default: report only
|
||||
* `--quarantine` moves candidates to quarantine
|
||||
|
||||
### 13.12 `prometeu packer quarantine <path> [--restore]`
|
||||
|
||||
Moves or restores files into/from quarantine.
|
||||
|
||||
---
|
||||
|
||||
## 14. Virtual Assets (Deep Explanation)
|
||||
|
||||
Virtual assets are a major capability.
|
||||
|
||||
### 14.1 Why Virtual Assets
|
||||
|
||||
* Most runtime formats should be derived from human-friendly authoring formats.
|
||||
* Example:
|
||||
|
||||
* author uses `source.png` and palette files
|
||||
* runtime expects indexed pixels + packed RGB565 palettes
|
||||
|
||||
### 14.2 Virtual Asset Contract
|
||||
|
||||
* Inputs are explicit.
|
||||
* Build steps are deterministic.
|
||||
* Outputs match a well-defined runtime payload format.
|
||||
|
||||
### 14.3 Examples of Future Virtual Assets
|
||||
|
||||
* `TILES/indexed_v1`: PNG + palette files → indexed pixels + packed palettes
|
||||
* `SOUNDS/pcm16le_v1`: WAV → PCM16LE
|
||||
* `SPRITESHEET/atlas_v1`: multiple PNG frames → atlas + metadata
|
||||
|
||||
Each `output.format` must have its own dedicated spec.
|
||||
|
||||
---
|
||||
|
||||
## 15. Integration with Cartridge Shipper
|
||||
|
||||
The Cartridge Shipper should:
|
||||
|
||||
1. Compile PBS into bytecode (e.g. `program.pbc` / `program.pbx`)
|
||||
2. Call `prometeu packer build`
|
||||
3. Consume `build/asset_table.json` and produce `cartridge/manifest.json`
|
||||
4. Copy artifacts into `cartridge/`
|
||||
5. Zip the cartridge into a distributable format (`.crt` / `.rom` / `.pro`)
|
||||
|
||||
The packer never touches `cartridge/`.
|
||||
|
||||
---
|
||||
|
||||
## 16. Compatibility and Versioning
|
||||
|
||||
* `assets/.prometeu/index.json` has `schema_version`
|
||||
* `asset.json` has `schema_version`
|
||||
* `build/asset_table.json` has `schema_version`
|
||||
|
||||
The Packer must be able to migrate older schema versions or emit actionable diagnostics.
|
||||
|
||||
---
|
||||
|
||||
## 17. Security and Trust Model
|
||||
|
||||
* The Packer is offline tooling.
|
||||
* It must never execute untrusted scripts.
|
||||
* It should treat external inputs as untrusted data.
|
||||
|
||||
---
|
||||
|
||||
## 18. Implementation Notes (Non-Normative)
|
||||
|
||||
* Rust implementation with a core crate + CLI wrapper.
|
||||
* Prefer structured JSON serde models.
|
||||
* Use stable diagnostic codes.
|
||||
* Keep the build deterministic.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — Glossary
|
||||
|
||||
* **ROM (`assets.pa`)**: packed asset payload used by runtime
|
||||
* **Descriptor (`asset_table.json`)**: mapping from logical assets to ROM slices
|
||||
* **Managed asset**: registered asset with stable identity and anchor file
|
||||
* **Virtual asset**: derived asset built from multiple inputs
|
||||
* **Quarantine**: safe area for suspected junk
|
||||
* **Doctor**: diagnostic command to keep sanity
|
||||
@ -15,24 +15,11 @@ docs/packer/
|
||||
├── learn/
|
||||
├── pull-requests/
|
||||
├── specs/
|
||||
├── Prometeu Packer.md
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Responsibilities
|
||||
|
||||
### `Prometeu Packer.md`
|
||||
|
||||
`Prometeu Packer.md` is the current bootstrap specification for the domain.
|
||||
|
||||
Use it to:
|
||||
|
||||
- understand the current packer scope,
|
||||
- recover the initial architecture and terminology,
|
||||
- seed the split into smaller normative specs under `specs/`.
|
||||
|
||||
This document is useful as a draft source, but it should not become a permanent dumping ground for every future change.
|
||||
|
||||
### `agendas/`
|
||||
|
||||
`agendas/` is a transient staging area for open packer topics.
|
||||
|
||||
@ -0,0 +1,146 @@
|
||||
# Metadata Convergence to AssetEntry.metadata Decision
|
||||
|
||||
Status: Accepted
|
||||
Date: 2026-03-17
|
||||
Domain Owner: `docs/packer`
|
||||
Cross-Domain Impact: `docs/vm-arch`, runtime asset consumers
|
||||
|
||||
## Context
|
||||
|
||||
The packer asset contract currently has multiple metadata-producing sources with different responsibilities:
|
||||
|
||||
- asset-level runtime contract metadata (authoring declaration);
|
||||
- codec-related metadata (codec configuration/effective codec parameters);
|
||||
- pipeline-derived metadata generated during build materialization (for example, indexed ranges for packed samples).
|
||||
|
||||
At runtime, consumers read one metadata sink from the asset table: `AssetEntry.metadata`.
|
||||
|
||||
Without an explicit decision, the system risks inconsistent behavior and documentation drift:
|
||||
|
||||
- metadata spread across sources without deterministic merge semantics;
|
||||
- ambiguity between storage-layout fields (`AssetEntry.offset`/`AssetEntry.size`) and pipeline-internal indexing data (`offset`/`length` per sample);
|
||||
- Studio, packer, and runtime docs diverging on where runtime consumers should read final values.
|
||||
|
||||
## Decision
|
||||
|
||||
The following direction is adopted:
|
||||
|
||||
1. All runtime-consumable metadata must converge to a single sink: `AssetEntry.metadata`.
|
||||
2. Source segmentation in `asset.json` is allowed for authoring clarity, but build materialization must normalize these sources into that single sink.
|
||||
3. Metadata normalization must be deterministic and testable.
|
||||
4. `AssetEntry.offset` and `AssetEntry.size` remain payload slicing fields and are not reinterpreted as pipeline-internal indexing metadata.
|
||||
5. Pipeline indexing metadata (for example, audio per-`sample_id` ranges) must live inside `AssetEntry.metadata` under explicit keys.
|
||||
|
||||
## Adopted Constraints
|
||||
|
||||
### 1. Source Segmentation vs Runtime Sink
|
||||
|
||||
- authoring sources may remain segmented (asset metadata, codec metadata, pipeline metadata);
|
||||
- runtime consumers must read effective values from `AssetEntry.metadata`;
|
||||
- packer build output is responsible for normalization.
|
||||
|
||||
### 2. Deterministic Convergence
|
||||
|
||||
- normalization must produce the same `AssetEntry.metadata` for the same effective declaration and build inputs;
|
||||
- metadata key collisions between independent sources must be rejected with a build-time error unless explicitly specified by family/format contract;
|
||||
- normalization order and collision policy must be documented by packer specs.
|
||||
|
||||
### 3. Audio Indexing Semantics
|
||||
|
||||
For multi-sample audio banks, sample indexing metadata belongs to `AssetEntry.metadata`, keyed by `sample_id`.
|
||||
|
||||
Illustrative shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"sample_rate": 22050,
|
||||
"channels": 1,
|
||||
"samples": {
|
||||
"1": { "offset": 0, "length": 100 }
|
||||
},
|
||||
"codec": {
|
||||
"parity": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This decision accepts either nested codec metadata (for example `metadata.codec.*`) or a flat equivalent only when the family/format spec declares that shape explicitly.
|
||||
|
||||
### 4. Offset Ambiguity Guardrail
|
||||
|
||||
- `AssetEntry.offset`/`AssetEntry.size` describe where one packed asset payload is stored in `assets.pa`;
|
||||
- `metadata.samples[*].offset`/`metadata.samples[*].length` describe internal layout/indexing inside that asset's runtime payload contract;
|
||||
- documentation and tests must keep these meanings separate.
|
||||
|
||||
## Why This Direction Was Chosen
|
||||
|
||||
- It keeps runtime consumption simple: one metadata sink.
|
||||
- It preserves authoring ergonomics: source metadata can stay segmented by concern.
|
||||
- It avoids semantic duplication between packer and runtime consumers.
|
||||
- It creates a clear path for bank-like assets (tiles/audio) that require indexed internal metadata.
|
||||
|
||||
## Explicit Non-Decisions
|
||||
|
||||
This decision does not define:
|
||||
|
||||
- the final complete metadata schema for every asset family;
|
||||
- the final canonical codec metadata shape (`nested` vs `flat`) for all formats;
|
||||
- multi-sample audio runtime loading implementation details;
|
||||
- exact binary container/header layout for audio banks.
|
||||
|
||||
## Implications
|
||||
|
||||
- packer specs must define normalization semantics and collision policy;
|
||||
- packer build/materialization must emit normalized metadata into `AssetEntry.metadata`;
|
||||
- runtime-facing docs must state that effective metadata is read from `AssetEntry.metadata`;
|
||||
- tests must cover convergence correctness and ambiguity boundaries for offset semantics.
|
||||
|
||||
## Propagation Targets
|
||||
|
||||
Specs:
|
||||
|
||||
- [`../specs/3. Asset Declaration and Virtual Asset Contract Specification.md`](../specs/3.%20Asset%20Declaration%20and%20Virtual%20Asset%20Contract%20Specification.md)
|
||||
- [`../specs/4. Build Artifacts and Deterministic Packing Specification.md`](../specs/4.%20Build%20Artifacts%20and%20Deterministic%20Packing%20Specification.md)
|
||||
- [`../../vm-arch/ARCHITECTURE.md`](../../vm-arch/ARCHITECTURE.md)
|
||||
|
||||
Plans:
|
||||
|
||||
- next packer PR/plan that introduces metadata normalization and multi-source merge validation
|
||||
|
||||
Code:
|
||||
|
||||
- packer asset declaration/materialization pipeline
|
||||
- asset table emission path (`AssetEntry.metadata` payload)
|
||||
|
||||
Tests:
|
||||
|
||||
- normalization unit tests (source merge determinism)
|
||||
- collision/ambiguity tests for offset semantics
|
||||
- regression tests for runtime-readable metadata shape
|
||||
|
||||
Docs:
|
||||
|
||||
- packer specs and learn artifacts covering metadata source-to-sink flow
|
||||
- runtime asset-management documentation referencing `AssetEntry.metadata` as sink
|
||||
|
||||
## References
|
||||
|
||||
Related specs:
|
||||
|
||||
- [`../specs/3. Asset Declaration and Virtual Asset Contract Specification.md`](../specs/3.%20Asset%20Declaration%20and%20Virtual%20Asset%20Contract%20Specification.md)
|
||||
- [`../specs/4. Build Artifacts and Deterministic Packing Specification.md`](../specs/4.%20Build%20Artifacts%20and%20Deterministic%20Packing%20Specification.md)
|
||||
|
||||
Related agendas:
|
||||
|
||||
- no formal agenda artifact yet for this specific topic; this decision consolidates current packer/runtime alignment discussion.
|
||||
|
||||
## Validation Notes
|
||||
|
||||
This decision is correctly implemented only when all of the following are true:
|
||||
|
||||
- runtime consumers can read final effective metadata exclusively from `AssetEntry.metadata`;
|
||||
- segmented metadata sources in authoring inputs converge deterministically during packing;
|
||||
- offset semantics remain unambiguous between asset-table payload slicing and pipeline-internal indexing;
|
||||
- documentation across packer and runtime-facing domains is consistent about this source-to-sink contract.
|
||||
@ -52,3 +52,4 @@ Start here:
|
||||
|
||||
1. [`mental-model-packer.md`](./mental-model-packer.md)
|
||||
2. [`mental-model-asset-identity-and-runtime-contract.md`](./mental-model-asset-identity-and-runtime-contract.md)
|
||||
3. [`mental-model-metadata-convergence-and-runtime-sink.md`](./mental-model-metadata-convergence-and-runtime-sink.md)
|
||||
|
||||
@ -0,0 +1,82 @@
|
||||
# Metadata Convergence and Runtime Sink (`AssetEntry.metadata`)
|
||||
|
||||
## Original Problem
|
||||
|
||||
Packer authoring contracts may carry metadata from different concerns:
|
||||
|
||||
- asset/runtime contract metadata;
|
||||
- codec-related metadata;
|
||||
- pipeline-derived metadata generated during materialization.
|
||||
|
||||
Without an explicit convergence rule, teams drift into ambiguous behavior:
|
||||
|
||||
- runtime readers are unsure where effective values must be read;
|
||||
- metadata merge behavior becomes accidental and non-deterministic;
|
||||
- payload slicing fields can be confused with internal indexing offsets.
|
||||
|
||||
## Consolidated Decision
|
||||
|
||||
The accepted direction is:
|
||||
|
||||
1. all runtime-consumable metadata converges to a single sink: `AssetEntry.metadata`;
|
||||
2. source segmentation in `asset.json` is allowed for authoring ergonomics;
|
||||
3. build/materialization must normalize sources deterministically;
|
||||
4. collisions require explicit contract rules or must fail build;
|
||||
5. `AssetEntry.offset`/`AssetEntry.size` keep payload slicing semantics and do not replace internal pipeline indexing.
|
||||
|
||||
Decision reference:
|
||||
|
||||
- [`../decisions/Metadata Convergence to AssetEntry.metadata Decision.md`](../decisions/Metadata%20Convergence%20to%20AssetEntry.metadata%20Decision.md)
|
||||
|
||||
## Final Model
|
||||
|
||||
Think in two layers:
|
||||
|
||||
- **Authoring layer**: source metadata may be segmented by concern;
|
||||
- **Runtime layer**: one effective metadata map in each emitted asset entry.
|
||||
|
||||
Practical rule for runtime consumers:
|
||||
|
||||
- if the value is runtime-consumable metadata, read it from `AssetEntry.metadata`.
|
||||
|
||||
Practical rule for packer implementation:
|
||||
|
||||
- normalize all relevant metadata sources into `AssetEntry.metadata` during build;
|
||||
- keep normalization deterministic and testable.
|
||||
|
||||
## Example (Audio Bank Indexing)
|
||||
|
||||
Illustrative normalized metadata shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"sample_rate": 22050,
|
||||
"channels": 1,
|
||||
"samples": {
|
||||
"1": { "offset": 0, "length": 100 }
|
||||
},
|
||||
"codec": {
|
||||
"parity": 10
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In this model:
|
||||
|
||||
- `samples[*].offset/length` are internal indexing values for the audio payload contract;
|
||||
- they are not the same thing as `AssetEntry.offset/size` in the asset table.
|
||||
|
||||
## Common Pitfalls and Anti-Patterns
|
||||
|
||||
- Treating segmented declaration metadata as multiple runtime sinks.
|
||||
- Allowing silent key overwrite when two metadata sources collide.
|
||||
- Mixing payload slicing semantics (`asset_table[].offset/size`) with internal indexing semantics (`metadata.samples[*].offset/length`).
|
||||
- Documenting convergence behavior only in code comments and not in normative specs.
|
||||
|
||||
## References
|
||||
|
||||
- Decision: [`../decisions/Metadata Convergence to AssetEntry.metadata Decision.md`](../decisions/Metadata%20Convergence%20to%20AssetEntry.metadata%20Decision.md)
|
||||
- Spec: [`../specs/3. Asset Declaration and Virtual Asset Contract Specification.md`](../specs/3.%20Asset%20Declaration%20and%20Virtual%20Asset%20Contract%20Specification.md)
|
||||
- Spec: [`../specs/4. Build Artifacts and Deterministic Packing Specification.md`](../specs/4.%20Build%20Artifacts%20and%20Deterministic%20Packing%20Specification.md)
|
||||
@ -94,6 +94,17 @@ Rules:
|
||||
- `output.metadata` carries runtime-relevant format-specific detail;
|
||||
- codec must remain explicit and must not be hidden inside format naming.
|
||||
|
||||
## Metadata Source Segmentation and Runtime Sink
|
||||
|
||||
`asset.json` may keep metadata segmented by concern during authoring.
|
||||
|
||||
Rules:
|
||||
|
||||
- declaration-time metadata may come from multiple sources under the asset contract (for example, format metadata, codec-related metadata, and build/pipeline-derived declarations);
|
||||
- this segmentation exists for authoring clarity and does not define multiple runtime sinks;
|
||||
- runtime consumers must read effective metadata from the runtime asset entry metadata sink (`AssetEntry.metadata`);
|
||||
- convergence/normalization behavior is normative in the build artifact specification.
|
||||
|
||||
## Build
|
||||
|
||||
`build` is optional and process-oriented.
|
||||
|
||||
@ -79,6 +79,17 @@ Rules:
|
||||
- no synthetic dense reindexing layer;
|
||||
- `asset_name` remains present as logical/API-facing metadata.
|
||||
|
||||
### Asset Entry Metadata Convergence
|
||||
|
||||
Each emitted asset entry has one runtime metadata sink: `AssetEntry.metadata`.
|
||||
|
||||
Rules:
|
||||
|
||||
- packer materialization must normalize all runtime-consumable metadata-producing sources into `asset_table[].metadata`;
|
||||
- equivalent declaration/build inputs must produce equivalent normalized metadata;
|
||||
- metadata key collisions across independent sources must fail build unless the family/format spec declares an explicit merge rule;
|
||||
- normalization behavior must be testable and covered by conformance-oriented tests.
|
||||
|
||||
### Preload
|
||||
|
||||
Preload is emitted deterministically from per-asset declaration.
|
||||
@ -114,6 +125,12 @@ Rules:
|
||||
- any required alignment must be normative and visible;
|
||||
- emitted offsets are always relative to the payload region, never the start of the full file.
|
||||
|
||||
Offset ambiguity guardrail:
|
||||
|
||||
- `asset_table[].offset`/`asset_table[].size` describe payload slicing inside `assets.pa`;
|
||||
- internal pipeline indexing data (for example per-sample ranges for audio banks) must live under `asset_table[].metadata`;
|
||||
- internal indexing fields must not be interpreted as payload slicing fields.
|
||||
|
||||
## Determinism
|
||||
|
||||
Equivalent build inputs must produce equivalent outputs.
|
||||
|
||||
@ -8,8 +8,7 @@ Specs define the official packer contract.
|
||||
|
||||
They exist to make behavior, constraints, interfaces, and artifact guarantees stable across implementation and review.
|
||||
|
||||
The current domain also has a bootstrap draft in [`../Prometeu Packer.md`](../Prometeu%20Packer.md).
|
||||
That draft can be used as source material, but long-term normative content should be decomposed into focused specs here.
|
||||
The packer normative content is maintained directly in this `specs/` corpus.
|
||||
|
||||
## Expected Format
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@ public final class AppContainer implements Container {
|
||||
this.mapper = new ObjectMapper();
|
||||
final ExecutorService backgroundExecutor = Executors.newFixedThreadPool(2, new StudioWorkerThreadFactory());
|
||||
this.backgroundTasks = new StudioBackgroundTasks(backgroundExecutor);
|
||||
final Packer packer = Packer.bootstrap(Container.mapper(), new StudioPackerEventAdapter(studioEventBus));
|
||||
final Packer packer = Packer.bootstrap(this.mapper, new StudioPackerEventAdapter(studioEventBus));
|
||||
this.embeddedPacker = new EmbeddedPacker(packer.workspaceService(), packer::close);
|
||||
}
|
||||
|
||||
|
||||
@ -698,6 +698,56 @@
|
||||
"message" : "Asset scan started",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "8 assets loaded",
|
||||
"severity" : "SUCCESS",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: test",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: bla",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: one-more-atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: ui_atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: one-more-atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: bbb2",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: ui_atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: Bigode",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Asset scan started",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Asset moved: bbb2 -> recovered/bbb2",
|
||||
@ -2448,54 +2498,4 @@
|
||||
"message" : "Discovered asset: one-more-atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: ui_atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: Bigode",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Asset scan started",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "7 assets loaded",
|
||||
"severity" : "SUCCESS",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: test",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: one-more-atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: ui_atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: bla",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: one-more-atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
}, {
|
||||
"source" : "Assets",
|
||||
"message" : "Discovered asset: ui_atlas",
|
||||
"severity" : "INFO",
|
||||
"sticky" : false
|
||||
} ]
|
||||
Loading…
x
Reference in New Issue
Block a user