565 lines
14 KiB
Markdown
565 lines
14 KiB
Markdown
# Prometeu Packer (prometeu-packer) — Specification (Draft)
|
|
|
|
> **Status:** Draft / Community-facing
|
|
>
|
|
> This document specifies the **Prometeu Packer**, the tooling responsible for **asset management** and producing two build artifacts:
|
|
>
|
|
> * `build/assets.pa` — the ROM payload containing packed asset bytes
|
|
> * `build/asset_table.json` — a machine-readable descriptor of the packed assets
|
|
>
|
|
> The Packer is deliberately **agnostic of cartridge building**. A separate **Cartridge Builder** (outside the Packer) consumes `build/asset_table.json` to generate the final `cartridge/manifest.json`, copy `assets.pa` to the cartridge directory, and zip the cartridge into a distributable format.
|
|
|
|
---
|
|
|
|
## 1. Goals and Non-Goals
|
|
|
|
### 1.1 Goals
|
|
|
|
1. **Be the guardian of sanity** in a constantly mutating `assets/` workspace.
|
|
|
|
* Users may be disorganized.
|
|
* The directory may contain WIP, junk, unused files, duplicates, outdated exports.
|
|
* The Packer must help users identify and fix mistakes.
|
|
|
|
2. Provide a robust, deterministic, **tooling-grade** asset pipeline.
|
|
|
|
* Stable asset identity.
|
|
* Deterministic packing order.
|
|
* Reproducible output bytes.
|
|
|
|
3. Support both **raw (direct) assets** and **virtual assets**.
|
|
|
|
* Raw assets: the payload in ROM is exactly the source bytes.
|
|
* Virtual assets: the payload is derived from multiple inputs via a build pipeline (e.g., PNG + palettes → `TILES` payload).
|
|
|
|
4. Produce an output descriptor (`build/asset_table.json`) suitable for:
|
|
|
|
* a Cartridge Builder to generate the runtime manifest
|
|
* CI checks
|
|
* a future IDE / GUI tooling
|
|
|
|
5. Provide an extensive **diagnostics chain** (doctor) with structured error codes and suggested fixes.
|
|
|
|
### 1.2 Non-Goals
|
|
|
|
* The Packer **does not**:
|
|
|
|
* generate `cartridge/manifest.json`
|
|
* decide preload slots
|
|
* copy files into `cartridge/`
|
|
* compile PBS bytecode
|
|
* zip cartridges
|
|
|
|
These responsibilities belong to a separate **Cartridge Builder**.
|
|
|
|
---
|
|
|
|
## 2. Repository / Project Layout (Golden Pipeline)
|
|
|
|
The Prometeu project uses the following canonical structure:
|
|
|
|
* `src/` — PBS scripts
|
|
* `assets/` — mutable asset workspace (WIP allowed)
|
|
* `build/` — generated artifacts and caches
|
|
* `cartridge/` — final cartridge directory (produced by Cartridge Builder)
|
|
* `prometeu.json` — project description (dependencies, version, etc.)
|
|
* `sdk/` — SDK/tooling and libraries
|
|
|
|
The Packer owns the asset workspace metadata under:
|
|
|
|
* `assets/.prometeu/` — Packer control directory (registry, cache, quarantine)
|
|
|
|
---
|
|
|
|
## 3. Crate Topology
|
|
|
|
### 3.1 Crates
|
|
|
|
* **`prometeu-packer`**
|
|
|
|
* **lib**: `prometeu_packer_core`
|
|
* **bin**: `prometeu-packer`
|
|
|
|
* a thin CLI wrapper delegating to `prometeu_packer_core::run()`
|
|
|
|
* **`prometeu` dispatcher**
|
|
|
|
* provides a wrapper command **`prometeup`** (or integrated subcommand)
|
|
* delegates to `prometeu-packer` for packer operations
|
|
|
|
### 3.2 Design Principle
|
|
|
|
Treat the Packer like the compiler: core library + CLI wrapper.
|
|
|
|
* The core library enables future integrations (IDE, GUI, watch mode) without shelling out.
|
|
* CLI is a stable interface for users and CI.
|
|
|
|
---
|
|
|
|
## 4. Mental Model: A “Git-like” Asset Workspace
|
|
|
|
The Packer treats `assets/` like a **dirty working tree**.
|
|
|
|
* `assets/` can contain *anything*.
|
|
* Only the assets registered in the Packer registry are considered part of the build.
|
|
|
|
This is analogous to Git:
|
|
|
|
* working tree (chaos) vs index (truth)
|
|
|
|
The **source of truth** for “what counts” is the registry:
|
|
|
|
* `assets/.prometeu/index.json`
|
|
|
|
---
|
|
|
|
## 5. Core Concepts
|
|
|
|
### 5.1 Managed Asset
|
|
|
|
A **managed asset** is an entry in `assets/.prometeu/index.json` pointing to an **asset root directory** that contains an anchor file:
|
|
|
|
* `asset.json` (the asset specification)
|
|
|
|
Everything else is free-form.
|
|
|
|
### 5.2 Asset Identity
|
|
|
|
Each asset has stable identity:
|
|
|
|
* `asset_id: u32` — stable within the project (used by runtime/tooling)
|
|
* `asset_uuid: string` — globally unique stable identifier (useful for IDE and future migrations)
|
|
|
|
Names and paths may change, but identity remains.
|
|
|
|
### 5.3 Asset Types (Bank Targets)
|
|
|
|
Assets ultimately target a **bank type** in the runtime:
|
|
|
|
* `TILES`
|
|
* `SOUNDS`
|
|
* (future) `SPRITESHEET`, `MAP`, `FONT`, `RAW_BLOB`, etc.
|
|
|
|
The Packer does **not** define bank memory semantics. It defines the *ROM payload* and its metadata.
|
|
|
|
### 5.4 Raw vs Virtual Assets
|
|
|
|
* **Raw assets**: ROM payload equals the source bytes.
|
|
* **Virtual assets**: ROM payload is derived from input(s) via deterministic build steps.
|
|
|
|
Examples:
|
|
|
|
* PNG + palette files → `TILES` payload (indexed pixels + packed palettes)
|
|
* WAV → PCM16LE payload
|
|
* Multiple PNGs → atlas spritesheet
|
|
|
|
---
|
|
|
|
## 6. Directory Structure and Control Files
|
|
|
|
### 6.1 Workspace
|
|
|
|
`assets/` is a mutable workspace:
|
|
|
|
* users may create nested organization trees
|
|
* junk files are allowed
|
|
|
|
### 6.2 Control Directory
|
|
|
|
The Packer stores its truth + tools state in:
|
|
|
|
```
|
|
assets/
|
|
.prometeu/
|
|
index.json
|
|
cache/
|
|
fingerprints.json
|
|
build-cache.json
|
|
quarantine/
|
|
...
|
|
```
|
|
|
|
* `index.json` — registry of managed assets
|
|
* `cache/` — fingerprints and incremental build cache
|
|
* `quarantine/` — optional area to move detected junk (only by explicit user action)
|
|
|
|
---
|
|
|
|
## 7. Asset Specification: `asset.json`
|
|
|
|
`asset.json` describes:
|
|
|
|
1. **the output ROM payload** expected by runtime
|
|
2. **the build pipeline** (for virtual assets)
|
|
3. **metadata** needed by runtime/builder
|
|
|
|
This spec is modular: **each asset format** (e.g. `TILES/indexed_v1`) has its own dedicated specification document.
|
|
|
|
### 7.1 Common Fields (All Assets)
|
|
|
|
* `schema_version`
|
|
* `name`
|
|
* `type` (bank type)
|
|
* `codec` (e.g. `RAW`; future: compression)
|
|
* `inputs` (for virtual assets)
|
|
* `output` (format + required metadata)
|
|
* `build` (optional pipeline configuration)
|
|
|
|
### 7.2 Virtual Asset Pipeline Declaration
|
|
|
|
Virtual assets must be declared in a way that is:
|
|
|
|
* deterministic
|
|
* fully materialized (no silent inference)
|
|
* explicit about defaults (defaults may exist, but must be written into `asset.json` or build outputs)
|
|
|
|
---
|
|
|
|
## 8. Build Artifacts Produced by the Packer
|
|
|
|
### 8.1 `build/assets.pa`
|
|
|
|
**`assets.pa`** is the ROM asset payload used by the runtime.
|
|
|
|
**Definition:** a contiguous binary blob where each managed asset contributes a payload region.
|
|
|
|
#### Key Properties
|
|
|
|
* Deterministic asset order (by `asset_id`)
|
|
* Offsets are recorded in `build/asset_table.json`
|
|
* Alignment rules (configurable by packer, default: no alignment unless required by a format)
|
|
|
|
**Note:** `assets.pa` is intentionally simple.
|
|
|
|
* No internal header is required.
|
|
* The authoritative structure comes from the descriptor (`asset_table.json`).
|
|
|
|
Future versions may introduce chunk tables, but v1 keeps ROM simple.
|
|
|
|
### 8.2 `build/asset_table.json`
|
|
|
|
**`asset_table.json`** is the canonical descriptor output of the Packer.
|
|
|
|
It contains:
|
|
|
|
* `assets_pa` file info (size, hash)
|
|
* `asset_table[]` entries describing each payload slice
|
|
* optional diagnostics/warnings
|
|
|
|
#### Asset Table Entry
|
|
|
|
An entry describes a ROM slice and its runtime meaning:
|
|
|
|
* `asset_id` — stable u32
|
|
* `asset_uuid` — stable UUID string
|
|
* `asset_name` — stable user-facing name
|
|
* `bank_type` — e.g. `TILES`, `SOUNDS`
|
|
* `offset` — byte offset in `assets.pa`
|
|
* `size` — bytes stored in ROM
|
|
* `decoded_size` — bytes after decode (equal to `size` when `codec=RAW`)
|
|
* `codec` — `RAW` (future: compression)
|
|
* `metadata` — format-specific metadata needed by runtime/builder
|
|
|
|
Additional tooling fields:
|
|
|
|
* `source_root` — path to asset dir
|
|
* `inputs` — resolved input paths
|
|
* `source_hashes` — stable fingerprints of inputs
|
|
|
|
`asset_table.json` is machine-readable and designed for:
|
|
|
|
* cartridge builder consumption
|
|
* IDE visualization
|
|
* debugging
|
|
|
|
---
|
|
|
|
## 9. Determinism Rules
|
|
|
|
1. Asset packing order MUST be deterministic.
|
|
|
|
* Default: increasing `asset_id`
|
|
|
|
2. All derived outputs MUST be deterministic.
|
|
|
|
* No random seeds unless explicitly declared
|
|
* Any seed must be written to output metadata
|
|
|
|
3. Default values MUST be materialized.
|
|
|
|
* If the packer infers something, it must be written into `asset.json` (via `--fix`) or recorded in build outputs.
|
|
|
|
---
|
|
|
|
## 10. Diagnostics and the “Sanity Guardian” Chain
|
|
|
|
The Packer provides structured diagnostics:
|
|
|
|
* `code` — stable diagnostic code
|
|
* `severity` — `error | warning | info`
|
|
* `path` — affected file
|
|
* `message` — human friendly
|
|
* `help` — extended context
|
|
* `fixes[]` — suggested automated or manual fixes
|
|
|
|
### 10.1 Diagnostic Classes
|
|
|
|
1. **Registered Errors** (break build)
|
|
|
|
* registry entry missing anchor file
|
|
* `asset.json` invalid
|
|
* missing inputs
|
|
* format/metadata mismatch
|
|
|
|
2. **Workspace Warnings** (does not break build)
|
|
|
|
* orphaned `asset.json` (not registered)
|
|
* unused large files
|
|
* duplicate inputs by hash
|
|
|
|
3. **Policy Hints** (optional)
|
|
|
|
* naming conventions
|
|
* missing preview
|
|
|
|
### 10.2 `doctor` Modes
|
|
|
|
* `doctor` (default) — validate registry only (fast)
|
|
* `doctor --workspace` — deep scan workspace (slow)
|
|
|
|
---
|
|
|
|
## 11. Incremental Build, Cache, and Fingerprints
|
|
|
|
The Packer maintains fingerprints of inputs:
|
|
|
|
* size
|
|
* mtime
|
|
* strong hash (sha256)
|
|
|
|
Stored in:
|
|
|
|
* `assets/.prometeu/cache/fingerprints.json`
|
|
|
|
This enables:
|
|
|
|
* detecting changes
|
|
* rebuild only what changed
|
|
* producing stable reports
|
|
|
|
The cache must never compromise determinism.
|
|
|
|
---
|
|
|
|
## 12. Quarantine and Garbage Collection
|
|
|
|
### 12.1 Quarantine
|
|
|
|
The Packer can optionally move suspected junk to:
|
|
|
|
* `assets/.prometeu/quarantine/`
|
|
|
|
Rules:
|
|
|
|
* Quarantine is **never automatic** without user consent.
|
|
* Packer must explain exactly what will be moved.
|
|
|
|
### 12.2 Garbage Collection (`gc`)
|
|
|
|
The Packer can report unused files:
|
|
|
|
* files not referenced by any registered asset
|
|
* orphaned asset dirs
|
|
|
|
Actions:
|
|
|
|
* list candidates
|
|
* optionally move to quarantine
|
|
* never delete without explicit user request
|
|
|
|
---
|
|
|
|
## 13. CLI Commands (Comprehensive)
|
|
|
|
> The CLI is a stable interface; all commands are implemented by calling `prometeu_packer_core`.
|
|
|
|
### 13.1 `prometeu packer init`
|
|
|
|
Creates the control directory and initial registry:
|
|
|
|
* creates `assets/.prometeu/index.json`
|
|
* creates caches directory
|
|
|
|
### 13.2 `prometeu packer add <path> [--name <name>] [--type <TILES|SOUNDS|...>]`
|
|
|
|
Registers a new managed asset.
|
|
|
|
* does not require moving files
|
|
* can create an asset root directory if desired
|
|
* generates `asset.json` with explicit defaults
|
|
* allocates `asset_id` and `asset_uuid`
|
|
|
|
Variants:
|
|
|
|
* `add --dir` creates a dedicated asset root dir
|
|
* `add --in-place` anchors next to the file
|
|
|
|
### 13.3 `prometeu packer adopt`
|
|
|
|
Scans workspace for unregistered `asset.json` anchors and offers to register them.
|
|
|
|
* default: dry-run list
|
|
* `--apply` registers them
|
|
|
|
### 13.4 `prometeu packer forget <name|id|uuid>`
|
|
|
|
Removes an asset from the registry without deleting files.
|
|
|
|
Useful for WIP and cleanup.
|
|
|
|
### 13.5 `prometeu packer rm <name|id|uuid> [--delete]`
|
|
|
|
Removes the asset from the registry.
|
|
|
|
* default: no deletion
|
|
* `--delete` can remove the asset root dir (dangerous; must confirm in UI tooling, or require a force flag in CLI)
|
|
|
|
### 13.6 `prometeu packer list`
|
|
|
|
Lists managed assets:
|
|
|
|
* id, uuid, name
|
|
* type
|
|
* status (ok/error)
|
|
|
|
### 13.7 `prometeu packer show <name|id|uuid>`
|
|
|
|
Shows detailed information:
|
|
|
|
* resolved inputs
|
|
* metadata
|
|
* fingerprints
|
|
* last build summary
|
|
|
|
### 13.8 `prometeu packer doctor [--workspace] [--strict] [--fix]`
|
|
|
|
Runs diagnostics:
|
|
|
|
* `--workspace` deep scan
|
|
* `--strict` warnings become errors
|
|
* `--fix` applies safe automatic fixes (materialize defaults, normalize paths)
|
|
|
|
### 13.9 `prometeu packer build [--out build/assets.pa] [--table build/asset_table.json]`
|
|
|
|
Builds:
|
|
|
|
* `build/assets.pa`
|
|
* `build/asset_table.json`
|
|
|
|
Key behaviors:
|
|
|
|
* validates registry before packing
|
|
* packs assets deterministically
|
|
* for virtual assets, runs build pipelines
|
|
* records all offsets and metadata
|
|
|
|
### 13.10 `prometeu packer watch`
|
|
|
|
Watches registered inputs and registry changes.
|
|
|
|
* emits events (future)
|
|
* rebuilds incrementally
|
|
|
|
`watch` is optional in v0 but recommended.
|
|
|
|
### 13.11 `prometeu packer gc [--workspace] [--quarantine]`
|
|
|
|
Reports unused files.
|
|
|
|
* default: report only
|
|
* `--quarantine` moves candidates to quarantine
|
|
|
|
### 13.12 `prometeu packer quarantine <path> [--restore]`
|
|
|
|
Moves or restores files into/from quarantine.
|
|
|
|
---
|
|
|
|
## 14. Virtual Assets (Deep Explanation)
|
|
|
|
Virtual assets are a major capability.
|
|
|
|
### 14.1 Why Virtual Assets
|
|
|
|
* Most runtime formats should be derived from human-friendly authoring formats.
|
|
* Example:
|
|
|
|
* author uses `source.png` and palette files
|
|
* runtime expects indexed pixels + packed RGB565 palettes
|
|
|
|
### 14.2 Virtual Asset Contract
|
|
|
|
* Inputs are explicit.
|
|
* Build steps are deterministic.
|
|
* Outputs match a well-defined runtime payload format.
|
|
|
|
### 14.3 Examples of Future Virtual Assets
|
|
|
|
* `TILES/indexed_v1`: PNG + palette files → indexed pixels + packed palettes
|
|
* `SOUNDS/pcm16le_v1`: WAV → PCM16LE
|
|
* `SPRITESHEET/atlas_v1`: multiple PNG frames → atlas + metadata
|
|
|
|
Each `output.format` must have its own dedicated spec.
|
|
|
|
---
|
|
|
|
## 15. Integration with Cartridge Builder
|
|
|
|
The Cartridge Builder should:
|
|
|
|
1. Compile PBS into bytecode (e.g. `program.pbc` / `program.pbx`)
|
|
2. Call `prometeu packer build`
|
|
3. Consume `build/asset_table.json` and produce `cartridge/manifest.json`
|
|
4. Copy artifacts into `cartridge/`
|
|
5. Zip the cartridge into a distributable format (`.crt` / `.rom` / `.pro`)
|
|
|
|
The packer never touches `cartridge/`.
|
|
|
|
---
|
|
|
|
## 16. Compatibility and Versioning
|
|
|
|
* `assets/.prometeu/index.json` has `schema_version`
|
|
* `asset.json` has `schema_version`
|
|
* `build/asset_table.json` has `schema_version`
|
|
|
|
The Packer must be able to migrate older schema versions or emit actionable diagnostics.
|
|
|
|
---
|
|
|
|
## 17. Security and Trust Model
|
|
|
|
* The Packer is offline tooling.
|
|
* It must never execute untrusted scripts.
|
|
* It should treat external inputs as untrusted data.
|
|
|
|
---
|
|
|
|
## 18. Implementation Notes (Non-Normative)
|
|
|
|
* Rust implementation with a core crate + CLI wrapper.
|
|
* Prefer structured JSON serde models.
|
|
* Use stable diagnostic codes.
|
|
* Keep the build deterministic.
|
|
|
|
---
|
|
|
|
## Appendix A — Glossary
|
|
|
|
* **ROM (`assets.pa`)**: packed asset payload used by runtime
|
|
* **Descriptor (`asset_table.json`)**: mapping from logical assets to ROM slices
|
|
* **Managed asset**: registered asset with stable identity and anchor file
|
|
* **Virtual asset**: derived asset built from multiple inputs
|
|
* **Quarantine**: safe area for suspected junk
|
|
* **Doctor**: diagnostic command to keep sanity
|