Files
prometeu-runtime/docs/specs/runtime/05a-audio-bank-format.md
T
bquarkz 8510c5b00d feat(audio): add 32-voice PCM mixing and native music sequencing
Add self-contained SoundBanks and MusicBanks with per-sample PCM16 and IMA ADPCM storage, materialized to resident PCM16. Integrate instrument envelopes, automation, shared DSP and host lifecycle controls while preserving staged real-time execution.
2026-11-20 22:15:24 +00:00

14 KiB

Audio bank and sequence format

Domain: virtual hardware / cartridge assets. Function: normative.

Banks and materialization

SOUNDS and MUSIC each have 16 independent resident slots. Loading is explicit. The outer asset codec is NONE; sample descriptors select the audio codec. The runtime validates and materializes the entire bank before publishing it. Every sample is immutable interleaved PCM16 in resident memory. Glyph, palette, and scene payload formats are unchanged. There is no legacy audio payload path.

All integers below are little endian. Offsets are relative to the bank start. Reserved bytes must be zero. Counts, lengths, arithmetic, UTF-8, enums, references, and nonoverlapping payload bounds are validated before allocation/publication. decoded_size is the canonical metadata/table bytes plus decoded PCM bytes; actual resident allocation overhead is reported separately. No per-bank byte cap is imposed; allocation failure is an operational load error.

Header (48 bytes)

Offset Type Meaning
0 bytes[4] SBNK or MBNK
4 u16 version = 1
6 u16 flags = 0
8 u32 sample count
12 u32 instrument-zone count
16 u32 event count
20 u32 PPQ = 960
24 u32 initial microseconds per quarter note, positive
28 u32 song duration in ticks
32 u32 song loop start tick, or 0xffffffff
36 u32 track count (0..256)
40 u32 UTF-8 song/bank name length
44 u32 reserved

The name follows the header, then sample descriptors, instrument zones, events, and sample payloads, in that order. A SoundBank has at least one sample and no instruments, tracks, events, duration, or song loop. A MusicBank contains exactly one composition; all references are bank-local. Duration must be positive, loop start must precede duration, and the compiled loop must span at least one output frame. Events are sorted by tick; array order breaks equal-time ties.

Sample descriptor (40 bytes)

Offset Type Meaning
0 u8 codec: 0 = PCM16_LE, 1 = IMA_ADPCM_WAV4
1 u8 channels: 1 or 2
2 u16 flags: bit 0 means loop bounds present
4 u32 source sample rate, positive
8 u32 decoded frames per channel
12 u32 loop start frame
16 u32 exclusive loop end frame
20 u32 payload offset
24 u32 payload bytes
28 u16 IMA block alignment in bytes; zero for PCM
30 u16 IMA decoded frames per block; zero for PCM
32 u8 default volume (0..255)
33 u8 default pan/balance (0..255, center 128)
34 u16 reserved
36 u32 default rate multiplier, unsigned Q16.16, positive

PCM payload size is exactly frames * channels * 2. Loops use decoded frames and must satisfy start < end <= frames. Without a loop flag both bounds are zero. Zero-length nonlooped samples are inert. No implicit downmix is performed.

IMA_ADPCM_WAV4 uses the WAVE IMA four-bit block layout, not RTP DVI4. Each block starts with a four-byte header per channel: initial PCM i16, step index u8 in 0..88, and reserved zero. The initial PCM frame is emitted. Codes are decoded low nibble first; stereo alternates four code bytes (eight frames) for left and four for right. Each channel has independent predictor/index state. Predictor is saturated to i16 and index to 0..88 after each code. Block alignment is a positive multiple of 4channels, at least 8channels. Frames/block equals 1 + (block_alignment - 4*channels)*2/channels. Every stored block is complete; only the final block may contain padding beyond the declared decoded frame count. Its unused codes are zero. The packer defaults to 256 bytes per channel (505 frames/block); loops need not align to blocks because playback uses decoded PCM. The codec is lossy and selected explicitly for each sample. Encoder/decoder code must have documented project-compatible provenance and independent conformance vectors. Runtime does not parse a WAV/RIFF container.

Instrument zone (64 bytes)

Offset Type Meaning
0 u16 instrument ID
2 u16 sample index
4 u8 lowest note (0..127)
5 u8 highest note inclusive
6 u8 sample root note
7 u8 flags: bit 0 enables looping
8 i16 fine tuning in cents
10 u8 pan/balance
11 u8 gain
12 u32 attack duration in microseconds
16 u32 decay duration in microseconds
20 u16 sustain gain (0..65535)
22 u16 reserved
24 u32 release duration in microseconds
28 u32 vibrato frequency in millihertz
32 u16 vibrato depth in cents
34 u16 tremolo depth (0..65535)
36 u32 tremolo frequency in millihertz
40 u16 default delay send (0..65535)
42 u8 curve: 0 = all linear, 1 = linear attack/exponential decay/release
43 bytes[21] reserved

Zones of the same instrument may not overlap. Note-on resolves exactly one zone. ADSR stages reach exact endpoints within their declared output-frame duration; zero duration transitions immediately. Sustain is constant. Note-off releases from the current gain. Exponential segments use the normalized curve (exp(-6*x)-exp(-6))/(1-exp(-6)), x in [0,1], with exact endpoints; table interpolation is permitted within 1e-4 absolute gain error.

Event (32 bytes)

The record contains tick:u32, kind:u8, track:u8, flags:u16, a:u32, b:u32, c:i32, d:i32, duration_frames:u32, curve:u32. Curve is 0 linear or 1 smoothstep. Durations run on audio time, independently of later musical tempo changes.

Kind Operation Operands
1 note-on a note identity, b instrument, c note, d velocity 0..255; flag 1 legato
2 note-off a note identity
3 tempo a positive microseconds/quarter; global
4 track gain ramp c destination 0..65535
5 track pan ramp c destination 0..255
6 track pitch ramp c bend in cents
7 vibrato a millihertz, c depth cents
8 tremolo a millihertz, c depth 0..65535
9 track delay send ramp c destination 0..65535
10 instrument selection a instrument ID; note-on b=0xffffffff inherits it

Note-on duration_frames is portamento duration when legato is set. Compatible legato retains cursor/envelope on the most recent active note of that track; incompatible instrument/zone/sample retriggers. It never creates a hidden voice. Stealing selects the lowest free local index, oldest release, then oldest note; Release age starts at note-off; note age starts at note-on. Serialized event order resolves same-frame ages, with local index as final tie-breaker. Obsolete note-off cannot affect a reused voice. Authored note identities must distinguish simultaneously live logical notes of a track.

Events and tempo segments compile into exact rational frame times using u128 checked arithmetic and denominator PPQ*1000000. Events execute at the first integer output frame not earlier than their represented time. Carry fractional remainder across tempo changes and song loops. At a loop boundary, notes from the previous iteration enter release and identities are renewed. A frame may execute at most 64 events per group; validate this bound, including the loop seam, during loading. Reject zero-time loops and malformed references.

Processing, effects, and lifecycle

Three fixed groups own 12/12/8 voices. A stereo sample consumes one voice. Mono pan uses equal-power gains; stereo balance preserves L/R at center and attenuates only the opposite side toward an extreme. Mono normalized pan is p/256 for p<=128 and 0.5+(p-128)/254 above center; gains are sqrt(1-pan) and sqrt(pan). Stereo gains are [1,p/128] left of center and [(255-p)/127,1] right of center. Gain and pitch changes do not reset a cursor. Sum in floating point and clamp only at final output.

Each group has a shared low-pass and stereo delay; selective sends contribute to that group's delay only. Group gain follows dry/wet recombination. Music A/B feed Music Bus; SFX feeds SFX Bus; bus and master gains are independent. Each track initially uses each instrument's default send. A track send event sets an absolute destination: interpolate from each voice's current send to that destination, including interrupted transitions. New notes resolve their instrument default through the same transition. SFX send defaults to zero on new playback and is explicitly set with audio.set_send.

The filter is a one-pole low-pass: y += a*(x-y), a=1-exp(-2picutoff/48000). Cutoff zero bypasses filtering. Delay receives unfiltered selective sends; feedback is local to the delay. Output is dryfiltered_input + wetdelayed_input. Changing delay length clears history in O(1); no per-voice filter or delay is allocated.

Linear interpolation is used for PCM. Vibrato/tremolo use deterministic LFO phase on audio time; gain ramps support linear and smoothstep curves.

Delay storage is prepared for 2000 ms per group at 48 kHz, stereo f32: 2304000 bytes total, independent of bank size. This is an explicit bounded processing budget, not a measured battery/ARM guarantee. Feedback must remain strictly inside (-1,1) after coefficient conversion. Reset invalidates history in O(1), without clearing all buffer bytes in the callback. Pause freezes group DSP and musical time. Music unload/replacement clears its group's history; stopping SFX of one bank allows already mixed shared delay history to finish naturally.

Resident banks carry invalidation tokens. Installing/unloading a slot invalidates its old instance; retained commands/voices check that token before generating more PCM. Failure to prepare a replacement leaves the old instance valid. Asset commit remains an owner-side frame-boundary operation, independent of audio consumption. No command may resurrect an invalid bank, including after pause. Samples, bank owners, command vectors and invalidation tokens are reclaimed by host service outside the callback; the callback never blocks or destroys a last owner. Already submitted device buffers are not recalled.

Retirement uses two preallocated packets. Each has capacity 2*(3*1024+32+2) resource owners plus two command-vector owners. A staged play retains at most a sample and invalidation token; music owns the whole immutable bank, so note references can be released while that bank is still retained. Retire the bank only after releasing its note references. No callback operation can grow the packet; packet starvation retains inactive owners until service.

The existing 1024 frame/callback quotas, independent lifecycle slot, FIFO prefix publication, aborted-frame cancellation, session invalidation and bounded backpressure remain in force. Musical events do not consume VM frame quota.

High-level ABI additions

All operations require AUDIO, return one status and cost five logical cycles. Existing audio play/control calls address only SFX local voices 0..7. Music groups are 0=A and 1=B; DSP groups are 0=A,1=B,2=SFX; buses are 0=Music,1=SFX,2=Master.

ID Operation Arguments
0x3010 music.play group, resident bank slot, loop 0/1
0x3011 music.stop group
0x3012 music.pause group
0x3013 music.resume group
0x3014 music.set_volume group, gain 0..255
0x3015 music.fade_to group, gain 0..255, milliseconds, curve 0/1
0x3016 music.crossfade source group, target group, milliseconds, curve 0/1
0x3020 audio.set_bus_volume bus, gain 0..255
0x3021 audio.set_effect group, cutoff Hz (0=bypass), delay ms 0..2000, feedback number, wet number 0..1, dry number 0..1
0x3022 audio.set_send SFX voice, gain 0..255
0x3023 audio.unload_bank type 0=SoundBank/1=MusicBank, slot
0x3024 audio.set_group_volume group 0/1/2, gain 0..255 (independent of bus gain)

Group/slot/voice errors and nonfinite/out-of-range arguments reject before admission without consuming quota. No command returns a playback handle or synchronously waits for the consumer. Crossfade schedules both ramps and stops the source at completion. If either group is inactive when the consumer applies the crossfade, it has no effect; admission success is not a consumer acknowledgement. Fade alone to zero keeps sequence time advancing. Unload is an independent bounded owner control, so an exhausted guest audio command quota cannot prevent invalidation. It returns AudioOpStatus, mapping unavailable owner control to BANK_INVALID. It is an asset-owner action: a later VM frame fault does not restore an unloaded bank. Music play/stop/replacement preserve the current group gain and cancel its old fade, so setting gain before play is effective. DSP configuration belongs to the group and persists across bank changes; its history follows the reset policy above. Session/epoch reset restores gains, buses and DSP configuration to defaults.

Codec references and diagnostics

The IMA WAVE block header includes the first uncompressed sample and uses low nibble first. RTP DVI4 has a different framing/order; the distinction is documented in RFC 3551 section 4.5.1. The implementation is project-owned code under the project MIT license; no third-party decoder or additional codec library is linked. PCM WAV source layout follows WAVEFORMATEX. Only the packer reads WAV containers.

Owner-side bank diagnostics expose cartridge bytes, canonical decoded bytes, PCM length/allocation bytes per resident bank, and unique PCM totals across slots. Accounted metadata/total includes resident structs, strings and vector capacities, excluding Arc/allocator headers and asset-manager caches; allocator peak observations are separate actual allocation measurements. Materialization elapsed time includes cooperative yields; pipeline load latency is separately recorded by bank type. Mixer diagnostics expose active and maximum voices, frames and render time; the service lane measures retained PCM separately from resident slots. Retained PCM can overlap resident PCM; those reference counts must not be summed as additional memory. These are host diagnostics, not new guest byte-budget quotas. The measurement executable includes allocator peak tracking for replacement/load observations. Prepared engine memory includes fixed DSP capacity; transport, retirement packets and bank memory are additional.

Integer event timing and codec PCM output are deterministic. Floating-point mixing/DSP replay is tested with identical buffer partitions on a host; bitwise identity across different CPU/libm implementations is not qualified here.