Add self-contained SoundBanks and MusicBanks with per-sample PCM16 and IMA ADPCM storage, materialized to resident PCM16. Integrate instrument envelopes, automation, shared DSP and host lifecycle controls while preserving staged real-time execution.
271 lines
14 KiB
Markdown
271 lines
14 KiB
Markdown
# Audio bank and sequence format
|
|
|
|
Domain: virtual hardware / cartridge assets. Function: normative.
|
|
|
|
## Banks and materialization
|
|
|
|
`SOUNDS` and `MUSIC` each have 16 independent resident slots. Loading is explicit.
|
|
The outer asset codec is `NONE`; sample descriptors select the audio codec.
|
|
The runtime validates and materializes the entire bank before publishing it.
|
|
Every sample is immutable interleaved PCM16 in resident memory. Glyph, palette,
|
|
and scene payload formats are unchanged. There is no legacy audio payload path.
|
|
|
|
All integers below are little endian. Offsets are relative to the bank start.
|
|
Reserved bytes must be zero. Counts, lengths, arithmetic, UTF-8, enums, references,
|
|
and nonoverlapping payload bounds are validated before allocation/publication.
|
|
`decoded_size` is the canonical metadata/table bytes plus decoded PCM bytes;
|
|
actual resident allocation overhead is reported separately. No per-bank byte cap
|
|
is imposed; allocation failure is an operational load error.
|
|
|
|
## Header (48 bytes)
|
|
|
|
| Offset | Type | Meaning |
|
|
|---:|---|---|
|
|
| 0 | bytes[4] | `SBNK` or `MBNK` |
|
|
| 4 | u16 | version = 1 |
|
|
| 6 | u16 | flags = 0 |
|
|
| 8 | u32 | sample count |
|
|
| 12 | u32 | instrument-zone count |
|
|
| 16 | u32 | event count |
|
|
| 20 | u32 | PPQ = 960 |
|
|
| 24 | u32 | initial microseconds per quarter note, positive |
|
|
| 28 | u32 | song duration in ticks |
|
|
| 32 | u32 | song loop start tick, or 0xffffffff |
|
|
| 36 | u32 | track count (0..256) |
|
|
| 40 | u32 | UTF-8 song/bank name length |
|
|
| 44 | u32 | reserved |
|
|
|
|
The name follows the header, then sample descriptors, instrument zones, events,
|
|
and sample payloads, in that order. A SoundBank has at least one sample and no
|
|
instruments, tracks, events, duration, or song loop. A MusicBank contains exactly
|
|
one composition; all references are bank-local. Duration must be positive, loop
|
|
start must precede duration, and the compiled loop must span at least one output
|
|
frame. Events are sorted by tick; array order breaks equal-time ties.
|
|
|
|
## Sample descriptor (40 bytes)
|
|
|
|
| Offset | Type | Meaning |
|
|
|---:|---|---|
|
|
| 0 | u8 | codec: 0 = PCM16_LE, 1 = IMA_ADPCM_WAV4 |
|
|
| 1 | u8 | channels: 1 or 2 |
|
|
| 2 | u16 | flags: bit 0 means loop bounds present |
|
|
| 4 | u32 | source sample rate, positive |
|
|
| 8 | u32 | decoded frames per channel |
|
|
| 12 | u32 | loop start frame |
|
|
| 16 | u32 | exclusive loop end frame |
|
|
| 20 | u32 | payload offset |
|
|
| 24 | u32 | payload bytes |
|
|
| 28 | u16 | IMA block alignment in bytes; zero for PCM |
|
|
| 30 | u16 | IMA decoded frames per block; zero for PCM |
|
|
| 32 | u8 | default volume (0..255) |
|
|
| 33 | u8 | default pan/balance (0..255, center 128) |
|
|
| 34 | u16 | reserved |
|
|
| 36 | u32 | default rate multiplier, unsigned Q16.16, positive |
|
|
|
|
PCM payload size is exactly frames * channels * 2. Loops use decoded frames and
|
|
must satisfy start < end <= frames. Without a loop flag both bounds are zero.
|
|
Zero-length nonlooped samples are inert. No implicit downmix is performed.
|
|
|
|
IMA_ADPCM_WAV4 uses the WAVE IMA four-bit block layout, not RTP DVI4. Each block
|
|
starts with a four-byte header per channel: initial PCM i16, step index u8 in
|
|
0..88, and reserved zero. The initial PCM frame is emitted. Codes are decoded
|
|
low nibble first; stereo alternates four code bytes (eight frames) for left and
|
|
four for right. Each channel has independent predictor/index state. Predictor is
|
|
saturated to i16 and index to 0..88 after each code. Block alignment is a positive
|
|
multiple of 4*channels, at least 8*channels. Frames/block equals
|
|
1 + (block_alignment - 4*channels)*2/channels. Every stored block is complete;
|
|
only the final block may contain padding beyond the declared decoded frame count.
|
|
Its unused codes are zero. The packer defaults to 256 bytes per channel (505
|
|
frames/block); loops need not align to blocks because playback uses decoded PCM.
|
|
The codec is lossy and selected explicitly for each sample. Encoder/decoder code
|
|
must have documented project-compatible provenance and independent conformance
|
|
vectors. Runtime does not parse a WAV/RIFF container.
|
|
|
|
## Instrument zone (64 bytes)
|
|
|
|
| Offset | Type | Meaning |
|
|
|---:|---|---|
|
|
| 0 | u16 | instrument ID |
|
|
| 2 | u16 | sample index |
|
|
| 4 | u8 | lowest note (0..127) |
|
|
| 5 | u8 | highest note inclusive |
|
|
| 6 | u8 | sample root note |
|
|
| 7 | u8 | flags: bit 0 enables looping |
|
|
| 8 | i16 | fine tuning in cents |
|
|
| 10 | u8 | pan/balance |
|
|
| 11 | u8 | gain |
|
|
| 12 | u32 | attack duration in microseconds |
|
|
| 16 | u32 | decay duration in microseconds |
|
|
| 20 | u16 | sustain gain (0..65535) |
|
|
| 22 | u16 | reserved |
|
|
| 24 | u32 | release duration in microseconds |
|
|
| 28 | u32 | vibrato frequency in millihertz |
|
|
| 32 | u16 | vibrato depth in cents |
|
|
| 34 | u16 | tremolo depth (0..65535) |
|
|
| 36 | u32 | tremolo frequency in millihertz |
|
|
| 40 | u16 | default delay send (0..65535) |
|
|
| 42 | u8 | curve: 0 = all linear, 1 = linear attack/exponential decay/release |
|
|
| 43 | bytes[21] | reserved |
|
|
|
|
Zones of the same instrument may not overlap. Note-on resolves exactly one zone.
|
|
ADSR stages reach exact endpoints within their declared output-frame duration;
|
|
zero duration transitions immediately. Sustain is constant. Note-off releases
|
|
from the current gain. Exponential segments use the normalized curve
|
|
(exp(-6*x)-exp(-6))/(1-exp(-6)), x in [0,1], with exact endpoints; table
|
|
interpolation is permitted within 1e-4 absolute gain error.
|
|
|
|
## Event (32 bytes)
|
|
|
|
The record contains tick:u32, kind:u8, track:u8, flags:u16, a:u32, b:u32,
|
|
c:i32, d:i32, duration_frames:u32, curve:u32. Curve is 0 linear or 1 smoothstep.
|
|
Durations run on audio time, independently of later musical tempo changes.
|
|
|
|
| Kind | Operation | Operands |
|
|
|---:|---|---|
|
|
| 1 | note-on | a note identity, b instrument, c note, d velocity 0..255; flag 1 legato |
|
|
| 2 | note-off | a note identity |
|
|
| 3 | tempo | a positive microseconds/quarter; global |
|
|
| 4 | track gain ramp | c destination 0..65535 |
|
|
| 5 | track pan ramp | c destination 0..255 |
|
|
| 6 | track pitch ramp | c bend in cents |
|
|
| 7 | vibrato | a millihertz, c depth cents |
|
|
| 8 | tremolo | a millihertz, c depth 0..65535 |
|
|
| 9 | track delay send ramp | c destination 0..65535 |
|
|
| 10 | instrument selection | a instrument ID; note-on b=0xffffffff inherits it |
|
|
|
|
Note-on duration_frames is portamento duration when legato is set. Compatible
|
|
legato retains cursor/envelope on the most recent active note of that track;
|
|
incompatible instrument/zone/sample retriggers. It never creates a hidden voice.
|
|
Stealing selects the lowest free local index, oldest release, then oldest note;
|
|
Release age starts at note-off; note age starts at note-on. Serialized event
|
|
order resolves same-frame ages, with local index as final tie-breaker.
|
|
Obsolete note-off cannot affect a reused voice. Authored note identities must
|
|
distinguish simultaneously live logical notes of a track.
|
|
|
|
Events and tempo segments compile into exact rational frame times using u128
|
|
checked arithmetic and denominator PPQ*1000000. Events execute at the first
|
|
integer output frame not earlier than their represented time. Carry fractional
|
|
remainder across tempo changes and song loops. At a loop boundary, notes from
|
|
the previous iteration enter release and identities are renewed. A frame may
|
|
execute at most 64 events per group; validate this bound, including the loop
|
|
seam, during loading. Reject zero-time loops and malformed references.
|
|
|
|
## Processing, effects, and lifecycle
|
|
|
|
Three fixed groups own 12/12/8 voices. A stereo sample consumes one voice.
|
|
Mono pan uses equal-power gains; stereo balance preserves L/R at center and
|
|
attenuates only the opposite side toward an extreme. Mono normalized pan is
|
|
p/256 for p<=128 and 0.5+(p-128)/254 above center; gains are sqrt(1-pan) and
|
|
sqrt(pan). Stereo gains are [1,p/128] left of center and [(255-p)/127,1]
|
|
right of center. Gain and pitch changes
|
|
do not reset a cursor. Sum in floating point and clamp only at final output.
|
|
|
|
Each group has a shared low-pass and stereo delay; selective sends contribute
|
|
to that group's delay only. Group gain follows dry/wet recombination. Music A/B
|
|
feed Music Bus; SFX feeds SFX Bus; bus and master gains are independent.
|
|
Each track initially uses each instrument's default send. A track send event
|
|
sets an absolute destination: interpolate from each voice's current send to
|
|
that destination, including interrupted transitions. New notes resolve their
|
|
instrument default through the same transition. SFX send defaults to zero on
|
|
new playback and is explicitly set with audio.set_send.
|
|
|
|
The filter is a one-pole low-pass: y += a*(x-y),
|
|
a=1-exp(-2*pi*cutoff/48000). Cutoff zero bypasses filtering. Delay receives
|
|
unfiltered selective sends; feedback is local to the delay. Output is
|
|
dry*filtered_input + wet*delayed_input. Changing delay length clears history
|
|
in O(1); no per-voice filter or delay is allocated.
|
|
|
|
Linear interpolation is used for PCM. Vibrato/tremolo use deterministic LFO
|
|
phase on audio time; gain ramps support linear and smoothstep curves.
|
|
|
|
Delay storage is prepared for 2000 ms per group at 48 kHz, stereo f32: 2304000
|
|
bytes total, independent of bank size. This is an explicit bounded processing
|
|
budget, not a measured battery/ARM guarantee. Feedback must remain strictly
|
|
inside (-1,1) after coefficient conversion. Reset invalidates history in O(1),
|
|
without clearing all buffer bytes in the callback. Pause freezes group DSP and
|
|
musical time. Music unload/replacement clears its group's history; stopping SFX
|
|
of one bank allows already mixed shared delay history to finish naturally.
|
|
|
|
Resident banks carry invalidation tokens. Installing/unloading a slot invalidates
|
|
its old instance; retained commands/voices check that token before generating
|
|
more PCM. Failure to prepare a replacement leaves the old instance valid. Asset
|
|
commit remains an owner-side frame-boundary operation, independent of audio
|
|
consumption. No command may resurrect an invalid bank, including after pause.
|
|
Samples, bank owners, command vectors and invalidation tokens are reclaimed by
|
|
host service outside the callback; the callback never blocks or destroys a last
|
|
owner. Already submitted device buffers are not recalled.
|
|
|
|
Retirement uses two preallocated packets. Each has capacity
|
|
2*(3*1024+32+2) resource owners plus two command-vector owners. A staged play
|
|
retains at most a sample and invalidation token; music owns the whole immutable
|
|
bank, so note references can be released while that bank is still retained.
|
|
Retire the bank only after releasing its note references. No callback operation
|
|
can grow the packet; packet starvation retains inactive owners until service.
|
|
|
|
The existing 1024 frame/callback quotas, independent lifecycle slot, FIFO prefix
|
|
publication, aborted-frame cancellation, session invalidation and bounded
|
|
backpressure remain in force. Musical events do not consume VM frame quota.
|
|
|
|
## High-level ABI additions
|
|
|
|
All operations require AUDIO, return one status and cost five logical cycles.
|
|
Existing audio play/control calls address only SFX local voices 0..7. Music groups
|
|
are 0=A and 1=B; DSP groups are 0=A,1=B,2=SFX; buses are 0=Music,1=SFX,2=Master.
|
|
|
|
| ID | Operation | Arguments |
|
|
|---|---|---|
|
|
| 0x3010 | music.play | group, resident bank slot, loop 0/1 |
|
|
| 0x3011 | music.stop | group |
|
|
| 0x3012 | music.pause | group |
|
|
| 0x3013 | music.resume | group |
|
|
| 0x3014 | music.set_volume | group, gain 0..255 |
|
|
| 0x3015 | music.fade_to | group, gain 0..255, milliseconds, curve 0/1 |
|
|
| 0x3016 | music.crossfade | source group, target group, milliseconds, curve 0/1 |
|
|
| 0x3020 | audio.set_bus_volume | bus, gain 0..255 |
|
|
| 0x3021 | audio.set_effect | group, cutoff Hz (0=bypass), delay ms 0..2000, feedback number, wet number 0..1, dry number 0..1 |
|
|
| 0x3022 | audio.set_send | SFX voice, gain 0..255 |
|
|
| 0x3023 | audio.unload_bank | type 0=SoundBank/1=MusicBank, slot |
|
|
| 0x3024 | audio.set_group_volume | group 0/1/2, gain 0..255 (independent of bus gain) |
|
|
|
|
Group/slot/voice errors and nonfinite/out-of-range arguments reject before
|
|
admission without consuming quota. No command returns a playback handle or
|
|
synchronously waits for the consumer. Crossfade schedules both ramps and stops
|
|
the source at completion. If either group is inactive when the consumer applies
|
|
the crossfade, it has no effect; admission success is not a consumer acknowledgement.
|
|
Fade alone to
|
|
zero keeps sequence time advancing. Unload is an independent bounded owner
|
|
control, so an exhausted guest audio command quota cannot prevent invalidation.
|
|
It returns AudioOpStatus, mapping unavailable owner control to BANK_INVALID. It
|
|
is an asset-owner action: a later VM frame fault does not restore an unloaded bank.
|
|
Music play/stop/replacement preserve the current group gain and cancel its old
|
|
fade, so setting gain before play is effective. DSP configuration belongs to the
|
|
group and persists across bank changes; its history follows the reset policy above.
|
|
Session/epoch reset restores gains, buses and DSP configuration to defaults.
|
|
|
|
## Codec references and diagnostics
|
|
|
|
The IMA WAVE block header includes the first uncompressed sample and uses low
|
|
nibble first. RTP DVI4 has a different framing/order; the distinction is
|
|
documented in [RFC 3551 section 4.5.1](https://www.rfc-editor.org/rfc/rfc3551#section-4.5.1).
|
|
The implementation is project-owned code under the project MIT license; no
|
|
third-party decoder or additional codec library is linked. PCM WAV source
|
|
layout follows [WAVEFORMATEX](https://learn.microsoft.com/en-us/windows/win32/api/mmreg/ns-mmreg-waveformatex).
|
|
Only the packer reads WAV containers.
|
|
|
|
Owner-side bank diagnostics expose cartridge bytes, canonical decoded bytes,
|
|
PCM length/allocation bytes per resident bank, and unique PCM totals across
|
|
slots. Accounted metadata/total includes resident structs, strings and vector
|
|
capacities, excluding Arc/allocator headers and asset-manager caches; allocator
|
|
peak observations are separate actual allocation measurements. Materialization elapsed time includes cooperative yields; pipeline load
|
|
latency is separately recorded by bank type. Mixer diagnostics expose active
|
|
and maximum voices, frames and render time; the service lane measures retained
|
|
PCM separately from resident slots. Retained PCM can overlap resident PCM; those
|
|
reference counts must not be summed as additional memory. These are host diagnostics, not new guest
|
|
byte-budget quotas. The measurement executable includes allocator peak tracking
|
|
for replacement/load observations. Prepared engine memory includes fixed DSP
|
|
capacity; transport, retirement packets and bank memory are additional.
|
|
|
|
Integer event timing and codec PCM output are deterministic. Floating-point
|
|
mixing/DSP replay is tested with identical buffer partitions on a host; bitwise
|
|
identity across different CPU/libm implementations is not qualified here.
|