feat(vm): add reference identity and deterministic value hashing

This commit is contained in:
2026-10-23 14:29:00 +01:00
parent 57992cf214
commit 69ad1931de
41 changed files with 2521 additions and 121 deletions
@@ -0,0 +1,146 @@
---
id: LSN-0071
ticket: vm-identity-and-value-hashing
title: Identity domains and borrowed intrinsic invocation
created: 2026-10-23
tags: [vm, identity, hashing, values, compiler]
---
## Context
The runtime needed stable identity queries and scalar/content hashing while
preserving typed values, faults, owners and execution accounting. Existing
Host intrinsics buffered arguments/results and exposed mutable context. That
path could not establish the required no-scratch/no-guest-allocation contract
by adding a purity flag alone.
The executed decision was DEC-0066, originating in AGD-0069; PLN-0197 delivered
the independent runtime provider. Their permanent lifecycle provenance is in
[the discussion index](../../index.ndjson). Workflow control artifacts are
removed after completion. Published authority is the
[canonical identity/hash specification](../../../docs/specs/runtime/02f-vm-identity-and-value-hashing.md),
with the [qualification record](../../../docs/vm-arch/IDENTITY_HASHING_QUALIFICATION.md)
and [producer handoff](../../../docs/vm-arch/identity-hashing-producer-handoff.md).
This lesson explains the published design; it does not replace its contract.
## Key decisions
### Equality belongs to a selected domain
**What:** The VM provides exact reference identity plus separate Int64, Float
and String hashes. SDK/producer policy selects equality, numeric canonicalization,
component composition and optional/Dynamic behavior.
**Why:** Sharing a physical carrier does not establish a semantic equality domain.
Two Trait views can share one receiver even with different implementation IDs;
two closures with identical code/captures still have separate instance identity.
Conversely, separately owned Strings can share content and therefore a content
hash. A universal equality layer would silently choose language policy.
**Trade-offs:** Producers need coherent wrappers and composition. They gain
explicit control without duplicating the machine's handle validation or hash
algorithm. A hypothetical Lua consumer was a design stress test; no Lua frontend
was implemented or needed for provider qualification.
### Improve intrinsic invocation with an executable profile
**What:** borrowed_inline_v1 borrows the original stack arguments, computes into
fixed native output slots, validates complete output and commits once. Its
handler receives no mutable HostContext or VM access. Invocation/scratch/context
metadata derive from the handler representation, preventing inconsistent claims.
The new profile is shared infrastructure, not a hash-namespace branch.
**Why:** deterministic, allocation and mutation flags describe separate concerns.
A deterministic function can still allocate scratch or mutate seeded state.
Purity metadata cannot make a buffered callback transactional. The invocation
interface and commit protocol must enforce the resource/ownership story.
**Trade-offs:** The profile deliberately supports bounded output replacement.
Unsupported capacities, ABI layouts and effect/cost combinations fail rather
than falling back to dynamic storage. Existing Host and VM-owned Random handlers
retain their established contract; they were not broadly migrated.
### Fault order and charging are part of semantics
**What:** Charge from safe carrier/Registry metadata before validation; preserve
both groups until complete validation and computation succeed. Validate left
before right even when a result seems obvious. String's known byte length is
charged before ownership refusal; overflow charges base and traps.
**Why:** Early false/null results can conceal malformed or stale references.
Popping operands before work makes failures change roots or drop String owners.
Partial fuel checks or duplicate generic INTRINSIC costs change deterministic
execution traces and compiler optimization obligations.
**Trade-offs:** An admitted instruction can overshoot positive fuel, and invalid
long Strings still pay traversal-sized virtual cost without being traversed.
The contract makes that explicit and testable. Zero fuel executes nothing.
## Patterns and algorithms
Reference identity is validated index plus generation, serialized as two LE u32s
for XXH64 seed zero. The hash excludes fields, captures, generator state, trait
metadata and native addresses. It is stable for a live object but creates no
root or persistent cross-session identity. GC may collect an object while an
Int64 hash remains; generation advancement still invalidates old handles.
Scalar hashing uses explicit LE messages. Float hashes binary64 bits, with both
zeros canonicalized, and rejects non-finite values. It never performs numerical
Float-to-integer conversion. String uses exact UTF-8 bytes, including NUL;
accepted absence returns zero directly and empty String hashes the empty message.
Signed Int64 carries the unchanged unsigned digest bits.
Schema 2 appends capability, nullability and cost fields to the original 18 CSV
columns. Runtime descriptors generate the catalog, validate consistency and feed
static per-entry nullability. Known null for hash.string does not relax older
String entries. Explicit legacy profile tokens document the existing contract
without claiming the bounded profile's guarantees for every intrinsic.
## Independent evidence
The Rust algorithm agrees with 118 vectors generated from the pinned official C
XXH64 reference. Raw instruction bytes are checked independently of round-trip
assembly. Serialized PBX exercises admission and actual VM dispatch; private
fault injection reaches stale handles, NaN payloads and deliberately broken
handlers that production admission prevents.
Ten warmed instruction regions each ran 1,000 iterations without native
allocation/deallocation or guest heap allocation. Resource refusal tests preserve
operands/owners/accounting. GC/reuse tests distinguish live identity, stale
identity and hashes that do not root their source. Distribution tests use reduced
buckets and deliberately observe collisions without substituting hash equality
for reference equality.
The default bytecode/VM suites passed 453 tests; the VM diagnostic suite passed
401; quality/layer checks passed 19; selected release suites passed 12. Local
target/toolchain, existing ignored tests, source fingerprints, catalog/constant
checks and exact commands are retained in the qualification record. Those
results do not assert cross-target or source frontend integration.
## Pitfalls
- Inferring no scratch, no context or purity from deterministic loses independent
capability information. Stateful Random is reproducible but still mutates.
- Comparing only a Trait receiver without validating its implementation pair
accepts malformed views; treating implementation zero as absence breaks valid views.
- Reusing physical HeapRef rules for String, Dynamic or arbitrary builtins expands
semantic domains without a contract. Eligibility needs its own precise predicate.
- Canonical stack null and HeapRef(0,0) have different validation behavior.
- Numerical conversion of Float, host-endian messages or native hashing defaults
breaks reproducible bit-level results.
- Allocation-free algorithms can still hide argument/result buffers in generic
invocation. Count the real instruction region, including reallocations.
- Updating a catalog mirror alone does not migrate the compiler or installed SDK.
Metadata, IR, emission, pinning, resources and normal/optimized source evidence
each need a named consumer acceptance target.
## Takeaways
Make the safe execution profile an implementation capability and validate its
metadata. Preserve original operands as roots through the last fallible step.
Separate identity, content and language equality policy. Qualify encoding,
algorithm, invocation, ownership and scheduling with independent evidence.
The runtime provider is complete independently of Studio source integration,
Set/Map, storage publication and future frontends. Consumer obligations remain
concrete handoff targets rather than an unbounded extension of this delivery.