feat(vm): add reference identity and deterministic value hashing
This commit is contained in:
+146
@@ -0,0 +1,146 @@
|
||||
---
|
||||
id: LSN-0071
|
||||
ticket: vm-identity-and-value-hashing
|
||||
title: Identity domains and borrowed intrinsic invocation
|
||||
created: 2026-10-23
|
||||
tags: [vm, identity, hashing, values, compiler]
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The runtime needed stable identity queries and scalar/content hashing while
|
||||
preserving typed values, faults, owners and execution accounting. Existing
|
||||
Host intrinsics buffered arguments/results and exposed mutable context. That
|
||||
path could not establish the required no-scratch/no-guest-allocation contract
|
||||
by adding a purity flag alone.
|
||||
|
||||
The executed decision was DEC-0066, originating in AGD-0069; PLN-0197 delivered
|
||||
the independent runtime provider. Their permanent lifecycle provenance is in
|
||||
[the discussion index](../../index.ndjson). Workflow control artifacts are
|
||||
removed after completion. Published authority is the
|
||||
[canonical identity/hash specification](../../../docs/specs/runtime/02f-vm-identity-and-value-hashing.md),
|
||||
with the [qualification record](../../../docs/vm-arch/IDENTITY_HASHING_QUALIFICATION.md)
|
||||
and [producer handoff](../../../docs/vm-arch/identity-hashing-producer-handoff.md).
|
||||
This lesson explains the published design; it does not replace its contract.
|
||||
|
||||
## Key decisions
|
||||
|
||||
### Equality belongs to a selected domain
|
||||
|
||||
**What:** The VM provides exact reference identity plus separate Int64, Float
|
||||
and String hashes. SDK/producer policy selects equality, numeric canonicalization,
|
||||
component composition and optional/Dynamic behavior.
|
||||
|
||||
**Why:** Sharing a physical carrier does not establish a semantic equality domain.
|
||||
Two Trait views can share one receiver even with different implementation IDs;
|
||||
two closures with identical code/captures still have separate instance identity.
|
||||
Conversely, separately owned Strings can share content and therefore a content
|
||||
hash. A universal equality layer would silently choose language policy.
|
||||
|
||||
**Trade-offs:** Producers need coherent wrappers and composition. They gain
|
||||
explicit control without duplicating the machine's handle validation or hash
|
||||
algorithm. A hypothetical Lua consumer was a design stress test; no Lua frontend
|
||||
was implemented or needed for provider qualification.
|
||||
|
||||
### Improve intrinsic invocation with an executable profile
|
||||
|
||||
**What:** borrowed_inline_v1 borrows the original stack arguments, computes into
|
||||
fixed native output slots, validates complete output and commits once. Its
|
||||
handler receives no mutable HostContext or VM access. Invocation/scratch/context
|
||||
metadata derive from the handler representation, preventing inconsistent claims.
|
||||
The new profile is shared infrastructure, not a hash-namespace branch.
|
||||
|
||||
**Why:** deterministic, allocation and mutation flags describe separate concerns.
|
||||
A deterministic function can still allocate scratch or mutate seeded state.
|
||||
Purity metadata cannot make a buffered callback transactional. The invocation
|
||||
interface and commit protocol must enforce the resource/ownership story.
|
||||
|
||||
**Trade-offs:** The profile deliberately supports bounded output replacement.
|
||||
Unsupported capacities, ABI layouts and effect/cost combinations fail rather
|
||||
than falling back to dynamic storage. Existing Host and VM-owned Random handlers
|
||||
retain their established contract; they were not broadly migrated.
|
||||
|
||||
### Fault order and charging are part of semantics
|
||||
|
||||
**What:** Charge from safe carrier/Registry metadata before validation; preserve
|
||||
both groups until complete validation and computation succeed. Validate left
|
||||
before right even when a result seems obvious. String's known byte length is
|
||||
charged before ownership refusal; overflow charges base and traps.
|
||||
|
||||
**Why:** Early false/null results can conceal malformed or stale references.
|
||||
Popping operands before work makes failures change roots or drop String owners.
|
||||
Partial fuel checks or duplicate generic INTRINSIC costs change deterministic
|
||||
execution traces and compiler optimization obligations.
|
||||
|
||||
**Trade-offs:** An admitted instruction can overshoot positive fuel, and invalid
|
||||
long Strings still pay traversal-sized virtual cost without being traversed.
|
||||
The contract makes that explicit and testable. Zero fuel executes nothing.
|
||||
|
||||
## Patterns and algorithms
|
||||
|
||||
Reference identity is validated index plus generation, serialized as two LE u32s
|
||||
for XXH64 seed zero. The hash excludes fields, captures, generator state, trait
|
||||
metadata and native addresses. It is stable for a live object but creates no
|
||||
root or persistent cross-session identity. GC may collect an object while an
|
||||
Int64 hash remains; generation advancement still invalidates old handles.
|
||||
|
||||
Scalar hashing uses explicit LE messages. Float hashes binary64 bits, with both
|
||||
zeros canonicalized, and rejects non-finite values. It never performs numerical
|
||||
Float-to-integer conversion. String uses exact UTF-8 bytes, including NUL;
|
||||
accepted absence returns zero directly and empty String hashes the empty message.
|
||||
Signed Int64 carries the unchanged unsigned digest bits.
|
||||
|
||||
Schema 2 appends capability, nullability and cost fields to the original 18 CSV
|
||||
columns. Runtime descriptors generate the catalog, validate consistency and feed
|
||||
static per-entry nullability. Known null for hash.string does not relax older
|
||||
String entries. Explicit legacy profile tokens document the existing contract
|
||||
without claiming the bounded profile's guarantees for every intrinsic.
|
||||
|
||||
## Independent evidence
|
||||
|
||||
The Rust algorithm agrees with 118 vectors generated from the pinned official C
|
||||
XXH64 reference. Raw instruction bytes are checked independently of round-trip
|
||||
assembly. Serialized PBX exercises admission and actual VM dispatch; private
|
||||
fault injection reaches stale handles, NaN payloads and deliberately broken
|
||||
handlers that production admission prevents.
|
||||
|
||||
Ten warmed instruction regions each ran 1,000 iterations without native
|
||||
allocation/deallocation or guest heap allocation. Resource refusal tests preserve
|
||||
operands/owners/accounting. GC/reuse tests distinguish live identity, stale
|
||||
identity and hashes that do not root their source. Distribution tests use reduced
|
||||
buckets and deliberately observe collisions without substituting hash equality
|
||||
for reference equality.
|
||||
|
||||
The default bytecode/VM suites passed 453 tests; the VM diagnostic suite passed
|
||||
401; quality/layer checks passed 19; selected release suites passed 12. Local
|
||||
target/toolchain, existing ignored tests, source fingerprints, catalog/constant
|
||||
checks and exact commands are retained in the qualification record. Those
|
||||
results do not assert cross-target or source frontend integration.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Inferring no scratch, no context or purity from deterministic loses independent
|
||||
capability information. Stateful Random is reproducible but still mutates.
|
||||
- Comparing only a Trait receiver without validating its implementation pair
|
||||
accepts malformed views; treating implementation zero as absence breaks valid views.
|
||||
- Reusing physical HeapRef rules for String, Dynamic or arbitrary builtins expands
|
||||
semantic domains without a contract. Eligibility needs its own precise predicate.
|
||||
- Canonical stack null and HeapRef(0,0) have different validation behavior.
|
||||
- Numerical conversion of Float, host-endian messages or native hashing defaults
|
||||
breaks reproducible bit-level results.
|
||||
- Allocation-free algorithms can still hide argument/result buffers in generic
|
||||
invocation. Count the real instruction region, including reallocations.
|
||||
- Updating a catalog mirror alone does not migrate the compiler or installed SDK.
|
||||
Metadata, IR, emission, pinning, resources and normal/optimized source evidence
|
||||
each need a named consumer acceptance target.
|
||||
|
||||
## Takeaways
|
||||
|
||||
Make the safe execution profile an implementation capability and validate its
|
||||
metadata. Preserve original operands as roots through the last fallible step.
|
||||
Separate identity, content and language equality policy. Qualify encoding,
|
||||
algorithm, invocation, ownership and scheduling with independent evidence.
|
||||
|
||||
The runtime provider is complete independently of Studio source integration,
|
||||
Set/Map, storage publication and future frontends. Consumer obligations remain
|
||||
concrete handoff targets rather than an unbounded extension of this delivery.
|
||||
Reference in New Issue
Block a user