prometeu-studio/docs/pbs/specs/13. Conformance Test Specification.md

PBS Conformance Test Specification

Status: Draft v1 (Temporary)
Applies to: conformance obligations, test categories, and acceptance criteria for PBS implementations

1. Purpose

This document defines the current PBS conformance-test baseline for v1.

Its purpose is to make the active specification operational by stating:

• which categories of behavior must be tested,
• which outcomes must be deterministic across conforming implementations,
• and which parts of the corpus are not yet mature enough to demand full conformance coverage.

2. Scope

This document defines:

• required positive and negative source-level conformance surfaces,
• runtime-oracle expectations already fixed by active semantics,
• minimum host-binding and capability-gating test surfaces,
• and the current limits of v1 conformance while diagnostics and lowering remain incomplete.

This document does not define:

• one specific test harness or repository layout,
• benchmark or performance certification,
• one artifact-level golden format,
• or product release governance policy.

3. Authority and Precedence

Normative precedence:

1. 1. Language Charter.md
2. 2. Governance and Versioning.md
3. 3. Core Syntax Specification.md
4. 4. Static Semantics Specification.md
5. 5. Manifest, Stdlib, and SDK Resolution Specification.md
6. 6.2. Host ABI Binding and Loader Resolution Specification.md
7. 7. Cartridge Manifest and Runtime Capabilities Specification.md
8. 9. Dynamic Semantics Specification.md
9. 10. Memory and Lifetime Specification.md
10. 11. Diagnostics Specification.md
11. 12. IR and Lowering Specification.md
12. This document

This document must not weaken a normative requirement imposed by higher-precedence specs.

4. Normative Inputs

This document depends on the active PBS v1 spec set.

For the currently integrated conformance baseline, the key inputs are:

• 3. Core Syntax Specification.md
• 4. Static Semantics Specification.md
• 5. Manifest, Stdlib, and SDK Resolution Specification.md
• 6.2. Host ABI Binding and Loader Resolution Specification.md
• 7. Cartridge Manifest and Runtime Capabilities Specification.md
• 9. Dynamic Semantics Specification.md
• 10. Memory and Lifetime Specification.md

Later expansion of this document will also depend on:

• 11. Diagnostics Specification.md
• 12. IR and Lowering Specification.md
• 15. Bytecode and PBX Mapping Specification.md

5. Already-Settled Inputs

The following inputs are already fixed elsewhere and must not be contradicted here:

• The parser must behave deterministically.
• Syntax and static semantics already enumerate minimum required rejection cases.
• Import/manifest failures are required to be deterministic.
• Loader failures for malformed or unauthorized host usage are required to be deterministic.
• Dynamic semantics are single-threaded and deterministic at the language-observable level.
• Memory semantics already distinguish identity-bearing values from pure-value and carrier forms.
• Traps are fatal runtime aborts rather than recoverable userland outcomes.

6. Current Conformance Baseline

At the current maturity level, a conforming PBS implementation must support tests in the following categories:

1. positive syntax and parsing tests,
2. negative syntax rejection tests,
3. positive binding and static-typing tests,
4. negative static-semantics rejection tests,
5. manifest/import-resolution tests,
6. host-binding and capability-gating tests,
7. source-level dynamic-semantics runtime tests,
8. source-level memory/identity behavior tests,
9. compatibility/regression tests for already-published behavior claims.

7. Source-Level Runtime Oracle

The current conformance oracle for runtime behavior is source-level and language-observable.

At minimum, conformance tests must cover:

1. strict left-to-right evaluation where the language does not define an exception,
2. exactly-once evaluation of:
   • call-target formation,
   • call arguments,
   • if conditions,
   • switch selectors,
   • some(expr) payload formation,
   • bind(context, fn_name) context capture,
3. optional behavior:
   • some(expr) is eager,
   • none is canonical absence,
   • opt else fallback evaluates fallback only on none,
4. result behavior:
   • expr! passes through success payload,
   • expr! immediately propagates same-domain error,
   • handle expr { ... } passes through success payload,
   • handle block arms terminate with ok(payload) or err(E.case),
   • short handle arms remap error as specified,
5. branch-selection behavior:
   • non-selected if and switch arms do not execute,
   • switch selects deterministically over admitted static discriminants,
6. trap behavior:
   • traps do not enter result flow,
   • traps are not intercepted by handle,
   • traps abort the running PBS program path.

Conformance does not currently require a userland-visible trap taxonomy. It requires only the fatal-abort boundary already defined by the runtime and dynamic-semantics specs.

8. Source-Level Memory and Identity Oracle

At minimum, conformance tests must cover the following language-observable memory/identity facts:

1. struct values are identity-bearing and alias through assignment, parameter passing, and return,
2. service values preserve singleton identity,
3. tuples, optional, and result do not create new userland identity of their own,
4. carriers preserve the identity behavior of their payloads,
5. bind(context, fn_name) captures the same runtime identity of the struct context rather than copying it,
6. callback invocation after bind observes mutation against that same captured context identity,
7. GC timing itself is not a conformance oracle,
8. reclamation is validated only indirectly through correctness and safety boundaries, not through collection timing.

9. Host-Binding and Boundary Tests

At minimum, conformance tests must cover:

1. deterministic rejection of malformed or unauthorized host usage,
2. deterministic acceptance of valid host-backed declarations admitted by the active capability/runtime line,
3. correct source-level behavior of host-backed calls that are already admitted by the Host ABI and capability specs,
4. the absence of userland-visible raw pointers or heap-shared host-memory surfaces at the PBS boundary.

This document does not yet require one artifact-level oracle for host-lowering shape. That remains dependent on later closure of lowering and PBX-mapping specifications.

10. TODO

The following items remain open before this document can be considered complete:

10.1 Depends on 11. Diagnostics Specification.md

• whether conformance checks diagnostic codes, spans, categories, wording classes, or only rejection phase and location;
• which warnings or cost diagnostics are in conformance scope.

10.2 Depends on 12. IR and Lowering Specification.md and 15. Bytecode and PBX Mapping Specification.md

• whether artifact-level conformance is required in addition to source-level conformance;
• which emitted host-binding and lowering invariants require golden tests;
• which source-to-artifact mapping hooks are part of the conformance contract.

10.3 Still to decide inside this document

• whether PBS v1 has one conformance level or staged claims such as frontend-only versus full toolchain;
• fixture strategy for stdlib environments, host registries, and capability-grant scenarios;
• how compatibility-matrix claims are validated across language/profile/stdlib domains.

11. Non-Goals

• Replacing the normative specs with tests as the primary authority.
• Forcing one repository layout or one test harness.
• Treating performance benchmarking as language conformance.
• Inventing new language rules through test-only precedent.

12. Exit Criteria

This document is ready to move beyond temporary status only when:

1. every active PBS v1 spec has a corresponding conformance surface,
2. positive and negative obligations are clearly separated,
3. diagnostics and artifact-level behavior have explicit oracle expectations where required,
4. compatibility and version claims can be tested rather than asserted informally,
5. and the document no longer relies on unresolved TODO items for the intended conformance level.