prometeu-studio/docs/pbs/specs/13. Conformance Test Specification.md

PBS Conformance Test Specification

Status: Draft v0 (Skeleton)
Applies to: conformance obligations, test categories, and acceptance criteria for PBS implementations

1. Purpose

This document will define the PBS conformance testing contract.

Its purpose is to make the specification operational by defining:

• what an implementation must demonstrate to claim conformance,
• which parts of the spec require positive versus negative tests,
• how deterministic diagnostics and runtime behavior are validated,
• how compatibility and version claims are checked.

2. Scope

This document is intended to define:

• conformance levels or profiles, if any,
• required test categories,
• expectations for syntax, static, manifest, lowering, loader, and runtime tests,
• diagnostic-conformance expectations,
• artifact and runtime fixtures needed for host-binding and capability-gating tests,
• compatibility-matrix conformance hooks.

This document does not define:

• one specific test runner implementation,
• CI policy for one repository,
• benchmark or performance certification,
• product release governance process.

3. Authority and Precedence

Normative precedence:

1. 1. Language Charter.md
2. 2. Governance and Versioning.md
3. 3. Core Syntax Specification.md
4. 4. Static Semantics Specification.md
5. 5. Manifest, Stdlib, and SDK Resolution Specification.md
6. 6.2. Host ABI Binding and Loader Resolution Specification.md
7. 7. Cartridge Manifest and Runtime Capabilities Specification.md
8. 9. Dynamic Semantics Specification.md
9. 10. Memory and Lifetime Specification.md
10. 11. Diagnostics Specification.md
11. 12. IR and Lowering Specification.md
12. This document

This document must not weaken a normative requirement imposed by higher-precedence specs.

4. Normative Inputs

This document depends on, at minimum, the full active PBS v1 spec set.

In the current phase, it should be drafted against:

• 3. Core Syntax Specification.md
• 4. Static Semantics Specification.md
• 5. Manifest, Stdlib, and SDK Resolution Specification.md
• 6.2. Host ABI Binding and Loader Resolution Specification.md
• 7. Cartridge Manifest and Runtime Capabilities Specification.md

It will require closure of:

• 9. Dynamic Semantics Specification.md
• 10. Memory and Lifetime Specification.md
• 11. Diagnostics Specification.md
• 12. IR and Lowering Specification.md

5. Already-Settled Inputs

The following inputs are already fixed elsewhere and must not be contradicted here:

• The parser must behave deterministically.
• Syntax and static semantics already enumerate minimum required rejection cases.
• Import/manifest failures are required to be deterministic.
• Loader failures for malformed or unauthorized host usage are required to be deterministic.
• Governance requires conformance updates before release.
• Compatibility claims must be reflected in a published compatibility matrix.

6. Initial Section Targets

At minimum, the completed document should contain normative sections for:

1. scope of conformance,
2. implementation claims and version domains,
3. required positive parsing/binding/type-checking tests,
4. required negative syntax/static tests,
5. manifest/import-resolution tests,
6. dynamic-semantics runtime tests,
7. host-binding and capability-gating tests,
8. diagnostics-conformance tests,
9. compatibility and regression test expectations.

7. A Ver

The following items remain to be closed before this document can be considered complete.

7.1 Depends on 9. Dynamic Semantics Specification.md and 10. Memory and Lifetime Specification.md

• Exact runtime-oracle model for observable evaluation order, traps, and effect behavior.
• Which memory/aliasing facts must be tested directly versus only through diagnostics/tooling hooks.

7.2 Depends on 11. Diagnostics Specification.md

• Whether conformance checks diagnostic codes, spans, categories, wording classes, or only rejection phase and location.
• Which warnings or cost diagnostics are in conformance scope.

7.3 Depends on 12. IR and Lowering Specification.md

• Whether artifact-level conformance is required in addition to source-level conformance.
• Which emitted host-binding and lowering invariants require golden tests.

7.4 Still to decide inside this document

• Whether PBS v1 has one conformance level or staged claims such as frontend-only versus full toolchain.
• Fixture strategy for stdlib environments, host registries, and capability-grant scenarios.
• How compatibility-matrix claims are validated across language/profile/stdlib domains.

8. Non-Goals

• Replacing the normative specs with tests as the primary authority.
• Forcing one repository layout or one test harness.
• Treating performance benchmarking as language conformance.
• Inventing new language rules through test-only precedent.

9. Exit Criteria

This document is ready to move beyond skeleton status only when:

1. every active PBS v1 spec has a corresponding conformance surface,
2. positive and negative obligations are clearly separated,
3. diagnostics and runtime behavior have explicit oracle expectations,
4. compatibility and version claims can be tested rather than asserted informally,
5. the document no longer relies on unresolved A Ver items for the intended conformance level.