139 lines
5.2 KiB
Markdown
139 lines
5.2 KiB
Markdown
# PBS Conformance Test Specification
|
|
|
|
Status: Draft v0 (Skeleton)
|
|
Applies to: conformance obligations, test categories, and acceptance criteria for PBS implementations
|
|
|
|
## 1. Purpose
|
|
|
|
This document will define the PBS conformance testing contract.
|
|
|
|
Its purpose is to make the specification operational by defining:
|
|
|
|
- what an implementation must demonstrate to claim conformance,
|
|
- which parts of the spec require positive versus negative tests,
|
|
- how deterministic diagnostics and runtime behavior are validated,
|
|
- how compatibility and version claims are checked.
|
|
|
|
## 2. Scope
|
|
|
|
This document is intended to define:
|
|
|
|
- conformance levels or profiles, if any,
|
|
- required test categories,
|
|
- expectations for syntax, static, manifest, lowering, loader, and runtime tests,
|
|
- diagnostic-conformance expectations,
|
|
- artifact and runtime fixtures needed for host-binding and capability-gating tests,
|
|
- compatibility-matrix conformance hooks.
|
|
|
|
This document does not define:
|
|
|
|
- one specific test runner implementation,
|
|
- CI policy for one repository,
|
|
- benchmark or performance certification,
|
|
- product release governance process.
|
|
|
|
## 3. Authority and Precedence
|
|
|
|
Normative precedence:
|
|
|
|
1. `1. Language Charter.md`
|
|
2. `2. Governance and Versioning.md`
|
|
3. `3. Core Syntax Specification.md`
|
|
4. `4. Static Semantics Specification.md`
|
|
5. `5. Manifest, Stdlib, and SDK Resolution Specification.md`
|
|
6. `6.2. Host ABI Binding and Loader Resolution Specification.md`
|
|
7. `7. Cartridge Manifest and Runtime Capabilities Specification.md`
|
|
8. `9. Dynamic Semantics Specification.md`
|
|
9. `10. Memory and Lifetime Specification.md`
|
|
10. `11. Diagnostics Specification.md`
|
|
11. `12. IR and Lowering Specification.md`
|
|
12. This document
|
|
|
|
This document must not weaken a normative requirement imposed by higher-precedence specs.
|
|
|
|
## 4. Normative Inputs
|
|
|
|
This document depends on, at minimum, the full active PBS v1 spec set.
|
|
|
|
In the current phase, it should be drafted against:
|
|
|
|
- `3. Core Syntax Specification.md`
|
|
- `4. Static Semantics Specification.md`
|
|
- `5. Manifest, Stdlib, and SDK Resolution Specification.md`
|
|
- `6.2. Host ABI Binding and Loader Resolution Specification.md`
|
|
- `7. Cartridge Manifest and Runtime Capabilities Specification.md`
|
|
|
|
It will require closure of:
|
|
|
|
- `9. Dynamic Semantics Specification.md`
|
|
- `10. Memory and Lifetime Specification.md`
|
|
- `11. Diagnostics Specification.md`
|
|
- `12. IR and Lowering Specification.md`
|
|
|
|
## 5. Already-Settled Inputs
|
|
|
|
The following inputs are already fixed elsewhere and must not be contradicted here:
|
|
|
|
- The parser must behave deterministically.
|
|
- Syntax and static semantics already enumerate minimum required rejection cases.
|
|
- Import/manifest failures are required to be deterministic.
|
|
- Loader failures for malformed or unauthorized host usage are required to be deterministic.
|
|
- Governance requires conformance updates before release.
|
|
- Compatibility claims must be reflected in a published compatibility matrix.
|
|
|
|
## 6. Initial Section Targets
|
|
|
|
At minimum, the completed document should contain normative sections for:
|
|
|
|
1. scope of conformance,
|
|
2. implementation claims and version domains,
|
|
3. required positive parsing/binding/type-checking tests,
|
|
4. required negative syntax/static tests,
|
|
5. manifest/import-resolution tests,
|
|
6. dynamic-semantics runtime tests,
|
|
7. host-binding and capability-gating tests,
|
|
8. diagnostics-conformance tests,
|
|
9. compatibility and regression test expectations.
|
|
|
|
## 7. A Ver
|
|
|
|
The following items remain to be closed before this document can be considered complete.
|
|
|
|
### 7.1 Depends on `9. Dynamic Semantics Specification.md` and `10. Memory and Lifetime Specification.md`
|
|
|
|
- Exact runtime-oracle model for observable evaluation order, traps, and effect behavior.
|
|
- Which memory/aliasing facts must be tested directly versus only through diagnostics/tooling hooks.
|
|
|
|
### 7.2 Depends on `11. Diagnostics Specification.md`
|
|
|
|
- Whether conformance checks diagnostic codes, spans, categories, wording classes, or only rejection phase and location.
|
|
- Which warnings or cost diagnostics are in conformance scope.
|
|
|
|
### 7.3 Depends on `12. IR and Lowering Specification.md`
|
|
|
|
- Whether artifact-level conformance is required in addition to source-level conformance.
|
|
- Which emitted host-binding and lowering invariants require golden tests.
|
|
|
|
### 7.4 Still to decide inside this document
|
|
|
|
- Whether PBS v1 has one conformance level or staged claims such as frontend-only versus full toolchain.
|
|
- Fixture strategy for stdlib environments, host registries, and capability-grant scenarios.
|
|
- How compatibility-matrix claims are validated across language/profile/stdlib domains.
|
|
|
|
## 8. Non-Goals
|
|
|
|
- Replacing the normative specs with tests as the primary authority.
|
|
- Forcing one repository layout or one test harness.
|
|
- Treating performance benchmarking as language conformance.
|
|
- Inventing new language rules through test-only precedent.
|
|
|
|
## 9. Exit Criteria
|
|
|
|
This document is ready to move beyond skeleton status only when:
|
|
|
|
1. every active PBS v1 spec has a corresponding conformance surface,
|
|
2. positive and negative obligations are clearly separated,
|
|
3. diagnostics and runtime behavior have explicit oracle expectations,
|
|
4. compatibility and version claims can be tested rather than asserted informally,
|
|
5. the document no longer relies on unresolved `A Ver` items for the intended conformance level.
|