66 lines
2.7 KiB
Markdown
66 lines
2.7 KiB
Markdown
# PBS Conformance Test Agenda
|
|
|
|
Status: Active
|
|
|
|
## Purpose
|
|
|
|
Drive the decisions needed to turn `13. Conformance Test Specification.md` into an executable conformance contract for PBS v1.
|
|
|
|
## Context
|
|
|
|
The current conformance spec already defines a source-level baseline, but it still leaves open:
|
|
|
|
- whether PBS has one conformance level or staged claims,
|
|
- how diagnostics participate in conformance,
|
|
- when artifact-level golden tests become mandatory,
|
|
- how fixtures for stdlib environments, host registries, and capability grants are modeled,
|
|
- and how compatibility promises are validated over time.
|
|
|
|
This agenda should keep conformance practical and layered so the project can evolve from frontend tests to full-toolchain claims without ambiguity.
|
|
|
|
## Decisions To Produce
|
|
|
|
1. Decide the conformance-claim model:
|
|
one level only or staged levels such as frontend-only and full toolchain.
|
|
2. Decide the normative oracle shape for diagnostics.
|
|
3. Decide when artifact-level conformance becomes required in addition to source-level behavior.
|
|
4. Decide the minimum fixture model for stdlib, host registry, capabilities, and runtime lines.
|
|
5. Decide how compatibility and regression claims are encoded in the conformance corpus.
|
|
|
|
## Core Questions
|
|
|
|
1. Is a parser plus binder implementation allowed to claim partial conformance, and under what label?
|
|
2. Do diagnostics tests assert codes, phases, spans, wording classes, or only acceptance and rejection?
|
|
3. Which lowering and host-binding invariants require golden tests once `12` and `15` are closed?
|
|
4. What is the smallest reusable fixture set that still exercises host-backed and stdlib-backed surfaces honestly?
|
|
5. How should compatibility expectations across language, stdlib, and cartridge domains be tested over time?
|
|
|
|
## Expected Spec Material
|
|
|
|
The resulting spec work should be able to add or close sections for:
|
|
|
|
- conformance levels or claim classes,
|
|
- required positive and negative suites,
|
|
- diagnostic oracle rules,
|
|
- artifact-level oracle rules,
|
|
- fixture model and environment assumptions,
|
|
- regression and compatibility matrices,
|
|
- and acceptance criteria for claiming PBS v1 support.
|
|
|
|
## Non-Goals
|
|
|
|
- Choosing one repository layout or test framework.
|
|
- Turning benchmarks into conformance.
|
|
- Replacing the normative specs with test precedent.
|
|
- Freezing every temporary implementation quirk as a golden artifact.
|
|
|
|
## Inputs
|
|
|
|
- `docs/pbs/specs/11. Diagnostics Specification.md`
|
|
- `docs/pbs/specs/12. IR and Lowering Specification.md`
|
|
- `docs/pbs/specs/13. Conformance Test Specification.md`
|
|
- `docs/pbs/specs/15. Bytecode and PBX Mapping Specification.md`
|
|
- `docs/pbs/specs/17. Compatibility and Evolution Policy.md`
|
|
- `docs/pbs/specs/19. Verification and Safety Checks Specification.md`
|
|
|