117 lines
3.9 KiB
Markdown
117 lines
3.9 KiB
Markdown
# PBS Conformance Test Agenda
|
|
|
|
Status: Active
|
|
|
|
## Purpose
|
|
|
|
Drive the decisions needed to turn `13. Conformance Test Specification.md` into an executable conformance contract for PBS v1.
|
|
|
|
## Context
|
|
|
|
The current conformance spec already defines a source-level baseline, but it still leaves open:
|
|
|
|
- whether PBS has one conformance level or staged claims,
|
|
- how diagnostics participate in conformance,
|
|
- when artifact-level golden tests become mandatory,
|
|
- how fixtures for stdlib environments, host registries, and capability grants are modeled,
|
|
- and how compatibility promises are validated over time.
|
|
|
|
This agenda should keep conformance practical and layered so the project can evolve from frontend tests to full-toolchain claims without ambiguity.
|
|
|
|
## Decisions To Produce
|
|
|
|
1. Decide the conformance-claim model:
|
|
one level only or staged levels such as frontend-only and full toolchain.
|
|
2. Decide the normative oracle shape for diagnostics.
|
|
3. Decide when artifact-level conformance becomes required in addition to source-level behavior.
|
|
4. Decide the minimum fixture model for stdlib, host registry, capabilities, and runtime lines.
|
|
5. Decide how compatibility and regression claims are encoded in the conformance corpus.
|
|
|
|
## Core Questions
|
|
|
|
1. Is a parser plus binder implementation allowed to claim partial conformance, and under what label?
|
|
2. Do diagnostics tests assert codes, phases, spans, wording classes, or only acceptance and rejection?
|
|
3. Which lowering and host-binding invariants require golden tests once `12` and `15` are closed?
|
|
4. What is the smallest reusable fixture set that still exercises host-backed and stdlib-backed surfaces honestly?
|
|
5. How should compatibility expectations across language, stdlib, and cartridge domains be tested over time?
|
|
|
|
## Proposed Workshop Sequence
|
|
|
|
### Workshop 1: Conformance Claim Levels
|
|
|
|
Purpose:
|
|
|
|
- decide whether PBS has one conformance level or staged claims.
|
|
|
|
Expected decisions:
|
|
|
|
- claim taxonomy,
|
|
- minimum requirements for each claim,
|
|
- and naming constraints for partial implementations.
|
|
|
|
### Workshop 2: Source-Level Oracle and Diagnostic Oracle
|
|
|
|
Purpose:
|
|
|
|
- close what source-level behavior must be asserted,
|
|
- and decide how diagnostics enter conformance.
|
|
|
|
Expected decisions:
|
|
|
|
- positive versus negative suite boundaries,
|
|
- diagnostic oracle granularity,
|
|
- and minimum deterministic assertions.
|
|
|
|
### Workshop 3: Artifact-Level Conformance and Fixtures
|
|
|
|
Purpose:
|
|
|
|
- decide when artifact-level tests become mandatory,
|
|
- and close the fixture model for stdlib, registry, capability, and runtime scenarios.
|
|
|
|
Expected decisions:
|
|
|
|
- artifact-golden boundary,
|
|
- fixture strategy,
|
|
- and environment assumptions for host-backed tests.
|
|
|
|
### Workshop 4: Regression and Compatibility Matrices
|
|
|
|
Purpose:
|
|
|
|
- decide how already-published behavior claims are preserved and tested over time.
|
|
|
|
Expected decisions:
|
|
|
|
- regression corpus policy,
|
|
- compatibility-matrix expectations,
|
|
- and alignment with `17`.
|
|
|
|
## Expected Spec Material
|
|
|
|
The resulting spec work should be able to add or close sections for:
|
|
|
|
- conformance levels or claim classes,
|
|
- required positive and negative suites,
|
|
- diagnostic oracle rules,
|
|
- artifact-level oracle rules,
|
|
- fixture model and environment assumptions,
|
|
- regression and compatibility matrices,
|
|
- and acceptance criteria for claiming PBS v1 support.
|
|
|
|
## Non-Goals
|
|
|
|
- Choosing one repository layout or test framework.
|
|
- Turning benchmarks into conformance.
|
|
- Replacing the normative specs with test precedent.
|
|
- Freezing every temporary implementation quirk as a golden artifact.
|
|
|
|
## Inputs
|
|
|
|
- `docs/pbs/specs/11. Diagnostics Specification.md`
|
|
- `docs/pbs/specs/12. IR and Lowering Specification.md`
|
|
- `docs/pbs/specs/13. Conformance Test Specification.md`
|
|
- `docs/pbs/specs/15. Bytecode and PBX Mapping Specification.md`
|
|
- `docs/pbs/specs/17. Compatibility and Evolution Policy.md`
|
|
- `docs/pbs/specs/19. Verification and Safety Checks Specification.md`
|