136 lines
5.1 KiB
Markdown
136 lines
5.1 KiB
Markdown
# PBS Conformance Test Specification
|
|
|
|
Status: Draft v1 (Quality-Gate Baseline)
|
|
Applies to: executable quality gates for PBS language frontends and language-level regression evidence
|
|
|
|
## 1. Purpose
|
|
|
|
This document defines the minimum quality-gate contract for PBS language implementations.
|
|
|
|
The goal is practical and test-driven:
|
|
|
|
- keep quality measurable while language, lowering, and runtime integration are still evolving,
|
|
- require only gates that can be proved with tests today,
|
|
- avoid freezing speculative backend policy too early.
|
|
|
|
## 2. Scope
|
|
|
|
This document defines:
|
|
|
|
- required unit-test gates for language frontend work,
|
|
- required regression gate behavior for published language behavior,
|
|
- and the boundary for runtime integration regression tests when a backend can emit executable artifacts.
|
|
|
|
This document does not define:
|
|
|
|
- runtime semantics or VM behavior taxonomy,
|
|
- full artifact-level compatibility matrices,
|
|
- one test harness layout,
|
|
- or one repository structure for fixtures.
|
|
|
|
## 3. Authority and Precedence
|
|
|
|
Normative precedence:
|
|
|
|
1. `1. Language Charter.md`
|
|
2. `2. Governance and Versioning.md`
|
|
3. `3. Core Syntax Specification.md`
|
|
4. `4. Static Semantics Specification.md`
|
|
5. `12. Diagnostics Specification.md`
|
|
6. `13. Lowering IRBackend Specification.md`
|
|
7. `20. IRBackend to IRVM Lowering Specification.md`
|
|
8. `21. IRVM Optimization Pipeline Specification.md`
|
|
9. This document
|
|
|
|
This document must not weaken normative requirements from higher-precedence specs.
|
|
|
|
## 4. Normative Inputs
|
|
|
|
This document depends on:
|
|
|
|
- `3. Core Syntax Specification.md`
|
|
- `4. Static Semantics Specification.md`
|
|
- `12. Diagnostics Specification.md`
|
|
- `13. Lowering IRBackend Specification.md`
|
|
- `20. IRBackend to IRVM Lowering Specification.md`
|
|
- `21. IRVM Optimization Pipeline Specification.md`
|
|
- `17. Compatibility and Evolution Policy.md`
|
|
|
|
## 5. Quality Gate Model
|
|
|
|
PBS v1 uses two practical gate families.
|
|
|
|
### 5.1 Gate U: Language Unit Quality Gate (Required)
|
|
|
|
Every language frontend integrated into Studio/compiler must have passing unit tests that cover, at minimum:
|
|
|
|
1. lexer behavior (valid tokenization and deterministic lexical rejection),
|
|
2. parser behavior (valid parse and deterministic syntax rejection),
|
|
3. AST shape for representative valid programs,
|
|
4. frontend-to-IR backend emission (`IRBackend`/`IRBackendFile`) for representative valid programs,
|
|
5. diagnostics surface for required rejection cases (`code`, `severity`, and primary attribution).
|
|
|
|
This gate is mandatory for each language regardless of backend maturity.
|
|
|
|
### 5.2 Gate I: Integration Regression Gate With Runtime (Conditional)
|
|
|
|
When a language backend can emit executable artifacts (for example bytecode), integration regression tests must be introduced for that language.
|
|
|
|
At minimum, those tests must:
|
|
|
|
1. compile selected fixtures through the executable pipeline for that language,
|
|
2. execute them against the runtime line declared for that backend,
|
|
3. assert deterministic observable outcomes for supported fixtures,
|
|
4. preserve fixed bugs and published supported examples as regression tests,
|
|
5. and cover deterministic behavior across `LowerToVM`, `OptimizeIRVM`, and `EmitBytecode` boundaries when those stages are present.
|
|
|
|
Until executable artifact emission exists for a language, Gate I is explicitly `deferred` for that language and does not block frontend progress.
|
|
|
|
## 6. Regression Policy
|
|
|
|
Published supported behavior must remain regression-protected.
|
|
|
|
At minimum, each language corpus must retain:
|
|
|
|
1. bug reproductions for fixes that remain in supported scope,
|
|
2. published supported examples,
|
|
3. and core gate fixtures required to justify the language support state.
|
|
|
|
Regression grouping by language is expected; one shared monolithic corpus is not required.
|
|
|
|
## 7. Claims and Evidence
|
|
|
|
For PBS quality claims at this stage:
|
|
|
|
- a language may be considered `frontend-ready` only with Gate U passing,
|
|
- a language may be considered `integration-ready` only with Gate U and Gate I passing where Gate I applies,
|
|
- and a published support claim must map to executable evidence from these gates.
|
|
- conformance-relevant FE/BE changes in executable backend paths MUST update `22. Backend Spec-to-Test Conformance Matrix.md` and pass matrix hard-gate policy checks.
|
|
|
|
The exact publication vocabulary is owned by `17. Compatibility and Evolution Policy.md`.
|
|
|
|
## 8. Explicit Deferrals
|
|
|
|
The following are intentionally deferred until backend/runtime contracts stabilize further:
|
|
|
|
- detailed artifact-level conformance taxonomy,
|
|
- verifier-facing conformance taxonomy,
|
|
- full compatibility matrix axes beyond the current minimum support table,
|
|
- and one mandatory fixture repository layout.
|
|
|
|
## 9. Non-Goals
|
|
|
|
- Replacing normative language specs with tests.
|
|
- Defining runtime semantics in this document.
|
|
- Freezing backend architecture decisions prematurely.
|
|
- Treating performance benchmarks as conformance gates.
|
|
|
|
## 10. Exit Criteria
|
|
|
|
This document is in a healthy state when:
|
|
|
|
1. every active language has an explicit Gate U status,
|
|
2. languages with executable backends have an explicit Gate I status,
|
|
3. published supported language behavior has regression evidence,
|
|
4. and no support claim exists without mapped test evidence.
|