discussion framework migration
This commit is contained in:
@@ -0,0 +1,135 @@
|
||||
# PBS Conformance Test Specification
|
||||
|
||||
Status: Draft v1 (Quality-Gate Baseline)
|
||||
Applies to: executable quality gates for PBS language frontends and language-level regression evidence
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the minimum quality-gate contract for PBS language implementations.
|
||||
|
||||
The goal is practical and test-driven:
|
||||
|
||||
- keep quality measurable while language, lowering, and runtime integration are still evolving,
|
||||
- require only gates that can be proved with tests today,
|
||||
- avoid freezing speculative backend policy too early.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
This document defines:
|
||||
|
||||
- required unit-test gates for language frontend work,
|
||||
- required regression gate behavior for published language behavior,
|
||||
- and the boundary for runtime integration regression tests when a backend can emit executable artifacts.
|
||||
|
||||
This document does not define:
|
||||
|
||||
- runtime semantics or VM behavior taxonomy,
|
||||
- full artifact-level compatibility matrices,
|
||||
- one test harness layout,
|
||||
- or one repository structure for fixtures.
|
||||
|
||||
## 3. Authority and Precedence
|
||||
|
||||
Normative precedence:
|
||||
|
||||
1. `1. Language Charter.md`
|
||||
2. `2. Governance and Versioning.md`
|
||||
3. `3. Core Syntax Specification.md`
|
||||
4. `4. Static Semantics Specification.md`
|
||||
5. `12. Diagnostics Specification.md`
|
||||
6. `13. Lowering IRBackend Specification.md`
|
||||
7. `20. IRBackend to IRVM Lowering Specification.md`
|
||||
8. `21. IRVM Optimization Pipeline Specification.md`
|
||||
9. This document
|
||||
|
||||
This document must not weaken normative requirements from higher-precedence specs.
|
||||
|
||||
## 4. Normative Inputs
|
||||
|
||||
This document depends on:
|
||||
|
||||
- `3. Core Syntax Specification.md`
|
||||
- `4. Static Semantics Specification.md`
|
||||
- `12. Diagnostics Specification.md`
|
||||
- `13. Lowering IRBackend Specification.md`
|
||||
- `20. IRBackend to IRVM Lowering Specification.md`
|
||||
- `21. IRVM Optimization Pipeline Specification.md`
|
||||
- `17. Compatibility and Evolution Policy.md`
|
||||
|
||||
## 5. Quality Gate Model
|
||||
|
||||
PBS v1 uses two practical gate families.
|
||||
|
||||
### 5.1 Gate U: Language Unit Quality Gate (Required)
|
||||
|
||||
Every language frontend integrated into Studio/compiler must have passing unit tests that cover, at minimum:
|
||||
|
||||
1. lexer behavior (valid tokenization and deterministic lexical rejection),
|
||||
2. parser behavior (valid parse and deterministic syntax rejection),
|
||||
3. AST shape for representative valid programs,
|
||||
4. frontend-to-IR backend emission (`IRBackend`/`IRBackendFile`) for representative valid programs,
|
||||
5. diagnostics surface for required rejection cases (`code`, `severity`, and primary attribution).
|
||||
|
||||
This gate is mandatory for each language regardless of backend maturity.
|
||||
|
||||
### 5.2 Gate I: Integration Regression Gate With Runtime (Conditional)
|
||||
|
||||
When a language backend can emit executable artifacts (for example bytecode), integration regression tests must be introduced for that language.
|
||||
|
||||
At minimum, those tests must:
|
||||
|
||||
1. compile selected fixtures through the executable pipeline for that language,
|
||||
2. execute them against the runtime line declared for that backend,
|
||||
3. assert deterministic observable outcomes for supported fixtures,
|
||||
4. preserve fixed bugs and published supported examples as regression tests,
|
||||
5. and cover deterministic behavior across `LowerToVM`, `OptimizeIRVM`, and `EmitBytecode` boundaries when those stages are present.
|
||||
|
||||
Until executable artifact emission exists for a language, Gate I is explicitly `deferred` for that language and does not block frontend progress.
|
||||
|
||||
## 6. Regression Policy
|
||||
|
||||
Published supported behavior must remain regression-protected.
|
||||
|
||||
At minimum, each language corpus must retain:
|
||||
|
||||
1. bug reproductions for fixes that remain in supported scope,
|
||||
2. published supported examples,
|
||||
3. and core gate fixtures required to justify the language support state.
|
||||
|
||||
Regression grouping by language is expected; one shared monolithic corpus is not required.
|
||||
|
||||
## 7. Claims and Evidence
|
||||
|
||||
For PBS quality claims at this stage:
|
||||
|
||||
- a language may be considered `frontend-ready` only with Gate U passing,
|
||||
- a language may be considered `integration-ready` only with Gate U and Gate I passing where Gate I applies,
|
||||
- and a published support claim must map to executable evidence from these gates.
|
||||
- conformance-relevant FE/BE changes in executable backend paths MUST update `22. Backend Spec-to-Test Conformance Matrix.md` and pass matrix hard-gate policy checks.
|
||||
|
||||
The exact publication vocabulary is owned by `17. Compatibility and Evolution Policy.md`.
|
||||
|
||||
## 8. Explicit Deferrals
|
||||
|
||||
The following are intentionally deferred until backend/runtime contracts stabilize further:
|
||||
|
||||
- detailed artifact-level conformance taxonomy,
|
||||
- verifier-facing conformance taxonomy,
|
||||
- full compatibility matrix axes beyond the current minimum support table,
|
||||
- and one mandatory fixture repository layout.
|
||||
|
||||
## 9. Non-Goals
|
||||
|
||||
- Replacing normative language specs with tests.
|
||||
- Defining runtime semantics in this document.
|
||||
- Freezing backend architecture decisions prematurely.
|
||||
- Treating performance benchmarks as conformance gates.
|
||||
|
||||
## 10. Exit Criteria
|
||||
|
||||
This document is in a healthy state when:
|
||||
|
||||
1. every active language has an explicit Gate U status,
|
||||
2. languages with executable backends have an explicit Gate I status,
|
||||
3. published supported language behavior has regression evidence,
|
||||
4. and no support claim exists without mapped test evidence.
|
||||
Reference in New Issue
Block a user