prometeu-studio/docs/general/specs/13. Conformance Test Specification.md

PBS Conformance Test Specification

Status: Draft v1 (Quality-Gate Baseline)
Applies to: executable quality gates for PBS language frontends and language-level regression evidence

1. Purpose

This document defines the minimum quality-gate contract for PBS language implementations.

The goal is practical and test-driven:

• keep quality measurable while language, lowering, and runtime integration are still evolving,
• require only gates that can be proved with tests today,
• avoid freezing speculative backend policy too early.

2. Scope

This document defines:

• required unit-test gates for language frontend work,
• required regression gate behavior for published language behavior,
• and the boundary for runtime integration regression tests when a backend can emit executable artifacts.

This document does not define:

• runtime semantics or VM behavior taxonomy,
• full artifact-level compatibility matrices,
• one test harness layout,
• or one repository structure for fixtures.

3. Authority and Precedence

Normative precedence:

1. 1. Language Charter.md
2. 2. Governance and Versioning.md
3. 3. Core Syntax Specification.md
4. 4. Static Semantics Specification.md
5. 12. Diagnostics Specification.md
6. 13. Lowering IRBackend Specification.md
7. This document

This document must not weaken normative requirements from higher-precedence specs.

4. Normative Inputs

This document depends on:

• 3. Core Syntax Specification.md
• 4. Static Semantics Specification.md
• 12. Diagnostics Specification.md
• 13. Lowering IRBackend Specification.md
• 17. Compatibility and Evolution Policy.md

5. Quality Gate Model

PBS v1 uses two practical gate families.

5.1 Gate U: Language Unit Quality Gate (Required)

Every language frontend integrated into Studio/compiler must have passing unit tests that cover, at minimum:

1. lexer behavior (valid tokenization and deterministic lexical rejection),
2. parser behavior (valid parse and deterministic syntax rejection),
3. AST shape for representative valid programs,
4. frontend-to-IR backend emission (IRBackend/IRBackendFile) for representative valid programs,
5. diagnostics surface for required rejection cases (code, severity, and primary attribution).

This gate is mandatory for each language regardless of backend maturity.

5.2 Gate I: Integration Regression Gate With Runtime (Conditional)

When a language backend can emit executable artifacts (for example bytecode), integration regression tests must be introduced for that language.

At minimum, those tests must:

1. compile selected fixtures through the executable pipeline for that language,
2. execute them against the runtime line declared for that backend,
3. assert deterministic observable outcomes for supported fixtures,
4. and preserve fixed bugs and published supported examples as regression tests.

Until executable artifact emission exists for a language, Gate I is explicitly deferred for that language and does not block frontend progress.

6. Regression Policy

Published supported behavior must remain regression-protected.

At minimum, each language corpus must retain:

1. bug reproductions for fixes that remain in supported scope,
2. published supported examples,
3. and core gate fixtures required to justify the language support state.

Regression grouping by language is expected; one shared monolithic corpus is not required.

7. Claims and Evidence

For PBS quality claims at this stage:

• a language may be considered frontend-ready only with Gate U passing,
• a language may be considered integration-ready only with Gate U and Gate I passing where Gate I applies,
• and a published support claim must map to executable evidence from these gates.

The exact publication vocabulary is owned by 17. Compatibility and Evolution Policy.md.

8. Explicit Deferrals

The following are intentionally deferred until backend/runtime contracts stabilize further:

• detailed artifact-level conformance taxonomy,
• verifier-facing conformance taxonomy,
• full compatibility matrix axes beyond the current minimum support table,
• and one mandatory fixture repository layout.

9. Non-Goals

• Replacing normative language specs with tests.
• Defining runtime semantics in this document.
• Freezing backend architecture decisions prematurely.
• Treating performance benchmarks as conformance gates.

10. Exit Criteria

This document is in a healthy state when:

1. every active language has an explicit Gate U status,
2. languages with executable backends have an explicit Gate I status,
3. published supported language behavior has regression evidence,
4. and no support claim exists without mapped test evidence.