Files
ai-engineering/docs/architecture.md
T
bquarkz 8f6cfa5d86 feat: import open Discussion Framework agendas into Works
Add deterministic preview and fingerprint-bound application for independent
DF sources under an AIE umbrella. Preserve complete agendas and original
evidence with source-scoped identities, validated receipts and safe retries.

Publish prepared Works atomically, require review for changed imports and
retain existing refinement without source edits or model calls.

Advertise protocol contract 2 while continuing to read contract-1 projects. Add JVM,
concurrency, recovery and native coverage; document usage and complete
WRK-0007 with verification evidence.
2026-11-16 14:53:05 +00:00

280 lines
18 KiB
Markdown

# Architecture
## Technology selection
The project deliberately stays on a compatible maintenance pair: **Spring Boot
4.0.9 and Spring Modulith 2.0.9**. The published Modulith 2.0.9 POM declares
Boot autoconfigure 4.0.9. Java compilation and bytecode target are both **25**,
without preview features. Gradle **9.3.1** supports running on Java 25; its Wrapper
distribution is pinned with SHA-256 verification.
The local development JDK is pinned through `.sdkmanrc` to SDKMAN candidate
`25.5.5.0+3-graalce` (GraalVM CE 25.5.5.0.3, OpenJDK 25.0.5.0.3). This
[GraalVM innovation release](https://www.graalvm.org/release-notes/25.5/) still uses
Java 25. It provides both JVM and Native Image tooling without changing compilation
target or application dependencies. Use `sdk env` for project-scoped activation;
the Gradle Wrapper remains responsible for the Gradle version.
Primary compatibility references checked during implementation:
- [Spring Boot 4.0 requirements](https://docs.spring.io/spring-boot/4.0/system-requirements.html): Java 25, Gradle 9 and GraalVM 25 are supported; Native Build Tools 0.11.5 is the documented version.
- [Published Modulith 2.0.9 dependencies](https://repo.maven.apache.org/maven2/org/springframework/modulith/spring-modulith-core/2.0.9/spring-modulith-core-2.0.9.pom): Boot 4.0.9 dependency.
- [Gradle Java compatibility](https://docs.gradle.org/9.3.1/userguide/compatibility.html): Java 25 support starts at Gradle 9.1.
- [Spring AOT Gradle integration](https://docs.spring.io/spring-boot/4.0/gradle-plugin/aot.html).
- [Picocli native configuration](https://picocli.info/#_graalvm_native_images): annotation-processor-generated reflection metadata.
- [GraalVM 25 release notes](https://www.graalvm.org/release-notes/JDK_25/): macOS x86_64 support was removed.
Picocli **4.7.7** handles argument validation, help and dispatch. Jackson 3 YAML/JSON
comes from Boot's BOM. JUnit is explicitly pinned to **5.13.4** because the requested
test baseline is JUnit 5; the Boot BOM otherwise selects a newer major generation.
AssertJ and Modulith/ArchUnit verify behavior and dependencies. There is no Spring AI,
Embabel, web starter, persistence server or workflow framework.
## Module boundaries
All packages below are under `net.intrepid.aiengineering`. A module's base package is
its explicit public API; implementation-only Git adapters live in `project.internal`.
`package-info.java` declares allowed dependencies. `ArchitectureTest` calls
`ApplicationModules.verify()` to enforce cycles, visibility and dependency rules.
| Module | Owns | May depend on |
|---|---|---|
| filesystem | Protocol codecs, safe paths, atomic file writes, locks, errors | none |
| project | Configuration, discovery, repository policy, read-only Git adapter | filesystem |
| work | Work identity/lifecycle, repository snapshots, references and versioned EPIC policy/graph | project, filesystem |
| discussion | Entries, topics, surveys, answer history and proposal dispositions | work, project, artifact, filesystem |
| decision | Draft, approval, revision and supersession records | work, filesystem |
| planning | Lightweight plans, dependencies, step results and revision history | work, decision, artifact, filesystem |
| legacyimport | DF preview, original agenda evidence, provenance and prepared Work publication | filesystem, project, work, discussion, artifact |
| artifact | Small evidence payloads and integrity metadata | work, filesystem |
| agent | Adapter contract, deterministic fixtures, routing and proposal application | project, work, discussion, decision, planning, artifact, filesystem |
| verification | Structural conformance, context, reviewed closure and deterministic EPIC coordination | all protocol modules |
| cli | Picocli commands, formats, exit codes | all APIs above |
Spring owns the process lifecycle and CLI bean, and supplies AOT processing. Protocol
services are ordinary Java classes with explicit path arguments and no retained Work
cache. This keeps direct tests and fresh instances simple. Infrastructure dependencies
remain inside services, outside immutable domain records. The small `GitAdapter` port
is internal to Project and can be replaced without changing Work contracts.
Context assembly, readiness and evidence-mapped closure belong to Verification because they combine independently persisted
modules and refuse to export structurally inconsistent state. Work does not depend
on Discussion, Decision or Artifact, avoiding cycles. It owns stable references only.
The engineering project is independent of repository count: a plain folder can use
the protocol with no Git dependency, a single Git repository can register the root
using `.`, and a larger workspace can register several child repositories. The same
configuration schema and services support each case; no separate project mode or
additional repository type is needed.
Work refinement uses an immutable update value that distinguishes omitted fields from
explicitly emptied lists. The domain preserves identity, lifecycle and references;
the filesystem service locks the project and replaces front matter while retaining
trailing notes. Discussion file input belongs to the CLI adapter: Picocli enforces
exclusive text sources and a bounded strict UTF-8 read supplies the existing domain
entry. Neither feature changes canonical record schemas or introduces new modules.
## Persistence and native execution
Writers take an OS file lock per project. Nested service calls on the same thread reuse
the outer lock; other threads/processes cannot interleave the proposal application.
Complete UTF-8 files are written to a sibling
temporary file, forced to disk and published atomically. New files use a hard link
to avoid overwriting an existing destination. Existing files use atomic replacement;
unsupported filesystems fail instead of quietly falling back to partial writes.
New Works, projects and artifact payload bundles are staged before directory rename.
See the protocol document for crash limits and recovery.
Jackson parses explicit records, rejects unknown fields, duplicate YAML keys and
unsupported schema versions. Native hints register those records; Picocli's annotation
processor provides command reflection metadata. The Spring AOT tasks are wired by the
native build plugin. No application arguments are passed through Spring's property
parser. The entry point explicitly chooses non-web mode and closes its context.
Discussion document version 2 separates typed YAML metadata from an opaque Markdown
body. The filesystem codec maps that body to the domain record's existing `text`
field; it does not infer structure from prose headings. The reader accepts legacy
YAML-only text as well. Explicit discussion migration backs up originals, preserves
semantic state and skips converted files. New decision narrative (`text`) and plan
description use the same Markdown-body codec; legacy draft DECs remain readable.
Work front matter remains canonical, with additive operational fields. Markdown
context renders discussion/decision prose without YAML escaping.
## Operational replacement baseline
The user expanded the original bootstrap scope to replace the existing discussion
workflow before introducing intelligent adapters or a legacy importer. Planning is
a concrete ninth module, not a custom workflow engine. A plan records editable scope,
dependencies and externally executed results. Starting/completing steps checks their
dependencies and accepted decisions. No repository command is executed by Planning.
Plan creation prepares deterministic draft-decision promotions through Decision,
validates every reference before publication and records SYSTEM_PLAN provenance
with the causal Plan ID and no fabricated human attestation. Existing accepted
decisions are unchanged. Decision exposes prepared promotion/recovery operations;
Planning coordinates them under the existing project lock. Ordinary failures roll
back the attempted writes; crash interruption remains subject to per-file atomicity
and structural validation. Verification checks promotion links against Plan history.
No adapter is invoked, and the Decision module has no dependency on Planning.
Plans/steps can be reopened and revised when observations change.
DEC approval is an explicit human-attestation command, with actor, reason and time.
Accepted records are immutable through the CLI; revision creates a new linked draft.
Acceptance supersedes the prior accepted DEC. Decision and plan changes preserve
the previous complete file under `.history`, linked by SHA-256. Hashes detect broken
chains; they are not signatures or protection against someone rewriting every file.
Cross-record supersession uses per-file atomic publication; validation diagnoses
incomplete links. There is no distributed transaction or automatic history repair.
Verification separates structural validity, mechanical readiness and human judgment.
PlanCoordinator consolidates valid Plan results without invoking an adapter, records
SYSTEM_PLANS receipts and invokes existing EPIC propagation. Planning owns content-bound
scope reviews of cancellations; Verification coordinates public mutations/reconciliation
without a reverse dependency. Reads expose the same pending reasons without writes.
Explicit reopening leaves time to change Plans. Legacy optional fields are omitted
to preserve historical digests. See [the contract](plan-consolidation.md).
For inspection-only Works without Plans, manual
readiness requires explicit criteria, resolved questions, reviewed DECs and terminal
plans. Completion records criterion-to-artifact mappings, a stable scope digest,
human approval, and canonical integration file digests or a reason no update is
needed. Canonical-file hashes are historical snapshots; later source evolution does
not invalidate an old delivery. Current completed Work records must still match
their frozen scope. Archive retains every file. Reopening preserves completion and
lifecycle history and permits new discussion/evidence.
Project owns provider-neutral entry resources copied into `.ai-engineering` by
`init`/`onboard`, including a machine-readable manifest. A managed root `AGENTS.md`
block points to those resources while preserving surrounding rules. Changed
instructions are backed up; repeated onboarding is idempotent. Context includes the
guide. This supports agents that inspect the project; it cannot force discovery by
an arbitrary agent. `status` and `next-actions` derive diagnostics from persisted
state without executing work or inventing human approval.
## Discussion refinement and adapter boundary
Discussion still owns refinement; DEC and Plan remain its downstream commitments.
Topics and surveys are functional records in Discussion. Legacy threads are derived
as implicit topics; reads never migrate files. Answers append explicit supersession
links. Archive/restore dispositions retain proposals and require policy/evidence.
There is no separate Idea entity. See [record contracts](discussion-slice.md).
Agent is the tenth module. `AgentAdapter` accepts immutable context/task/executor
values and returns a typed `AgentProposal`; it receives no filesystem Path, persistence
service or execution tools. Only two deterministic adapters are registered. General
selects an activity when omitted and provides fallback for unspecialized activities;
an explicit activity selects its configured executor. Classification in fixtures uses
task markers, not natural-language model interpretation.
AgentService validates the entire proposal before applying it through domain APIs.
It rechecks the source fingerprint under the project lock to reject concurrent stale
application. A RUN preserves source digest, selected executor, requester, task,
proposal, applied references and terminal/error state. Completed request IDs provide
idempotent replay of identical input. An interrupted APPLYING run blocks new runs
pending inspection. Publication is atomic per file, not a rollback transaction;
FAILED records and validation expose partial application for deliberate repair.
Verification reads run provenance and computes staleness from canonical semantic
sources, excluding run bookkeeping. State checks never invoke an adapter. A summary
is a recorded result; fresh context does not implicitly rerun it. Adapter output
cannot provide human survey answers, approve DECs, complete Works or execute commands.
Typed records and nested proposal values have Native Image reflection hints.
The initial slice did not add provider integration; the local CLI slice below extends it.
## Local provider CLI transport
PLN-0003 implements the human-selected Codex/Claude CLI transport for Discussion and
synthesis. The agent module retains the same public adapter boundary. Its internal
LocalProcess uses fixed ProcessBuilder vectors, bounded streams, virtual threads,
timeout/descendant termination and private temporary workspaces. It receives no
engineering root. Provider tool/configuration controls constrain proposals, while
OS/provider policy remains separate. The full contract and verified CLI references
are in [cli-adapters.md](cli-adapters.md).
Versioned wire schemas normalize provider output into existing AgentProposal values;
the framework assigns identity and performs ownership/policy preflight. Classification
can request clarify surveys. An explicit activity skips that provider call. Immutable
previous RUNs travel in context, but derived run bookkeeping stays out of semantic
source fingerprints. Only validated results publish canonical RUNs. Earlier transport
failures return safe diagnostics without mutating Work state or storing raw stderr.
Optional invocation provenance captures observed version, configured role/model,
duration and output digest. Absent optional fields are omitted to preserve older
fingerprints/digests. Setup collects/validates all answers before initialization and
does not invoke providers. No Node SDK, HTTP client framework or external service is
required by tests. Deterministic JDK subprocess fixtures use the production transport.
## EPIC coordination
PLN-0002 implements DEC-0003 within the existing modules. Work owns the EPIC marker,
definition, forest topology, revision history and explicit policy authorization.
Verification coordinates lifecycle writes and checks affected ancestors, using
Artifact for owned aggregate receipts. No extra module, broker, workflow engine or
agent invocation is needed. Low-level storage callers must reconcile explicitly;
public CLI/Verification lifecycle entry points propagate within the project lock.
Policy authority binds parent/important-child scopes and links, excluding execution
status. Important child completions and local readiness control RESULTS completion;
DECOMPOSITION remains a reviewed human delivery. System completions record their
origin, policy delegation and receipt, with humanApproved=false. Historical direct
human completions keep their existing digest/approval semantics.
Frozen EPIC scope digests include current important delivery records; optional
execution can continue independently. Deterministic serialization uses ordered maps
and canonical records across JVM/native processes. Automatic receipts exclude prior
receipts from their input key, allowing reuse after interrupted publication.
Reconciliation derives state from files; no mutable in-memory index is authoritative.
Details and explicit reopening/reauthorization rules are in [epics.md](epics.md).
## Compatibility inspection
The project module owns the protocol contract revision independently of the CLI
release and manifest schema. Filesystem owns the supported storage-format catalog;
existing typed readers continue interpreting older representations in memory.
Verification's compatibility service inspects actual metadata and typed content,
including record revisions and system completion receipts. The CLI exposes a full
read-only report through `project compatibility` and a summary in `status`.
This adds no automatic conversion or blanket normalization. Onboarding refreshes
entry instructions with backups and refuses to overwrite an unsupported contract.
Frozen prior-release fixtures protect historical approvals, completion provenance,
scope hashes and bytes. See [compatibility](compatibility.md) for the public contract
and requirements before a future incompatible writer or migration is introduced.
## Earlier framework review and migration
Both implementations were inspected read-only:
- The user-identified Python repository at
`/Users/niltonconstantino/personal/workspace.personal/intrepid/discussion-framework`:
README, model/locking/index persistence and structural validation.
- The sibling Java `discussion-framework`: `DiscussionOrchestrator`,
`WorkflowValidator`, `WorkPacket`, `ArtifactDocumentParser` and
`FilesystemDiscussionRepository` concepts and implementations.
Reused concepts: deterministic context as the handoff surface, stable IDs, explicit
references, file locks, atomic replacement, duplicate/reference checks and separating
content from mechanical CLI state management. No Python code or dependency was copied.
The old generic Markdown inference was replaced by typed, versioned front matter;
semantic sections are explicit and cannot silently disappear from context parsing.
No legacy HTTP adapter, NDJSON global index, accepted-agenda gate, automatic housekeeping
deletion or Lesson model was retained. Work is the project-scoped aggregate; Discussion
is refinement within a Work. Source repositories were not changed.
The deterministic DF importer reads each independent source's NDJSON index and open
AGDs. It creates one Work per source+DSC, retaining agenda Markdown as refinement
NOTEs and original bytes as evidence. A versioned owned receipt links source IDs,
DSCs, AGDs, target descriptions and generated AIE references; existing Work schemas
and historical data remain unchanged. Receipt validation is available without the
source directory and compatibility inspection recognizes the receipt format.
Work, Discussion and Artifact expose prepared publication helpers using their normal
formats. The importer seeds a complete hidden Work folder before the normal atomic
directory publication and monotonic ID reservation. Revalidation under the destination
lock binds application to the preview fingerprint; immutable receipts make retry
recognize already published Works. A multi-Work batch is deliberately not a global
transaction. No legacy decision/Plan/Lesson is imported, source files are untouched
and no agent adapter is involved. See [the import contract](discussion-import.md).
Legacy Lessons remain candidates for manual integration into canonical docs.