Files
ai-engineering/docs/architecture.md
T
Nilton Constantino e7e8d95415
Test / Tests and coverage occurred while executing withChecks step.
Intrepid/AI Engineering/pipeline/head There was a failure building this commit
build / jvm (push) Canceled after 0s
build / native (macos-15, macos-arm64) (push) Canceled after 0s
build / native (ubuntu-24.04, linux-x86_64) (push) Canceled after 0s
build / native (windows-2025, windows-x86_64) (push) Canceled after 0s
feat: add Work guidance and adaptive Plan experiments
Persist explicit human discussion choices, DEC requests and approval policies. Add optional evidence-backed Plan experiments and align context, adapters and next actions under protocol contract 3.

Include WRK-000B decisions, plan history and verification evidence: 167 JVM tests, 25 focused checks, Jenkins fixtures and native smoke coverage.
2026-11-16 17:33:24 +00:00

19 KiB

Architecture

Technology selection

The project deliberately stays on a compatible maintenance pair: Spring Boot 4.0.9 and Spring Modulith 2.0.9. The published Modulith 2.0.9 POM declares Boot autoconfigure 4.0.9. Java compilation and bytecode target are both 25, without preview features. Gradle 9.3.1 supports running on Java 25; its Wrapper distribution is pinned with SHA-256 verification.

The local development JDK is pinned through .sdkmanrc to SDKMAN candidate 25.5.5.0+3-graalce (GraalVM CE 25.5.5.0.3, OpenJDK 25.0.5.0.3). This GraalVM innovation release still uses Java 25. It provides both JVM and Native Image tooling without changing compilation target or application dependencies. Use sdk env for project-scoped activation; the Gradle Wrapper remains responsible for the Gradle version.

Primary compatibility references checked during implementation:

Picocli 4.7.7 handles argument validation, help and dispatch. Jackson 3 YAML/JSON comes from Boot's BOM. JUnit is explicitly pinned to 5.13.4 because the requested test baseline is JUnit 5; the Boot BOM otherwise selects a newer major generation. AssertJ and Modulith/ArchUnit verify behavior and dependencies. There is no Spring AI, Embabel, web starter, persistence server or workflow framework.

Module boundaries

All packages below are under net.intrepid.aiengineering. A module's base package is its explicit public API; implementation-only Git adapters live in project.internal. package-info.java declares allowed dependencies. ArchitectureTest calls ApplicationModules.verify() to enforce cycles, visibility and dependency rules.

Module Owns May depend on
filesystem Protocol codecs, safe paths, atomic file writes, locks, errors none
project Configuration, discovery, repository policy, read-only Git adapter filesystem
work Work identity/lifecycle, human guidance/history, repository snapshots, references and EPIC policy/graph project, filesystem
discussion Entries, topics, surveys, answer history and proposal dispositions work, project, artifact, filesystem
decision Draft, approval, revision and supersession records work, filesystem
planning Plans, dependencies, step results, experiments and revision history work, decision, discussion, artifact, filesystem
legacyimport DF preview, original agenda evidence, provenance and prepared Work publication filesystem, project, work, discussion, artifact
artifact Small evidence payloads and integrity metadata work, filesystem
agent Adapter contract, deterministic fixtures, routing and proposal application project, work, discussion, decision, planning, artifact, filesystem
verification Structural conformance, context, reviewed closure and deterministic EPIC coordination all protocol modules
cli Picocli commands, formats, exit codes all APIs above

Spring owns the process lifecycle and CLI bean, and supplies AOT processing. Protocol services are ordinary Java classes with explicit path arguments and no retained Work cache. This keeps direct tests and fresh instances simple. Infrastructure dependencies remain inside services, outside immutable domain records. The small GitAdapter port is internal to Project and can be replaced without changing Work contracts.

Context assembly, readiness and evidence-mapped closure belong to Verification because they combine independently persisted modules and refuse to export structurally inconsistent state. Work does not depend on Discussion, Decision or Artifact, avoiding cycles. It owns stable references only.

The engineering project is independent of repository count: a plain folder can use the protocol with no Git dependency, a single Git repository can register the root using ., and a larger workspace can register several child repositories. The same configuration schema and services support each case; no separate project mode or additional repository type is needed.

Work refinement uses an immutable update value that distinguishes omitted fields from explicitly emptied lists. The domain preserves identity, lifecycle and references; the filesystem service locks the project and replaces front matter while retaining trailing notes. Discussion file input belongs to the CLI adapter: Picocli enforces exclusive text sources and a bounded strict UTF-8 read supplies the existing domain entry. Neither feature changes canonical record schemas or introduces new modules.

Persistence and native execution

Writers take an OS file lock per project. Nested service calls on the same thread reuse the outer lock; other threads/processes cannot interleave the proposal application. Complete UTF-8 files are written to a sibling temporary file, forced to disk and published atomically. New files use a hard link to avoid overwriting an existing destination. Existing files use atomic replacement; unsupported filesystems fail instead of quietly falling back to partial writes. New Works, projects and artifact payload bundles are staged before directory rename. See the protocol document for crash limits and recovery.

Jackson parses explicit records, rejects unknown fields, duplicate YAML keys and unsupported schema versions. Native hints register those records; Picocli's annotation processor provides command reflection metadata. The Spring AOT tasks are wired by the native build plugin. No application arguments are passed through Spring's property parser. The entry point explicitly chooses non-web mode and closes its context.

Discussion document version 2 separates typed YAML metadata from an opaque Markdown body. The filesystem codec maps that body to the domain record's existing text field; it does not infer structure from prose headings. The reader accepts legacy YAML-only text as well. Explicit discussion migration backs up originals, preserves semantic state and skips converted files. New decision narrative (text) and plan description use the same Markdown-body codec; legacy draft DECs remain readable. Work front matter remains canonical, with additive operational fields. Markdown context renders discussion/decision prose without YAML escaping.

Operational replacement baseline

The user expanded the original bootstrap scope to replace the existing discussion workflow before introducing intelligent adapters or a legacy importer. Planning is a concrete ninth module, not a custom workflow engine. A plan records editable scope, dependencies and externally executed results. Starting/completing steps checks their dependencies and accepted decisions. No repository command is executed by Planning.

Work owns optional human guidance and its choice/request history. New contract-3 Works begin without an inferred discussion mode and require human DEC approval. Decision and Planning enforce that authority; explicit Work-wide waiver retains system promotion provenance. Planning owns optional experiments inside its existing revision chain and depends on Discussion to check unresolved planning commitments. Verification and local CLI adapters expose the applicable conduct and OODA guidance. Legacy absent fields are omitted to preserve historical digests.

Plan creation prepares eligible deterministic draft-decision promotions through Decision, validates every reference before publication and records SYSTEM_PLAN provenance with the causal Plan ID and no fabricated human attestation. Existing accepted decisions are unchanged. Decision exposes prepared promotion/recovery operations; Planning coordinates them under the existing project lock. Ordinary failures roll back the attempted writes; crash interruption remains subject to per-file atomicity and structural validation. Verification checks promotion links against Plan history. No adapter is invoked, and the Decision module has no dependency on Planning. Plans/steps can be reopened and revised when observations change.

DEC approval is an explicit human-attestation command, with actor, reason and time. Accepted records are immutable through the CLI; revision creates a new linked draft. Acceptance supersedes the prior accepted DEC. Decision and plan changes preserve the previous complete file under .history, linked by SHA-256. Hashes detect broken chains; they are not signatures or protection against someone rewriting every file. Cross-record supersession uses per-file atomic publication; validation diagnoses incomplete links. There is no distributed transaction or automatic history repair.

Verification separates structural validity, mechanical readiness and human judgment. PlanCoordinator consolidates valid Plan results without invoking an adapter, records SYSTEM_PLANS receipts and invokes existing EPIC propagation. Planning owns content-bound scope reviews of cancellations; Verification coordinates public mutations/reconciliation without a reverse dependency. Reads expose the same pending reasons without writes. Explicit reopening leaves time to change Plans. Legacy optional fields are omitted to preserve historical digests. See the contract.

For inspection-only Works without Plans, manual readiness requires explicit criteria, resolved questions, reviewed DECs and terminal plans. Completion records criterion-to-artifact mappings, a stable scope digest, human approval, and canonical integration file digests or a reason no update is needed. Canonical-file hashes are historical snapshots; later source evolution does not invalidate an old delivery. Current completed Work records must still match their frozen scope. Archive retains every file. Reopening preserves completion and lifecycle history and permits new discussion/evidence.

Project owns provider-neutral entry resources copied into .ai-engineering by init/onboard, including a machine-readable manifest. A managed root AGENTS.md block points to those resources while preserving surrounding rules. Changed instructions are backed up; repeated onboarding is idempotent. Context includes the guide. This supports agents that inspect the project; it cannot force discovery by an arbitrary agent. status and next-actions derive diagnostics from persisted state without executing work or inventing human approval.

Discussion refinement and adapter boundary

Discussion still owns refinement; DEC and Plan remain its downstream commitments. Topics and surveys are functional records in Discussion. Legacy threads are derived as implicit topics; reads never migrate files. Answers append explicit supersession links. Archive/restore dispositions retain proposals and require policy/evidence. There is no separate Idea entity. See record contracts.

Agent is the tenth module. AgentAdapter accepts immutable context/task/executor values and returns a typed AgentProposal; it receives no filesystem Path, persistence service or execution tools. Only two deterministic adapters are registered. General selects an activity when omitted and provides fallback for unspecialized activities; an explicit activity selects its configured executor. Classification in fixtures uses task markers, not natural-language model interpretation.

AgentService validates the entire proposal before applying it through domain APIs. It rechecks the source fingerprint under the project lock to reject concurrent stale application. A RUN preserves source digest, selected executor, requester, task, proposal, applied references and terminal/error state. Completed request IDs provide idempotent replay of identical input. An interrupted APPLYING run blocks new runs pending inspection. Publication is atomic per file, not a rollback transaction; FAILED records and validation expose partial application for deliberate repair.

Verification reads run provenance and computes staleness from canonical semantic sources, excluding run bookkeeping. State checks never invoke an adapter. A summary is a recorded result; fresh context does not implicitly rerun it. Adapter output cannot provide human survey answers, approve DECs, complete Works or execute commands. Typed records and nested proposal values have Native Image reflection hints. The initial slice did not add provider integration; the local CLI slice below extends it.

Local provider CLI transport

PLN-0003 implements the human-selected Codex/Claude CLI transport for Discussion and synthesis. The agent module retains the same public adapter boundary. Its internal LocalProcess uses fixed ProcessBuilder vectors, bounded streams, virtual threads, timeout/descendant termination and private temporary workspaces. It receives no engineering root. Provider tool/configuration controls constrain proposals, while OS/provider policy remains separate. The full contract and verified CLI references are in cli-adapters.md.

Versioned wire schemas normalize provider output into existing AgentProposal values; the framework assigns identity and performs ownership/policy preflight. Classification can request clarify surveys. An explicit activity skips that provider call. Immutable previous RUNs travel in context, but derived run bookkeeping stays out of semantic source fingerprints. Only validated results publish canonical RUNs. Earlier transport failures return safe diagnostics without mutating Work state or storing raw stderr.

Optional invocation provenance captures observed version, configured role/model, duration and output digest. Absent optional fields are omitted to preserve older fingerprints/digests. Setup collects/validates all answers before initialization and does not invoke providers. No Node SDK, HTTP client framework or external service is required by tests. Deterministic JDK subprocess fixtures use the production transport.

EPIC coordination

PLN-0002 implements DEC-0003 within the existing modules. Work owns the EPIC marker, definition, forest topology, revision history and explicit policy authorization. Verification coordinates lifecycle writes and checks affected ancestors, using Artifact for owned aggregate receipts. No extra module, broker, workflow engine or agent invocation is needed. Low-level storage callers must reconcile explicitly; public CLI/Verification lifecycle entry points propagate within the project lock.

Policy authority binds parent/important-child scopes and links, excluding execution status. Important child completions and local readiness control RESULTS completion; DECOMPOSITION remains a reviewed human delivery. System completions record their origin, policy delegation and receipt, with humanApproved=false. Historical direct human completions keep their existing digest/approval semantics.

Frozen EPIC scope digests include current important delivery records; optional execution can continue independently. Deterministic serialization uses ordered maps and canonical records across JVM/native processes. Automatic receipts exclude prior receipts from their input key, allowing reuse after interrupted publication. Reconciliation derives state from files; no mutable in-memory index is authoritative. Details and explicit reopening/reauthorization rules are in epics.md.

Compatibility inspection

The project module owns the protocol contract revision independently of the CLI release and manifest schema. Filesystem owns the supported storage-format catalog; existing typed readers continue interpreting older representations in memory. Verification's compatibility service inspects actual metadata and typed content, including record revisions and system completion receipts. The CLI exposes a full read-only report through project compatibility and a summary in status.

This adds no automatic conversion or blanket normalization. Onboarding refreshes entry instructions with backups and refuses to overwrite an unsupported contract. Frozen prior-release fixtures protect historical approvals, completion provenance, scope hashes and bytes. See compatibility for the public contract and requirements before a future incompatible writer or migration is introduced.

Earlier framework review and migration

Both implementations were inspected read-only:

  • The user-identified Python repository at /Users/niltonconstantino/personal/workspace.personal/intrepid/discussion-framework: README, model/locking/index persistence and structural validation.
  • The sibling Java discussion-framework: DiscussionOrchestrator, WorkflowValidator, WorkPacket, ArtifactDocumentParser and FilesystemDiscussionRepository concepts and implementations.

Reused concepts: deterministic context as the handoff surface, stable IDs, explicit references, file locks, atomic replacement, duplicate/reference checks and separating content from mechanical CLI state management. No Python code or dependency was copied. The old generic Markdown inference was replaced by typed, versioned front matter; semantic sections are explicit and cannot silently disappear from context parsing.

No legacy HTTP adapter, NDJSON global index, accepted-agenda gate, automatic housekeeping deletion or Lesson model was retained. Work is the project-scoped aggregate; Discussion is refinement within a Work. Source repositories were not changed.

The deterministic DF importer reads each independent source's NDJSON index and open AGDs. It creates one Work per source+DSC, retaining agenda Markdown as refinement NOTEs and original bytes as evidence. A versioned owned receipt links source IDs, DSCs, AGDs, target descriptions and generated AIE references; existing Work schemas and historical data remain unchanged. Receipt validation is available without the source directory and compatibility inspection recognizes the receipt format.

Work, Discussion and Artifact expose prepared publication helpers using their normal formats. The importer seeds a complete hidden Work folder before the normal atomic directory publication and monotonic ID reservation. Revalidation under the destination lock binds application to the preview fingerprint; immutable receipts make retry recognize already published Works. A multi-Work batch is deliberately not a global transaction. No legacy decision/Plan/Lesson is imported, source files are untouched and no agent adapter is involved. See the import contract. Legacy Lessons remain candidates for manual integration into canonical docs.