Files
ai-engineering/docs/experiments.md
T
Nilton Constantino c84bb8d36f feat: store discussion prose in Markdown bodies
Add versioned Markdown discussion documents with legacy YAML reading and explicit migration that backs up originals and preserves semantic history.

Render readable discussion and decision prose in Markdown context, document compatibility, and extend native smoke coverage for migration.

Validated with 29 passing tests, macOS ARM64 native smoke, and context-preserving migration of the Zelda pilot Work.
2026-11-11 19:50:36 +00:00

11 KiB

Initial experiments

Execution host: macOS ARM64, 2026-11-11. JVM: Amazon Corretto 25.0.4+7. Native toolchain: Oracle GraalVM for JDK 25.0.4+7.1 and Apple C compiler. Resolved Jackson YAML/Databind: 3.1.7 (Boot BOM). No Python, Node, model API, database or external runtime service was used.

H1 — One non-web modular application is enough

Observation: the packaged Spring Boot application prints Picocli help/version and terminates. Automated process tests enforce a 30-second termination limit. There are no web/server dependencies. Modulith verifies eight modules and their declared dependency graph.

Result: verified by ArchitectureTest, CLI smoke tests and independent JVM processes. Spring handles process lifecycle and AOT; the portable protocol stays plain Java.

H2 — Files alone preserve meaningful engineering state

The main integration fixture initializes an engineering project with two independent Git repositories (runtime, studio), creates a Work referencing both, supplies goal, constraints and acceptance criteria, appends a question/conclusion, persists a draft DEC and attaches evidence. A fresh CLI instance invoked from studio/src/editor reconstructs exactly the same exported context.

ProcessContinuityTest repeats the scenario through separate java -jar processes. Each source repository contains a real fixture commit. The fresh context includes both revision snapshots, author, discussion, DRAFT status and evidence SHA-256. JSON exports from independent processes compare byte-for-byte. Project validation succeeds.

Result: verified without shared application memory or conversation history. The native smoke script reproduces the same scenario with a separate native process per command.

H3 — Structural safety and integrity are testable without a model

Verified cases: multiple Works, separate artifact workspaces, missing repositories, unknown configuration fields/schema, malformed YAML and duplicate YAML keys, traversal and symlink attempts, malformed/missing/duplicate Work identifiers, missing decision files and evidence payloads, orphan records, required directories, SHA-256 corruption, explicit question replies, reversible status, default read-only access, immutable models, atomic no-overwrite writes and persistent ID reservations.

A real submodule fixture additionally verifies registration without rewriting .gitmodules. It initially exposed a path equality bug between macOS /var and /private/var. Comparing both real paths fixed it. A numeric-enum negative fixture initially failed to alter Jackson's quoted YAML; the corrected fixture now verifies that ordinal lifecycle values are rejected.

Result: final clean build has 20 tests, 0 failures, 0 errors, 0 skipped. The first test-first run intentionally failed because implementation classes did not exist. Intermediate compilation caught interruption handling and a Jackson 2 feature constant moved to Jackson 3's EnumFeature; both were corrected. These failures are resolved in the final tested source.

Commands and evidence

./gradlew clean test bootJar --console=plain
./gradlew dependencies --configuration runtimeClasspath --console=plain
JAVA_HOME=<graalvm25-home> GRAALVM_HOME=<graalvm25-home> \
  ./gradlew nativeCompile nativeDistZip nativeDistTar \
  -PtargetPlatform=macos-arm64 --console=plain

The clean test/JAR build passed. Generated test reports are under build/reports/tests/test; raw JUnit XML is under build/test-results/test. Runtime dependency inspection confirmed the pinned Spring pair and Jackson 3.1.7. An initial Native Image compilation succeeded, but only the final rebuilt binary's execution results count as native functionality verification.

Native result: compiled and executed successfully on macOS ARM64. The initial verified Native Image build completed in approximately 1m 13s (Gradle invocation: 1m 23s). ai-engineering is a Mach-O ARM64 executable. ZIP and tar.gz archives were generated with executable permissions and inspected.

The native continuity smoke passed with no JDK on the configured execution path:

JAVA_HOME=/nonexistent PATH=/usr/bin:/bin:/usr/sbin:/sbin AIE_KEEP_FIXTURE=1 \
  bash scripts/native-smoke.sh "$PWD/build/native/nativeCompile/ai-engineering"

Observed output: Native multi-repository continuity smoke passed. Every CLI command in the script starts a separate process. Help/version, exit code 2 for invalid input, both repository registrations, Work creation, discussion, evidence, DEC draft, nested discovery, JSON byte equality, Markdown export and final structural validation all succeeded. The engineering fixture root had no .git directory.

The retained local demonstration fixture is at /var/folders/0t/tfsqc14j5wbgh9h4s4dytwb00000gq/T/ai-engineering-smoke.ziQUkw. Inspect .ai-engineering/works/WRK-0001-improve-scene-editor/, before.json, after.json, context.md and validation.json there. The JSON validation report contains valid: true and an empty issue list. Temporary fixtures are not required for future use; the committed script can recreate the experiment.

Nonblocking Native Image build warnings: Picocli's generated proxy metadata uses a deprecated configuration resource option; SLF4J reports no provider during image analysis. Neither prevented compilation or the full native execution scenario. An earlier exploratory native invocation was terminated with exit 137 during a clean rebuild; it was not treated as a functional pass. The final rebuilt binary passed the full smoke scenario above.

H4 — SDKMAN GraalVM Community toolchain

The local project JDK is now pinned in .sdkmanrc to SDKMAN candidate 25.5.5.0+3-graalce. Direct inspection identifies GraalVM Community Edition 25.5.5.0.3+1.1, based on OpenJDK 25.0.5.0.3, with Native Image included. Gradle 9.3.1 recognizes it as both launcher and daemon JVM. A clean Java/AOT/JAR build and all 20 tests passed again using this installation. The language and bytecode target remain Java 25. CI native builds select the Community 25.5 series; CI itself has not been executed locally.

Native compilation with Native Build Tools 0.11.5 also succeeded (image generation approximately 1m 8s; complete clean test/JAR/native invocation 1m 40s). The full native continuity smoke passed with the CE-built executable, including project validation and byte-identical context exports across processes. An isolated copy containing only the executable passed the same scenario with both JAVA_HOME and GRAALVM_HOME set to nonexistent paths and no JDK on the configured PATH. Native Image also emitted optional desktop libraries beside the build output; the implemented CLI commands were verified without those libraries present beside the isolated executable.

The compiler reports nonblocking deprecation warnings for --no-fallback, FallbackThreshold and Picocli's proxy configuration. No application dependency or native plugin upgrade was needed for the tested functionality.

Reproduction after sdk env:

export GRAALVM_HOME="$JAVA_HOME"
./gradlew clean test bootJar nativeCompile --console=plain
JAVA_HOME=/nonexistent GRAALVM_HOME=/nonexistent PATH=/usr/bin:/bin:/usr/sbin:/sbin \
  bash scripts/native-smoke.sh "$PWD/build/native/nativeCompile/ai-engineering"

H5 — Simple projects and a single root repository

A simple project can initialize .ai-engineering directly beside its existing source files. Repository registration is optional; a project with no registered repositories has a complete Work/discussion/decision/artifact workflow and no Git revision snapshots. The separate-process test runs those commands with no Git on PATH, reconstructs the same context, validates the project and confirms no .git or source changes appear.

For projects already using one Git repository, repo add app --path . now registers the root repository. A test-first run reproduced the earlier root-path rejection; removing that restriction preserves traversal, symlink and .ai-engineering exclusions. Regression tests confirm read-only defaults, nested discovery, deterministic context, duplicate-path rejection and unchanged Git configuration/source content. Existing multi-repository and real-submodule fixtures still pass.

Result: 23 tests, 0 failures, 0 errors, 0 skipped on the SDKMAN GraalVM CE toolchain. The native smoke script now exercises all three layouts: no Git, a single root Git repository, and multiple child Git repositories. The README starts with the simple project workflow. These changes do not require a new configuration schema or project mode.

Native compilation succeeded on macOS ARM64. With JAVA_HOME and GRAALVM_HOME pointing to nonexistent paths, the expanded smoke passed both the simple-project scenarios (including no Git on PATH) and the multi-repository scenario.

H6 — Discussion readability and compatible migration

The Zelda pilot produced a long suggestion whose YAML string was hard to inspect in an editor. New discussions now store metadata in front matter with documentVersion: 2 and canonical prose in the Markdown body. Context and discussion-list Markdown output render prose directly. Work/draft DEC storage and JSON record fields are unchanged.

Six new tests verify exact Unicode/Markdown/whitespace round trips, legacy read-only loading, context-preserving migration and backups, idempotence, refusal of legacy trailing notes, ambiguous/unknown formats, and CRLF front matter. The complete suite passed with 29 tests, 0 failures, 0 errors, 0 skipped, including Modulith.

A copy of the actual Zelda Work was staged at /tmp/aie-zelda-markdown-preview.z3YttK. Both legacy discussion entries converted successfully, with originals backed up. JSON and Markdown contexts before/after migration compared byte-for-byte equal. The long DSC-0002 proposal can now be read directly as ordinary paragraphs and lists, without escaped newlines or split words. The source project was not modified during this rehearsal.

The native smoke additionally constructs a legacy entry, migrates it, verifies the new format, repeats migration to confirm no-op behavior, and compares exported context. The complete native smoke passed on macOS ARM64, including migration without Git on PATH and the existing single-/multi-repository layouts. The converted Zelda copy also validated with the native binary; repeating migration reported zero conversions.

After the rehearsal, both discussions in the original Zelda Work were explicitly migrated with the native CLI. Original files were retained under the reported .local backup directory. JSON context, Markdown context and work.md compared byte-for-byte equal before/after conversion; project validation returned valid: true, no issues.

Limits and next experiment

CI workflows are provided but have not been executed from this local workspace. Linux x86_64 and Windows x86_64 binaries are not locally verified. GraalVM 25 removed macOS x86_64 support, so that release target has no claimed native artifact.

Filesystem atomicity is per file/directory publication, not a transaction across the child record and parent manifest. Orphans after interruption are detected, not automatically repaired. Manual concurrent editors and adversarial path replacement require external coordination/sandboxing. Artifact attachment is limited to small files, and no full provenance/environment capture or agent intelligence is implemented.

Next recommended experiment: define a deterministic AgentAdapter test contract and demonstrate two fake adapters continuing the same Work. Keep that work in Milestone 2; no adapter implementation is included here.