docs: consolidate Studio AI-first acceptance and lessons
Test / Build occurred while executing withChecks step.
Intrepid/Prometeu/Studio/pipeline/pr-master There was a failure building this commit
Intrepid/Prometeu/Studio/pipeline/head There was a failure building this commit

This commit is contained in:
2026-09-29 15:55:09 +01:00
parent 4c3597a2fb
commit 546c6d1ee5
3 changed files with 320 additions and 3 deletions
+3
View File
@@ -139,6 +139,9 @@ correctly returns `NO_FRAME_AVAILABLE`; it is not a runtime defect. Noninteracti
clients must also provide an approval path for MCP work calls; the acceptance used
Codex's approval review with a project-scoped workspace-write sandbox.
The completed baseline's architecture, acceptance evidence and post-implementation
follow-ups are consolidated in [LSN-0076](discussion/lessons/DSC-0068-studio-headless-runtime-ai-first/LSN-0076-ai-first-studio-through-shared-authority-and-headless-evidence.md).
## Requirements
- JDK 21 or newer recommended for local development
@@ -0,0 +1,303 @@
---
id: LSN-0076
ticket: studio-headless-runtime-ai-first
title: AI-first Studio Through Shared Authority and Headless Evidence
created: 2026-09-29
tags: [studio, runtime, headless, ai, mcp, automation]
---
## Context
Owner: **studio**, with runtime preparation in `../runtime` and language-owned PBS
documentation/LSP integration. DSC-0068, accepted DEC-0060 and PLN-0139 delivered
28 ordered steps: runtime (01–07), DEV/shared services (08–14), MCP (15–21), and
real integration/acceptance (22–28). Those workflow identifiers are historical
references; maintained contracts live in the specs linked below.
The original headless host was an in-process runtime API. Studio's human Debug
was useful, but neither a public automation protocol nor a safe authority boundary
for another editor. The goal was an external coding agent able to learn PBS,
edit a game, build/run it, see its pixels and iterate while the developer watches.
A complete gameplay library or embedded AI chat was not a prerequisite.
## Key Decisions
### Separate ownership, not parallel implementations
The external coding client owns conversation, model/provider, native file edits
and budget. Studio owns project authority, build orchestration, process supervision
and the shared observation model. Runtime owns execution and framebuffer semantics.
The language frontend/SDK owns its references; the existing project LSP owns semantic
queries. DEV and MCP consume the same neutral services.
```text
External coding agent -- local MCP HTTP --> Studio API/Core <-- DEV (JavaFX)
|
compiler/packer + existing LSP
|
supervised runtime JSONL/stdio
|
canonical PNG + frame identity
```
The configured installation provides `prometeu headless --stdio`. UTF-8 JSONL is
versioned, bounded and protocol-only on stdout; logs belong on stderr. A process
contains one session, with no implicit conflicting-work queue. Handshake establishes
runtime/protocol versions, capabilities and effective limits. Human Debug keeps its
own host/protocol. API/Core do not depend on JavaFX, Spring, HTTP/MCP or wire DTOs.
This avoids a provider adapter inside Studio and avoids automating the human debugger
as the execution API. The deliberate cost is an explicit process boundary and
reconfiguration of the external client after Studio restarts.
### Connection, authority and operation receipts are different identities
Authentication to loopback `/mcp` permits discovery/context/reference reads, not
editing authority. A human grants a specific project in DEV; acquisition then
returns a task identity. Manual Debug and agent work exclude each other in services,
not only through disabled UI buttons. Waiting for acquisition already blocks
conflicting manual work. A disconnected client does not release the project.
An operation's `clientRequestId` deduplicates a build or advance within the task.
The same ID and parameters return the same operation; changing parameters fails.
Lookup recovers a lost receipt without replaying frames. These records are in-memory,
not a source journal or restart-recovery mechanism. Runtime wire correlation has
its own `requestId`; neither identity is a substitute for valid task authority.
### Return is a cleanup barrier
The sequence is: invalidate task authority → stop managed work/processes → refresh
open documents from saved files → enable manual control. A late result cannot publish
a new capture or restore authority. Closing only DEV does not end the task; closing
the project/app does. Reopening starts manual, never with recovered agent authority.
Compile errors, guest faults and runtime loss remain distinct outcomes. The first
two permit correction within the same task. Runtime loss requires explicit session
replacement, not replay. Forced cleanup reports infrastructure failure first; a
later explicit reclaim may acknowledge success only after the process tree is dead.
The adversarial gate exposed a real app-teardown gap: UI closure alone did not own
every project session. The project-session factory now tracks and closes sessions
before dependent executors/LSP are destroyed, attempts other sessions after a failure,
and rejects new opens once closure begins. Cancellation also tolerates a concurrent
writer/timer shutdown so that it cannot bypass cleanup waiting.
### Direct edits require truthful boundaries
The agent edits actual project files. No worktree, merge/apply stage, code rollback,
Git cleanliness gate or automatic commit is imposed. Studio offers save-all-and-grant
for its dirty buffers and refreshes from disk on return. External unsaved buffers,
concurrent native writes and Git discipline remain the developer's responsibility.
Revocation cannot stop an external process from writing files directly; the developer
must also stop that client. SDK/external libraries are readable, not editable by
the game task. Read-only semantic refactoring scopes are enforced by the adapter.
### Pixels are evidence; metadata explains which evidence
MCP returns the lossless canonical PNG as native image content, plus session/build,
logical frame, render FrameId, dimensions, stride and pixel format. DEV projects
the same capture, with an explicit stale label after edits/rebuild/session changes.
Logical frames and render frames are independent counters. Neither a hash nor a
desktop screenshot replaces framebuffer pixels. Captures do not travel through
temporary image files, and there is no continuous video requirement.
Default bounds are 600 frames/advance, 64 KiB/command, 32 MiB/response including
image/envelope, 5 s handshake, 60 s call, and 5 s cancel/shutdown grace. Oversized
requests fail rather than clamp. Deadlines supervise processes, not simulated time.
## Final Result and Acceptance Evidence
All 28 plan steps completed on the tested platform. Production exposes 17 MCP
tools, live DEV configuration-copy, shared task authority, existing LSP/build,
indexed offline language references and the supervised real Rust headless host.
There is no production fake, embedded agent/chat, generic MCP editor/shell or
provider-specific runtime behavior.
### Requirement-to-evidence map
| Requirement and steps | Evidence retained in code or acceptance record |
| --- | --- |
| Runtime contract, commands, limits, determinism; 01–07 | Runtime `automation_process.rs`, headless tests, published request/response schemas and packaged-installation smoke |
| Neutral architecture; 08, 14, 15, 27 | Studio specs 9–11; architecture/Modulith tests, 17 passing cases in final full rerun |
| Grant/revoke, symmetric locks, dirty-buffer handoff; 09–14 | `DefaultAutomationSessionTest`, Core execution/session tests, `DevDesktopTest`, `DevWorkspaceProjectionTest` |
| HTTP authentication/Origin/restart and real config; 16, 21 | `LocalMcpServerTest`, `McpContractTest`, `McpConnectionGuideTest`; packaged restart rejects prior credential |
| Manifest context, unknown languages/versions, indexed offline docs; 17–18 | `CompilerProjectKnowledgeTest`, `CompilerProjectLanguageCatalogTest`, `InstalledKnowledgeTest` |
| Saved-source LSP and dependency navigation; 19 | `AutomationLspQueriesTest`, including `externalDependencyNavigationIsReadableButRefactoringIsReadOnly`; SDK limitation remains explicit below |
| Task operations, lost receipts, deduplication; 20–21 | `McpTaskToolsTest`, Core operation tests; reconnect/duplicate advance in real desktop HTTP gate |
| Real supervision/build/corrective iterations; 22–23 | `HeadlessProcessSessionTest`, `HeadlessInstalledRuntimeTest`, `HeadlessTaskIntegrationTest` |
| Runtime/MCP/DEV pixel and identity parity; 24 | `HeadlessMcpDesktopTest` + independent `DirectHeadlessOracle`; compare every ImageView pixel, PNG bytes, metadata, reset/fresh-session/rebuild |
| Interruption/cleanup/Debug regression; 25 | Nine real HTTP/runtime/DEV tests, including crash, forced shutdown, app close, stale-result rejection and real manual Debug; controlled document participant makes refresh barrier observable |
| Real coding client and human confirmation; 26 | Codex CLI 0.158.0 run described below; developer confirmed DEV and manual Debug Play |
| Full regression and distribution; 27 | Commands/counts below; installed references and two extracted-ZIP application lifecycle cycles |
| Final propagation and handoff; 28 | This lesson, updated spec 11 delivery boundary, README troubleshooting, CLI validation and housekeeping |
### Real external-client example
The developer granted control in `../ai-first-smoke` and copied the descriptor.
Codex CLI 0.158.0 used a fresh ephemeral session, only the configured Studio MCP,
project-root workspace-write isolation and its approval-review mechanism. Default
noninteractive approval policy initially blocked acquisition before any edit; the
retry retained sandboxing and did not bypass the human grant. The configured default
model was used; this record does not assert an independently verified model identity.
Credentials stayed out of versioned files and reports.
The agent consulted language/SDK references, edited only `src/main.pbs`, queried
LSP, built/packed and consumed native MCP image blocks. It changed the bundled
16x16 orange rectangle to one-pixel/frame directional movement in a 480x270 frame,
with no idle movement, opposing directions canceled before clamping, and unchanged
colors. Both builds succeeded; diagnostics were empty; final guest fault was null.
The 77 MCP calls included rejected/recovered calls and 13 native image blocks.
| Scenario | Logical frame / FrameId | Client-reported observed position |
| --- | --- | --- |
| Idle for 10 frames | 10 / 9 | (0,32) |
| Right 20; down 20 | 30 / 29; 50 / 49 | (20,32); (20,52) |
| Left 10; up 10 | 60 / 59; 70 / 69 | (10,52); (10,42) |
| Horizontal then vertical opposites, 10 each | 80 / 79; 90 / 89 | (10,42), unchanged |
| Right + down, 500 | 590 / 589 | (464,254), intact at bottom-right |
| All directions, 10 | 600 / 599 | (464,254), unchanged at edge |
| Left + up, 500; idle 10 | 1100 / 1099; 1110 / 1109 | (0,0), intact at top-left |
Build ID: `da441f65-479b-4099-8b86-fe48c7750e76`; runtime session:
`aa089209ec22c9a07286e5b01e967a57`. Frame identities/statuses were checked in tool
evidence; positions are the coding client's visual observations, not an invented
independent pixel measurement. Step 24 separately supplies exhaustive pixel parity.
Release returned MANUAL, no headless host remained, and game edits were preserved.
The developer confirmed the DEV result and successfully played the game through
normal Debug afterward. No game commit was made automatically.
### Builds, commands and scope
Studio executable code was `e6a83c94`; acceptance/troubleshooting documentation
was committed as `4c3597a2`. Runtime is merged at `01fab56`, distribution 0.1.0,
protocol 1.0. Native acceptance was **macOS ARM64 only**; no other coding client,
operating system or native-installer matrix is claimed.
Studio final commands:
```sh
./gradlew test build :prometeu-app:installDist --max-workers=2 --console=plain
./gradlew test :prometeu-app:installedKnowledgeTest :prometeu-app:headlessProcessTest --rerun-tasks -PstudioRuntimeInstall=/absolute/path/to/runtime --max-workers=2 --console=plain
```
The forced rerun executed all 91 tasks: normal suite **803 passed, 15 pre-existing
disabled tests, zero failures/errors**; installed knowledge **1 passed**; real
headless integration **4 passed**. The disabled compiler classes were
`IRVMIntrinsicRegistryParityTest`, `BackendGateIIntegrationTest` and
`MainProjectPipelineIntegrationTest`; none was disabled for this delivery.
The earlier step 25 targeted run had **254 passing tests**, including 9 real
HTTP/runtime/DEV and 4 DEV/Debug desktop tests. These overlap the full suite and
must not be added together as unique tests.
The Studio ZIP was extracted outside the checkout. A test-only JavaFX probe loaded
only its distribution JARs, delegated to the real App init/start/stop, used isolated
preferences, and passed two shown-launcher/closed-listener cycles, credential
renewal and old-token rejection. It did not operate the developer's open window.
ZIP SHA-256: `256f80b25416c284dca58e5519fdde0be36e9a6a2a450cb58f9c453862188c8d`.
Runtime final commands (cwd `../runtime`, separate temporary Cargo target):
```sh
cargo test --workspace --offline --quiet
PROMETEU_PROTOCOL_TEST_INSTALL=/absolute/path/to/runtime cargo test -p prometeu-cli --offline --test automation_process -- --include-ignored
cargo test -p prometeu-host-desktop-winit --offline --lib -- --ignored
```
Workspace **678 passed**, with six opt-in tests ignored there; separate runs passed
all **13** process/protocol cases including the installed package, and all **5**
localhost Debug cases. Twelve process cases overlap the workspace result. The
runtime package was built during preparatory delivery; the final gate reused and
validated it rather than claiming a fresh distribution build. Installed headless
SHA-256: `24b74712397545ddd7f745daded859c12d19102f81c5b51bf46c202fb8d6b65d`.
## Deferred Findings — Return Here After Implementation
The user requested deferred investigation, not silent removal of these findings.
| Finding | Owner and next investigation |
| --- | --- |
| `Input.pad()` SDK definition returns no location | Investigated on 2026-09-29: editorial lookup succeeds, but the physical-file policy filters the virtual SDK source. A separate chained-call editorial limitation was reproduced; see below. No runtime defect or fix is claimed. |
| Oversized reference page produces generic `-32602` | Studio MCP: improve parameter-limit discoverability/error specificity; default pagination already works. |
| Concurrent capture/fault/state may return BUSY | Studio MCP/Core: review concurrency guidance and clarity. Sequential calls work; do not weaken accepted exclusivity implicitly. |
| Framebuffer dimensions were discovered from capture | Language/runtime documentation: improve discovery; this does not prove dimensions are absent from every reference. |
| JDK 25/Kotlin target-24 fallback, JavaFX module/native-access/Unsafe warnings | Build/desktop tooling: review supported toolchain alignment and startup options separately. Passing tests do not erase warnings. |
| UI automation helper required newer Node than 22.13.1 | Environment tooling, not Studio: upgrade only as separate authorized work; human DEV confirmation supplied acceptance. |
| macOS ARM64 + Codex is the validated matrix | Future interoperability/platform testing; no uniform Grok/Windows/Linux promise. |
`NO_FRAME_AVAILABLE` before the first completed frame and client permission prompts
are expected boundaries, not guest defects. The shutdown ownership/reclaim bugs
found during step 25 are fixed and tested, not outstanding issues.
### Follow-up Investigation: SDK Navigation and Chained Calls
Owner: `compiler/pbs`, with propagation to compiler tooling contracts and Studio/LSP
consumers. This diagnosis does not reopen the completed AI-first delivery or change
its accepted requirements.
On 2026-09-29, a temporary Java probe exercised `CompilerLanguageServiceBridge`
with a request-local PBS overlay on `test-projects/main/src/main.pbs`, using the
application runtime classpath. It compared raw PBS editorial locations, exported
frontend locations, final LSP definitions, and hover. No production source was
changed. The comparison included imports from `@sdk:input` and `@sdk:gfx`, a local
`helper()`, an explicitly typed `pad: InputPad`, and `button: InputButton`.
| Cursor target | Raw PBS locations | Frontend / LSP locations | Hover |
| --- | --- | --- | --- |
| Local `helper()` | 1 | 1 / 1 | Resolved |
| `pad` in `Input.pad()` | 1 | 0 / 0 | `pad() -> InputPad` |
| `left` in `pad.left()` | 1 | 0 / 0 | `left() -> InputButton` |
| `down` in `button.down()` | 1 | 0 / 0 | `down() -> bool` |
| `down` in `pad.left().down()` | 0 | 0 / 0 | No symbol information |
| `left` and `down` in `Input.pad().left().down()` | 0 each | 0 / 0 each | No symbol information |
| `clear` in `Gfx.clear(0)` | 1 | 0 / 0 | Resolved |
There are two separate causes:
1. **SDK destination policy, not failed lookup.** `InterfaceModuleLoader` registers
SDK declarations under `/virtual/stdlib/sdk/input/main.pbs` (and the corresponding
`gfx` path). The source handle can read the declaration text, but the path is not
a physical file. `PBSFrontendLanguageService.toFrontendLocation` explicitly
rejects it via `Files.isRegularFile`. The LSP bridge also reads destination text
from disk; removing the frontend check alone would not deliver a usable target.
This behavior follows Compiler Spec 23 §8.3 and the physical-file rule preserved
in `LSN-0068`, and already has an explicit `Gfx.clear()` regression assertion in
`CompilerLanguageServiceBridgeTest.definitionUsesEditorialLookupAndPhysicalStdlibFiles`.
2. **Chained-call editorial receiver resolution.**
`PbsEditorialSupportService.resolveReceiverTypeBeforeDot` uses
`resolveChainSegmentsEndingAt`, which walks identifier/dot chains, not a call
expression ending in `)`. Thus the receiver of `down` in `pad.left().down()` is
unknown before destination mapping even begins. An explicitly typed intermediate
`button` restores raw lookup and hover, while the virtual-source policy still
prevents navigation. This is an editorial limitation, not evidence that executing
the same expression fails.
Recommended follow-up, not an accepted implementation decision: address chained
receiver resolution in the PBS editorial layer, and separately decide how to expose
the exact SDK sources as navigable read-only files. A version-scoped physical source
cache is compatible with the existing destination policy; virtual-document support
would require an explicit contract revision. Neither approach should duplicate SDK
semantics in MCP, permit SDK edits, or point to an unrelated developer checkout.
## Pitfalls and Takeaways
- A successful HTTP fixture is not an external-agent acceptance test. Prove both.
- Tab disposal is not task disposal; application teardown must own hidden sessions.
- Never unlock before process termination and document refresh acknowledge completion.
- Do not replay an advance because a connection or receipt was lost.
- A matching hash/blank canvas is weak visual evidence. Compare actual pixels,
metadata and an input timeline that changes a known object.
- An empty SDK definition is a language limitation, not permission to edit SDK code.
- Packaging language knowledge is part of making an unfamiliar language usable by
agents; adding another hidden language configuration is not.
- Keep provider permissions separate from Studio authority. Neither bypasses the other.
- Preserve explicit platform/client/test exclusions and deferred findings at closure.
## References
- Historical accepted decision DEC-0060 and completed plan PLN-0139, DSC-0068.
- [Studio architecture](../../../docs/specs/studio/9.%20Studio%20Core%20and%20Adapter%20Architecture%20Specification.md).
- [DEV and authority contract](../../../docs/specs/studio/10.%20DEV%20and%20Automation%20Specification.md).
- [MCP tools and knowledge contract](../../../docs/specs/studio/11.%20MCP%20Tools%20and%20Project%20Knowledge%20Contract.md).
- [Usage and troubleshooting](../../../README.md#external-coding-agents-mcp).
- Runtime-owned contract: `../runtime/docs/specs/runtime/11a-headless-automation-protocol.md`.
- Previous foundation: [Services must outlive UI](../DSC-0067-studio-core-javafx-decoupling/LSN-0075-studio-services-must-outlive-the-current-ui.md).
@@ -479,6 +479,17 @@ The normal desktop Debug process starts after release and blocks a new grant.
The suspension/manual-Debug scenarios require a POSIX desktop and a free Debug
port 7777; the gate never attaches to an existing user's Debug process.
The automated client is not an external coding agent. Real-agent compatibility
and final packaging acceptance remain later gates. API/Core and JavaFX MUST NOT
import MCP/HTTP implementations; architecture tests enforce it.
The automated client is not an external coding agent. Separate acceptance on
2026-09-29 used Codex CLI 0.158.0 against the packaged Studio and runtime 0.1.0
on macOS ARM64. The agent consulted references/LSP, edited saved PBS source,
built/packed, consumed native MCP PNGs, tested 1,110 frames of directional input,
and explicitly released authority. The developer confirmed DEV and then ran the
edited game through manual Debug Play. This establishes compatibility for that
client/platform, not uniform compatibility across providers or operating systems.
Full regression and distribution gates also passed: installed offline references,
real runtime protocol tests, architecture/Modulith, and two isolated startup/stop
cycles using the extracted Studio ZIP. The latter verified listener closure and
old-credential rejection after restart. API/Core and JavaFX MUST NOT import
MCP/HTTP implementations; architecture tests enforce it. Consolidated acceptance
evidence and deferred findings live in the DSC-0068 lesson, not in temporary logs.