docs: consolidate Studio AI-first acceptance and lessons
This commit is contained in:
@@ -139,6 +139,9 @@ correctly returns `NO_FRAME_AVAILABLE`; it is not a runtime defect. Noninteracti
|
||||
clients must also provide an approval path for MCP work calls; the acceptance used
|
||||
Codex's approval review with a project-scoped workspace-write sandbox.
|
||||
|
||||
The completed baseline's architecture, acceptance evidence and post-implementation
|
||||
follow-ups are consolidated in [LSN-0076](discussion/lessons/DSC-0068-studio-headless-runtime-ai-first/LSN-0076-ai-first-studio-through-shared-authority-and-headless-evidence.md).
|
||||
|
||||
## Requirements
|
||||
|
||||
- JDK 21 or newer recommended for local development
|
||||
|
||||
+303
@@ -0,0 +1,303 @@
|
||||
---
|
||||
id: LSN-0076
|
||||
ticket: studio-headless-runtime-ai-first
|
||||
title: AI-first Studio Through Shared Authority and Headless Evidence
|
||||
created: 2026-09-29
|
||||
tags: [studio, runtime, headless, ai, mcp, automation]
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Owner: **studio**, with runtime preparation in `../runtime` and language-owned PBS
|
||||
documentation/LSP integration. DSC-0068, accepted DEC-0060 and PLN-0139 delivered
|
||||
28 ordered steps: runtime (01–07), DEV/shared services (08–14), MCP (15–21), and
|
||||
real integration/acceptance (22–28). Those workflow identifiers are historical
|
||||
references; maintained contracts live in the specs linked below.
|
||||
|
||||
The original headless host was an in-process runtime API. Studio's human Debug
|
||||
was useful, but neither a public automation protocol nor a safe authority boundary
|
||||
for another editor. The goal was an external coding agent able to learn PBS,
|
||||
edit a game, build/run it, see its pixels and iterate while the developer watches.
|
||||
A complete gameplay library or embedded AI chat was not a prerequisite.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
### Separate ownership, not parallel implementations
|
||||
|
||||
The external coding client owns conversation, model/provider, native file edits
|
||||
and budget. Studio owns project authority, build orchestration, process supervision
|
||||
and the shared observation model. Runtime owns execution and framebuffer semantics.
|
||||
The language frontend/SDK owns its references; the existing project LSP owns semantic
|
||||
queries. DEV and MCP consume the same neutral services.
|
||||
|
||||
```text
|
||||
External coding agent -- local MCP HTTP --> Studio API/Core <-- DEV (JavaFX)
|
||||
|
|
||||
compiler/packer + existing LSP
|
||||
|
|
||||
supervised runtime JSONL/stdio
|
||||
|
|
||||
canonical PNG + frame identity
|
||||
```
|
||||
|
||||
The configured installation provides `prometeu headless --stdio`. UTF-8 JSONL is
|
||||
versioned, bounded and protocol-only on stdout; logs belong on stderr. A process
|
||||
contains one session, with no implicit conflicting-work queue. Handshake establishes
|
||||
runtime/protocol versions, capabilities and effective limits. Human Debug keeps its
|
||||
own host/protocol. API/Core do not depend on JavaFX, Spring, HTTP/MCP or wire DTOs.
|
||||
|
||||
This avoids a provider adapter inside Studio and avoids automating the human debugger
|
||||
as the execution API. The deliberate cost is an explicit process boundary and
|
||||
reconfiguration of the external client after Studio restarts.
|
||||
|
||||
### Connection, authority and operation receipts are different identities
|
||||
|
||||
Authentication to loopback `/mcp` permits discovery/context/reference reads, not
|
||||
editing authority. A human grants a specific project in DEV; acquisition then
|
||||
returns a task identity. Manual Debug and agent work exclude each other in services,
|
||||
not only through disabled UI buttons. Waiting for acquisition already blocks
|
||||
conflicting manual work. A disconnected client does not release the project.
|
||||
|
||||
An operation's `clientRequestId` deduplicates a build or advance within the task.
|
||||
The same ID and parameters return the same operation; changing parameters fails.
|
||||
Lookup recovers a lost receipt without replaying frames. These records are in-memory,
|
||||
not a source journal or restart-recovery mechanism. Runtime wire correlation has
|
||||
its own `requestId`; neither identity is a substitute for valid task authority.
|
||||
|
||||
### Return is a cleanup barrier
|
||||
|
||||
The sequence is: invalidate task authority → stop managed work/processes → refresh
|
||||
open documents from saved files → enable manual control. A late result cannot publish
|
||||
a new capture or restore authority. Closing only DEV does not end the task; closing
|
||||
the project/app does. Reopening starts manual, never with recovered agent authority.
|
||||
|
||||
Compile errors, guest faults and runtime loss remain distinct outcomes. The first
|
||||
two permit correction within the same task. Runtime loss requires explicit session
|
||||
replacement, not replay. Forced cleanup reports infrastructure failure first; a
|
||||
later explicit reclaim may acknowledge success only after the process tree is dead.
|
||||
|
||||
The adversarial gate exposed a real app-teardown gap: UI closure alone did not own
|
||||
every project session. The project-session factory now tracks and closes sessions
|
||||
before dependent executors/LSP are destroyed, attempts other sessions after a failure,
|
||||
and rejects new opens once closure begins. Cancellation also tolerates a concurrent
|
||||
writer/timer shutdown so that it cannot bypass cleanup waiting.
|
||||
|
||||
### Direct edits require truthful boundaries
|
||||
|
||||
The agent edits actual project files. No worktree, merge/apply stage, code rollback,
|
||||
Git cleanliness gate or automatic commit is imposed. Studio offers save-all-and-grant
|
||||
for its dirty buffers and refreshes from disk on return. External unsaved buffers,
|
||||
concurrent native writes and Git discipline remain the developer's responsibility.
|
||||
Revocation cannot stop an external process from writing files directly; the developer
|
||||
must also stop that client. SDK/external libraries are readable, not editable by
|
||||
the game task. Read-only semantic refactoring scopes are enforced by the adapter.
|
||||
|
||||
### Pixels are evidence; metadata explains which evidence
|
||||
|
||||
MCP returns the lossless canonical PNG as native image content, plus session/build,
|
||||
logical frame, render FrameId, dimensions, stride and pixel format. DEV projects
|
||||
the same capture, with an explicit stale label after edits/rebuild/session changes.
|
||||
Logical frames and render frames are independent counters. Neither a hash nor a
|
||||
desktop screenshot replaces framebuffer pixels. Captures do not travel through
|
||||
temporary image files, and there is no continuous video requirement.
|
||||
|
||||
Default bounds are 600 frames/advance, 64 KiB/command, 32 MiB/response including
|
||||
image/envelope, 5 s handshake, 60 s call, and 5 s cancel/shutdown grace. Oversized
|
||||
requests fail rather than clamp. Deadlines supervise processes, not simulated time.
|
||||
|
||||
## Final Result and Acceptance Evidence
|
||||
|
||||
All 28 plan steps completed on the tested platform. Production exposes 17 MCP
|
||||
tools, live DEV configuration-copy, shared task authority, existing LSP/build,
|
||||
indexed offline language references and the supervised real Rust headless host.
|
||||
There is no production fake, embedded agent/chat, generic MCP editor/shell or
|
||||
provider-specific runtime behavior.
|
||||
|
||||
### Requirement-to-evidence map
|
||||
|
||||
| Requirement and steps | Evidence retained in code or acceptance record |
|
||||
| --- | --- |
|
||||
| Runtime contract, commands, limits, determinism; 01–07 | Runtime `automation_process.rs`, headless tests, published request/response schemas and packaged-installation smoke |
|
||||
| Neutral architecture; 08, 14, 15, 27 | Studio specs 9–11; architecture/Modulith tests, 17 passing cases in final full rerun |
|
||||
| Grant/revoke, symmetric locks, dirty-buffer handoff; 09–14 | `DefaultAutomationSessionTest`, Core execution/session tests, `DevDesktopTest`, `DevWorkspaceProjectionTest` |
|
||||
| HTTP authentication/Origin/restart and real config; 16, 21 | `LocalMcpServerTest`, `McpContractTest`, `McpConnectionGuideTest`; packaged restart rejects prior credential |
|
||||
| Manifest context, unknown languages/versions, indexed offline docs; 17–18 | `CompilerProjectKnowledgeTest`, `CompilerProjectLanguageCatalogTest`, `InstalledKnowledgeTest` |
|
||||
| Saved-source LSP and dependency navigation; 19 | `AutomationLspQueriesTest`, including `externalDependencyNavigationIsReadableButRefactoringIsReadOnly`; SDK limitation remains explicit below |
|
||||
| Task operations, lost receipts, deduplication; 20–21 | `McpTaskToolsTest`, Core operation tests; reconnect/duplicate advance in real desktop HTTP gate |
|
||||
| Real supervision/build/corrective iterations; 22–23 | `HeadlessProcessSessionTest`, `HeadlessInstalledRuntimeTest`, `HeadlessTaskIntegrationTest` |
|
||||
| Runtime/MCP/DEV pixel and identity parity; 24 | `HeadlessMcpDesktopTest` + independent `DirectHeadlessOracle`; compare every ImageView pixel, PNG bytes, metadata, reset/fresh-session/rebuild |
|
||||
| Interruption/cleanup/Debug regression; 25 | Nine real HTTP/runtime/DEV tests, including crash, forced shutdown, app close, stale-result rejection and real manual Debug; controlled document participant makes refresh barrier observable |
|
||||
| Real coding client and human confirmation; 26 | Codex CLI 0.158.0 run described below; developer confirmed DEV and manual Debug Play |
|
||||
| Full regression and distribution; 27 | Commands/counts below; installed references and two extracted-ZIP application lifecycle cycles |
|
||||
| Final propagation and handoff; 28 | This lesson, updated spec 11 delivery boundary, README troubleshooting, CLI validation and housekeeping |
|
||||
|
||||
### Real external-client example
|
||||
|
||||
The developer granted control in `../ai-first-smoke` and copied the descriptor.
|
||||
Codex CLI 0.158.0 used a fresh ephemeral session, only the configured Studio MCP,
|
||||
project-root workspace-write isolation and its approval-review mechanism. Default
|
||||
noninteractive approval policy initially blocked acquisition before any edit; the
|
||||
retry retained sandboxing and did not bypass the human grant. The configured default
|
||||
model was used; this record does not assert an independently verified model identity.
|
||||
Credentials stayed out of versioned files and reports.
|
||||
|
||||
The agent consulted language/SDK references, edited only `src/main.pbs`, queried
|
||||
LSP, built/packed and consumed native MCP image blocks. It changed the bundled
|
||||
16x16 orange rectangle to one-pixel/frame directional movement in a 480x270 frame,
|
||||
with no idle movement, opposing directions canceled before clamping, and unchanged
|
||||
colors. Both builds succeeded; diagnostics were empty; final guest fault was null.
|
||||
The 77 MCP calls included rejected/recovered calls and 13 native image blocks.
|
||||
|
||||
| Scenario | Logical frame / FrameId | Client-reported observed position |
|
||||
| --- | --- | --- |
|
||||
| Idle for 10 frames | 10 / 9 | (0,32) |
|
||||
| Right 20; down 20 | 30 / 29; 50 / 49 | (20,32); (20,52) |
|
||||
| Left 10; up 10 | 60 / 59; 70 / 69 | (10,52); (10,42) |
|
||||
| Horizontal then vertical opposites, 10 each | 80 / 79; 90 / 89 | (10,42), unchanged |
|
||||
| Right + down, 500 | 590 / 589 | (464,254), intact at bottom-right |
|
||||
| All directions, 10 | 600 / 599 | (464,254), unchanged at edge |
|
||||
| Left + up, 500; idle 10 | 1100 / 1099; 1110 / 1109 | (0,0), intact at top-left |
|
||||
|
||||
Build ID: `da441f65-479b-4099-8b86-fe48c7750e76`; runtime session:
|
||||
`aa089209ec22c9a07286e5b01e967a57`. Frame identities/statuses were checked in tool
|
||||
evidence; positions are the coding client's visual observations, not an invented
|
||||
independent pixel measurement. Step 24 separately supplies exhaustive pixel parity.
|
||||
Release returned MANUAL, no headless host remained, and game edits were preserved.
|
||||
The developer confirmed the DEV result and successfully played the game through
|
||||
normal Debug afterward. No game commit was made automatically.
|
||||
|
||||
### Builds, commands and scope
|
||||
|
||||
Studio executable code was `e6a83c94`; acceptance/troubleshooting documentation
|
||||
was committed as `4c3597a2`. Runtime is merged at `01fab56`, distribution 0.1.0,
|
||||
protocol 1.0. Native acceptance was **macOS ARM64 only**; no other coding client,
|
||||
operating system or native-installer matrix is claimed.
|
||||
|
||||
Studio final commands:
|
||||
|
||||
```sh
|
||||
./gradlew test build :prometeu-app:installDist --max-workers=2 --console=plain
|
||||
./gradlew test :prometeu-app:installedKnowledgeTest :prometeu-app:headlessProcessTest --rerun-tasks -PstudioRuntimeInstall=/absolute/path/to/runtime --max-workers=2 --console=plain
|
||||
```
|
||||
|
||||
The forced rerun executed all 91 tasks: normal suite **803 passed, 15 pre-existing
|
||||
disabled tests, zero failures/errors**; installed knowledge **1 passed**; real
|
||||
headless integration **4 passed**. The disabled compiler classes were
|
||||
`IRVMIntrinsicRegistryParityTest`, `BackendGateIIntegrationTest` and
|
||||
`MainProjectPipelineIntegrationTest`; none was disabled for this delivery.
|
||||
The earlier step 25 targeted run had **254 passing tests**, including 9 real
|
||||
HTTP/runtime/DEV and 4 DEV/Debug desktop tests. These overlap the full suite and
|
||||
must not be added together as unique tests.
|
||||
|
||||
The Studio ZIP was extracted outside the checkout. A test-only JavaFX probe loaded
|
||||
only its distribution JARs, delegated to the real App init/start/stop, used isolated
|
||||
preferences, and passed two shown-launcher/closed-listener cycles, credential
|
||||
renewal and old-token rejection. It did not operate the developer's open window.
|
||||
ZIP SHA-256: `256f80b25416c284dca58e5519fdde0be36e9a6a2a450cb58f9c453862188c8d`.
|
||||
|
||||
Runtime final commands (cwd `../runtime`, separate temporary Cargo target):
|
||||
|
||||
```sh
|
||||
cargo test --workspace --offline --quiet
|
||||
PROMETEU_PROTOCOL_TEST_INSTALL=/absolute/path/to/runtime cargo test -p prometeu-cli --offline --test automation_process -- --include-ignored
|
||||
cargo test -p prometeu-host-desktop-winit --offline --lib -- --ignored
|
||||
```
|
||||
|
||||
Workspace **678 passed**, with six opt-in tests ignored there; separate runs passed
|
||||
all **13** process/protocol cases including the installed package, and all **5**
|
||||
localhost Debug cases. Twelve process cases overlap the workspace result. The
|
||||
runtime package was built during preparatory delivery; the final gate reused and
|
||||
validated it rather than claiming a fresh distribution build. Installed headless
|
||||
SHA-256: `24b74712397545ddd7f745daded859c12d19102f81c5b51bf46c202fb8d6b65d`.
|
||||
|
||||
## Deferred Findings — Return Here After Implementation
|
||||
|
||||
The user requested deferred investigation, not silent removal of these findings.
|
||||
|
||||
| Finding | Owner and next investigation |
|
||||
| --- | --- |
|
||||
| `Input.pad()` SDK definition returns no location | Investigated on 2026-09-29: editorial lookup succeeds, but the physical-file policy filters the virtual SDK source. A separate chained-call editorial limitation was reproduced; see below. No runtime defect or fix is claimed. |
|
||||
| Oversized reference page produces generic `-32602` | Studio MCP: improve parameter-limit discoverability/error specificity; default pagination already works. |
|
||||
| Concurrent capture/fault/state may return BUSY | Studio MCP/Core: review concurrency guidance and clarity. Sequential calls work; do not weaken accepted exclusivity implicitly. |
|
||||
| Framebuffer dimensions were discovered from capture | Language/runtime documentation: improve discovery; this does not prove dimensions are absent from every reference. |
|
||||
| JDK 25/Kotlin target-24 fallback, JavaFX module/native-access/Unsafe warnings | Build/desktop tooling: review supported toolchain alignment and startup options separately. Passing tests do not erase warnings. |
|
||||
| UI automation helper required newer Node than 22.13.1 | Environment tooling, not Studio: upgrade only as separate authorized work; human DEV confirmation supplied acceptance. |
|
||||
| macOS ARM64 + Codex is the validated matrix | Future interoperability/platform testing; no uniform Grok/Windows/Linux promise. |
|
||||
|
||||
`NO_FRAME_AVAILABLE` before the first completed frame and client permission prompts
|
||||
are expected boundaries, not guest defects. The shutdown ownership/reclaim bugs
|
||||
found during step 25 are fixed and tested, not outstanding issues.
|
||||
|
||||
### Follow-up Investigation: SDK Navigation and Chained Calls
|
||||
|
||||
Owner: `compiler/pbs`, with propagation to compiler tooling contracts and Studio/LSP
|
||||
consumers. This diagnosis does not reopen the completed AI-first delivery or change
|
||||
its accepted requirements.
|
||||
|
||||
On 2026-09-29, a temporary Java probe exercised `CompilerLanguageServiceBridge`
|
||||
with a request-local PBS overlay on `test-projects/main/src/main.pbs`, using the
|
||||
application runtime classpath. It compared raw PBS editorial locations, exported
|
||||
frontend locations, final LSP definitions, and hover. No production source was
|
||||
changed. The comparison included imports from `@sdk:input` and `@sdk:gfx`, a local
|
||||
`helper()`, an explicitly typed `pad: InputPad`, and `button: InputButton`.
|
||||
|
||||
| Cursor target | Raw PBS locations | Frontend / LSP locations | Hover |
|
||||
| --- | --- | --- | --- |
|
||||
| Local `helper()` | 1 | 1 / 1 | Resolved |
|
||||
| `pad` in `Input.pad()` | 1 | 0 / 0 | `pad() -> InputPad` |
|
||||
| `left` in `pad.left()` | 1 | 0 / 0 | `left() -> InputButton` |
|
||||
| `down` in `button.down()` | 1 | 0 / 0 | `down() -> bool` |
|
||||
| `down` in `pad.left().down()` | 0 | 0 / 0 | No symbol information |
|
||||
| `left` and `down` in `Input.pad().left().down()` | 0 each | 0 / 0 each | No symbol information |
|
||||
| `clear` in `Gfx.clear(0)` | 1 | 0 / 0 | Resolved |
|
||||
|
||||
There are two separate causes:
|
||||
|
||||
1. **SDK destination policy, not failed lookup.** `InterfaceModuleLoader` registers
|
||||
SDK declarations under `/virtual/stdlib/sdk/input/main.pbs` (and the corresponding
|
||||
`gfx` path). The source handle can read the declaration text, but the path is not
|
||||
a physical file. `PBSFrontendLanguageService.toFrontendLocation` explicitly
|
||||
rejects it via `Files.isRegularFile`. The LSP bridge also reads destination text
|
||||
from disk; removing the frontend check alone would not deliver a usable target.
|
||||
This behavior follows Compiler Spec 23 §8.3 and the physical-file rule preserved
|
||||
in `LSN-0068`, and already has an explicit `Gfx.clear()` regression assertion in
|
||||
`CompilerLanguageServiceBridgeTest.definitionUsesEditorialLookupAndPhysicalStdlibFiles`.
|
||||
2. **Chained-call editorial receiver resolution.**
|
||||
`PbsEditorialSupportService.resolveReceiverTypeBeforeDot` uses
|
||||
`resolveChainSegmentsEndingAt`, which walks identifier/dot chains, not a call
|
||||
expression ending in `)`. Thus the receiver of `down` in `pad.left().down()` is
|
||||
unknown before destination mapping even begins. An explicitly typed intermediate
|
||||
`button` restores raw lookup and hover, while the virtual-source policy still
|
||||
prevents navigation. This is an editorial limitation, not evidence that executing
|
||||
the same expression fails.
|
||||
|
||||
Recommended follow-up, not an accepted implementation decision: address chained
|
||||
receiver resolution in the PBS editorial layer, and separately decide how to expose
|
||||
the exact SDK sources as navigable read-only files. A version-scoped physical source
|
||||
cache is compatible with the existing destination policy; virtual-document support
|
||||
would require an explicit contract revision. Neither approach should duplicate SDK
|
||||
semantics in MCP, permit SDK edits, or point to an unrelated developer checkout.
|
||||
|
||||
## Pitfalls and Takeaways
|
||||
|
||||
- A successful HTTP fixture is not an external-agent acceptance test. Prove both.
|
||||
- Tab disposal is not task disposal; application teardown must own hidden sessions.
|
||||
- Never unlock before process termination and document refresh acknowledge completion.
|
||||
- Do not replay an advance because a connection or receipt was lost.
|
||||
- A matching hash/blank canvas is weak visual evidence. Compare actual pixels,
|
||||
metadata and an input timeline that changes a known object.
|
||||
- An empty SDK definition is a language limitation, not permission to edit SDK code.
|
||||
- Packaging language knowledge is part of making an unfamiliar language usable by
|
||||
agents; adding another hidden language configuration is not.
|
||||
- Keep provider permissions separate from Studio authority. Neither bypasses the other.
|
||||
- Preserve explicit platform/client/test exclusions and deferred findings at closure.
|
||||
|
||||
## References
|
||||
|
||||
- Historical accepted decision DEC-0060 and completed plan PLN-0139, DSC-0068.
|
||||
- [Studio architecture](../../../docs/specs/studio/9.%20Studio%20Core%20and%20Adapter%20Architecture%20Specification.md).
|
||||
- [DEV and authority contract](../../../docs/specs/studio/10.%20DEV%20and%20Automation%20Specification.md).
|
||||
- [MCP tools and knowledge contract](../../../docs/specs/studio/11.%20MCP%20Tools%20and%20Project%20Knowledge%20Contract.md).
|
||||
- [Usage and troubleshooting](../../../README.md#external-coding-agents-mcp).
|
||||
- Runtime-owned contract: `../runtime/docs/specs/runtime/11a-headless-automation-protocol.md`.
|
||||
- Previous foundation: [Services must outlive UI](../DSC-0067-studio-core-javafx-decoupling/LSN-0075-studio-services-must-outlive-the-current-ui.md).
|
||||
@@ -479,6 +479,17 @@ The normal desktop Debug process starts after release and blocks a new grant.
|
||||
The suspension/manual-Debug scenarios require a POSIX desktop and a free Debug
|
||||
port 7777; the gate never attaches to an existing user's Debug process.
|
||||
|
||||
The automated client is not an external coding agent. Real-agent compatibility
|
||||
and final packaging acceptance remain later gates. API/Core and JavaFX MUST NOT
|
||||
import MCP/HTTP implementations; architecture tests enforce it.
|
||||
The automated client is not an external coding agent. Separate acceptance on
|
||||
2026-09-29 used Codex CLI 0.158.0 against the packaged Studio and runtime 0.1.0
|
||||
on macOS ARM64. The agent consulted references/LSP, edited saved PBS source,
|
||||
built/packed, consumed native MCP PNGs, tested 1,110 frames of directional input,
|
||||
and explicitly released authority. The developer confirmed DEV and then ran the
|
||||
edited game through manual Debug Play. This establishes compatibility for that
|
||||
client/platform, not uniform compatibility across providers or operating systems.
|
||||
|
||||
Full regression and distribution gates also passed: installed offline references,
|
||||
real runtime protocol tests, architecture/Modulith, and two isolated startup/stop
|
||||
cycles using the extracted Studio ZIP. The latter verified listener closure and
|
||||
old-credential rejection after restart. API/Core and JavaFX MUST NOT import
|
||||
MCP/HTTP implementations; architecture tests enforce it. Consolidated acceptance
|
||||
evidence and deferred findings live in the DSC-0068 lesson, not in temporary logs.
|
||||
|
||||
Reference in New Issue
Block a user