package PBS references and expose paginated MCP reads

This commit is contained in:
2026-09-29 08:48:33 +01:00
parent 58f11a8123
commit feb57c5298
23 changed files with 738 additions and 21 deletions
@@ -323,3 +323,36 @@ The current Studio direction is:
4. the compiler should use explicit abstractions such as `StdlibEnvironment` and `StdlibModuleResolver`,
5. stdlib interface modules are parsed with the normal PBS parser and validated in interface-module mode,
6. reserved attributes such as `[Host(...)]` are extracted as compile-time metadata for later lowering.
## 13. Installed Language and SDK Reference Bundle
The PBS frontend distribution MUST provide local, indexed documentation through
the frontend reference-provider contract. This packaging does not create a new
normative specification: the numbered PBS documents in their owner directory
remain authoritative, and the quickstart/example remain informative.
The build MUST generate the reference index from owner-maintained sources:
- this directory's numbered specifications and README quickstart;
- the frontend's example manifest, PBS source and barrel;
- the actual packaged SDK `.pbs` and `.barrel` files, including their documentation.
Generated copies MUST NOT be edited independently. SDK declarations MUST NOT be
rewritten to manufacture a documentation-only API. The index MUST contain stable
reference identities, titles, kinds, packaged-resource identities, content hashes
and applicable SDK major/target. The loader MUST reject missing/corrupted indexed
content instead of advertising unreadable references or consulting a checkout.
For the current distribution, SDK/example selection is Game / major 1. The general
PBS specifications remain available for other selections, but Game SDK 1 references
MUST NOT be presented as another SDK version or target. An absent SDK reference
set MUST be explicit to consumers.
Language, compiler, SDK and index-schema versions remain independent. General
specification references use an unknown release version when owner release
metadata is absent; SDK-associated references identify the installed SDK major.
Content hashes identify the packaged texts, not a semantic language release.
Conformance MUST include reading all indexed references from installed JARs outside
the repository and compiling the shipped example against the installed SDK. No
network access or source checkout may be required by that reference-read path.
@@ -2,6 +2,73 @@
This directory contains the normative PBS specification set.
## Installed Quickstart (Informative)
This guide and the numbered specifications are packaged with the PBS frontend.
They describe the installed distribution, not a promise of a numbered language
release. A spec's `Draft v1` status is not the compiler version or SDK version.
The project manifest selects the SDK major with `stdlib`; current packaged
Game SDK resources use major `1`. Unknown language/compiler versions remain
unknown in Studio's context.
For an external coding agent:
1. Request `studio_project_context` for the absolute root of the open project.
2. Read `pbs:quickstart`, `pbs:spec:3` (syntax), `pbs:spec:4` (static semantics)
and the relevant SDK reference IDs returned in that context. Continue pages
until `nextOffset` is null. References are IDs, not filesystem paths.
3. Inspect the project and its dependencies before changing source. Only edit
the project's editable scope, excluding read-only SDK/external libraries.
4. Use the installed LSP and compiler to check changes. When the corresponding
MCP tools are available, acquire project control after the developer's DEV
grant, then use the advertised language/build tools. Documentation reads do
not require that grant. Missing tools/capabilities must not be assumed.
5. Build and test small changes. Runtime execution also requires a compatible
configured runtime; reading references does not install one.
### Small Game Project
The installed example has three separately indexed files:
- `pbs:example:game:1:prometeu.json`: a Game manifest with `language: pbs`,
`stdlib: 1` (serialized as a string), and no external dependencies;
- `pbs:example:game:1:src/main.pbs`: `[Init]` and `[Frame]` functions drawing a
moving rectangle through `Gfx` and constructing colors through `Color`;
- `pbs:example:game:1:src/mod.barrel`: the module's public declarations.
Copy these reference texts to the corresponding paths only when creating that
example in your editable project. The example is compiled in the test suite;
it is not a complete game or a claim that every requested gameplay abstraction
already exists. If a project has no external dependencies, no registry is needed.
For dependencies, follow the installed compiler's manifest/registry requirements.
PBS source lives under `src` for the installed provider. Functions use `fn`,
typed parameters and `->` return types. `let` declares a local binding;
`declare global` declares persistent global state. Imports name exports and a
module address, for example `import { Gfx } from @sdk:gfx;`. A `mod.barrel`
describes a module's public surface. Read the syntax/static-semantics chapters
for exact rules rather than inferring compatibility with another language.
### SDK Navigation and Limits
`pbs:sdk:game:1:sdk/gfx/main.pbs` exposes the actual installed Gfx source,
including documentation comments and signatures. Its `mod.barrel` reference
defines exports. Equivalent references exist for `sdk/asset`, `sdk/composer`,
`sdk/input`, `sdk/log`, and `core/color`. Read those interfaces before calling
them. Game SDK 1 does not currently ship `audio` or `core/math` modules.
Ordinary game source uses public services such as `Gfx`; it must not manufacture
host bindings or mutate SDK declarations. Asset/composer workflows require their
real asset/scene data and runtime capabilities. Reference availability does not
guarantee those project resources exist. SDK or tooling limitations should be
reported for separate work, not patched in read-only dependencies.
The reference index is generated from these owner-maintained documents and the
frontend's SDK/example resources. The index schema revision is not a language
version. Source hashes bind texts to the installed bundle; no repository checkout
or network is needed to read them. The numbered specifications remain normative;
this onboarding section and example are explanatory, not a second specification.
## Purpose
Specs define the official PBS language/frontend contract.
@@ -184,7 +184,8 @@ Reference reads take an opaque ID from that index, not a path or arbitrary URI.
Offset defaults to 0; limit defaults to 4000 and accepts 1–16000 Unicode code
points. A page returns referenceId, offset, total, text and nextOffset (null at
end). Invalid pagination fails; unknown/traversal-like IDs cannot escape the
index. Installed packaging and real provider resolution are steps 17–18.
index. Installed reference content is resolved through the frontend's owner-maintained
reference provider, not a Studio-maintained language specification.
### 5.1 Manifest and Provider Resolution
@@ -230,9 +231,44 @@ symlink targets. SDK entries are always read-only; packaged SDK resources need n
have filesystem roots. Scope metadata does not sandbox an external agent's own
filesystem tools; the developer remains responsible for those tools and Git.
Paths in MCP results are absolute filesystem strings, not `file:` URIs. Until
step 18 packages the owner-maintained index, references/reference IDs are empty
with `DOCUMENTATION_UNAVAILABLE`; no guessed path or reference is advertised.
Paths in MCP results are absolute filesystem strings, not `file:` URIs.
### 5.2 Installed References and Pagination
An installed frontend MAY implement `FrontendReferenceProvider` through
`FrontendProvider.documentation()`. That compiler-side contract supplies indexed,
owner-maintained texts for an SDK major and project target. It does not depend on
Studio API or MCP. Studio projects these into neutral reference descriptors;
full texts MUST NOT be included in project context. Reference IDs are namespaced
by language (`<languageId>:`), unique within the selected project index and opaque
to callers. Studio MUST NOT interpret an ID as a filename, URI or URL.
PBS packages its numbered specifications and informative quickstart from their
owner directory, plus a compilable Game example and actual SDK sources/barrels.
A build-generated index binds these texts to the installed frontend using content
hashes. Distribution-only reads MUST work without a checkout or network. SDK and
example references identify their SDK major and target; unsupported selections
MUST NOT silently reuse another SDK/target. Numbered spec/draft revisions and the
reference-index schema revision MUST NOT be reported as language/compiler versions.
`studio_reference_read` is authenticated but does not require a control grant or
task. It resolves the explicit open root and its current saved manifest selection
on every read. Changing SDK/target can invalidate previously advertised IDs.
Only IDs in that project's installed index can be read, including when other
languages/providers are installed globally. SDK and external-library texts remain
consultable, not editable through this surface.
Pagination counts Unicode code points: an astral character occupies one offset,
not two UTF-16 units. Offset 0/limit 4000 are defaults; limit MUST be 1–16000.
An offset equal to total returns empty text and null nextOffset. Offsets beyond
total, negative offsets, nonintegral values and integers beyond the implementation's
signed 32-bit range are invalid parameters, never wrapped/truncated to another page.
Missing or unreadable provider documentation yields `DOCUMENTATION_UNAVAILABLE`
limitations, not promised references. An uninstalled primary language yields
`UNSUPPORTED_LANGUAGE` for reference reads; an available index without the requested
ID yields `REFERENCE_NOT_FOUND`. Closed projects yield `PROJECT_NOT_OPEN` before
reference lookup. These failures do not grant/revoke task authority.
## 6. Image Contract
@@ -258,7 +294,8 @@ authentication, Origin, endpoint/method/version handling, tool dispatch, malform
or oversized input, closure, restart and old-token rejection. Provider doubles
exist only in tests. Composition starts/closes transport with Studio.
The production registry composes `studio_project_context` as of step 17; other
The production registry composes `studio_project_context` and `studio_reference_read`
as of step 18; other
tools are advertised only when their functional handlers are connected.
DEV configuration-copy stays unavailable until step 21. Passing this transport
gate is not the phase 3 or final external-agent acceptance gate. API/Core and
+14 -1
View File
@@ -31,7 +31,7 @@ javafx {
}
// Opt-in desktop gate: real JavaFX toolkit and windows, no display required by normal unit tests.
tasks.named<Test>("test") { exclude("**/DevDesktopTest.class") }
tasks.named<Test>("test") { exclude("**/DevDesktopTest.class", "**/InstalledKnowledgeTest.class") }
tasks.register<Test>("devDesktopTest") {
group = "verification"
description = "Runs the DEV/Debug desktop lifecycle and handoff smoke tests (requires a display)."
@@ -41,3 +41,16 @@ tasks.register<Test>("devDesktopTest") {
include("**/DevDesktopTest.class")
systemProperty("java.awt.headless", "false")
}
tasks.register<Test>("installedKnowledgeTest") {
group = "verification"
description = "Reads installed references and compiles their example in an isolated JVM outside the checkout."
dependsOn(tasks.named("installDist"))
testClassesDirs = sourceSets["test"].output.classesDirs
classpath = sourceSets["test"].runtimeClasspath
useJUnitPlatform()
include("**/InstalledKnowledgeTest.class")
val distribution = layout.buildDirectory.dir("install/prometeu-app")
inputs.dir(distribution)
systemProperty("studio.test.distribution", distribution.get().asFile.absolutePath)
}
@@ -304,7 +304,7 @@ public class StudioApplicationConfiguration {
@Bean
p.studio.app.mcp.McpToolRegistry mcpToolRegistry(p.studio.app.mcp.McpToolCatalog catalog, ObjectMapper mapper,
p.studio.api.automation.ProjectKnowledge knowledge) {
// Further providers are connected in PLN-0139 steps 18–20. Never advertise placeholders.
// Further providers are connected in PLN-0139 steps 19–20. Never advertise placeholders.
return new p.studio.app.mcp.McpToolRegistry(catalog, mapper,
p.studio.app.mcp.McpProjectKnowledgeTools.handlers(knowledge, mapper));
}
@@ -20,6 +20,8 @@ import java.util.*;
/** Read-only manifest/provider discovery. Never selects a default frontend or starts a build. */
public final class CompilerProjectKnowledge implements ProjectKnowledge {
private record OwnedReference(String owner, p.studio.compiler.services.FrontendReferenceProvider.Reference reference) {}
private record Discovery(Context context, Map<String, OwnedReference> references) {}
private static final int MAX_PROJECTS = 256;
private final ProjectAccessRegistry openProjects;
private final ProjectCatalog projects;
@@ -33,6 +35,10 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
}
@Override public Context context(Path openProjectRoot) {
return discover(openProjectRoot).context();
}
private Discovery discover(Path openProjectRoot) {
Path root = openProjects.requireOpenRoot(openProjectRoot);
var limitations = new ArrayList<String>();
var discovered = new LinkedHashMap<Path, ProjectReference>();
@@ -128,12 +134,15 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
}
}
libraries.addAll(unresolved);
var references = new LinkedHashMap<String, OwnedReference>();
discovered.values().forEach(project -> collectReferences(project, references, limitations));
var languages = new ArrayList<Language>();
String primary = discovered.get(root).languageId();
if (byLanguage.size() > 1) limitations.add("The installed build pipeline requires the same language across the dependency graph");
for (var entry : byLanguage.entrySet()) {
var language = language(entry.getKey(), entry.getValue(), primary.equals(entry.getKey()),
dependenciesComplete && byLanguage.size() == 1, limitations);
dependenciesComplete && byLanguage.size() == 1, limitations,
references.entrySet().stream().filter(ref -> ref.getValue().owner().equals(entry.getKey())).map(Map.Entry::getKey).toList());
languages.add(language);
for (Path source : language.sourceRoots()) if (!source.startsWith(root)) readOnly.add(source);
for (var project : entry.getValue()) {
@@ -144,15 +153,19 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
}
}
limitations.add("SDK resources are provider-owned, read-only resources, not editable project files");
limitations.add("DOCUMENTATION_UNAVAILABLE: installed language/SDK reference index is not yet available");
// Closing a project during discovery must not publish a successful context for that closed project.
openProjects.requireOpenRoot(root);
return new Context(root, discovered.get(root).name(), languages, libraries,
List.of(root), List.copyOf(readOnly), List.of(), limitations);
var descriptors = references.values().stream().map(owned -> {
var ref = owned.reference();
return new Reference(ref.id(), ref.title(), owned.owner(), ReferenceKind.valueOf(ref.kind().name()),
new Version(ref.version(), ref.versionSource()));
}).toList();
return new Discovery(new Context(root, discovered.get(root).name(), languages, libraries,
List.of(root), List.copyOf(readOnly), descriptors, limitations.stream().distinct().toList()), Map.copyOf(references));
}
private Language language(String id, List<ProjectReference> projects, boolean primary,
boolean graphBuildable, List<String> limitations) {
boolean graphBuildable, List<String> limitations, List<String> referenceIds) {
var sdkVersions = projects.stream().map(ProjectReference::stdlibVersion).distinct().toList();
Version sdk = sdkVersions.size() == 1
? new Version(sdkVersions.getFirst().toString(), "prometeu.json#stdlib (requested)") : Version.unknown();
@@ -161,7 +174,7 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
if (provider.isEmpty()) {
limitations.add("UNSUPPORTED_LANGUAGE: no installed frontend for " + id);
return new Language(id, primary, Version.unknown(), Version.unknown(), sdk, List.of(), List.of(),
List.of(capability("build", false), capability("lsp", false)), List.of());
List.of(capability("build", false), capability("lsp", false)), referenceIds);
}
var spec = provider.get().specification();
var roots = new LinkedHashSet<Path>();
@@ -176,7 +189,35 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
String compilerVersion = provider.get().getClass().getPackage().getImplementationVersion();
return new Language(id, primary, Version.unknown(), compilerVersion == null || compilerVersion.isBlank()
? Version.unknown() : new Version(compilerVersion, "installed frontend package Implementation-Version"), sdk,
spec.getAllowedExtensions().stream().sorted().toList(), List.copyOf(roots), capabilities, List.of());
spec.getAllowedExtensions().stream().sorted().toList(), List.copyOf(roots), capabilities, referenceIds);
}
private void collectReferences(ProjectReference project, Map<String, OwnedReference> references, List<String> limitations) {
String owner = project.languageId();
String selection = owner + " / " + project.target() + " / SDK " + project.stdlibVersion();
try {
var documentation = FrontendRegistryService.getProvider(owner).flatMap(p.studio.compiler.services.FrontendProvider::documentation);
if (documentation.isEmpty()) {
limitations.add("DOCUMENTATION_UNAVAILABLE: no installed reference provider for " + owner);
return;
}
var selected = documentation.get().references(project.stdlibVersion(), p.studio.AppMode.valueOf(project.target().name()));
var additions = new LinkedHashMap<String, OwnedReference>();
for (var reference : selected) {
var owned = new OwnedReference(owner, reference);
if (!reference.id().startsWith(owner + ":")
|| references.containsKey(reference.id()) && !references.get(reference.id()).equals(owned)
|| additions.putIfAbsent(reference.id(), owned) != null)
throw new IllegalStateException("Ambiguous or invalid reference ID");
}
references.putAll(additions);
if (selected.isEmpty()) limitations.add("DOCUMENTATION_UNAVAILABLE: no installed references for " + selection);
if (selected.stream().noneMatch(ref -> ref.kind() == p.studio.compiler.services.FrontendReferenceProvider.Kind.SDK
&& Integer.toString(project.stdlibVersion()).equals(ref.version())))
limitations.add("DOCUMENTATION_UNAVAILABLE: no installed SDK reference for " + selection);
} catch (RuntimeException unavailable) {
limitations.add("DOCUMENTATION_UNAVAILABLE: cannot load installed references for " + selection);
}
}
private void addLspCapabilities(String id, ProjectReference project, List<Capability> capabilities, List<String> limitations) {
@@ -224,6 +265,21 @@ public final class CompilerProjectKnowledge implements ProjectKnowledge {
@Override public ReferencePage readReference(Path openProjectRoot, String referenceId, int offset, int limit) {
openProjects.requireOpenRoot(openProjectRoot);
throw new AutomationException(AutomationFailure.Code.DOCUMENTATION_UNAVAILABLE, "No installed reference index is available");
if (referenceId == null || referenceId.isBlank() || offset < 0 || limit < 1 || limit > 16000)
throw new IllegalArgumentException("Invalid reference ID or pagination");
var discovery = discover(openProjectRoot);
var reference = discovery.references().get(referenceId);
if (reference == null) {
var primary = discovery.context().languages().stream().filter(Language::primary).findFirst().orElseThrow();
var code = FrontendRegistryService.getProvider(primary.id()).isEmpty() ? AutomationFailure.Code.UNSUPPORTED_LANGUAGE
: discovery.references().isEmpty() ? AutomationFailure.Code.DOCUMENTATION_UNAVAILABLE : AutomationFailure.Code.REFERENCE_NOT_FOUND;
throw new AutomationException(code, "Reference is not available in this project's installed index");
}
String text = reference.reference().text();
int total = text.codePointCount(0, text.length());
if (offset > total) throw new IllegalArgumentException("Reference offset exceeds its length");
int next = offset + Math.min(limit, total - offset);
String page = text.substring(text.offsetByCodePoints(0, offset), text.offsetByCodePoints(0, next));
return new ReferencePage(referenceId, offset, total, next == total ? null : next, page);
}
}
@@ -19,6 +19,16 @@ public final class McpProjectKnowledgeTools {
var paths = new SimpleModule().addSerializer(Path.class, new ToStringSerializer());
var wireMapper = mapper.copy().registerModule(paths);
return Map.of("studio_project_context", arguments -> McpToolResult.success(mapper,
wireMapper.valueToTree(knowledge.context(Path.of(arguments.path("projectRoot").asText())))));
wireMapper.valueToTree(knowledge.context(Path.of(arguments.path("projectRoot").asText())))),
"studio_reference_read", arguments -> McpToolResult.success(mapper, wireMapper.valueToTree(
knowledge.readReference(Path.of(arguments.path("projectRoot").asText()), arguments.path("referenceId").asText(),
integer(arguments, "offset", 0), integer(arguments, "limit", 4000)))));
}
private static int integer(JsonNode arguments, String name, int fallback) {
var value = arguments.get(name);
if (value == null) return fallback;
if (!value.isIntegralNumber() || !value.canConvertToInt()) throw new IllegalArgumentException("Invalid " + name);
return value.intValue();
}
}
@@ -111,6 +111,7 @@
"offset": {
"type": "integer",
"minimum": 0,
"maximum": 2147483647,
"default": 0
},
"limit": {
@@ -183,12 +184,12 @@
},
"exampleArguments": {
"projectRoot": "/projects/game",
"referenceId": "pbs.quickstart",
"referenceId": "pbs:quickstart",
"offset": 0,
"limit": 4000
},
"exampleValue": {
"referenceId": "pbs.quickstart",
"referenceId": "pbs:quickstart",
"offset": 0,
"total": 7,
"nextOffset": null,
@@ -30,8 +30,9 @@ class StudioApplicationConfigurationTest {
assertFalse(executor.isShutdown());
assertTrue(mcp.isRunning());
var tools = context.getBean(p.studio.app.mcp.McpToolRegistry.class).descriptors();
org.junit.jupiter.api.Assertions.assertEquals(1, tools.size());
org.junit.jupiter.api.Assertions.assertEquals(2, tools.size());
org.junit.jupiter.api.Assertions.assertEquals("studio_project_context", tools.get(0).path("name").asText());
org.junit.jupiter.api.Assertions.assertEquals("studio_reference_read", tools.get(1).path("name").asText());
}
assertTrue(executor.isShutdown());
assertFalse(mcp.isRunning());
@@ -45,8 +45,10 @@ class CompilerProjectKnowledgeTest {
assertEquals(List.of(root), result.editableRoots());
assertFalse(result.libraries().getFirst().editable());
assertTrue(result.libraries().getFirst().id().startsWith("sdk:pbs:"));
assertTrue(result.references().isEmpty());
assertTrue(result.limitations().stream().anyMatch(value -> value.contains("DOCUMENTATION_UNAVAILABLE")));
assertFalse(result.references().isEmpty());
assertTrue(result.references().stream().anyMatch(value -> value.id().equals("pbs:quickstart")));
assertTrue(language.referenceIds().contains("pbs:quickstart"));
assertFalse(result.limitations().stream().anyMatch(value -> value.contains("DOCUMENTATION_UNAVAILABLE")));
assertFalse(Files.exists(root.resolve(".workspace")));
assertFalse(Files.exists(root.resolve(".studio")));
}
@@ -0,0 +1,62 @@
package p.studio.app.compiler;
import com.fasterxml.jackson.databind.ObjectMapper;
import p.studio.api.automation.ProjectKnowledge;
import p.studio.compiler.FrontendRegistryService;
import p.studio.compiler.messages.BuilderPipelineConfig;
import p.studio.compiler.models.BuilderPipelineContext;
import p.studio.compiler.workspaces.BuilderPipelineService;
import p.studio.core.project.*;
import p.studio.utilities.logs.LogAggregator;
import java.nio.file.Files;
import java.nio.file.Path;
/** Separate JVM gate: product classes/resources must resolve exclusively from installDist JARs. */
public final class InstalledKnowledgeProbe {
public static void main(String[] arguments) throws Exception {
var provider = FrontendRegistryService.require("pbs");
check(provider.getClass().getProtectionDomain().getCodeSource().getLocation().getPath().endsWith(".jar"), "Frontend must come from installed JAR");
check(CompilerProjectKnowledge.class.getProtectionDomain().getCodeSource().getLocation().getPath().endsWith(".jar"), "App must come from installed JAR");
check(!Files.exists(Path.of("settings.gradle.kts")) && !Files.exists(Path.of("docs")), "No repository in working directory");
Path root = Files.createDirectory(Path.of("example")).toRealPath();
var docs = provider.documentation().orElseThrow().references(1, p.studio.AppMode.Game);
for (var reference : docs) {
if (reference.id().startsWith("pbs:example:game:1:")) {
Path file = root.resolve(reference.id().substring("pbs:example:game:1:".length()));
Files.createDirectories(file.getParent());
Files.writeString(file, reference.text());
}
}
var mapper = new ObjectMapper();
var access = new ProjectAccessRegistry();
var registration = access.register(root, () -> { throw new AssertionError("References require no grant"); }, () -> true);
var catalog = new ProjectCatalogService(root, mapper, new ProjectLocalStudioSetupStore(mapper));
var knowledge = new CompilerProjectKnowledge(access, catalog, mapper);
var context = knowledge.context(root);
for (var kind : java.util.List.of(ProjectKnowledge.ReferenceKind.QUICKSTART, ProjectKnowledge.ReferenceKind.SYNTAX,
ProjectKnowledge.ReferenceKind.SEMANTICS, ProjectKnowledge.ReferenceKind.EXAMPLE, ProjectKnowledge.ReferenceKind.SDK))
check(context.references().stream().anyMatch(ref -> ref.kind() == kind), "Missing " + kind);
for (var reference : docs) {
var actual = new StringBuilder();
int offset = 0;
while (true) {
var page = knowledge.readReference(root, reference.id(), offset, 997);
actual.append(page.text());
if (page.nextOffset() == null) break;
offset = page.nextOffset();
}
check(actual.toString().equals(reference.text()), "Incorrect installed reference: " + reference.id());
}
var result = BuilderPipelineService.INSTANCE.compile(
BuilderPipelineContext.fromConfig(new BuilderPipelineConfig(false, root.toString())), LogAggregator.empty());
check(result.diagnostics().stream().noneMatch(issue -> issue.isError()), "Example must compile without errors");
check(result.bytecodeBytes().length > 0, "Example must produce bytecode");
registration.unsubscribe();
System.out.println("Installed knowledge verified: " + docs.size() + " references; example compiled");
}
private static void check(boolean condition, String message) {
if (!condition) throw new AssertionError(message);
}
}
@@ -0,0 +1,33 @@
package p.studio.app.compiler;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static org.junit.jupiter.api.Assertions.*;
class InstalledKnowledgeTest {
@TempDir Path outsideRepository;
@Test void installedJarsProvideAllReferencesAndCompileTheExampleOutsideTheCheckout() throws Exception {
Path distribution = Path.of(System.getProperty("studio.test.distribution"));
String jars;
try (var files = Files.list(distribution.resolve("lib"))) {
jars = files.filter(file -> file.toString().endsWith(".jar")).sorted()
.map(Path::toString).collect(Collectors.joining(java.io.File.pathSeparator));
}
String probeClasses = Path.of(InstalledKnowledgeProbe.class.getProtectionDomain().getCodeSource().getLocation().toURI()).toString();
Path output = outsideRepository.resolve("probe.log");
var process = new ProcessBuilder(Path.of(System.getProperty("java.home"), "bin", "java").toString(),
"-cp", jars + java.io.File.pathSeparator + probeClasses, InstalledKnowledgeProbe.class.getName())
.directory(outsideRepository.toFile()).redirectErrorStream(true).redirectOutput(output.toFile()).start();
try {
assertTrue(process.waitFor(45, TimeUnit.SECONDS), "Installed knowledge probe timed out");
String log = Files.readString(output);
assertEquals(0, process.exitValue(), log);
assertTrue(log.contains("references; example compiled"), log);
} finally { if (process.isAlive()) process.destroyForcibly().waitFor(); }
}
}
@@ -0,0 +1,143 @@
package p.studio.app.compiler;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import p.studio.api.automation.*;
import p.studio.compiler.FrontendRegistryService;
import p.studio.compiler.models.FrontendSpec;
import p.studio.compiler.services.*;
import p.studio.core.project.*;
import p.studio.utilities.structures.ReadOnlySet;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import static org.junit.jupiter.api.Assertions.*;
class ProjectReferencesTest {
@TempDir Path root;
final ObjectMapper mapper = new ObjectMapper();
final ProjectAccessRegistry access = new ProjectAccessRegistry();
CompilerProjectKnowledge knowledge(String language, String sdk) throws Exception {
writeManifest(language, sdk);
access.register(root, () -> { throw new AssertionError("Read needs no grant"); }, () -> true);
return new CompilerProjectKnowledge(access, new ProjectCatalogService(root, mapper, new ProjectLocalStudioSetupStore(mapper)), mapper);
}
void writeManifest(String language, String sdk) throws Exception {
mapper.writeValue(root.resolve("prometeu.json").toFile(), Map.of("name", "References", "version", "1.0.0",
"language", language, "stdlib", sdk, "target", "Game", "dependencies", List.of()));
}
@Test void pbsReferenceKindsAndVersionsMatchTheInstalledProviderAndPaginateExactly() throws Exception {
var knowledge = knowledge("pbs", "1");
var context = knowledge.context(root);
var kinds = context.references().stream().map(ProjectKnowledge.Reference::kind).collect(java.util.stream.Collectors.toSet());
assertTrue(kinds.containsAll(List.of(ProjectKnowledge.ReferenceKind.QUICKSTART, ProjectKnowledge.ReferenceKind.SYNTAX,
ProjectKnowledge.ReferenceKind.SEMANTICS, ProjectKnowledge.ReferenceKind.EXAMPLE, ProjectKnowledge.ReferenceKind.SDK)));
assertEquals(context.references().size(), context.languages().getFirst().referenceIds().size());
for (var reference : context.references()) {
assertEquals("pbs", reference.owner());
if (reference.kind() == ProjectKnowledge.ReferenceKind.SDK || reference.kind() == ProjectKnowledge.ReferenceKind.EXAMPLE) {
assertEquals("1", reference.version().value());
assertTrue(reference.version().source().contains("SDK"));
} else assertFalse(reference.version().known());
}
var sdk = "pbs:sdk:game:1:sdk/gfx/main.pbs";
var expected = FrontendRegistryService.require("pbs").documentation().orElseThrow().references(1, p.studio.AppMode.Game)
.stream().filter(ref -> ref.id().equals(sdk)).findFirst().orElseThrow().text();
var actual = new StringBuilder();
int offset = 0;
do {
var page = knowledge.readReference(root, sdk, offset, 97);
assertEquals(expected.codePointCount(0, expected.length()), page.total());
actual.append(page.text());
if (page.nextOffset() == null) break;
offset = page.nextOffset();
} while (true);
assertEquals(expected, actual.toString());
assertTrue(actual.toString().contains("fn fill_rect"));
}
@Test void unknownAndTraversalLikeIdsNeverReadFilesOrUrls() throws Exception {
var knowledge = knowledge("pbs", "1");
Path secret = root.resolve("secret.txt");
Files.writeString(secret, "not a reference");
for (String id : List.of(secret.toString(), secret.toUri().toString(), "../secret.txt", "pbs:../secret.txt",
"pbs-knowledge/index.json", "pbs:%2e%2e/secret.txt", "https://example.invalid", "pbs:unknown")) {
assertEquals(AutomationFailure.Code.REFERENCE_NOT_FOUND,
assertThrows(AutomationException.class, () -> knowledge.readReference(root, id, 0, 100)).failure().code());
}
}
@Test void changedSdkCannotReadPreviouslyAdvertisedSdkOrExampleIds() throws Exception {
var knowledge = knowledge("pbs", "1");
assertTrue(knowledge.context(root).languages().getFirst().referenceIds().contains("pbs:sdk:game:1:sdk/gfx/main.pbs"));
writeManifest("pbs", "999");
var context = knowledge.context(root);
assertTrue(context.references().stream().noneMatch(ref -> ref.kind() == ProjectKnowledge.ReferenceKind.SDK || ref.kind() == ProjectKnowledge.ReferenceKind.EXAMPLE));
assertTrue(context.limitations().stream().anyMatch(value -> value.contains("no installed SDK reference")));
assertEquals(AutomationFailure.Code.REFERENCE_NOT_FOUND, assertThrows(AutomationException.class,
() -> knowledge.readReference(root, "pbs:sdk:game:1:sdk/gfx/main.pbs", 0, 100)).failure().code());
assertFalse(knowledge.readReference(root, "pbs:quickstart", 0, 100).text().isEmpty());
}
@Test void paginationCountsUnicodeCodePointsAndAcceptsTheExactEnd() throws Exception {
String id = "reference-test";
register(id, Optional.of((sdk, target) -> List.of(new FrontendReferenceProvider.Reference(
id + ":text", "Unicode text", FrontendReferenceProvider.Kind.QUICKSTART, null, "unknown", "A😀é\nB"))));
try {
var knowledge = knowledge(id, "1");
var emoji = knowledge.readReference(root, id + ":text", 1, 1);
assertEquals("😀", emoji.text());
assertEquals(5, emoji.total());
assertEquals(2, emoji.nextOffset());
var end = knowledge.readReference(root, id + ":text", 5, 100);
assertEquals("", end.text()); assertNull(end.nextOffset());
for (int offset : List.of(-1, 6, Integer.MAX_VALUE))
assertThrows(IllegalArgumentException.class, () -> knowledge.readReference(root, id + ":text", offset, 1));
for (int limit : List.of(0, -1, 16001))
assertThrows(IllegalArgumentException.class, () -> knowledge.readReference(root, id + ":text", 0, limit));
} finally { FrontendRegistryService.unregister(id); }
}
@Test void missingDocumentationAndUnsupportedLanguageHaveExplicitFailures() throws Exception {
String id = "reference-test";
register(id, Optional.empty());
try {
var knowledge = knowledge(id, "1");
assertTrue(knowledge.context(root).references().isEmpty());
assertEquals(AutomationFailure.Code.DOCUMENTATION_UNAVAILABLE, assertThrows(AutomationException.class,
() -> knowledge.readReference(root, id + ":text", 0, 100)).failure().code());
writeManifest("uninstalled", "1");
assertEquals(AutomationFailure.Code.UNSUPPORTED_LANGUAGE, assertThrows(AutomationException.class,
() -> knowledge.readReference(root, id + ":text", 0, 100)).failure().code());
} finally { FrontendRegistryService.unregister(id); }
}
@Test void failingProviderDoesNotAdvertiseUnreadableReferences() throws Exception {
String id = "reference-test";
register(id, Optional.of((sdk, target) -> { throw new IllegalStateException("Missing installed index"); }));
try {
var knowledge = knowledge(id, "1");
var context = knowledge.context(root);
assertTrue(context.references().isEmpty());
assertTrue(context.limitations().stream().anyMatch(value -> value.contains("cannot load installed references")));
assertEquals(AutomationFailure.Code.DOCUMENTATION_UNAVAILABLE, assertThrows(AutomationException.class,
() -> knowledge.readReference(root, id + ":text", 0, 100)).failure().code());
} finally { FrontendRegistryService.unregister(id); }
}
private void register(String id, Optional<FrontendReferenceProvider> documentation) {
FrontendRegistryService.register(new FrontendProvider() {
@Override public FrontendSpec specification() {
return FrontendSpec.builder().languageId(id).allowedExtensions(ReadOnlySet.from("alt"))
.sourceRoots(ReadOnlySet.from("src")).stdlibVersions(List.of(FrontendSpec.Stdlib.of(1))).build();
}
@Override public FrontendPhaseService compiler() { throw new AssertionError("No compilation for reference reads"); }
@Override public Optional<FrontendReferenceProvider> documentation() { return documentation; }
});
}
}
@@ -129,6 +129,20 @@ class LocalMcpServerTest {
assertEquals("pbs", value.path("languages").get(0).path("id").asText());
assertTrue(value.path("languages").get(0).path("languageVersion").path("value").isNull());
assertFalse(java.nio.file.Files.exists(root.resolve(".studio")));
var params = (com.fasterxml.jackson.databind.node.ObjectNode) call.path("params");
params.put("name", "studio_reference_read");
var arguments = (com.fasterxml.jackson.databind.node.ObjectNode) params.path("arguments");
arguments.put("referenceId", "pbs:quickstart");
var page = mapper.readTree(post(call.toString()).body()).path("result").path("structuredContent").path("value");
assertEquals(0, page.path("offset").asInt());
assertEquals(4000, page.path("text").asText().codePointCount(0, page.path("text").asText().length()));
assertEquals(4000, page.path("nextOffset").asInt());
arguments.put("offset", 4294967296L);
assertEquals(-32602, mapper.readTree(post(call.toString()).body()).path("error").path("code").asInt());
arguments.remove("offset");
arguments.put("referenceId", "../../prometeu.json");
var missing = mapper.readTree(post(call.toString()).body()).path("result");
assertEquals("REFERENCE_NOT_FOUND", missing.path("structuredContent").path("error").path("code").asText());
registration.unsubscribe();
var denied = mapper.readTree(post(call.toString()).body()).path("result");
assertTrue(denied.path("isError").asBoolean());
@@ -1,3 +1,5 @@
import java.security.MessageDigest
plugins {
id("gradle.java-library-conventions")
}
@@ -8,3 +10,47 @@ dependencies {
implementation(project(":prometeu-compiler:prometeu-frontend-api"))
implementation(project(":prometeu-compiler:prometeu-compiler-core"))
}
// Build-time projection of owner-maintained sources; no second normative copy is edited by Studio.
val pbsSpecs = rootProject.layout.projectDirectory.dir("docs/specs/compiler-languages/pbs")
val pbsResources = layout.projectDirectory.dir("src/main/resources")
val generatePbsKnowledge by tasks.registering(Sync::class) {
from(pbsSpecs) { include("*.md"); into("pbs-knowledge/specs") }
into(layout.buildDirectory.dir("generated/pbsKnowledge"))
inputs.files(fileTree(pbsResources) { include("game/stdlib/**/*.pbs", "game/stdlib/**/*.barrel", "pbs-knowledge/examples/**") })
doLast {
val entries = mutableListOf<Map<String, Any?>>()
fun entry(id: String, title: String, kind: String, resource: String, source: File, sdk: Int? = null, target: String? = null) {
check(source.isFile) { "Missing PBS reference: $source" }
val digest = MessageDigest.getInstance("SHA-256").digest(source.readBytes())
.joinToString("") { "%02x".format(it) }
entries.add(linkedMapOf("id" to id, "title" to title, "kind" to kind,
"resource" to resource, "sha256" to digest, "sdkVersion" to sdk, "target" to target))
}
pbsSpecs.asFile.listFiles()!!.filter { it.extension == "md" }.sortedBy { it.name }.forEach { source ->
val chapter = source.name.substringBefore(". ")
val quickstart = source.name == "README.md"
entry(if (quickstart) "pbs:quickstart" else "pbs:spec:$chapter",
source.readLines().first { it.startsWith("# ") }.removePrefix("# "),
if (quickstart) "QUICKSTART" else if (chapter == "3") "SYNTAX" else "SEMANTICS",
"pbs-knowledge/specs/${source.name}", source)
}
fileTree(pbsResources) { include("game/stdlib/**/*.pbs", "game/stdlib/**/*.barrel") }.files.sorted().forEach { source ->
val path = source.relativeTo(pbsResources.asFile).invariantSeparatorsPath
val parts = path.split('/')
val sdk = parts[2].toInt()
val module = parts.drop(3).dropLast(1).joinToString(":")
entry("pbs:sdk:game:$sdk:${parts.drop(3).joinToString("/")}", "PBS SDK $sdk @$module / ${source.name}",
"SDK", path, source, sdk, "Game")
}
fileTree(pbsResources.dir("pbs-knowledge/examples/game/1")) { include("**/*") }.files.sorted().forEach { source ->
val relative = source.relativeTo(pbsResources.dir("pbs-knowledge/examples/game/1").asFile).invariantSeparatorsPath
entry("pbs:example:game:1:$relative", "PBS Game / SDK 1 example: $relative", "EXAMPLE",
"pbs-knowledge/examples/game/1/$relative", source, 1, "Game")
}
check(entries.map { it["id"] }.distinct().size == entries.size) { "Duplicate PBS reference IDs" }
destinationDir.resolve("pbs-knowledge/index.json").writeText(groovy.json.JsonOutput.prettyPrint(
groovy.json.JsonOutput.toJson(mapOf("schemaVersion" to 1, "owner" to "pbs", "references" to entries))))
}
}
sourceSets.main { resources.srcDir(generatePbsKnowledge) }
@@ -11,6 +11,8 @@ import java.util.Optional;
public final class PBSFrontendProvider implements FrontendProvider {
private final FrontendPhaseService compiler;
private final PBSFrontendLanguageService languageService;
private final p.studio.compiler.services.FrontendReferenceProvider documentation =
new p.studio.compiler.pbs.documentation.PbsDocumentation();
public PBSFrontendProvider() {
this(new PBSFrontendPhaseService(), new PBSFrontendLanguageService());
@@ -41,4 +43,9 @@ public final class PBSFrontendProvider implements FrontendProvider {
public Optional<FrontendLanguageService> languageService() {
return Optional.of(languageService);
}
@Override
public Optional<p.studio.compiler.services.FrontendReferenceProvider> documentation() {
return Optional.of(documentation);
}
}
@@ -0,0 +1,78 @@
package p.studio.compiler.pbs.documentation;
import com.fasterxml.jackson.databind.ObjectMapper;
import p.studio.AppMode;
import p.studio.compiler.services.FrontendReferenceProvider;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.*;
import java.io.InputStream;
import java.util.function.Function;
/** Loads only the build-generated PBS index and its packaged resources, never user-supplied paths. */
public final class PbsDocumentation implements FrontendReferenceProvider {
private record Index(int schemaVersion, String owner, List<Entry> references) {}
private record Entry(String id, String title, Kind kind, String resource, String sha256,
Integer sdkVersion, String target) {}
private record Installed(Entry entry, Reference reference) {}
private final Function<String, InputStream> resources;
private volatile List<Installed> installed;
public PbsDocumentation() { this(PbsDocumentation.class.getClassLoader()::getResourceAsStream); }
PbsDocumentation(Function<String, InputStream> resources) { this.resources = Objects.requireNonNull(resources); }
@Override public List<Reference> references(int stdlibVersion, AppMode target) {
Objects.requireNonNull(target);
return installed().stream()
.filter(item -> item.entry().sdkVersion() == null || item.entry().sdkVersion() == stdlibVersion)
.filter(item -> item.entry().target() == null || item.entry().target().equals(target.name()))
.map(Installed::reference).toList();
}
private List<Installed> installed() {
if (installed != null) return installed;
synchronized (this) {
if (installed != null) return installed;
try {
var index = new ObjectMapper().readValue(resource("pbs-knowledge/index.json"), Index.class);
if (index.schemaVersion() != 1 || !"pbs".equals(index.owner()) || index.references() == null)
throw new IOException("Invalid PBS reference index");
var entries = new ArrayList<Installed>();
var ids = new HashSet<String>();
for (var entry : index.references()) {
if (entry.id() == null || !entry.id().startsWith("pbs:") || !ids.add(entry.id())
|| entry.sdkVersion() != null && entry.sdkVersion() < 1)
throw new IOException("Invalid or duplicate PBS reference identity");
byte[] bytes = resource(entry.resource());
if (!HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)).equals(entry.sha256()))
throw new IOException("PBS reference does not match its installed index");
String content = StandardCharsets.UTF_8.newDecoder().onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT).decode(ByteBuffer.wrap(bytes)).toString();
entries.add(new Installed(entry, new Reference(entry.id(), entry.title(), entry.kind(),
entry.sdkVersion() == null ? null : entry.sdkVersion().toString(),
entry.sdkVersion() == null ? "unknown" : "installed PBS SDK major; target=" + entry.target(), content)));
}
installed = List.copyOf(entries);
return installed;
} catch (Exception failure) {
throw new IllegalStateException("Installed PBS documentation is unavailable", failure);
}
}
}
private byte[] resource(String path) throws IOException {
if (path == null || path.contains("..") || path.contains(":") || path.contains("\\")
|| !(path.startsWith("pbs-knowledge/") || path.startsWith("game/stdlib/")))
throw new IOException("Invalid PBS reference resource");
try (var stream = resources.apply(path)) {
if (stream == null) throw new IOException("Missing PBS reference resource");
byte[] bytes = stream.readNBytes(4 * 1024 * 1024 + 1);
if (bytes.length > 4 * 1024 * 1024) throw new IOException("PBS reference resource is too large");
return bytes;
}
}
}
@@ -0,0 +1,8 @@
{
"name": "pbs-reference-example",
"version": "1.0.0",
"language": "pbs",
"stdlib": "1",
"target": "Game",
"dependencies": []
}
@@ -0,0 +1,19 @@
import { Color } from @core:color;
import { Gfx } from @sdk:gfx;
declare global x: int = 0;
[Init]
fn init() -> void {
x = 0;
}
[Frame]
fn frame() -> void {
Gfx.clear(Color.rgb(16, 20, 28));
Gfx.fill_rect(x, 32, 16, 16, Color.rgb(240, 160, 48));
x = x + 1;
if (x > 120) {
x = 0;
}
}
@@ -0,0 +1,2 @@
pub fn init() -> void;
pub fn frame() -> void;
@@ -0,0 +1,60 @@
package p.studio.compiler.pbs.documentation;
import org.junit.jupiter.api.Test;
import p.studio.AppMode;
import p.studio.compiler.services.FrontendReferenceProvider;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.*;
import static org.junit.jupiter.api.Assertions.*;
class PbsDocumentationTest {
@Test void installedIndexHasAllKindsUniqueIdsAndExactSdkSources() throws Exception {
var references = new PbsDocumentation().references(1, AppMode.Game);
assertEquals(references.size(), references.stream().map(FrontendReferenceProvider.Reference::id).distinct().count());
for (var kind : List.of(FrontendReferenceProvider.Kind.QUICKSTART, FrontendReferenceProvider.Kind.SYNTAX,
FrontendReferenceProvider.Kind.SEMANTICS, FrontendReferenceProvider.Kind.EXAMPLE, FrontendReferenceProvider.Kind.SDK))
assertTrue(references.stream().anyMatch(ref -> ref.kind() == kind));
for (var reference : references) {
assertFalse(reference.text().isBlank());
if (reference.kind() != FrontendReferenceProvider.Kind.SDK) continue;
try (var stream = getClass().getClassLoader().getResourceAsStream("game/stdlib/1/" + reference.id().substring("pbs:sdk:game:1:".length()))) {
assertNotNull(stream);
assertEquals(new String(stream.readAllBytes(), StandardCharsets.UTF_8), reference.text());
assertEquals("1", reference.version());
}
}
}
@Test void unsupportedSdkOrTargetDoesNotBorrowAnotherSdkOrExample() {
var docs = new PbsDocumentation();
for (var selection : List.of(docs.references(999, AppMode.Game), docs.references(1, AppMode.Shell))) {
assertFalse(selection.isEmpty());
assertTrue(selection.stream().noneMatch(ref -> ref.kind() == FrontendReferenceProvider.Kind.SDK || ref.kind() == FrontendReferenceProvider.Kind.EXAMPLE));
assertTrue(selection.stream().allMatch(ref -> ref.version() == null && ref.versionSource().equals("unknown")));
}
}
@Test void missingIndexAndCorruptedContentFailExplicitly() {
assertThrows(IllegalStateException.class, () -> new PbsDocumentation(path -> null).references(1, AppMode.Game));
var corrupt = new PbsDocumentation(path -> path.equals("pbs-knowledge/index.json")
? getClass().getClassLoader().getResourceAsStream(path)
: new ByteArrayInputStream("corrupted resource".getBytes(StandardCharsets.UTF_8)));
assertThrows(IllegalStateException.class, () -> corrupt.references(1, AppMode.Game));
}
@Test void invalidIndexResourceCannotReachTheResourceLoader() {
var opened = new ArrayList<String>();
String invalidIndex = """
{"schemaVersion":1,"owner":"pbs","references":[
{"id":"pbs:bad","title":"Bad","kind":"SYNTAX","resource":"../private.txt","sha256":"invalid"}
]}
""";
var docs = new PbsDocumentation(path -> {
opened.add(path);
return new ByteArrayInputStream(invalidIndex.getBytes(StandardCharsets.UTF_8));
});
assertThrows(IllegalStateException.class, () -> docs.references(1, AppMode.Game));
assertEquals(List.of("pbs-knowledge/index.json"), opened);
}
}
@@ -12,4 +12,8 @@ public interface FrontendProvider {
default Optional<FrontendLanguageService> languageService() {
return Optional.empty();
}
default Optional<FrontendReferenceProvider> documentation() {
return Optional.empty();
}
}
@@ -0,0 +1,21 @@
package p.studio.compiler.services;
import p.studio.AppMode;
import java.util.List;
import java.util.Objects;
/** Owner-maintained, installed reference texts. This is not a filesystem or network reader. */
public interface FrontendReferenceProvider {
List<Reference> references(int stdlibVersion, AppMode target);
enum Kind { QUICKSTART, SYNTAX, SEMANTICS, EXAMPLE, SDK, LIBRARY }
record Reference(String id, String title, Kind kind, String version, String versionSource, String text) {
public Reference {
Objects.requireNonNull(id); Objects.requireNonNull(title); Objects.requireNonNull(kind);
Objects.requireNonNull(versionSource); Objects.requireNonNull(text);
if (id.isBlank() || title.isBlank() || versionSource.isBlank() || version != null && version.isBlank())
throw new IllegalArgumentException("Blank reference metadata");
}
}
}