Document Agentbox runtime and model workflow

Co-authored-by: codegirl007 <codegirl-007@users.noreply.github.com>
2026-08-17 16:47:22 +00:00
parent 94146fcae2
commit e9197798f9
2 changed files with 123 additions and 53 deletions
@@ -1,13 +1,16 @@
-# Agentbox Phase 1
+# Agentbox

 This is a local technical spike for a programmable graphical Linux
 environment. It proves that a controller can launch an ordinary GUI process
-headlessly, observe pixels, inject keyboard and mouse input, and shut the
+headlessly, observe pixels and logs, inject keyboard and mouse input, drive the
+application through a replaceable agent, record a trace, and shut the
 environment down.

-Phase 1 does not contain an LLM, model provider, cloud control plane, or
-production sandbox. See [the architecture decision](docs/architecture.md) for
-the choices and security boundary.
+The default agent is deterministic and requires no API credentials. One
+optional OpenAI Responses adapter demonstrates screenshot + text model control.
+This is still a local spike, not a cloud control plane or production sandbox.
+See [the architecture decision](docs/architecture.md) for the choices and
+security boundary.

 ## Requirements

@@ -19,41 +22,78 @@ the choices and security boundary.
 No X server, window manager, or C compiler is required on the host; those
 dependencies are built into the container image.

-## Run the proof
+## One-command demo

 From this directory:

 ```sh
-make phase1
+make demo
 ```

 That one command:

-1. builds the local environment image;
-2. starts a restricted container with Xvfb and Openbox;
-3. launches the mover application;
-4. captures `before.png`;
-5. moves and clicks the mouse;
-6. holds the RIGHT key for one second;
-7. captures `after.png`;
-8. locates the green square in both images and fails unless it moved at least
-   100 pixels right;
-9. stops and removes the container, including on failure or interruption.
+1. builds the `agentbox` CLI and mover executable;
+2. creates a restricted Xvfb/Openbox environment;
+3. stages and launches the executable;
+4. runs the deterministic observe/press/wait/release/observe loop;
+5. records screenshots, actions, decisions, logs, and final status;
+6. stops and removes the environment, including on failure or interruption.

-A passing run ends with output similar to:
+The equivalent explicit commands are:

-```text
-Capturing screenshot before input...
-Sending synthetic mouse input...
-Holding RIGHT for one second...
-Capturing screenshot after input...
-Verified: square moved 180.0 pixels to the right.
-Phase 1 passed. Artifacts: .../.agentbox/runs/20260817T...
-Stopping environment...
-Environment stopped cleanly.
+```sh
+make build demo-binary
+./agentbox run ./examples/mover \
+  --task "Launch the application, move the character to the right, and describe what happened."
 ```

-## Artifacts
+Flags can appear before or after the path.
+
+## Run a private executable
+
+Pass a prebuilt Linux executable:
+
+```sh
+./agentbox run ./path/to/application --task "Open the menu and click Settings."
+```
+
+For a directory, add `agentbox.json`:
+
+```json
+{
+  "command": "game",
+  "args": ["--windowed"],
+  "env": {"EXAMPLE": "value"},
+  "window_title": "My Game"
+}
+```
+
+The command is resolved relative to the manifest. The current runtime is
+Debian-based, so dynamically linked executables must have compatible libraries.
+
+## Model agent
+
+The model adapter is opt-in:
+
+```sh
+export OPENAI_API_KEY="..."
+./agentbox run ./path/to/application \
+  --task "Move the character right and report the result." \
+  --agent openai \
+  --model gpt-5
+```
+
+Each request contains the task, current screenshot, previous actions, step
+history, and recent application logs. Strict structured output allows one
+backend-neutral input action or completion. Use `--max-steps` to bound a run.
+The environment itself still has no network access.
+
+## Runs and inspection
+
+```sh
+./agentbox runs
+./agentbox inspect <run-id>
+```

 Each run writes:

@@ -61,17 +101,25 @@ Each run writes:
 .agentbox/runs/<run-id>/
 ├── run.json
 ├── actions.jsonl
+├── steps.jsonl
 ├── screenshots/
-│   ├── before.png
-│   └── after.png
+│   ├── 0001.png
+│   └── ...
 ├── stdout.log
 └── stderr.log
 ```

-`run.json` contains the measured before/after centroids, movement distance,
-container-engine version, timestamps, and pass/fail status. `actions.jsonl`
-contains the backend-neutral input actions sent during this proof. Application
-output is copied from the container before it is destroyed.
+`run.json` has `schema_version: "1"` plus task, application, agent, timestamps,
+status, and step count. `steps.jsonl` records each observation reference, agent
+message, and action. `actions.jsonl` is a compact action-only stream.
+
+## Original environment proof
+
+The visual centroid test from Phase 1 remains available:
+
+```sh
+make phase1
+```

 ## Development checks

@@ -79,6 +127,3 @@ output is copied from the container before it is destroyed.
 make test
 go vet ./...
 ```
-
-The future `agentbox run <path> --task ...` flow belongs to Phases 2 and 3.
-This branch intentionally stops after proving the environment mechanism.
@@ -1,6 +1,6 @@
-# Phase 1 architecture decision
+# Agentbox spike architecture decision

-Status: accepted for the local spike
+Status: implemented for Phases 1–4 of the local spike

 ## Decision

@@ -12,10 +12,10 @@ Run one application environment per Docker container. Inside the container:
 - `scrot` captures the complete display as PNG.
 - `xdotool` injects keyboard and mouse events through X11's XTEST extension.

-The Go process on the host owns the lifecycle. It builds and creates the
+The Go control plane on the host owns the lifecycle. It builds and creates the
 container, starts the display, launches the application as a separate step,
 captures images, sends input, copies artifacts out, and removes the container.
-The first spike intentionally uses the Docker CLI as its narrow adapter rather
+The spike intentionally uses the Docker CLI as its narrow adapter rather
 than adding a Docker SDK dependency.

 ```text
@@ -79,23 +79,48 @@ source or package dependency without changing the tested path. The runtime is
 not coupled to Xlib: any executable in a future staged image can use SDL,
 Raylib, Qt, GTK, a browser, or another X11-compatible toolkit.

-## Separation preserved for later phases
+## Runtime separation

-Phase 1 contains concrete orchestration, but its data flow already keeps these
-roles distinct:
+The implemented data flow keeps these roles distinct:

 ```text
-CLI -> lifecycle controller -> container environment -> application
-                                      |
-                                observation/input
-                                      |
-                              deterministic driver
+CLI -> runtime controller -> Environment -> application
+               |                |
+               |         observation/input/logs
+               |
+             Agent
+               |
+       deterministic or OpenAI
+               |
+          versioned trace
 ```

-Phase 2 should extract lifecycle, screenshot, input, and logs into an
-`Environment` interface with backend-neutral actions. Phase 3 should consume
-that interface through an `Agent`; neither a deterministic agent nor a model
-adapter should import Docker or X11 details.
+`internal/environment` defines lifecycle, screenshot, input, and log contracts
+with backend-neutral actions. `internal/environment/dockerx11` is the only
+package that knows about Docker, X11, `scrot`, or `xdotool`.
+
+`internal/agent` defines observations, history, and decisions. The deterministic
+agent and optional OpenAI Responses adapter both implement that interface; they
+do not import the Docker backend. The model adapter receives the task, current
+PNG, action history, and recent logs, then returns one schema-constrained
+action. It is intentionally one concrete adapter, not a provider framework.
+
+`internal/runtime` connects those interfaces and contains no provider or X11
+logic. `internal/trace` records versioned metadata plus append-only step and
+action streams. Screenshots remain separate PNG files referenced by relative
+path, which keeps traces readable and suitable for later replay or comparison.
+
+## Application staging
+
+`agentbox run` accepts one prebuilt Linux executable. A directory can contain
+an `agentbox.json` manifest naming that executable, arguments, environment
+variables, and an optional window title. The Docker backend streams the file
+into the environment's tmpfs and makes it executable; no host directory is
+mounted into the container.
+
+The executable and its libraries must be compatible with the Debian-based
+runtime image. Packaging arbitrary dependency trees is a separate upload/build
+format problem and is not hidden by this spike.

 ## Security boundary