diff --git a/agentbox/README.md b/agentbox/README.md index 3203d7f..dbbfd8b 100644 --- a/agentbox/README.md +++ b/agentbox/README.md @@ -1,13 +1,16 @@ -# Agentbox Phase 1 +# Agentbox This is a local technical spike for a programmable graphical Linux environment. It proves that a controller can launch an ordinary GUI process -headlessly, observe pixels, inject keyboard and mouse input, and shut the +headlessly, observe pixels and logs, inject keyboard and mouse input, drive the +application through a replaceable agent, record a trace, and shut the environment down. -Phase 1 does not contain an LLM, model provider, cloud control plane, or -production sandbox. See [the architecture decision](docs/architecture.md) for -the choices and security boundary. +The default agent is deterministic and requires no API credentials. One +optional OpenAI Responses adapter demonstrates screenshot + text model control. +This is still a local spike, not a cloud control plane or production sandbox. +See [the architecture decision](docs/architecture.md) for the choices and +security boundary. ## Requirements @@ -19,41 +22,78 @@ the choices and security boundary. No X server, window manager, or C compiler is required on the host; those dependencies are built into the container image. -## Run the proof +## One-command demo From this directory: ```sh -make phase1 +make demo ``` That one command: -1. builds the local environment image; -2. starts a restricted container with Xvfb and Openbox; -3. launches the mover application; -4. captures `before.png`; -5. moves and clicks the mouse; -6. holds the RIGHT key for one second; -7. captures `after.png`; -8. locates the green square in both images and fails unless it moved at least - 100 pixels right; -9. stops and removes the container, including on failure or interruption. +1. builds the `agentbox` CLI and mover executable; +2. creates a restricted Xvfb/Openbox environment; +3. stages and launches the executable; +4. runs the deterministic observe/press/wait/release/observe loop; +5. records screenshots, actions, decisions, logs, and final status; +6. stops and removes the environment, including on failure or interruption. -A passing run ends with output similar to: +The equivalent explicit commands are: -```text -Capturing screenshot before input... -Sending synthetic mouse input... -Holding RIGHT for one second... -Capturing screenshot after input... -Verified: square moved 180.0 pixels to the right. -Phase 1 passed. Artifacts: .../.agentbox/runs/20260817T... -Stopping environment... -Environment stopped cleanly. +```sh +make build demo-binary +./agentbox run ./examples/mover \ + --task "Launch the application, move the character to the right, and describe what happened." ``` -## Artifacts +Flags can appear before or after the path. + +## Run a private executable + +Pass a prebuilt Linux executable: + +```sh +./agentbox run ./path/to/application --task "Open the menu and click Settings." +``` + +For a directory, add `agentbox.json`: + +```json +{ + "command": "game", + "args": ["--windowed"], + "env": {"EXAMPLE": "value"}, + "window_title": "My Game" +} +``` + +The command is resolved relative to the manifest. The current runtime is +Debian-based, so dynamically linked executables must have compatible libraries. + +## Model agent + +The model adapter is opt-in: + +```sh +export OPENAI_API_KEY="..." +./agentbox run ./path/to/application \ + --task "Move the character right and report the result." \ + --agent openai \ + --model gpt-5 +``` + +Each request contains the task, current screenshot, previous actions, step +history, and recent application logs. Strict structured output allows one +backend-neutral input action or completion. Use `--max-steps` to bound a run. +The environment itself still has no network access. + +## Runs and inspection + +```sh +./agentbox runs +./agentbox inspect +``` Each run writes: @@ -61,17 +101,25 @@ Each run writes: .agentbox/runs// ├── run.json ├── actions.jsonl +├── steps.jsonl ├── screenshots/ -│ ├── before.png -│ └── after.png +│ ├── 0001.png +│ └── ... ├── stdout.log └── stderr.log ``` -`run.json` contains the measured before/after centroids, movement distance, -container-engine version, timestamps, and pass/fail status. `actions.jsonl` -contains the backend-neutral input actions sent during this proof. Application -output is copied from the container before it is destroyed. +`run.json` has `schema_version: "1"` plus task, application, agent, timestamps, +status, and step count. `steps.jsonl` records each observation reference, agent +message, and action. `actions.jsonl` is a compact action-only stream. + +## Original environment proof + +The visual centroid test from Phase 1 remains available: + +```sh +make phase1 +``` ## Development checks @@ -79,6 +127,3 @@ output is copied from the container before it is destroyed. make test go vet ./... ``` - -The future `agentbox run --task ...` flow belongs to Phases 2 and 3. -This branch intentionally stops after proving the environment mechanism. diff --git a/agentbox/docs/architecture.md b/agentbox/docs/architecture.md index e728564..c3d4397 100644 --- a/agentbox/docs/architecture.md +++ b/agentbox/docs/architecture.md @@ -1,6 +1,6 @@ -# Phase 1 architecture decision +# Agentbox spike architecture decision -Status: accepted for the local spike +Status: implemented for Phases 1–4 of the local spike ## Decision @@ -12,10 +12,10 @@ Run one application environment per Docker container. Inside the container: - `scrot` captures the complete display as PNG. - `xdotool` injects keyboard and mouse events through X11's XTEST extension. -The Go process on the host owns the lifecycle. It builds and creates the +The Go control plane on the host owns the lifecycle. It builds and creates the container, starts the display, launches the application as a separate step, captures images, sends input, copies artifacts out, and removes the container. -The first spike intentionally uses the Docker CLI as its narrow adapter rather +The spike intentionally uses the Docker CLI as its narrow adapter rather than adding a Docker SDK dependency. ```text @@ -79,23 +79,48 @@ source or package dependency without changing the tested path. The runtime is not coupled to Xlib: any executable in a future staged image can use SDL, Raylib, Qt, GTK, a browser, or another X11-compatible toolkit. -## Separation preserved for later phases +## Runtime separation -Phase 1 contains concrete orchestration, but its data flow already keeps these -roles distinct: +The implemented data flow keeps these roles distinct: ```text -CLI -> lifecycle controller -> container environment -> application - | - observation/input - | - deterministic driver +CLI -> runtime controller -> Environment -> application + | | + | observation/input/logs + | + Agent + | + deterministic or OpenAI + | + versioned trace ``` -Phase 2 should extract lifecycle, screenshot, input, and logs into an -`Environment` interface with backend-neutral actions. Phase 3 should consume -that interface through an `Agent`; neither a deterministic agent nor a model -adapter should import Docker or X11 details. +`internal/environment` defines lifecycle, screenshot, input, and log contracts +with backend-neutral actions. `internal/environment/dockerx11` is the only +package that knows about Docker, X11, `scrot`, or `xdotool`. + +`internal/agent` defines observations, history, and decisions. The deterministic +agent and optional OpenAI Responses adapter both implement that interface; they +do not import the Docker backend. The model adapter receives the task, current +PNG, action history, and recent logs, then returns one schema-constrained +action. It is intentionally one concrete adapter, not a provider framework. + +`internal/runtime` connects those interfaces and contains no provider or X11 +logic. `internal/trace` records versioned metadata plus append-only step and +action streams. Screenshots remain separate PNG files referenced by relative +path, which keeps traces readable and suitable for later replay or comparison. + +## Application staging + +`agentbox run` accepts one prebuilt Linux executable. A directory can contain +an `agentbox.json` manifest naming that executable, arguments, environment +variables, and an optional window title. The Docker backend streams the file +into the environment's tmpfs and makes it executable; no host directory is +mounted into the container. + +The executable and its libraries must be compatible with the Debian-based +runtime image. Packaging arbitrary dependency trees is a separate upload/build +format problem and is not hidden by this spike. ## Security boundary