Add headless Agentbox environment spike
Co-authored-by: codegirl007 <codegirl-007@users.noreply.github.com>
This commit is contained in:
Cursor Agent
co-authored by
codegirl007
codegirl007
parent
98f56833a0
commit
96dade334d
@@ -0,0 +1,2 @@
|
|||||||
|
.agentbox/
|
||||||
|
agentbox
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
.PHONY: phase1 test
|
||||||
|
|
||||||
|
phase1:
|
||||||
|
go run ./cmd/agentbox phase1
|
||||||
|
|
||||||
|
test:
|
||||||
|
go test ./...
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
# Agentbox Phase 1
|
||||||
|
|
||||||
|
This is a local technical spike for a programmable graphical Linux
|
||||||
|
environment. It proves that a controller can launch an ordinary GUI process
|
||||||
|
headlessly, observe pixels, inject keyboard and mouse input, and shut the
|
||||||
|
environment down.
|
||||||
|
|
||||||
|
Phase 1 does not contain an LLM, model provider, cloud control plane, or
|
||||||
|
production sandbox. See [the architecture decision](docs/architecture.md) for
|
||||||
|
the choices and security boundary.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Linux
|
||||||
|
- Go 1.22 or newer
|
||||||
|
- Docker Engine with a running daemon
|
||||||
|
- permission to use Docker without an interactive `sudo` prompt
|
||||||
|
|
||||||
|
No X server, window manager, or C compiler is required on the host; those
|
||||||
|
dependencies are built into the container image.
|
||||||
|
|
||||||
|
## Run the proof
|
||||||
|
|
||||||
|
From this directory:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
make phase1
|
||||||
|
```
|
||||||
|
|
||||||
|
That one command:
|
||||||
|
|
||||||
|
1. builds the local environment image;
|
||||||
|
2. starts a restricted container with Xvfb and Openbox;
|
||||||
|
3. launches the mover application;
|
||||||
|
4. captures `before.png`;
|
||||||
|
5. moves and clicks the mouse;
|
||||||
|
6. holds the RIGHT key for one second;
|
||||||
|
7. captures `after.png`;
|
||||||
|
8. locates the green square in both images and fails unless it moved at least
|
||||||
|
100 pixels right;
|
||||||
|
9. stops and removes the container, including on failure or interruption.
|
||||||
|
|
||||||
|
A passing run ends with output similar to:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Capturing screenshot before input...
|
||||||
|
Sending synthetic mouse input...
|
||||||
|
Holding RIGHT for one second...
|
||||||
|
Capturing screenshot after input...
|
||||||
|
Verified: square moved 180.0 pixels to the right.
|
||||||
|
Phase 1 passed. Artifacts: .../.agentbox/runs/20260817T...
|
||||||
|
Stopping environment...
|
||||||
|
Environment stopped cleanly.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
Each run writes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
.agentbox/runs/<run-id>/
|
||||||
|
├── run.json
|
||||||
|
├── actions.jsonl
|
||||||
|
├── screenshots/
|
||||||
|
│ ├── before.png
|
||||||
|
│ └── after.png
|
||||||
|
├── stdout.log
|
||||||
|
└── stderr.log
|
||||||
|
```
|
||||||
|
|
||||||
|
`run.json` contains the measured before/after centroids, movement distance,
|
||||||
|
container-engine version, timestamps, and pass/fail status. `actions.jsonl`
|
||||||
|
contains the backend-neutral input actions sent during this proof. Application
|
||||||
|
output is copied from the container before it is destroyed.
|
||||||
|
|
||||||
|
## Development checks
|
||||||
|
|
||||||
|
```sh
|
||||||
|
make test
|
||||||
|
go vet ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
The future `agentbox run <path> --task ...` flow belongs to Phases 2 and 3.
|
||||||
|
This branch intentionally stops after proving the environment mechanism.
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
# Phase 1 architecture decision
|
||||||
|
|
||||||
|
Status: accepted for the local spike
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Run one application environment per Docker container. Inside the container:
|
||||||
|
|
||||||
|
- Xvfb provides a 640×480, 24-bit, in-memory X11 display.
|
||||||
|
- Openbox gives ordinary desktop windows focus and placement behavior.
|
||||||
|
- The application renders to that display without knowing it is headless.
|
||||||
|
- `scrot` captures the complete display as PNG.
|
||||||
|
- `xdotool` injects keyboard and mouse events through X11's XTEST extension.
|
||||||
|
|
||||||
|
The Go process on the host owns the lifecycle. It builds and creates the
|
||||||
|
container, starts the display, launches the application as a separate step,
|
||||||
|
captures images, sends input, copies artifacts out, and removes the container.
|
||||||
|
The first spike intentionally uses the Docker CLI as its narrow adapter rather
|
||||||
|
than adding a Docker SDK dependency.
|
||||||
|
|
||||||
|
```text
|
||||||
|
Go phase-1 runner
|
||||||
|
|
|
||||||
|
+-- Docker lifecycle and process execution
|
||||||
|
|
|
||||||
|
+-- Xvfb display :99
|
||||||
|
| |
|
||||||
|
| +-- Openbox
|
||||||
|
| +-- arbitrary X11 application
|
||||||
|
|
|
||||||
|
+-- scrot observation
|
||||||
|
+-- xdotool keyboard/mouse input
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the smallest stack that exercises a real desktop input and rendering
|
||||||
|
path. Xvfb is an X server backed by memory instead of display hardware.
|
||||||
|
`xdotool` synthesizes standard X11 input, so the demo is not instrumented with
|
||||||
|
a private control API. The verification reads the before and after PNGs and
|
||||||
|
checks that the green square's pixel centroid moved, rather than trusting
|
||||||
|
application state or logs.
|
||||||
|
|
||||||
|
## Why this instead of the alternatives
|
||||||
|
|
||||||
|
### X11/Xvfb rather than Wayland
|
||||||
|
|
||||||
|
Wayland deliberately gives clients less global authority. Synthetic input and
|
||||||
|
whole-desktop capture are compositor-mediated, so a Wayland implementation
|
||||||
|
would require selecting and configuring a headless compositor plus its
|
||||||
|
specific control protocol. That is a good future backend, but it adds no
|
||||||
|
evidence to this first question. Xvfb, XTEST, and framebuffer capture are
|
||||||
|
mature, software-only, and replaceable behind the later `Environment`
|
||||||
|
interface.
|
||||||
|
|
||||||
|
X11 is not a security boundary. Processes sharing one X server can generally
|
||||||
|
observe or affect one another. The design therefore uses one display and one
|
||||||
|
container per environment.
|
||||||
|
|
||||||
|
### No VNC/noVNC yet
|
||||||
|
|
||||||
|
VNC is useful for a human live viewer, but it is not needed for programmatic
|
||||||
|
screenshots or input. Adding an RFB server and browser client would introduce
|
||||||
|
more processes, ports, encoding, and latency without improving the Phase 1
|
||||||
|
proof. A later observer can attach x11vnc, or the display backend can become an
|
||||||
|
Xvnc server, without changing agent actions.
|
||||||
|
|
||||||
|
### Software rendering first
|
||||||
|
|
||||||
|
Xvfb does not provide a modern GPU. Many toolkit applications can use software
|
||||||
|
rendering, which is sufficient for this proof. GPU-heavy games may require a
|
||||||
|
different environment backend using headless DRM/EGL, a virtual GPU, or GPU
|
||||||
|
passthrough. That compatibility question is explicitly not answered by this
|
||||||
|
spike.
|
||||||
|
|
||||||
|
### A small Xlib demo rather than Raylib
|
||||||
|
|
||||||
|
The demo is C/Xlib so the image has only distribution packages and proves the
|
||||||
|
display/input mechanism directly. Raylib would make a nicer example but adds a
|
||||||
|
source or package dependency without changing the tested path. The runtime is
|
||||||
|
not coupled to Xlib: any executable in a future staged image can use SDL,
|
||||||
|
Raylib, Qt, GTK, a browser, or another X11-compatible toolkit.
|
||||||
|
|
||||||
|
## Separation preserved for later phases
|
||||||
|
|
||||||
|
Phase 1 contains concrete orchestration, but its data flow already keeps these
|
||||||
|
roles distinct:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CLI -> lifecycle controller -> container environment -> application
|
||||||
|
|
|
||||||
|
observation/input
|
||||||
|
|
|
||||||
|
deterministic driver
|
||||||
|
```
|
||||||
|
|
||||||
|
Phase 2 should extract lifecycle, screenshot, input, and logs into an
|
||||||
|
`Environment` interface with backend-neutral actions. Phase 3 should consume
|
||||||
|
that interface through an `Agent`; neither a deterministic agent nor a model
|
||||||
|
adapter should import Docker or X11 details.
|
||||||
|
|
||||||
|
## Security boundary
|
||||||
|
|
||||||
|
The container flags reduce accidental damage: no network, read-only root
|
||||||
|
filesystem, a bounded tmpfs, a non-root user, all Linux capabilities dropped,
|
||||||
|
`no-new-privileges`, and CPU, memory, and PID limits. They are useful
|
||||||
|
defense-in-depth, not a production hostile-code sandbox.
|
||||||
|
|
||||||
|
Ordinary Docker containers share the host kernel. A kernel or container-runtime
|
||||||
|
escape can cross this boundary, resource-exhaustion controls are incomplete,
|
||||||
|
and image/build processing also handles attacker-controlled content. Do not run
|
||||||
|
arbitrary customer binaries with this spike on a valuable or multi-tenant
|
||||||
|
host.
|
||||||
|
|
||||||
|
Before production use, at minimum:
|
||||||
|
|
||||||
|
- put each untrusted workload behind a hardware-backed microVM boundary
|
||||||
|
(Firecracker or Kata Containers), or evaluate gVisor where its syscall and
|
||||||
|
graphics compatibility is sufficient;
|
||||||
|
- isolate image building from runtime hosts and verify limits on uploaded and
|
||||||
|
expanded content;
|
||||||
|
- enforce outbound network policy, ephemeral storage quotas, wall-clock
|
||||||
|
deadlines, and host-level CPU/memory/PID/I/O controls;
|
||||||
|
- use immutable, patched base images and a minimal guest kernel/filesystem;
|
||||||
|
- authenticate control operations and separate each tenant's artifacts,
|
||||||
|
credentials, logs, and encryption keys;
|
||||||
|
- destroy the environment after every run and monitor the host boundary.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- [X.Org Xvfb manual](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml)
|
||||||
|
- [xdotool project and XTEST behavior](https://github.com/jordansissel/xdotool)
|
||||||
|
- [Docker Engine security](https://docs.docker.com/engine/security/)
|
||||||
|
- [gVisor security model](https://gvisor.dev/docs/architecture_guide/security/)
|
||||||
|
- [Firecracker design](https://firecracker-microvm.github.io/)
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
FROM debian:bookworm-slim AS builder
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||||
|
build-essential \
|
||||||
|
libx11-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
COPY examples/mover/main.c /src/main.c
|
||||||
|
RUN cc -std=c11 -O2 -Wall -Wextra -Werror /src/main.c -lX11 -o /mover
|
||||||
|
|
||||||
|
FROM debian:bookworm-slim
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||||
|
libx11-6 \
|
||||||
|
openbox \
|
||||||
|
scrot \
|
||||||
|
x11-utils \
|
||||||
|
xdotool \
|
||||||
|
xvfb \
|
||||||
|
&& rm -rf /var/lib/apt/lists/* \
|
||||||
|
&& useradd --create-home --uid 10001 agentbox
|
||||||
|
|
||||||
|
COPY --from=builder /mover /opt/agentbox/mover
|
||||||
|
COPY environment/entrypoint.sh /opt/agentbox/entrypoint.sh
|
||||||
|
RUN chmod 0755 /opt/agentbox/entrypoint.sh /opt/agentbox/mover
|
||||||
|
|
||||||
|
USER agentbox
|
||||||
|
ENV DISPLAY=:99
|
||||||
|
ENTRYPOINT ["/opt/agentbox/entrypoint.sh"]
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
export HOME=/tmp/home
|
||||||
|
export XDG_RUNTIME_DIR=/tmp/runtime
|
||||||
|
mkdir -p "$HOME" "$XDG_RUNTIME_DIR"
|
||||||
|
chmod 0700 "$XDG_RUNTIME_DIR"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
rm -f /tmp/agentbox-ready
|
||||||
|
kill "${OPENBOX_PID:-}" "${XVFB_PID:-}" 2>/dev/null || true
|
||||||
|
wait "${OPENBOX_PID:-}" "${XVFB_PID:-}" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
terminate() {
|
||||||
|
trap - EXIT INT TERM
|
||||||
|
cleanup
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
trap cleanup EXIT
|
||||||
|
trap terminate INT TERM
|
||||||
|
|
||||||
|
Xvfb "$DISPLAY" -screen 0 640x480x24 -nolisten tcp -noreset &
|
||||||
|
XVFB_PID=$!
|
||||||
|
|
||||||
|
attempt=0
|
||||||
|
until xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; do
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
if [ "$attempt" -ge 100 ]; then
|
||||||
|
echo "Xvfb did not become ready" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
|
||||||
|
openbox --sm-disable >/tmp/openbox.log 2>&1 &
|
||||||
|
OPENBOX_PID=$!
|
||||||
|
touch /tmp/agentbox-ready
|
||||||
|
|
||||||
|
while :; do
|
||||||
|
sleep 3600 &
|
||||||
|
wait $!
|
||||||
|
done
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
#define _POSIX_C_SOURCE 199309L
|
||||||
|
|
||||||
|
#include <X11/Xlib.h>
|
||||||
|
#include <X11/keysym.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <time.h>
|
||||||
|
|
||||||
|
enum {
|
||||||
|
WINDOW_WIDTH = 640,
|
||||||
|
WINDOW_HEIGHT = 480,
|
||||||
|
SQUARE_SIZE = 50,
|
||||||
|
};
|
||||||
|
|
||||||
|
static double monotonic_seconds(void) {
|
||||||
|
struct timespec now;
|
||||||
|
if (clock_gettime(CLOCK_MONOTONIC, &now) != 0) {
|
||||||
|
perror("clock_gettime");
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
return (double)now.tv_sec + (double)now.tv_nsec / 1000000000.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void set_key_state(KeySym key, bool down, bool *left, bool *right,
|
||||||
|
bool *up, bool *down_key) {
|
||||||
|
switch (key) {
|
||||||
|
case XK_Left:
|
||||||
|
*left = down;
|
||||||
|
break;
|
||||||
|
case XK_Right:
|
||||||
|
*right = down;
|
||||||
|
break;
|
||||||
|
case XK_Up:
|
||||||
|
*up = down;
|
||||||
|
break;
|
||||||
|
case XK_Down:
|
||||||
|
*down_key = down;
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
setvbuf(stdout, NULL, _IOLBF, 0);
|
||||||
|
|
||||||
|
Display *display = XOpenDisplay(NULL);
|
||||||
|
if (display == NULL) {
|
||||||
|
fputs("unable to open X display\n", stderr);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int screen = DefaultScreen(display);
|
||||||
|
Window window = XCreateSimpleWindow(
|
||||||
|
display, RootWindow(display, screen), 0, 0, WINDOW_WIDTH, WINDOW_HEIGHT,
|
||||||
|
0, BlackPixel(display, screen), BlackPixel(display, screen));
|
||||||
|
XStoreName(display, window, "Agentbox Mover");
|
||||||
|
XSelectInput(display, window,
|
||||||
|
ExposureMask | KeyPressMask | KeyReleaseMask | ButtonPressMask);
|
||||||
|
XMapWindow(display, window);
|
||||||
|
|
||||||
|
GC gc = XCreateGC(display, window, 0, NULL);
|
||||||
|
Colormap colors = DefaultColormap(display, screen);
|
||||||
|
XColor green;
|
||||||
|
XColor red;
|
||||||
|
if (!XParseColor(display, colors, "#00ff66", &green) ||
|
||||||
|
!XAllocColor(display, colors, &green) ||
|
||||||
|
!XParseColor(display, colors, "#ff3355", &red) ||
|
||||||
|
!XAllocColor(display, colors, &red)) {
|
||||||
|
fputs("unable to allocate colors\n", stderr);
|
||||||
|
XCloseDisplay(display);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
double x = (WINDOW_WIDTH - SQUARE_SIZE) / 2.0;
|
||||||
|
double y = (WINDOW_HEIGHT - SQUARE_SIZE) / 2.0;
|
||||||
|
bool left = false;
|
||||||
|
bool right = false;
|
||||||
|
bool up = false;
|
||||||
|
bool down = false;
|
||||||
|
bool marker_visible = false;
|
||||||
|
int marker_x = 0;
|
||||||
|
int marker_y = 0;
|
||||||
|
double previous = monotonic_seconds();
|
||||||
|
|
||||||
|
printf("mover started at x=%.1f y=%.1f\n", x, y);
|
||||||
|
|
||||||
|
for (;;) {
|
||||||
|
while (XPending(display) > 0) {
|
||||||
|
XEvent event;
|
||||||
|
XNextEvent(display, &event);
|
||||||
|
if (event.type == KeyPress || event.type == KeyRelease) {
|
||||||
|
set_key_state(XLookupKeysym(&event.xkey, 0),
|
||||||
|
event.type == KeyPress, &left, &right, &up, &down);
|
||||||
|
} else if (event.type == ButtonPress) {
|
||||||
|
marker_visible = true;
|
||||||
|
marker_x = event.xbutton.x;
|
||||||
|
marker_y = event.xbutton.y;
|
||||||
|
printf("mouse click at x=%d y=%d\n", marker_x, marker_y);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
double now = monotonic_seconds();
|
||||||
|
double delta = now - previous;
|
||||||
|
previous = now;
|
||||||
|
const double speed = 180.0;
|
||||||
|
x += ((right ? 1.0 : 0.0) - (left ? 1.0 : 0.0)) * speed * delta;
|
||||||
|
y += ((down ? 1.0 : 0.0) - (up ? 1.0 : 0.0)) * speed * delta;
|
||||||
|
|
||||||
|
if (x < 0.0) {
|
||||||
|
x = 0.0;
|
||||||
|
} else if (x > WINDOW_WIDTH - SQUARE_SIZE) {
|
||||||
|
x = WINDOW_WIDTH - SQUARE_SIZE;
|
||||||
|
}
|
||||||
|
if (y < 0.0) {
|
||||||
|
y = 0.0;
|
||||||
|
} else if (y > WINDOW_HEIGHT - SQUARE_SIZE) {
|
||||||
|
y = WINDOW_HEIGHT - SQUARE_SIZE;
|
||||||
|
}
|
||||||
|
|
||||||
|
XClearWindow(display, window);
|
||||||
|
XSetForeground(display, gc, green.pixel);
|
||||||
|
XFillRectangle(display, window, gc, (int)x, (int)y, SQUARE_SIZE,
|
||||||
|
SQUARE_SIZE);
|
||||||
|
if (marker_visible) {
|
||||||
|
XSetForeground(display, gc, red.pixel);
|
||||||
|
XFillRectangle(display, window, gc, marker_x - 4, marker_y - 4, 9,
|
||||||
|
9);
|
||||||
|
}
|
||||||
|
XFlush(display);
|
||||||
|
|
||||||
|
struct timespec frame = {.tv_sec = 0, .tv_nsec = 16000000};
|
||||||
|
nanosleep(&frame, NULL);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
module agentbox
|
||||||
|
|
||||||
|
go 1.22
|
||||||
@@ -0,0 +1,401 @@
|
|||||||
|
package phase1
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"image/png"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const imageName = "agentbox-phase1:local"
|
||||||
|
|
||||||
|
type point struct {
|
||||||
|
X float64 `json:"x"`
|
||||||
|
Y float64 `json:"y"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type runRecord struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Phase string `json:"phase"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
StartedAt time.Time `json:"started_at"`
|
||||||
|
FinishedAt time.Time `json:"finished_at"`
|
||||||
|
ContainerEngine string `json:"container_engine"`
|
||||||
|
Before *point `json:"square_before,omitempty"`
|
||||||
|
After *point `json:"square_after,omitempty"`
|
||||||
|
MovementPixels float64 `json:"movement_pixels,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type actionRecord struct {
|
||||||
|
Timestamp time.Time `json:"timestamp"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
Key string `json:"key,omitempty"`
|
||||||
|
X int `json:"x,omitempty"`
|
||||||
|
Y int `json:"y,omitempty"`
|
||||||
|
Button int `json:"button,omitempty"`
|
||||||
|
Duration string `json:"duration,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type runner struct {
|
||||||
|
root string
|
||||||
|
runDir string
|
||||||
|
containerName string
|
||||||
|
containerMade bool
|
||||||
|
actions *os.File
|
||||||
|
record runRecord
|
||||||
|
}
|
||||||
|
|
||||||
|
func Run(ctx context.Context) (runErr error) {
|
||||||
|
root, err := findRoot()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
id, err := runID()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("create run ID: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
runDir := filepath.Join(root, ".agentbox", "runs", id)
|
||||||
|
if err := os.MkdirAll(filepath.Join(runDir, "screenshots"), 0o755); err != nil {
|
||||||
|
return fmt.Errorf("create run directory: %w", err)
|
||||||
|
}
|
||||||
|
actions, err := os.Create(filepath.Join(runDir, "actions.jsonl"))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("create action trace: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
r := &runner{
|
||||||
|
root: root,
|
||||||
|
runDir: runDir,
|
||||||
|
containerName: "agentbox-phase1-" + id,
|
||||||
|
actions: actions,
|
||||||
|
record: runRecord{
|
||||||
|
ID: id,
|
||||||
|
Phase: "environment-prototype",
|
||||||
|
Status: "running",
|
||||||
|
StartedAt: time.Now().UTC(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
cleanupErr := r.cleanup()
|
||||||
|
r.record.FinishedAt = time.Now().UTC()
|
||||||
|
if runErr == nil && cleanupErr != nil {
|
||||||
|
runErr = cleanupErr
|
||||||
|
}
|
||||||
|
if runErr != nil {
|
||||||
|
r.record.Status = "failed"
|
||||||
|
r.record.Error = runErr.Error()
|
||||||
|
} else {
|
||||||
|
r.record.Status = "passed"
|
||||||
|
}
|
||||||
|
if err := r.writeRecord(); err != nil && runErr == nil {
|
||||||
|
runErr = err
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
fmt.Println("Checking container engine...")
|
||||||
|
version, err := r.docker(ctx, "version", "--format", "{{.Server.Version}}")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Docker is required and the daemon must be accessible: %w", err)
|
||||||
|
}
|
||||||
|
r.record.ContainerEngine = "Docker " + strings.TrimSpace(string(version))
|
||||||
|
|
||||||
|
fmt.Println("Building Phase 1 environment...")
|
||||||
|
if err := r.dockerStream(ctx, "build", "-t", imageName, "-f",
|
||||||
|
filepath.Join(root, "environment", "Dockerfile"), root); err != nil {
|
||||||
|
return fmt.Errorf("build environment image: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println("Creating isolated graphical environment...")
|
||||||
|
_, err = r.docker(ctx,
|
||||||
|
"create",
|
||||||
|
"--name", r.containerName,
|
||||||
|
"--init",
|
||||||
|
"--network=none",
|
||||||
|
"--read-only",
|
||||||
|
"--tmpfs=/tmp:rw,nosuid,nodev,size=64m",
|
||||||
|
"--cap-drop=ALL",
|
||||||
|
"--security-opt=no-new-privileges",
|
||||||
|
"--pids-limit=128",
|
||||||
|
"--memory=512m",
|
||||||
|
"--cpus=1",
|
||||||
|
imageName,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("create environment: %w", err)
|
||||||
|
}
|
||||||
|
r.containerMade = true
|
||||||
|
if _, err := r.docker(ctx, "start", r.containerName); err != nil {
|
||||||
|
return fmt.Errorf("start environment: %w", err)
|
||||||
|
}
|
||||||
|
if err := r.waitFor(ctx, 10*time.Second, func() bool {
|
||||||
|
_, readyErr := r.docker(ctx, "exec", r.containerName, "test", "-f", "/tmp/agentbox-ready")
|
||||||
|
return readyErr == nil
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("wait for graphical environment: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println("Launching graphical mover demo...")
|
||||||
|
if _, err := r.docker(ctx, "exec", "-d", r.containerName, "sh", "-c",
|
||||||
|
"exec /opt/agentbox/mover >/tmp/stdout.log 2>/tmp/stderr.log"); err != nil {
|
||||||
|
return fmt.Errorf("launch demo: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var windowID string
|
||||||
|
if err := r.waitFor(ctx, 10*time.Second, func() bool {
|
||||||
|
output, searchErr := r.docker(ctx, "exec", r.containerName, "xdotool",
|
||||||
|
"search", "--onlyvisible", "--name", "Agentbox Mover")
|
||||||
|
if searchErr != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
windowID = strings.TrimSpace(strings.Split(string(output), "\n")[0])
|
||||||
|
return windowID != ""
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("wait for demo window: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := r.docker(ctx, "exec", r.containerName, "xdotool",
|
||||||
|
"windowactivate", "--sync", windowID); err != nil {
|
||||||
|
return fmt.Errorf("focus demo window: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
beforePath := filepath.Join(r.runDir, "screenshots", "before.png")
|
||||||
|
fmt.Println("Capturing screenshot before input...")
|
||||||
|
if err := r.screenshot(ctx, "/tmp/before.png", beforePath); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
before, err := squareCentroid(beforePath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("inspect before screenshot: %w", err)
|
||||||
|
}
|
||||||
|
r.record.Before = &before
|
||||||
|
|
||||||
|
fmt.Println("Sending synthetic mouse input...")
|
||||||
|
if _, err := r.docker(ctx, "exec", r.containerName, "xdotool",
|
||||||
|
"mousemove", "--window", windowID, "100", "100", "mousedown", "1", "mouseup", "1"); err != nil {
|
||||||
|
return fmt.Errorf("send mouse input: %w", err)
|
||||||
|
}
|
||||||
|
if err := r.trace(actionRecord{
|
||||||
|
Timestamp: time.Now().UTC(), Type: "mouse_move", X: 100, Y: 100,
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := r.trace(actionRecord{
|
||||||
|
Timestamp: time.Now().UTC(), Type: "mouse_down", Button: 1,
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := r.trace(actionRecord{
|
||||||
|
Timestamp: time.Now().UTC(), Type: "mouse_up", Button: 1,
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println("Holding RIGHT for one second...")
|
||||||
|
if _, err := r.docker(ctx, "exec", r.containerName, "xdotool", "keydown", "Right"); err != nil {
|
||||||
|
return fmt.Errorf("send key down: %w", err)
|
||||||
|
}
|
||||||
|
if err := r.trace(actionRecord{
|
||||||
|
Timestamp: time.Now().UTC(), Type: "key_down", Key: "RIGHT",
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
}
|
||||||
|
if _, err := r.docker(ctx, "exec", r.containerName, "xdotool", "keyup", "Right"); err != nil {
|
||||||
|
return fmt.Errorf("send key up: %w", err)
|
||||||
|
}
|
||||||
|
if err := r.trace(actionRecord{
|
||||||
|
Timestamp: time.Now().UTC(), Type: "key_up", Key: "RIGHT", Duration: "1s",
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
|
||||||
|
afterPath := filepath.Join(r.runDir, "screenshots", "after.png")
|
||||||
|
fmt.Println("Capturing screenshot after input...")
|
||||||
|
if err := r.screenshot(ctx, "/tmp/after.png", afterPath); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
after, err := squareCentroid(afterPath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("inspect after screenshot: %w", err)
|
||||||
|
}
|
||||||
|
r.record.After = &after
|
||||||
|
r.record.MovementPixels = after.X - before.X
|
||||||
|
if r.record.MovementPixels < 100 {
|
||||||
|
return fmt.Errorf("visual verification failed: square moved %.1f pixels right, want at least 100", r.record.MovementPixels)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("Verified: square moved %.1f pixels to the right.\n", r.record.MovementPixels)
|
||||||
|
fmt.Printf("Phase 1 passed. Artifacts: %s\n", runDir)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func findRoot() (string, error) {
|
||||||
|
current, err := os.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("get working directory: %w", err)
|
||||||
|
}
|
||||||
|
for {
|
||||||
|
data, readErr := os.ReadFile(filepath.Join(current, "go.mod"))
|
||||||
|
if readErr == nil && bytes.Contains(data, []byte("module agentbox")) {
|
||||||
|
return current, nil
|
||||||
|
}
|
||||||
|
parent := filepath.Dir(current)
|
||||||
|
if parent == current {
|
||||||
|
return "", errors.New("run from the agentbox module directory")
|
||||||
|
}
|
||||||
|
current = parent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func runID() (string, error) {
|
||||||
|
random := make([]byte, 3)
|
||||||
|
if _, err := rand.Read(random); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return time.Now().UTC().Format("20060102T150405") + "-" + hex.EncodeToString(random), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) waitFor(ctx context.Context, timeout time.Duration, check func() bool) error {
|
||||||
|
timer := time.NewTimer(timeout)
|
||||||
|
defer timer.Stop()
|
||||||
|
ticker := time.NewTicker(100 * time.Millisecond)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
if check() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-timer.C:
|
||||||
|
return errors.New("timed out")
|
||||||
|
case <-ticker.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) screenshot(ctx context.Context, containerPath, hostPath string) error {
|
||||||
|
if _, err := r.docker(ctx, "exec", r.containerName, "scrot", "-o", containerPath); err != nil {
|
||||||
|
return fmt.Errorf("capture screenshot: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := r.docker(ctx, "cp", r.containerName+":"+containerPath, hostPath); err != nil {
|
||||||
|
return fmt.Errorf("copy screenshot: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func squareCentroid(path string) (point, error) {
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return point{}, err
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
img, err := png.Decode(file)
|
||||||
|
if err != nil {
|
||||||
|
return point{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var sumX, sumY, count uint64
|
||||||
|
bounds := img.Bounds()
|
||||||
|
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
|
||||||
|
for x := bounds.Min.X; x < bounds.Max.X; x++ {
|
||||||
|
red, green, blue, _ := img.At(x, y).RGBA()
|
||||||
|
if green > 0xc000 && red < 0x4000 && blue < 0x8000 {
|
||||||
|
sumX += uint64(x)
|
||||||
|
sumY += uint64(y)
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if count < 1000 {
|
||||||
|
return point{}, fmt.Errorf("found only %d green square pixels", count)
|
||||||
|
}
|
||||||
|
return point{
|
||||||
|
X: float64(sumX) / float64(count),
|
||||||
|
Y: float64(sumY) / float64(count),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) trace(action actionRecord) error {
|
||||||
|
encoded, err := json.Marshal(action)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("encode action: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := r.actions.Write(append(encoded, '\n')); err != nil {
|
||||||
|
return fmt.Errorf("write action trace: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) cleanup() error {
|
||||||
|
if r.actions != nil {
|
||||||
|
_ = r.actions.Close()
|
||||||
|
}
|
||||||
|
if !r.containerMade {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println("Stopping environment...")
|
||||||
|
stopCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
_, stopErr := r.docker(stopCtx, "stop", "--time=3", r.containerName)
|
||||||
|
for _, name := range []string{"stdout.log", "stderr.log"} {
|
||||||
|
_, _ = r.docker(stopCtx, "cp", r.containerName+":/tmp/"+name, filepath.Join(r.runDir, name))
|
||||||
|
}
|
||||||
|
_, removeErr := r.docker(stopCtx, "rm", "-f", r.containerName)
|
||||||
|
if stopErr != nil {
|
||||||
|
return fmt.Errorf("stop environment: %w", stopErr)
|
||||||
|
}
|
||||||
|
if removeErr != nil {
|
||||||
|
return fmt.Errorf("remove environment: %w", removeErr)
|
||||||
|
}
|
||||||
|
fmt.Println("Environment stopped cleanly.")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) writeRecord() error {
|
||||||
|
data, err := json.MarshalIndent(r.record, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("encode run record: %w", err)
|
||||||
|
}
|
||||||
|
data = append(data, '\n')
|
||||||
|
if err := os.WriteFile(filepath.Join(r.runDir, "run.json"), data, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write run record: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) docker(ctx context.Context, args ...string) ([]byte, error) {
|
||||||
|
command := exec.CommandContext(ctx, "docker", args...)
|
||||||
|
output, err := command.CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
return output, fmt.Errorf("docker %s: %w: %s", args[0], err, strings.TrimSpace(string(output)))
|
||||||
|
}
|
||||||
|
return output, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *runner) dockerStream(ctx context.Context, args ...string) error {
|
||||||
|
command := exec.CommandContext(ctx, "docker", args...)
|
||||||
|
command.Stdout = os.Stdout
|
||||||
|
command.Stderr = os.Stderr
|
||||||
|
if err := command.Run(); err != nil {
|
||||||
|
return fmt.Errorf("docker %s: %w", strconv.Quote(strings.Join(args, " ")), err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user