# Agentbox spike architecture decision Status: implemented for Phases 1–4 of the local spike ## Decision Run one application environment per Docker container. Inside the container: - Xvfb provides a 640×480, 24-bit, in-memory X11 display. - Openbox gives ordinary desktop windows focus and placement behavior. - The application renders to that display without knowing it is headless. - `scrot` captures the complete display as PNG. - `xdotool` injects keyboard and mouse events through X11's XTEST extension. The Go control plane on the host owns the lifecycle. It builds and creates the container, starts the display, launches the application as a separate step, captures images, sends input, copies artifacts out, and removes the container. The spike intentionally uses the Docker CLI as its narrow adapter rather than adding a Docker SDK dependency. ```text Go phase-1 runner | +-- Docker lifecycle and process execution | +-- Xvfb display :99 | | | +-- Openbox | +-- arbitrary X11 application | +-- scrot observation +-- xdotool keyboard/mouse input ``` This is the smallest stack that exercises a real desktop input and rendering path. Xvfb is an X server backed by memory instead of display hardware. `xdotool` synthesizes standard X11 input, so the demo is not instrumented with a private control API. The verification reads the before and after PNGs and checks that the green square's pixel centroid moved, rather than trusting application state or logs. ## Why this instead of the alternatives ### X11/Xvfb rather than Wayland Wayland deliberately gives clients less global authority. Synthetic input and whole-desktop capture are compositor-mediated, so a Wayland implementation would require selecting and configuring a headless compositor plus its specific control protocol. That is a good future backend, but it adds no evidence to this first question. Xvfb, XTEST, and framebuffer capture are mature, software-only, and replaceable behind the later `Environment` interface. X11 is not a security boundary. Processes sharing one X server can generally observe or affect one another. The design therefore uses one display and one container per environment. ### No VNC/noVNC yet VNC is useful for a human live viewer, but it is not needed for programmatic screenshots or input. Adding an RFB server and browser client would introduce more processes, ports, encoding, and latency without improving the Phase 1 proof. A later observer can attach x11vnc, or the display backend can become an Xvnc server, without changing agent actions. ### Software rendering first Xvfb does not provide a modern GPU. Many toolkit applications can use software rendering, which is sufficient for this proof. GPU-heavy games may require a different environment backend using headless DRM/EGL, a virtual GPU, or GPU passthrough. That compatibility question is explicitly not answered by this spike. ### A small Xlib demo rather than Raylib The demo is C/Xlib so the image has only distribution packages and proves the display/input mechanism directly. Raylib would make a nicer example but adds a source or package dependency without changing the tested path. The runtime is not coupled to Xlib: any executable in a future staged image can use SDL, Raylib, Qt, GTK, a browser, or another X11-compatible toolkit. ## Runtime separation The implemented data flow keeps these roles distinct: ```text CLI -> runtime controller -> Environment -> application | | | observation/input/logs | Agent | deterministic or OpenAI | versioned trace ``` `internal/environment` defines lifecycle, screenshot, input, and log contracts with backend-neutral actions. `internal/environment/dockerx11` is the only package that knows about Docker, X11, `scrot`, or `xdotool`. `internal/agent` defines observations, history, and decisions. The deterministic agent and optional OpenAI Responses adapter both implement that interface; they do not import the Docker backend. The model adapter receives the task, current PNG, action history, and recent logs, then returns one schema-constrained action. It is intentionally one concrete adapter, not a provider framework. `internal/runtime` connects those interfaces and contains no provider or X11 logic. `internal/trace` records versioned metadata plus append-only step and action streams. Screenshots remain separate PNG files referenced by relative path, which keeps traces readable and suitable for later replay or comparison. ## Application staging `agentbox run` accepts one prebuilt Linux executable. A directory can contain an `agentbox.json` manifest naming that executable, arguments, environment variables, and an optional window title. The Docker backend streams the file into the environment's tmpfs and makes it executable; no host directory is mounted into the container. The executable and its libraries must be compatible with the Debian-based runtime image. Packaging arbitrary dependency trees is a separate upload/build format problem and is not hidden by this spike. ## Security boundary The container flags reduce accidental damage: no network, read-only root filesystem, a bounded tmpfs, a non-root user, all Linux capabilities dropped, `no-new-privileges`, and CPU, memory, and PID limits. They are useful defense-in-depth, not a production hostile-code sandbox. Ordinary Docker containers share the host kernel. A kernel or container-runtime escape can cross this boundary, resource-exhaustion controls are incomplete, and image/build processing also handles attacker-controlled content. Do not run arbitrary customer binaries with this spike on a valuable or multi-tenant host. Before production use, at minimum: - put each untrusted workload behind a hardware-backed microVM boundary (Firecracker or Kata Containers), or evaluate gVisor where its syscall and graphics compatibility is sufficient; - isolate image building from runtime hosts and verify limits on uploaded and expanded content; - enforce outbound network policy, ephemeral storage quotas, wall-clock deadlines, and host-level CPU/memory/PID/I/O controls; - use immutable, patched base images and a minimal guest kernel/filesystem; - authenticate control operations and separate each tenant's artifacts, credentials, logs, and encryption keys; - destroy the environment after every run and monitor the host boundary. ## Sources - [X.Org Xvfb manual](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml) - [xdotool project and XTEST behavior](https://github.com/jordansissel/xdotool) - [Docker Engine security](https://docs.docker.com/engine/security/) - [gVisor security model](https://gvisor.dev/docs/architecture_guide/security/) - [Firecracker design](https://firecracker-microvm.github.io/)