codegirl-games/agentbox/docs/architecture.md

Phase 1 architecture decision

Status: accepted for the local spike

Decision

Run one application environment per Docker container. Inside the container:

• Xvfb provides a 640×480, 24-bit, in-memory X11 display.
• Openbox gives ordinary desktop windows focus and placement behavior.
• The application renders to that display without knowing it is headless.
• scrot captures the complete display as PNG.
• xdotool injects keyboard and mouse events through X11's XTEST extension.

The Go process on the host owns the lifecycle. It builds and creates the container, starts the display, launches the application as a separate step, captures images, sends input, copies artifacts out, and removes the container. The first spike intentionally uses the Docker CLI as its narrow adapter rather than adding a Docker SDK dependency.

Go phase-1 runner
  |
  +-- Docker lifecycle and process execution
        |
        +-- Xvfb display :99
        |     |
        |     +-- Openbox
        |     +-- arbitrary X11 application
        |
        +-- scrot observation
        +-- xdotool keyboard/mouse input

This is the smallest stack that exercises a real desktop input and rendering path. Xvfb is an X server backed by memory instead of display hardware. xdotool synthesizes standard X11 input, so the demo is not instrumented with a private control API. The verification reads the before and after PNGs and checks that the green square's pixel centroid moved, rather than trusting application state or logs.

Why this instead of the alternatives

X11/Xvfb rather than Wayland

Wayland deliberately gives clients less global authority. Synthetic input and whole-desktop capture are compositor-mediated, so a Wayland implementation would require selecting and configuring a headless compositor plus its specific control protocol. That is a good future backend, but it adds no evidence to this first question. Xvfb, XTEST, and framebuffer capture are mature, software-only, and replaceable behind the later Environment interface.

X11 is not a security boundary. Processes sharing one X server can generally observe or affect one another. The design therefore uses one display and one container per environment.

No VNC/noVNC yet

VNC is useful for a human live viewer, but it is not needed for programmatic screenshots or input. Adding an RFB server and browser client would introduce more processes, ports, encoding, and latency without improving the Phase 1 proof. A later observer can attach x11vnc, or the display backend can become an Xvnc server, without changing agent actions.

Software rendering first

Xvfb does not provide a modern GPU. Many toolkit applications can use software rendering, which is sufficient for this proof. GPU-heavy games may require a different environment backend using headless DRM/EGL, a virtual GPU, or GPU passthrough. That compatibility question is explicitly not answered by this spike.

A small Xlib demo rather than Raylib

The demo is C/Xlib so the image has only distribution packages and proves the display/input mechanism directly. Raylib would make a nicer example but adds a source or package dependency without changing the tested path. The runtime is not coupled to Xlib: any executable in a future staged image can use SDL, Raylib, Qt, GTK, a browser, or another X11-compatible toolkit.

Separation preserved for later phases

Phase 1 contains concrete orchestration, but its data flow already keeps these roles distinct:

CLI -> lifecycle controller -> container environment -> application
                                      |
                                observation/input
                                      |
                              deterministic driver

Phase 2 should extract lifecycle, screenshot, input, and logs into an Environment interface with backend-neutral actions. Phase 3 should consume that interface through an Agent; neither a deterministic agent nor a model adapter should import Docker or X11 details.

Security boundary

The container flags reduce accidental damage: no network, read-only root filesystem, a bounded tmpfs, a non-root user, all Linux capabilities dropped, no-new-privileges, and CPU, memory, and PID limits. They are useful defense-in-depth, not a production hostile-code sandbox.

Ordinary Docker containers share the host kernel. A kernel or container-runtime escape can cross this boundary, resource-exhaustion controls are incomplete, and image/build processing also handles attacker-controlled content. Do not run arbitrary customer binaries with this spike on a valuable or multi-tenant host.

Before production use, at minimum:

• put each untrusted workload behind a hardware-backed microVM boundary (Firecracker or Kata Containers), or evaluate gVisor where its syscall and graphics compatibility is sufficient;
• isolate image building from runtime hosts and verify limits on uploaded and expanded content;
• enforce outbound network policy, ephemeral storage quotas, wall-clock deadlines, and host-level CPU/memory/PID/I/O controls;
• use immutable, patched base images and a minimal guest kernel/filesystem;
• authenticate control operations and separate each tenant's artifacts, credentials, logs, and encryption keys;
• destroy the environment after every run and monitor the host boundary.

Sources

• X.Org Xvfb manual
• xdotool project and XTEST behavior
• Docker Engine security
• gVisor security model
• Firecracker design