UI testing an agent can write and run itself

Read this before testing a UI, and particularly if you are tempted to reach for screenshot-and-mouse control. There is almost always a faster and more reliable approach — and one an agent can author and run itself, headlessly, in CI, across many cases at once.

The line to remember: drive the UI by stable selector or accessibility identity, never by pixels and a cursor. Reserve screen control for the few surfaces that have no DOM and no accessibility tree.

Why not just let the agent click the screen

Screenshot-and-mouse control is real and occasionally necessary, but for in-content UI it is the worst available tool:

• Brittle and slow. The agent must locate the window, infer its size, aim the cursor, and re-aim whenever layout, theme, display density or window size changes. That “getting a feel for the window” phase is pure overhead, repeated every session.
• Non-deterministic. Pixel matching and coordinate arithmetic fail in ways that are hard to debug and impossible to trust as a regression gate.
• It does not scale. One agent, one screen, one slow run. No parallelism, no CI, no fan-out.

It is the right tool for exactly one category: surfaces with no DOM and no accessibility tree — native window chrome, system tray, operating-system file and permission dialogs, installers, global hotkeys, GPU-rendered canvases you cannot introspect. Everything else has a better handle available.

The anchor convention, which is the whole trick

What makes selector-driven testing reliable is a stable semantic handle on each interactive element, instead of a coordinate:

• Web — a test id attribute, and/or an ARIA role plus accessible name.
• iOS — an accessibility identifier, selected by name in UI tests.
• Android — a test tag or resource id.
• Game engines — stable object names or tags, driven through the engine’s own play-mode test framework, never a screen-space click.

Add anchors as you build. It is nearly free at the time and a tax to retrofit. They survive redesigns, so tests stop rotting every time the layout moves, and they are what lets an agent target elements without ever seeing the screen. Lean on roles and accessible names where you can and the accessibility of the product improves for free.

The tiers

Prefer the cheapest tier that catches the class of bug you care about. Most of the value is in the middle three, all headless and all runnable from a command line.

Logic, called directly with no UI

Do not render anything. Exercise the business logic as plain functions or a throwaway binary that drives the real code path and prints a transcript. Deterministic, fast, and the agent runs it itself. Anything expressible without the view layer belongs here.

Components in a headless DOM, faked at your own seam

Render the real components in a headless DOM, drive them by anchor, and fake the backend at your own single API seam — not at the framework’s internals. This catches rendering, state, event wiring and flow logic in milliseconds.

Two moves make it work:

Mock your own wrapper module, keeping every other export real, so stores and helpers still load and the faked surface stays tiny:

vi.mock('../../lib/platform', async (importOriginal) => ({
  ...(await importOriginal()),
  startJob: async () => 'run-123',
  onJobStep: async (_id, cb) => { captured.step = cb; return () => {}; },
}));

Capture the callbacks the component registers, then fire them yourself. That is how you simulate server-push in an event-driven UI without a server. Because the mock sits at your boundary rather than the framework’s, the surface is small and the test is stable.

A real browser, with the platform bridge stubbed

Same idea in a real browser, so you get real layout, CSS and events, and you exercise your real API-client layer — which the previous tier’s boundary mock skips.

For a webview application, stub the platform bridge the client talks to, in the page. The contract worth understanding for any such framework is how calls and events are marshalled: typically a global invoke function, and event subscription implemented as an invoke with a transformed callback id. A browser-side stub keeps a callback registry and an event-to-handler map, and exposes a helper to emit events into it.

An isolated mount harness is what keeps this tier cheap — a dev-only page that renders one component chosen by a query parameter. You then stub a handful of commands rather than every command a full application boot fires.

Full stack, real backend

Drive the real application against the real backend through the platform’s end-to-end driver. Reserve this for a few critical paths; it is the slowest and flakiest tier.

Platform caveat worth knowing before you fight it: on macOS the system webview’s WebDriver support is too weak for this to work well. Put real-backend end-to-end runs in a Linux CI job rather than trying to make them work on a Mac.

There is an escape hatch if you need full-stack coverage locally: run a dev-only HTTP bridge inside the application that shares live application state and exposes the real commands over a request endpoint plus a server-sent-events stream, and give the frontend a transport switch. An ordinary browser then drives the real backend with no WebDriver involved.

Why this multiplies an agent

Every tier below full-stack is just a command line. That means an agent authors the tests and runs them itself, with no vision tooling, and gets a deterministic pass or fail plus a trace on failure to diagnose from.

It also means scale: independent cases fan out across parallel runs, far beyond what one person clicking — or one agent aiming a cursor — could cover. That is exactly what a large regression suite needs.

Gotchas already paid for

• Vet test dependencies like any other. Exact versions, and avoid releases less than 72 hours old. Two testing helpers were dropped from one setup purely because their latest builds were hours old; the basic event helpers covered the work without them.
• Keep the runners’ globs separate. Component and browser test runners both match similar filename patterns. Scope each explicitly or they will try to run each other’s files.
• Set the DOM environment per file, so existing render-to-string tests in a plain environment are untouched.
• The mount harness must not ship. Confirm your bundler’s production entry excludes it rather than assuming.
• Browser binaries are not dependencies. They install to a user cache, not the project’s modules directory. Do not commit them; install them explicitly in CI.
• Ignore the artefact directories — results, reports and browser caches.

CI, split by the constraint that actually differs

Two jobs, not one:

A cheap Linux job for the web layer — install, typecheck, component tests, browser tests. Fast, gates every change, and is the thing that stops the harness rotting.

A platform-locked job only for a native build that genuinely requires it. If a native dependency is pinned to a platform-specific feature, that job runs on that platform with a build cache, and nothing else does.

Two non-obvious lessons from running this:

• Split CI by the native-build constraint, so expensive runners are used only for the part that cannot run anywhere else.
• Scope native tests to the hermetic subset. A full native test run will flake when tests needing network, GPU or downloaded models are not tagged to be skipped. Filter to the parts that are genuinely self-contained, and widen the filter as tests get tagged — rather than gating on a suite that cannot pass.

Cancel superseded runs on new pushes; it saves the expensive minutes.

Adapting to any stack

The tiers and the anchor-plus-mock-at-your-boundary principle transfer; only the tools change. When a stack is not one you have done before, find its two seams:

1. The stable identifier you can use to select elements.
2. The boundary where you can substitute a fake backend.

Those two are the whole pattern. Everything else is tooling.

The companion problem

This drives the near side — the interface, by anchor. Faking the far side, an external service you do not own, is a separate discipline with its own document. Together they let a full user journey run end to end with no live external calls and no cursor.