What to observe
A screenshot, an accessibility tree (AX Tree), the output of a command, the response of an API, or any combination. The model decides which it needs, including whether it needs a new screenshot at all.
MAI-UI TEAM/FIRST RELEASE
OpenGCUA
An ongoing project to develop and open the full stack for generalist computer-use agents: model, harness, SDK, trajectories and benchmarks.
A GCUA is an agent system designed to complete diverse computer tasks through both visual interaction and programmatic access, in which the model decides how to observe, act, and organize its work as the task requires.
BENCHMARKOSWorld-v282 offline tasks
+19.6ptsMean partial score over GUI-only execution with batched actions45.3 → 64.9
−53.0%Steps per task, averaged over each model’s relative reduction8 frontier models
−27.2%Output tokens per taskfewer for 6 of 8 models
TESTED WITH 8 FRONTIER MODELSEVALUATION TRAJECTORIES RELEASED
GPT-6
Astra
Claude
Opus 5
Gemini 3.8
Flash
GLM-5.3
Flash
Qwen3.8
Max- Qwen3.8
Flash 
Kimi
K3
DeepSeek-V4.1
Flash
INTERACTIVE LAB
Try the main ideas of OpenGCUA,
one question at a time.
Seven labs, each built around one question from the paper. Every demo starts with one button, and the details are folded below it. Simulations and real runs are labeled as such.
Loading the lab…
01ModelNext release02Harness + SDKReleased03TrajectoriesReleased04EfficiencyOngoing research05EvaluationPlanned
01 / POSITIONING
A generalist decides
how to use the computer.
We define a generalist CUA (GCUA) by the decisions it makes at each step, not by the components it contains: an agent system that completes diverse computer tasks through both visual interaction and programmatic access, in which the model decides how to observe, act, and organize its work as the task requires.
How to act
A GUI operation at a screen coordinate, an operation on an accessibility element, a shell command, an API call, or a program that combines several of these.
How to organize
Across steps, keep a plan, track what has been done, and decide which past observations stay in context. Most harnesses fix these as rules set by the developer; in a GCUA, the model decides as the task requires.
FOUR INTERFACES · EACH ONE BOTH OBSERVES AND ACTS
GUI
Sees a screenshot and clicks, drags, scrolls or types at a screen coordinate. The interface that always works, since every application has pixels.
AX Tree
Reads a window’s elements with their roles, names and values, and operates on an element directly: no coordinate, no bringing the window to the front, and on real devices the user’s pointer never moves.
CLI
Runs shell commands and reads their output. The natural route for files, data and software.
API
Calls external services, such as web search, and reads their responses. This reaches what neither the screen nor the shell can.
No single interface is best for every step. What matters is choosing the right one at each step, and combining them when a step needs more than one.
OVERVIEW / OPEN GENERALIST CUA
FROM CUA TO GCUA
Existing agents fix some decisions in advance.
Written in the same form, each existing family fixes some of the three decisions instead of letting the model make them at every step. GUI+CLI agents with batched actions, including our earlier Qwen-UI-Agent, are a special case of GCUA.
(qt, pt, mt+1) ∼ πθ(· | mt, ot)| Family | What to observe | How to act | How to organize | Interface choice |
|---|---|---|---|---|
| GUI agent | Screenshot onlyqt ≡ {screenshot} | One GUI actionpt ∈ AGUI | Harness rule, e.g. last k screenshotsmt+1 = h(mt, ot) | No choice (GUI only) |
| CLI agent | Command output onlyqt ≡ ∅, only et | Shell commands and scriptsprogram over ACLI | Partly by the model, e.g. context summaries | No choice (shell only) |
| Multi-agent hybrid | Fixed for each sub-agentqt ≡ Qk | Fixed for each sub-agentpt over Ak | Split between orchestrator and sub-agents | Orchestrator, per subtask |
| Qwen-UI-Agent (GUI+CLI) | Screenshot every step, plus command output when a command runsqt ≡ {screenshot}, plus et | A batch of GUI and CLI actionssequence over AGUI ∪ ACLI | Harness rulemt+1 = h(mt, ot) | Model, per action (GUI or shell) |
| GCUA | Chosen per step, plus program outputqt ⊆ Q, plus et | Program over all interfacesprogram over A | Decided by the modelmt+1 ∼ πθ | Model, per step |
q: views requested (screenshot, AX tree); p: the program; m: the working state; e: execution output. Highlighted rows: the path this report follows.
WHY NOW · CODE EXECUTION VS. BATCHED ACTIONS
A batch is a program without control flow.
With batched actions, the model must decide every click before it sees the result of the first one. With code execution, each model call yields a program, and control flow is decided while the program runs.
Three things become possible in a single call. A task that took a dozen round trips becomes one.
Condition
if the dialog is visible, dismiss itLoop
for each row, fill the three fieldsVerification
assert the status now reads Saved02 / BENCHMARK EVIDENCE
Same models.
A more capable harness.
Changing the execution interface and workflow improves task scores and cuts steps for all eight existing models, with no additional model training. The gap between the strongest and weakest model narrows by more than half.
OSWorld-v2 v2.1 · 82 offline tasksMEAN TASK SCORE GAIN+19.6pts45.3 → 64.9 · partial credit
MEAN STEP REDUCTION53.0%Average of within-model relative changes
MODELS EVALUATED8modelsFrontier and open models · unchanged weights
OSWORLD-V2 / BENCHMARK RESULTS
Task score
Mean partial-credit score · higher is better
GUI-only + batched actionsFull GCUA
Detailed benchmark values
| Model | Task score | Steps | Full success (%) | Output kTok |
|---|
82 offline tasks.
GUI+CLI HYBRID CUA VS. GCUA
Same GUI and shell access. Higher scores, fewer steps.
Batched GUI+CLI agents, including our earlier Qwen-UI-Agent, are a special case of GCUA. With the same interfaces, OpenGCUA adds code execution and lets the model decide what to observe and how to organize its work.
+4.4 to +9.1 task-score pointsAll four models, with fewer reported steps per task
| Model | Task score ↑ | Steps / task ↓ | ||
|---|---|---|---|---|
| GUI+CLI | GCUA | GUI+CLI | GCUA | |
| Qwen3.8-Max | 57.2 | 64.0 | 66.1 | 57.0 |
| DeepSeek-V4.1-Flash | 50.9 | 57.7 | 86.7 | 84.9 |
| GLM-5.3-Flash | 55.8 | 64.9 | 81.7 | 75.5 |
| Claude Opus 5 | 65.7 | 70.1 | 61.6 | 61.4 |
Selected runs on the 82 offline tasks.
RESEARCH / 01
One program.
More work per step.
Code execution combines GUI operations, shell commands, and computation in one executable interface. Loops, conditions, and local checks respond to intermediate results within a step, and models bring their existing programming ability to computer interaction.
+6.1 task-score pointsMean of three models vs. GUI+CLI batching14.8% fewer stepsSame GUI and shell access, three models
+15.5 to +33.6 pointsvs. batched GUI actions, three models55.3–63.1% fewer stepsvs. batched GUI actions, three models
CODE EXECUTION / ABLATION
RESEARCH / 02
The mechanism in the harness.
The decision in the model.
Harness-to-model decision transfer moves selected workflow decisions from fixed harness rules to the model: when to request a screenshot, what task state to record, and how to state a program’s intent.
screenshot()Choose when to look again.todoTrack progress and unverified changes.<intent>Make the purpose of a program explicit.
Micro-compaction and the rolling screenshot buffer remain harness policies. On-demand screenshots raise cache hit for both models, todo raises task score for both, and intent targets user understanding rather than benchmark score.
+10.5 / +14.4 pp cache hitOn-demand screenshots alone, Qwen3.8-Max / DeepSeek-V4.1-Flash, without lowering task scores
DECISION TRANSFER / ABLATION
ConfigurationTask scoreStepsCache hit
Read the numerical ablation table
| Configuration | Score | Steps | Cache hit (%) |
|---|
Each mechanism is added separately to code execution; the final row combines all mechanisms. Cache hit is the share of input tokens served from cache; dashes mark omitted rates.
RESEARCH / 03
What can money and time buy?
It depends on the budget.
Final scores alone do not show how much performance is available under limited resources. We apply a common per-task spending cap, or a time limit, to the same full-setting runs: a task counts only if its whole trajectory fits.
60.5 points at ≈ $0.38 per taskQwen3.8-Flash75.7 points at ≈ $15.26 per taskGPT-6 Astra, the highest score
As the budget grows, the highest-scoring model shifts from Qwen3.8-Flash to GLM-5.3-Flash, Gemini 3.8 Flash, and finally GPT-6 Astra. Under recorded time limits, GPT-6 Astra scores higher at most limits; time conditions were not controlled across models, so the time curves are exploratory.
MONEY AND TIME / OSWORLD-V2
OPEN TRAJECTORIES
See how the work gets done.
We release the prompts and complete OSWorld-v2 execution trajectories of all eight evaluated models, including GPT-6 Astra, Claude Opus 5, and Qwen3.8-Max: model-written code, observations, and execution feedback.
Explore evaluation trajectories →REPRESENTATIVE PATTERNSGPT-6 ASTRA
- Cross-interface compositionGUI clicks alternate with checks of generated contracts and drafts
- Loop-based executionone loop visits seven webpage sections and checks each through the DOM
- Result-dependent executiona draft comparison decides whether to click Send
- Observation preparation25 views assembled into one numbered contact sheet
- Executable verificationassertions check the edits and that 64 other package entries are unchanged
03 / REAL-DEVICE HARNESS
From a benchmark
to your own machine.
Users run agents on their own computers: nothing is prepared for the task, windows overlap, the user keeps working, and requests change along the way. A five-layer architecture connects model decisions to native applications; each layer has a clear job and can be replaced on its own.
THE GAP BETWEEN BENCHMARKS AND REAL MACHINES
The user and agent share control.
The agent cannot assume exclusive use of the pointer, keyboard or focused window, and must not mistake the user’s activity for its own.
The screen keeps changing.
Windows open, close, move and cover each other, even while the agent is reasoning; a coordinate or element seen a moment ago may already be wrong.
Requests evolve during execution.
Users clarify, correct the approach or add constraints as work proceeds; the agent keeps the overall goal and progress across interruptions and turns.
Targets must be explicit.
The relevant app, window or tab may sit in the background. A desktop screenshot shows what is visible, not every target, so the model chooses both what to inspect and how.
Tasks begin in an existing environment.
Files, tabs, windows and account sessions from prior use remain; the agent must find the right inputs, and expired sessions or missing permissions may need the user.
↓ Requests / actions↑ Observations / feedback / errors
Model
Decides how to observe, act and organize
Expressed as Python programs against a common SDKRuntime
Agent loop · persistent interpreter · next model input
Variables, imports and the working directory persist across turnsDevice SDK
Binds observations and actions to explicit resources
Element IDs come from delivered observations · reset / rebind invalidates themDriver
Translates SDK requests into native operations
AX actions · coordinate input · foreground / background delivery · typed errorsReal applications & OS
Determine what can be observed and controlled
Permissions · application state · accessibility supportMinimize interference.
Prefer element-level accessibility actions, then window-targeted coordinate input, then foreground input, and coordinate with the user before taking the foreground.
Select observations explicitly.
The agent chooses when to observe and which target to inspect. One program can capture, process and emit only what the next decision needs.
Keep AX information manageable.
Irrelevant nodes are filtered into a flat list linked by parent_id; search, page and read query a retained snapshot instead of capturing the window again. Vision fills in where AX falls short.
Keep the person in the loop.
ask_user resolves ambiguous targets, asks for help with login or verification, and gets authorization for consequential actions. Interpreter state, bindings and history carry across these exchanges.
Preliminary real-device results
Four macOS tasks · workbook reasoning, PDF form filling, shopping research, hotel comparison
| Model | Steps / task | Output kTok / task | Est. USD / task | Complete |
|---|---|---|---|---|
| GPT-6 Astra | 34.5 | 12.9 | 18.290 | 4/4 |
| Claude Opus 5 | 52.3 | 42.3 | 17.117 | 3/4 |
| Gemini 3.8 Flash | 36.0 | 31.9 | 0.662 | 2/4 |
| Qwen3.8-Max | 50.8 | 73.4 | 1.494 | 3/4 |
| Qwen3.8-Flash | 67.3 | 75.7 | 0.120 | 2/4 |
Run 1 for each task and model. Costs are four-task means estimated from recorded usage, cache hits and public API prices. DeepSeek lacks a hotel trajectory, so its aggregates are unavailable. With four tasks, these results characterize the evaluated configurations rather than establish a general model ranking.
1.5×steps
5.7×output tokens
19.4×reasoning tokens
Qwen3.8-Max versus GPT-6 Astra under the same harness: GPT-6 Astra averages 34.5 steps and 12.9K output tokens per task (2.6K reasoning), Qwen3.8-Max 50.8 steps and 73.4K (50.2K reasoning). Token usage and cost need not rank models the same way: Qwen3.8-Max costs $1.49 per task, GPT-6 Astra $18.29.
Per-task results and task descriptions
Workbook reasoning
A workbook with race rules, a track layout, seven worked examples and 125 Level questions: compute every answer, enter it in the right cell, and save a completed copy without altering the original.
| Model | Steps | Out K | USD | Success |
|---|---|---|---|---|
| GPT-6 Astra | 22 | 8.8 | 8.873 | ✓ |
| Claude Opus 5 | 72 | 46.0 | 29.827 | ✓ |
| Gemini 3.8 Flash | 40 | 27.6 | 0.479 | ✗ |
| Qwen3.8-Max | 37 | 120.8 | 1.572 | ✓ |
| Qwen3.8-Flash | 80 | 128.9 | 0.158 | ✓ |
| DeepSeek-Flash | 81 | 215.3 | 0.797 | ✓ |
| GLM-5.3-Flash | 21 | — | 0.070 | ✗ |
PDF form filling
A blank four-page Schengen application form and synthetic identity, employment, travel and accommodation records: fill every applicable field and save a new PDF with all entries visible and correct.
| Model | Steps | Out K | USD | Success |
|---|---|---|---|---|
| GPT-6 Astra | 28 | 10.0 | 9.499 | ✓ |
| Claude Opus 5 | 19 | 10.3 | 2.792 | ✓ |
| Gemini 3.8 Flash | 15 | 19.4 | 0.258 | ✓ |
| Qwen3.8-Max | 17 | 28.3 | 0.401 | ✓ |
| Qwen3.8-Flash | 19 | 24.3 | 0.025 | ✗ |
| DeepSeek-Flash | 19 | 16.3 | 0.050 | ✓ |
| GLM-5.3-Flash | 11 | — | 0.018 | ✗ |
Shopping research
22 snack candidates and a dietary-restriction list: verify exact products on Amazon (or official pages), and write a DOCX covering every candidate with evidence and explicit uncertainty. No purchase is permitted.
| Model | Steps | Out K | USD | Success |
|---|---|---|---|---|
| GPT-6 Astra | 28 | 12.8 | 12.279 | ✓ |
| Claude Opus 5 | 40 | 49.5 | 10.566 | ✗ |
| Gemini 3.8 Flash | 19 | 27.3 | 0.264 | ✗ |
| Qwen3.8-Max | 31 | 41.9 | 0.777 | ✗ |
| Qwen3.8-Flash | 43 | 41.2 | 0.057 | ✗ |
| DeepSeek-Flash | 33 | 35.5 | 0.097 | ✗ |
| GLM-5.3-Flash | 37 | — | 0.056 | ✗ |
Hotel comparison
Two Paris hotels for two adults over December 20–30, 2026, with one hotel change: list ten feasible plans, research four on the official sites, Expedia and Booking.com, and document prices in a four-sheet workbook. No reservation is permitted.
| Model | Steps | Out K | USD | Success |
|---|---|---|---|---|
| GPT-6 Astra | 60 | 20.1 | 42.510 | ✓ |
| Claude Opus 5 | 78 | 63.2 | 25.282 | ✓ |
| Gemini 3.8 Flash | 70 | 53.4 | 1.648 | ✓ |
| Qwen3.8-Max | 118 | 102.4 | 3.229 | ✓ |
| Qwen3.8-Flash | 127 | 108.4 | 0.239 | ✓ |
| DeepSeek-Flash | — | — | — | — |
| GLM-5.3-Flash | 152 | — | 0.400 | ✗ |
Run 1 for every entry. Out K is output tokens in thousands; USD is the estimated response cost; ✓ is the adjudicated binary outcome. GLM-5.3-Flash output cells await the updated per-task breakdown; other dashes mark a missing trajectory.
Case studies use earlier runs under the same harness, separate from the selected runs above.
Simple steps, different costs.
In a shopping task, Qwen3.8-Max writes a four-line program that captures a desktop screenshot; its response uses 2,105 output tokens, 2,030 of them reasoning. A GPT-6 Astra screenshot step uses 124 output tokens, 11 of them reasoning.
Observe the page you work on.
Qwen captures the desktop and sees a foreground Taobao login page instead of its Amazon tab. GPT captures the selected browser page directly.
Cover the whole task in code.
GPT-6 Astra builds a reusable solver, checks it against the seven worked examples, finds question rows from the workbook contents and fills all 125 answers. Qwen3.8-Max processes fixed row ranges covering 85 questions and leaves 40 empty, without checking for omissions.
REAL-DEVICE DEMONSTRATIONS
Watch the workflows.
Inspect the recordings.
Compare four approaches to the same visa form, then explore additional desktop workflows.
One visa form, four workflows
Edited side-by-side comparison of OpenGCUA, OpenGCUA (prefer GUI), GUI + batch, and Codex. Playback speeds vary; use the in-video clocks for elapsed task time.
Open video ↗Durations describe the video files, not normalized task execution times. Open a video in full screen to inspect small text and interface details.
04 / CROSS-DEVICE SDK
Several devices.
One programming interface.
A task may span a mobile app, desktop tools and a browser-based service. The Cross-Device SDK gives one agent a common programming interface to all of them: shared operations with explicit targets, structured and visual observation, and cross-device workflows within a single task.
CROSS-DEVICE SDK / SYSTEM ARCHITECTURE
mu.device(…)Shared operations, explicit targets.
Every operation is scoped to a device, application, window or tab. Handles keep their targets, so selecting a new target never redirects earlier bindings. The macOS adapter uses the unmodified Cua Driver from trycua.
ui.ax · ui.vision · mu.emitObserve in code, deliver what matters.
ui.ax gives structured elements for conditions and checks; ui.vision gives images and coordinates when semantics fall short. ui.ax.observe() returns data to the program, and mu.emit delivers only the selected observations or results to the model.
device.shell.run · mu.files.copyOne task, several devices.
Within one session, a program can retrieve a report through a phone app, copy it to a computer for analysis, and enter the results in a browser-based service, reusing variables and bindings across turns.
.android · .macos · .browserExplicit platform capabilities.
A common interface does not mean identical capabilities. Each resource declares what it supports, and platform-specific features such as Android navigation or browser page evaluation keep their own namespaces.
05 / PROJECT ROADMAP
Opening the stack.
One layer at a time.
RELEASE PATHFIRST RELEASE → GCUA MODEL → RESEARCH PRIORITIES
First release
Released with this report
- OpenGCUA harness implementation and prompts
- OSWorld-v2 results and ablations across eight frontier models
- Evaluation trajectories of all eight models
- Real-device harness and Cross-Device SDK
A GCUA model
Turn the designs studied here into learned capabilities
- One model for computer use and browser use
- Coordinate visual and programmatic operations through code
- Request observations when needed
- Maintain task state throughout execution
Research priorities
Efficiency, real devices, evaluation
- EFFICIENCYMore concise reasoning and efficient execution, with tokens, latency and cost measured separately
- DEVICESReliable work across computers, browsers and phones: target selection, failure recovery, task continuity
- DECISIONSMove more workflow decisions from harness rules to the model
- BENCHMARKSReal-device environments and benchmarks, especially on macOS
These priorities guide planned releases without fixing a delivery timeline; the agenda will evolve with experimental evidence and community proposals.
06 / BUILD WITH US
The next frontier
should be open.
Help make generalist computer use reproducible, extensible, and useful on real devices.
Contribution guideImprove the harness and SDK
Platform adapters, more reliable observation and control, execution optimizations, real-device testing and documentation.
Build real-device evaluation
Environments, realistic tasks, evaluation criteria and benchmark infrastructure, including execution traces that expose current limitations.
Propose research goals
Workflows agents should support, capabilities worth developing, or questions to investigate with the released systems and trajectories.