Semantic commitment before implementation, source acceptance afterward

Trace2World

Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training

Zezhou Wang1,2* Hanzhang Zhou2‡ Panrong Tong2 Quyu Kong2 Pengxiang Li2 Steven Hoi2 Zhuzhong Qian1† Yue Wang2†‡
1State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China 2Alibaba Token Hub, Alibaba Group
*Work was done during internship at Alibaba Token Hub, Alibaba Group. †Corresponding authors. ‡Project leads.

One resettable world per app, built by us from real traces. The montage shows recorded rollouts from Trace2World's 50 reconstructed apps and 1,570 source traces, each playing its own unmarked observation sequence.

1,570
real-app source traces
50
resettable environments
across 10 domains
91.1%
Workflow fidelity
vs 59.8% strongest existing route
94.0%
of simulated gain kept on real devices
vs 54.7% with self-validation
61.1%
Trace2World-Bench success, learner-guided
vs 56.9% static generation
+16.3
points on a 100-task
real-device panel
Panel (a): the real Meituan home next to CUA-Gym, PhoneWorld, GUI-GENESIS and Trace2World reconstructions, and the icons of all 50 apps. Panel (b): a 20-step rollout of an augmented Meituan task.
Fifty real apps rebuilt from traces into faithful, resettable worlds. (a) From the same Meituan traces, topic-, task-, and screen-level construction (our CUA-Gym, PhoneWorld, and GUI-GENESIS reimplementations) each yield a different Home; Trace2World reproduces the real one. Right: all 50 apps. (b) One world, many tasks: a 20-step rollout of an augmented task no source trace performs; three stages replay observed support, three add a state check, a derived instance, and order checks.

Abstract

Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce Trace2World, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4–16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it. The code and resources are released.

Automating implementation does not automate reconstruction

Existing routes hand coding agents both the code and the semantics: CUA-Gym codes mocks from topic specifications, GUI-GENESIS codes a task-local site from one trace, and PhoneWorld codes mocks from recovered screen types and transitions. None asks which shared identities, initial states, data, guards, or conflicting effects the traces actually support. Two failures follow.

1Structural reconstruction is not semantic reconstruction

Each trace observes one path from one initial state. Xi'an and Beijing may share a Home page with different city values, while one control can have state-dependent effects. A transition graph cannot identify the state condition behind an effect, so coding before reconciliation hardens one interpretation into an app. Across our matched builds, routes reach 50.4–94.8% of endpoints yet preserve only 16.7–59.8% of source workflows.

2A runnable app cannot certify its own fidelity

An implementation and the tests derived from the same interpretation can agree while both diverge from the source. In Meituan, tapping Disable quick pay opens a confirmation without changing the saved setting. Our PhoneWorld-style reconstruction disables it immediately yet passes a final-value check, rewarding a policy for a transition the real app does not perform.

Which app semantics do partial traces actually support, and what evidence may accept an implementation of them?
Real Meituan quick-pay flow confirms before saving; a PhoneWorld-style build saves on tap; both end disabled.
Same terminal value, wrong transition. The source confirms before saving; our PhoneWorld-style build saves on tap. Both end disabled, so a final-value check passes both.

Method

Trace2World answers with two boundaries. It fixes app semantics from source evidence before any code is written, and accepts an implementation only by independent replay of the source workflows afterward. Learner feedback then decides which supported tasks are practiced, never what the app means.

Trace2World pipeline: real app traces, reconcile and index app semantics, build a standalone environment checked by source replay, and learner-guided tasks.
Semantic commitment before implementation, source acceptance afterward. Trace2World reconciles partial traces into one source-linked model of shared pages, entities, state, and effects, indexed as a semantic inventory; unresolved behavior remains non-executable. A builder implements the fixed model, and source replay checks recorded workflows. Failures return to code repair or semantic reconciliation; learner-guided tasks are composed from the validated inventory.
1

Reconcile app semantics across traces

Two observations may differ because they show different pages, different values of the same state, or one entity on different surfaces. Trace2World matches repeated pages, action hosts, and entities using containment, routes, lifecycle, and observed transitions, then separates stable structure from mutable state and reusable data. Each effect links to the source step that shows its precondition and result; if no observed condition explains a conflict, the conflict stays unresolved instead of being guessed.

2

Index a frozen semantic inventory

Supported relations become a source-linked inventory of canonical surfaces, hosts, entities, state and data fields, legal actions, guards, effects, and observable outcomes, plus an information-dependency graph. Indexing adds no behavior. Traces are reconciled in bounded shards: across all 50 apps the largest model-visible call is 33.2K tokens under a 40K-token budget, with 99.1% audited semantic coverage.

3

Build, then validate against the source

The builder implements the supported model as a standalone DOM-backed browser app with a central state kernel; unsupported controls stay visible but inert. A read-only checker replays every recorded workflow from reset and compares information, intermediate effects, and terminal state with the source, without seeing the builder's model, verifier, or repair log. Implementation errors are repaired (up to three waves); semantic mismatches return to reconciliation. No human QA is needed for acceptance.

4

Generate learner-guided tasks

The validated inventory becomes the task-authoring substrate. Generation may compose relations or rebind observed entities and initial states, but cannot add a surface, field, guard, transition, or dependency. A fixed stronger solver filters unsolvable candidates, and four current-learner attempts keep tasks whose reward varies (SD > 0.1). One semantic reconstruction is amortized across many tasks.

Two Meituan traces with Xi'an and Beijing share one Home page specification; city is reset state selected by task id.
Two traces, one Home page. Xi'an and Beijing are supported initial values of the same city state field. The task selects a reset value; subsequent behavior uses the shared page and state-dependent action rules.

What the model commits to

Every supported interaction has a page and control, an action and guard, state or data reads and writes, a destination, and a visible postcondition. The same rule serves multiple workflows without copying a task-specific page.

What the builder may change

Rendering, routes, and storage. Source traces determine the app model; the builder cannot invent app behavior. Compilation checks show that the app runs, source replay checks that it matches.

Cost is not the reason it wins

Trace2World reconciles with GPT-5.2 and implements with GPT-5.5; compared builders receive GPT-5.5 in every role they define. A Trace2World seed build averages an estimated 3.6 hours and 1,160 model calls per app, below GUI-GENESIS-style (5.9 h, 2,415 calls) and near PhoneWorld-style (5.0 h, 1,118 calls).

Learner-guided task generation loop over a frozen semantic inventory: learner gaps, supported task generation, validation and selection, training.
Learner-guided tasks from a frozen semantic inventory. Task generation queries the validated inventory rather than re-inferring the app: supported surfaces, fields, actions, and dependencies compose a harder food-delivery task; learner gaps guide which patterns to sample, and source, GUI, solver, and learner-signal checks screen candidates. Implementation repair fills supported paths without changing the inventory.

Results

End-to-end reconstruction fidelity

All conditions use the same 200-workflow, ten-app set. Direct Agent, CUA-Gym, PhoneWorld-style, and GUI-GENESIS-style are reimplemented on the same substrate, tools, runtime, and recordings. A blinded reviewer compares anonymous executions with source recordings and assigns pass/fail to six diagnostics; Workflow requires all six, Reach only the endpoint.

Workflow fidelity and its six diagnostics (%). Ten-app macro means over 200 paired workflows.
MethodOverallStructureStateData
WorkflowReachSurfaceStateEffectTerminalData
Direct w/o traces5.228.414.822.715.015.912.8
Direct + traces16.753.635.341.234.633.731.7
CUA-Gym w/o traces8.234.420.029.022.621.519.6
CUA-Gym + traces19.950.437.538.035.035.335.0
PhoneWorld-style22.457.039.346.841.539.838.5
GUI-GENESIS-style59.894.883.976.677.079.478.4
Trace2World91.198.093.793.393.893.794.1
Control (not our method)
Self-validation79.198.590.487.085.088.089.1

The task-local GUI-GENESIS-style build reaches 94.8% but passes 59.8%, with the State that workflows share at 76.6%; Trace2World reaches 98.0% and passes 91.1%. The ordering is not explained by a stronger constructor or reviewer bias: blind human relabeling agrees with the automated reviewer on 90.2% of the same 1,200 packets, moves every route by only 1.4–3.8 pp, and keeps Trace2World 29.9 pp above GUI-GENESIS-style and 12.8 pp above self-validation.

Route recovery, evidence selection and Workflow fidelity under Low, Mid and Full trace budgets.
More source evidence expands the semantic inventory without collapsing fidelity. From Low to Full, 100→290 traces expand supported routes 270→512 and visual obligations 553→1,076, while Workflow moves from 92.6% to 91.1%. Across all 1,570 source workflows, 74.6% pass source replay on the first attempt and all close within the three-wave automated repair cap, without human QA.

Source-grounded validation decides how much gain transfers

The self-validation control changes only the validation evidence under the same construction stages and inputs. Replacing source-trace validation with self-validation lowers Workflow from 91.1% to 79.1% while Reach stays near 98%. Independent source rechecks reject 22/160 (13.8%) self-accepted executions versus 1/176 (0.6%) source-accepted ones.

On a shared 100-task, 20-app real-device panel, source validation retains 94.0% of its 16.7-pp simulated gain on real devices (15.7 pp), versus 54.7% of 17.7 pp under self-validation (9.7 pp). With training matched and simulated gains nearly equal, the rise in simulation-only passes from 10.0% to 18.3% locates the loss in behavior the simulator accepts but the device rejects.

Real-device transfer: gains over base on 100 paired tasks from 20 apps (pp; three-seed mean ± SD).
MetricMatched stage-1 sourceStage 3
Self-validationSource-trace validationOurs
Real gain9.7 ± 2.115.7 ± 1.516.3 ± 2.1
Sim. gain17.7 ± 1.516.7 ± 2.118.7 ± 1.5
Real / sim gain54.7%94.0%87.5%
Sim-only pass18.3%10.0%10.7%

Learner-guided task generation

Four branches fork from the stage-1 policy for the same 20 RL iterations, environment revisions, sampler, and evaluator. Across the two batches, 76.8/75.3% of guided tasks carry learning signal versus 47.9/44.0% for the static control. At iteration 52, Trace2World-Bench SR is 61.1% guided, 56.9% static, 55.3% entity rebinding, and 52.2% seed continuation. Budget does not substitute for feedback: training on all validated candidates for 54 iterations reaches 57.8%.

Selected tasks with learning signal per branch and Trace2World-Bench SR/PR over RL iterations.
Learner-guided task supply. (a) Selected tasks with/without learning signal (four-attempt reward SD > 0.1). (b, c) Trace2World-Bench SR/PR for iteration- and rollout-matched continuations. Error bars show three-seed SD.

Downstream policy performance

Qwen3.5-9B is trained with DAPO across three frozen stages: 1,280 seed tasks, then two 384-task learner-guided batches (Ours). No task from the four external suites enters task generation, screening, RL, or checkpoint selection. Our final policy gains 20.2 pp SR on Trace2World-Bench, 13.6 pp on apps excluded from policy training, and 24.7 pp on evaluation-only two-app tasks although every training task is single-app.

Trace2World-Bench and MobileGym (MG) report SR / PR; AndroidWorld (AW) and AndroidDaily (AD) report SR; MobileWorld (MW) reports GUI-only SR@50, all in percent.
ModelTrace2World-Bench (SR / PR)MG
SR / PR
AW
SR
MW
SR@50
AD
SR
Entity repl.Held-out workflowCross-appUnseen appsOverall
Proprietary models
Gemini 3.1 Pro75.1 / 82.269.0 / 76.258.0 / 67.065.7 / 71.968.9 / 76.131.3 / 44.6–––
GPT-5.574.5 / 82.568.3 / 76.857.3 / 66.366.1 / 73.868.4 / 76.7––62.4–
Qwen3.7-Plus76.3 / 83.470.6 / 78.459.0 / 67.868.8 / 76.470.6 / 78.332.8 / 46.081.062.379.8
Open-source GUI agents
UI-TARS-1.5-7B54.4 / 66.045.2 / 58.530.3 / 44.239.6 / 50.845.1 / 57.713.8 / 26.330.0––
MAI-UI-8B66.3 / 76.958.1 / 69.845.0 / 57.551.9 / 62.757.8 / 69.221.5 / –70.727.5–
UI-Venus-2-9B64.2 / 75.056.8 / 67.544.7 / 56.052.8 / 61.456.8 / 67.252.7 / –80.265.8–
Controlled Trace2World environment training (Qwen3.5-9B)
Base policy49.2 / 63.540.2 / 55.126.0 / 41.339.2 / 52.640.9 / 55.49.0 / 19.757.818.547.2
Stage-1 policy57.0 / 72.648.6 / 65.135.3 / 52.044.3 / 58.948.7 / 64.615.8 / 28.460.121.951.9
Ours68.4 / 78.962.4 / 71.850.7 / 60.552.8 / 63.761.1 / 71.125.2 / 40.763.227.458.3

Fifty environments, ten domains

All 50 apps provide construction traces and all 50 are reconstructed. Forty apps supply policy-training tasks; ten held-out apps, one per domain, supply construction traces but no seed, candidate, screening, or RL-update task. Collection follows a quota fixed before construction: 50 traces for each of the 19 apps with at least three independent service lines, 20 for each of the other 31 (950 + 620 = 1,570). Every environment is a browser reimplementation with synthetic content; it connects to no real service.

DomainPolicy-training apps (40)Held-out eval app (10)
Social & communityWeChat · QQ · Weibo · XiaohongshuZhihu
News & short videoKuaishou · Bilibili · Toutiao · Tencent NewsDouyin
Long video & audioiQIYI · Youku · NetEase Cloud Music · Kugou Music · XimalayaQQ Music
Commerce & resaleTaobao · JD · Xianyu · DewuPinduoduo
Local services & foodMeituan · Dianping · Dingdong Maicai · KFC · HemaEle.me
Mobility, travel & ticketsAmap · Ctrip · Didi · Railway 12306 · DamaiTaopiaopiao
Auto, housing & jobs58Tongcheng · Autohome · Dongchedi · BOSS ZhipinBeike
Work, learning & searchFeishu · WPS Office · Baidu · DingTalk · Youdao Dictionary · Baidu NetdiskTencent Meeting
Payment & logisticsAlipay · SF ExpressCainiao
Health & fitnessBoohee HealthKeep

Generated-app home screens

Default states of twenty formal browser environments (the human-audit roster), captured in fresh browsers at a common viewport with no task-specific initialization. Click to enlarge.

Trace2World-Bench

One frozen evaluation view over the portfolio: 1,044 tasks, disjoint from every optimization batch, with SR averaged over three attempts per task.

256
Entity replacementtraining-side workflow skeletons, new entities
512
Held-out workflow20% signature split (320 seed + 192 aug.)
100
Cross-app10 app pairs, return-dependent A→B→A
176
Unseen apps10 apps excluded from policy training

A source trace, end to end

Each environment is reconstructed from recorded traces of the real app. This is one of the 1,570: twenty steps in Meituan, from the home screen to the Flash-buy order page. Every frame is the original screenshot with the recorded action marked, and its caption describes that action. A source trace is construction evidence, so it carries no task checks and no reward.

· ·

Task instruction (zh)

Source

    Click a frame to enlarge it.

    Case studies: step through two tasks

    Two tasks in reconstructed environments: a 22-step Bilibili task within one app, and a Railway 12306 → Didi → Railway 12306 task whose return to the train order depends on the fare read in Didi. Step through each frame with its rationale and the task checks met so far.

    The Bilibili task shows the marked observations and step rationale of an evaluation rollout; frames with incorrect media bindings are replaced by marked replay captures from the repaired publication package, and the last frame is its visible outcome. The Railway–Didi task presents the appendix's marked sequence with synthesized step rationale and eight ordered task checks.

    ·

    Task instruction (zh)

    Source & verification

    Original observation Current action marked

    Step rationale step
    Verification

    ○ Not met✓ Met+ Newly met− Lost this step

      Verification reward

      What is released

      Trace2World is released as an environment-construction resource, not as a benchmark alone: the source evidence, the reconstructed environments, the training-task supply, the evaluation sets, and the fidelity records of the paper. Code lives on GitHub; the environments live in a Hugging Face dataset.

      ComponentScaleRole
      Source traces1,570 traces / 50 appsconstruction evidence; external fidelity authority
      Construction pipelineone shared pipelinetrace → reconciled app specification → environment
      Environment portfolio50 apps / 10 domainsresettable GUI RL substrate with state and evaluator manifests
      Environment hostone process, all appsserve, step, evaluate (t2w-serve, t2w-rollout, t2w-smoke)
      Training-task supply1,280 seed + 768 learner-guidedpolicy optimization
      Trace2World-Bench1,044 tasksfrozen policy evaluation
      Fidelity study200 workflows / 10 appsreconstruction correctness, reviewer labels, human adjudication
      Real-device panel100 tasks / 20 appspaired sim-to-real evidence

      Quick start

      git clone https://github.com/Tongyi-MAI/trace2app.git && cd trace2app
      python3.12 -m venv .venv && source .venv/bin/activate
      pip install ./trace2world_env huggingface_hub && playwright install chromium
      
      python scripts/download_environments.py            # -> data/trace2world, verified against the manifest
      t2w-serve --environments-dir data/trace2world \
                --state-root /tmp/t2w-state --host 127.0.0.1 --port 6800
      t2w-rollout --server http://127.0.0.1:6800 --task-id <task_id> \
                  --llm-base-url http://<model-host>:<port>/v1 --model <served-model-name> \
                  --output trajectory.json

      One process hosts the whole portfolio; every task starts from a task-conditioned initial state and is fully reset between episodes. Rewards come from the frozen task evaluator; the source-trace validation that certifies each environment is a separate reviewer and never supplies training reward. See the repository README for the HTTP API, action space, and the constructor (t2w-construct) that builds an environment from a trace set.

      Citation

      @misc{wang2026trace2worldreconstructingappworlds,
            title={Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training},
            author={Zezhou Wang and Hanzhang Zhou and Panrong Tong and Quyu Kong and Pengxiang Li and Steven Hoi and Zhuzhong Qian and Yue Wang},
            year={2026},
            eprint={2610.30792},
            archivePrefix={arXiv},
            primaryClass={cs.LG},
            url={https://arxiv.org/abs/2610.30792},
      }