[Paper] A History-Aware Visually Grounded Critic for Computer Use Agents
Source: arXiv - 2606.11078v1
Overview
Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy’s completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.
Key Contributions
This paper presents research in the following areas:
- cs.AI
- cs.CL
- cs.CV
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.AI.
Authors
- Jaewoo Lee
- Zaid Khan
- Archiki Prasad
- Justin Chih-Yao Chen
- Supriyo Chakraborty
- Kartik Balasubramaniam
- Sambit Sahu
- Elias Stengel-Eskin
- Hyunji Lee
- Mohit Bansal
Paper Information
- arXiv ID: 2606.11078v1
- Categories: cs.AI, cs.CL, cs.CV
- Published: June 9, 2026
- PDF: Download PDF