ProactiveMobile Benchmark for Proactive Mobile AI

Post LinkedIn

📄Read original on ArXiv AI

#mobile-agents#mllm-benchmarksproactivemobileproactivemobileqwen2.5-vl-7b-instructo1gpt-5

💡New benchmark shows MLLMs lack proactivity; Qwen beats o1/GPT-5 at 19% success

⚡ 30-Second TL;DR

What Changed

New benchmark with 3,660 instances in 14 real-world mobile scenarios

Why It Matters

This benchmark exposes proactivity gaps in current MLLMs, spurring development of autonomous mobile agents. It enables standardized, executable evaluations critical for advancing beyond reactive paradigms.

What To Do Next

Download ProactiveMobile from arXiv:2602.21858 and evaluate your MLLM on its proactive tasks.

Who should care:Researchers & Academics

Key Points

•New benchmark with 3,660 instances in 14 real-world mobile scenarios
•Inf ers user intent from 4 on-device signals, generates sequences from 63 APIs
•Audited by 30 experts for accuracy and feasibility
•Fine-tuned Qwen2.5-VL-7B-Instruct: 19.15% success vs o1 (15.71%) and GPT-5 (7.39%)

🧠 Deep Insight

Background and context from public sources — not the original article. 10 sources cited.

🔑 Enhanced Key Takeaways

•ProactiveMobile benchmark is open-sourced, enabling community access to its 3,660 instances and code for further research and model development.[1][2]
•The paper was authored by a team of 15 researchers including Dezhi Kong, Zhengzhao Feng, and others affiliated with institutions like Xiaomi Corporation's HyperAI Team.[6][7]
•ProactiveMobile emphasizes multi-intent instances and reference-based plus LLM-as-a-judge metrics to ensure robust evaluation of complex, multi-step proactive tasks.[1][2]

🛠️ Technical Deep Dive

•Four contextual signal dimensions: user profile, device status, world information, and behavioral trajectories, used to infer latent user intent.[1]
•Task formalization requires generating executable function sequences from a predefined pool of 63 APIs, enabling objective and deployable evaluation.[1][2]
•Benchmark construction involved mapping intents to function sequences with expert audit by 30 specialists for factual accuracy, logical consistency, and feasibility.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

ProactiveMobile will drive fine-tuning improvements in MLLMs for mobile proactivity beyond 20% success rates

Low absolute scores under 20% across top models confirm the benchmark's difficulty and its role as a testbed for breakthroughs in learnable proactivity.[1]

It establishes a standard for evaluating multi-step, realistic mobile agent tasks over simplistic single-step benchmarks

Prior benchmarks lack real-world multi-dimensional context and executable sequences, which ProactiveMobile addresses through 14 diverse scenarios.[1]

⏳ Timeline

2026-02

ProactiveMobile paper submitted to arXiv as v1 on February 25 by Dezhi Kong et al.

📎 Sources (10)

Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.

📄Read original article on ArXiv AI

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #mobile-agents

Same product