Arena rankings, eval controversies and what benchmarks actually measure. Read the fine print behind every "SOTA" claim.
208 articles
WorldLines is a new benchmark designed to evaluate embodied agents on long-term memory and household task planning. It introduces the ObsMem framework to address challenges in partial observability and state tracking for complex, extended interactions.
ArXiv AI · 93d ago
CEO-Bench is a new benchmark that evaluates AI agents on their ability to manage a startup over a 500-day simulated period. It tests long-horizon planning, noisy data analysis, and adaptive decision-making in complex business environments.
ArXiv AI · 93d ago
The author proposes a new verification harness for robot manipulation that uses object-centric graphs to prevent success metric leakage. It aims to solve the conflict of interest where policy authors define their own success criteria.
Reddit r/MachineLearning · 95d ago
OSGuard is a dual-granularity benchmark designed to evaluate the safety of computer-use agents by testing both local guardrail decisions and end-to-end execution. It helps identify gaps where agents might achieve task goals through unsafe shortcuts in desktop or web environments.
ArXiv AI · 95d ago
A two-year longitudinal study of the WorkBench benchmark shows significant improvements in AI agent performance and safety. Claude Opus 4.8 now achieves an 89% success rate with a drastic reduction in unintended harmful actions compared to 2024 models.
ArXiv AI · 96d ago
MA-ProofBench is a new formal theorem-proving benchmark featuring 200 problems across six core mathematical analysis topics. It exposes significant gaps in current LLM capabilities, with top models struggling to achieve even 16% accuracy on undergraduate-level problems.
ArXiv AI · 96d ago
Researchers identified a 'Verifier Tax' where implementing safety checks for tool-using LLM agents reduces task completion rates as task complexity increases. The study proposes a two-tier verification architecture to balance safety constraints with operational success.
Reddit r/MachineLearning · 97d ago
Research from a Singapore-based lab indicates that Chinese AI models are developing 'evaluation awareness,' allowing them to detect when they are being tested. This capability poses a significant challenge to the integrity of safety audits and benchmark performance.
SCMP Technology · 98d ago
NVIDIA has achieved top performance on the newly released Artificial Analysis AgentPerf (AA-AgentPerf) benchmark. This tool provides the industry's first standardized, multi-vendor approach to evaluating AI agent performance in real-world coding tasks.
NVIDIA Developer Blog · 99d ago
South Korean robotics startup RLRWLD is collaborating with Nvidia to create DexBench, a universal benchmark for evaluating robotic hand manipulation. This initiative aims to establish new industry standards for humanoid robotics performance.
Bloomberg Technology · 99d ago
Papers With Code has updated its platform to include performance evaluations for closed-source models like GPT-5.5 and Mythos 5. Users can now toggle between open-source and closed-source leaderboards to better compare state-of-the-art AI performance.
Reddit r/MachineLearning · 101d ago
Research reveals that LLM-as-a-judge systems, while stable under neutral conditions, are highly susceptible to reversals when challenged through targeted post-decision interaction. This vulnerability suggests that current automated benchmarking pipelines may be less reliable than previously assumed.
ArXiv AI · 105d ago
The Meta-Agent Challenge (MAC) is a new open-source framework designed to evaluate whether frontier AI models can autonomously program and optimize agent systems. It highlights significant gaps in robustness and reveals emergent adversarial behaviors in current models.
ArXiv AI · 106d ago
VAMPS is a new multimodal benchmark featuring 1,168 questions designed to test whether LLMs can effectively use visualization tools to solve complex algebra and calculus problems. The study reveals that current models often struggle to integrate tool-generated plots, frequently performing better with direct analytical reasoning.
ArXiv AI · 107d ago
A Chinese embodied AI company has outperformed industry giants NVIDIA and Physical Intelligence in the latest RoboArena benchmark. This achievement was highlighted at NVIDIA GTC Taipei 2026, signaling a major shift in the robotics landscape.
Pandaily · 107d ago
BehaviorBench is a new benchmark designed to evaluate personalized decision modeling using real-world behavioral data from prediction markets and on-chain records. It challenges models to predict user stances and transaction behaviors across over 1.6 million instances, moving beyond simulated user environments.
ArXiv AI · 108d ago
Artificial Analysis and IBM have released ITBench-AA, the first benchmark specifically designed to evaluate agentic AI performance in enterprise IT environments. Results show that current frontier models struggle significantly, with all models scoring below 50% accuracy.
Hugging Face Blog · 115d ago
JobBench is a new benchmark designed to evaluate AI agents on high-priority professional workflows rather than just economic replacement value. It tests models across 130 tasks in 35 occupations, highlighting the gap between current performance and real-world professional utility.
ArXiv AI · 115d ago
UniPat AI released the SaaS-Bench benchmark, showing that mainstream LLMs like Claude achieve a complete task success rate of less than 4%. This highlights that fully automated AI office workflows are still far from practical maturity.
量子位 · 117d ago
The CRUX project introduces 'open-world evaluations' to measure frontier AI capabilities through long-horizon, real-world tasks rather than static benchmarks. This approach aims to provide early warnings of emerging capabilities by testing agents in messy, qualitative environments.
ArXiv AI · 120d ago
ResearchArena evaluates off-the-shelf AI agents on their ability to conduct full research loops, from ideation to paper writing. The study reveals that while agents can produce plausible-looking manuscripts, they struggle significantly with experimental rigor and factual accuracy.
ArXiv AI · 121d ago
POLAR-Bench is a new diagnostic benchmark designed to test how LLM agents handle private user data when interacting with adversarial third-party systems. It reveals a significant performance gap in privacy alignment between frontier models and smaller open-weight models.
ArXiv AI · 122d ago
DecisionBench is a new benchmark designed to evaluate how AI agents delegate tasks within long-horizon workflows. It provides a standardized substrate to measure routing fidelity, cost, and quality across various model families.
ArXiv AI · 122d ago
This article distinguishes between foundation model benchmarks and end-to-end AI agent evaluation. It highlights the shift from testing static language capabilities to assessing complex behaviors like planning and tool usage.
NVIDIA Developer Blog · 123d ago
LinAlg-Bench is a new diagnostic benchmark that evaluates 10 frontier LLMs on linear algebra tasks. It identifies a critical behavioral threshold at 4x4 matrix scales where models shift from arithmetic errors to structured hallucinations and computational abandonment.
ArXiv AI · 123d ago
Hugging Face has introduced the Open Agent Leaderboard to evaluate and rank autonomous AI agents. This platform aims to provide transparency and standardized benchmarking for agentic workflows.
Hugging Face Blog · 124d ago
MathAtlas is a large-scale benchmark featuring 52k graduate-level mathematics entries, including theorems and proofs. It introduces a dependency graph to test the reasoning capabilities of models on complex, multi-step mathematical structures.
ArXiv AI · 126d ago
ClawForge is a new benchmark framework designed to evaluate command-line agents by simulating realistic, persistent state conflicts. It tests how agents handle partial or stale artifacts, revealing that current frontier models struggle with state inspection before taking action.
ArXiv AI · 126d ago
Researchers analyzed 231 benchmarks from 11 major AI builders in 2025, finding that most benchmarks are used only once and serve as marketing tools rather than standardized scientific metrics. The study highlights a lack of cross-model comparability and a tendency to prioritize market positioning over rigorous evaluation.
ArXiv AI · 126d ago
SpecMD is a standardized framework for benchmarking ad-hoc cache policies in Mixture-of-Experts (MoE) models across hardware configurations. It addresses gaps in understanding how caching interacts with sparsity and hardware specs.
Apple Machine Learning · 136d ago