Arena rankings, eval controversies and what benchmarks actually measure. Read the fine print behind every "SOTA" claim.
208 articles
AOE Tech Labs reported that its Floatboat Harness enabled DeepSeek-V4-Flash to beat Claude Opus 4.8 on all five tested benchmarks. The single-variable experiment changed only the execution Harness, while using the same $0.14 model at 57.1 times lower cost.
Pandaily · 37d ago
Xiaomi has open-sourced Xiaomi-Robotics-1, an embodied-AI foundation model. The release includes the full workflow from real-robot post-training to deployment, along with code for related benchmark evaluations.
TechNode · 44d ago
The new 'Schema' harness improves performance on the ARC-AGI-3 benchmark by optimizing the interaction process between models and game environments. It uses Claude Opus 4.8 and Fable 5 to achieve 99% accuracy without modifying model weights.
Reddit r/MachineLearning · 63d ago
A comparative test of 63 complex prompts shows GPT-5.6 Sol Ultra significantly outperforming other models in generating detailed, interactive 3D web environments.
虎嗅 · 68d ago
Current safety benchmarks are failing to accurately measure the evolving hacking capabilities of frontier AI models. This leaves security teams and regulators without reliable tools to assess the risks posed by advanced systems.
The Next Web (TNW) · 72d ago
MemoBench is a new diagnostic benchmark designed to test world models' ability to maintain object permanence and state evolution in dynamic environments. It reveals that current video generation models struggle to accurately track and update object states during occlusion.
虎嗅 · 74d ago
Beijing-based Zhipu AI has released GLM-5.2, a model that demonstrates competitive performance against Anthropic’s Claude Opus 4.8 in cybersecurity bug-hunting tasks. This development highlights the narrowing gap between Chinese AI models and top-tier US counterparts.
SCMP Technology · 81d ago
The First Proof project tested four AI systems against original, unpublished research-level math problems, finding that while AI shows promise, it still lags behind top human researchers.
虎嗅 · 86d ago
Sina Weibo researchers introduced VibeThinker-3B, a 3-billion parameter model that allegedly matches or exceeds the reasoning performance of massive flagship models. The release has sparked intense debate within the AI community regarding the validity of current benchmarks.
VentureBeat · 93d ago
UC Berkeley researchers launched the Agents’ Last Exam (ALE), a rigorous benchmark designed to evaluate AI agents on real-world, long-horizon professional workflows. GPT-5.5 achieved the top score, outperforming Anthropic's Claude Fable 5 in a test that emphasizes practical labor impact over static coding puzzles.
VentureBeat · 99d ago
Chinese startup Spirit AI has surpassed Nvidia's robotics models on the RoboArena leaderboard. Their Spirit v1.6 model achieved a score of 1,924, outperforming Nvidia's Cosmos3-Nano-Policy.
The Next Web (TNW) · 105d ago
Datacurve launched DeepSWE, a new coding benchmark that reveals significant performance gaps between frontier models, crowning GPT-5.5 as the leader. The report also exposes a 32% error rate in existing SWE-Bench Pro evaluations, questioning the reliability of current industry standards.
VentureBeat · 114d ago
AI pioneer Fei-Fei Li has introduced a new benchmark designed to evaluate embodied spatial intelligence. This initiative aims to provide a standardized dataset and evaluation framework for the next generation of robotic and spatial AI systems.
量子位 · 119d ago
BenchJack is an automated red-teaming system designed to identify reward-hacking exploits in AI agent benchmarks. By applying it to 10 major benchmarks, researchers discovered 219 flaws, demonstrating that many current evaluation metrics are easily gamed without actual task completion.
ArXiv AI · 127d ago
HarvestBench is a reproducible benchmark that measures whether LLM agents will spend resources to avoid killing animals during a simulated corn harvest. Across nine models, kill rates ranged from 0.4% to 98.8%, while moral briefings dramatically reduced harmful behavior.
ArXiv AI · 11d ago
OpenAI reportedly declared the AGI era after GPT-6 Astra achieved a 99.9% score, but the same model received 62.7% when run through the benchmark’s official software. ARC Prize published both figures, raising questions about evaluation methodology and the reproducibility of the AGI claim.
The Next Web (TNW) · 12d ago
DS-Lighting is an open-source harness toolkit that makes the infrastructure behind data-science agents explicit and reusable. It organizes data, workflow, execution, and evaluation layers while standardizing benchmark tasks for controlled comparison.
ArXiv AI · 17d ago
Terminal Bench 4.0 reports that GLM-5.3 performs at roughly the same level as Fable 5 when accounting for the margin of error. The discussion also highlights the benchmark’s rapid iteration and the high cost of evaluating coding agents at scale.
Reddit r/LocalLLaMA · 20d ago
Together AI evaluated GLM-5.3 and GLM-5.3 Flash across 900 DeepSWE rollouts. GLM-5.3 Flash delivered 5.6 fewer pass@1 points at 17x lower cost, while the pass@4 gap narrowed to 2.6 points.
Together AI Blog · 21d ago
Agent Seer proposes synthesizing realistic evaluation scenarios for tool-using AI agents directly from tool specifications. By using function names, natural-language descriptions, and typed parameter schemas, it aims to reduce manual curation and keep benchmarks aligned with evolving APIs.
Apple Machine Learning · 21d ago
HarnessOpt-Bench evaluates whether an LLM can improve another agent’s coding harness without accessing held-out tests, credentials, or budgets. Across 111 runs, model selection influenced performance more than harness selection, while Claude Opus 5 led three of four tasks under OpenCode.
Reddit r/MachineLearning · 22d ago
Nvidia presented the architecture of the Groq 3 LPX rack at Hot Chips 2026. The company also published its first third-party inference benchmark and said an LP30-based rack is already in production.
Tom's Hardware · 23d ago
ESQ-Bench is a new Oracle-first benchmark designed to test NL2SQL dialect generalization and silent semantic divergence in realistic enterprise schemas. Across three complexity tiers, even execution-passing queries frequently produced incorrect results, while Claude Sonnet 4.6 outperformed GPT-4o and local Llama 3.2 by a wide margin.
ArXiv AI · 23d ago
OpenAI says its new Jalapeno chips outperformed Nvidia’s current lineup in testing. The chips reportedly led in AI workload per unit of power and response speed.
Bloomberg Technology · 24d ago
OpenAI’s Jalapeño chip reportedly delivers strong inference performance at scale. On Semianalysis’s InferenceX benchmark, it achieved both higher tokens per user and greater throughput per kilowatt than currently available state-of-the-art systems.
TechCrunch AI · 24d ago
Unbounded Labs released Bart, a 2.82B-parameter language model trained from scratch on 20.1B English tokens published before 1931. The team also open-sourced its curated datasets, Vintage CORE benchmarks, training code, evaluations, and research runs.
Reddit r/MachineLearning · 25d ago
A new CSE benchmark shows that LLMs handle individual instructions reasonably well but struggle when many constraints must be satisfied simultaneously. Across 15 models and 369,753 checks, all-constraint success collapses rapidly beyond roughly five to six simultaneous requirements.
ArXiv AI · 35d ago
SAPO is a segment-level Automatic Prompt Optimization method that separately improves prompt roles, context, tasks, and output formats. Across five benchmarks and GPT-3.5-Turbo and GPT-4o-mini, it reports the best average performance against zero-shot prompting and several strong APO baselines.
ArXiv AI · 36d ago
SAI Review attempted to reproduce 105 of ICML 2026’s Oral papers and found that only eight reproduced more than 80% of their verifiable claims. The review exposed broken code, missing files, mismatched results, unavailable models, and extremely high reproduction costs.
虎嗅 · 40d ago
PPIO has officially launched its Fusion model, claiming it can outperform leading models in intelligence while costing one-tenth as much. The announcement positions Fusion as a potentially more cost-efficient option for production AI workloads.
量子位 · 42d ago