Search

Tag: #llm-benchmark24 results

Alibaba's Qwen3.7-Max Outperforms OpenAI and Google in Coding

Alibaba's latest AI model, Qwen3.7-Max, has secured the fourth position on the global Code Arena leaderboard. It is the only developer besides Anthropic to break into the top five, signaling a significant shift in the competitive landscape of coding-focused LLMs.

SCMP TechnologyMedia•May 27#llm-benchmark#coding-assistant#qwen-model

Qwen 3.7 Ranks Second Globally in Coding Benchmarks

Alibaba's Qwen 3.7 model has achieved a top-tier ranking in global coding benchmarks. It is currently positioned just behind Claude, marking a significant milestone for Chinese LLMs.

量子位Media•May 26#llm-benchmark#coding-assistant#chinese-ai

ThermoQA LLM Thermodynamic Benchmark Launch

ThermoQA introduces 293 thermodynamics problems in three tiers: property lookups, component analysis, and full cycle analysis. It evaluates six frontier LLMs, led by Claude Opus (94.1%), GPT-5.4 (93.1%), and Gemini 3.1 Pro (92.5%). The open-source dataset reveals LLMs' reasoning gaps beyond memorization.

ArXiv AIResearch•Apr 23#llm-benchmark#thermodynamics#open-source

GTO Wizard Poker AI Benchmark

GTO Wizard Benchmark launches a public API and framework for evaluating Heads-Up No-Limit Texas Hold'em agents against superhuman GTO Wizard AI, which outperforms Slumbot by 19.4 bb/100. It employs AIVAT for 10x variance reduction efficiency. Benchmarks reveal LLM progress but all models lag far behind the baseline.

ArXiv AIResearch•Mar 26#poker-ai#llm-benchmark#multi-agent

DeIllusionLLM Bridges LLM Know-Act Gap

LLMs show a know-act gap: they detect input flaws discriminatively but generate flawed answers. Researchers introduce FaultyScience benchmark and DeIllusionLLM, a task-level autoregressive framework using self-distillation to unify judgment and reasoning. It reduces error-ignoring responses while preserving reasoning ability.

ArXiv AIResearch•Mar 25#know-act-gap#task-autoregression#self-distillation

New Causal Reasoning Benchmark Launches

CausalReasoningBenchmark is a new benchmark with 173 queries from 138 real-world datasets across 85 papers and 4 textbooks. It separately evaluates causal identification (structured specs) and estimation (numerical outputs). Baseline LLM results show 84% high-level strategy success but only 30% full identification accuracy.

ArXiv AIResearch•Feb 25#causal-inference#llm-benchmark#causal-evaluation

GPSBench Tests LLM GPS Reasoning

Researchers launch GPSBench, a 57,800-sample dataset across 17 tasks to probe LLMs' geospatial reasoning without tools. 14 SOTA LLMs show reliability in geographic knowledge but struggle with geometric computations like distance and bearing. Dataset, code, and findings reveal trade-offs in finetuning and augmentation benefits.

ArXiv AIResearch•Feb 19#geospatial-reasoning#gps-coordinates#llm-benchmark

AI Outsmarts Humans in 40% Yield Scam Test

Nanjing University's latest research shows AI models remain more rational than humans when confronted with a scam promising 40% annualized returns. Under simulated investor pressure, seven mainstream large models demonstrated superior financial scam resistance by holding their底线 better. This real-world test highlights AI's edge in fraud detection.

钛媒体Media•Apr 25#llm-benchmark#financial-safety#scam-detection

LLMs Struggle in Clue Reasoning Test

Researchers built a text-based multi-agent Clue game to evaluate LLMs' multi-step deductive reasoning with GPT-4o-mini and Gemini-2.5-Flash agents. Across 18 simulated games, agents secured only four correct wins, struggling with consistent reasoning. Fine-tuning on logic puzzles did not reliably enhance performance and sometimes increased verbosity without precision gains.

ArXiv AIResearch•Mar 19#deductive-reasoning#llm-benchmark#game-eval