Tag: #coding-benchmarks11 results
Moonshot and ZAI are reportedly approaching the performance of models from OpenAI and Anthropic. The models can reportedly build passable websites at a cost 75% lower than Claude, highlighting potential advantages in cost-sensitive coding workflows.
Z.ai released GLM-5.1, a 754B parameter open-source MoE LLM under MIT license on Hugging Face, capable of autonomous work for up to 8 hours. It outperforms Claude Opus 4.6 and GPT 5.4 on SWE-Bench Pro via a 'staircase pattern' optimization. The model handles 1,700 steps and thousands of tool calls without strategy drift.
Google has launched Gemini 3.7 Flash with significant gains on coding benchmarks and introductory pricing of $0.75 per million input tokens. Meanwhile, Gemini 3.5 Pro remains months behind schedule, with Google declining to confirm whether it will still launch.
References to 'models/gemma-4' found in AI Studio code, hinting at imminent release with Kaggle page live. Two test models—Significant-Otter (120B, unstable but factual) and Pteronura (27B dense, strong vision/coding)—perform well on LMArena.
MiniMax platform will introduce dynamic rate limiting during peak hours due to overwhelming popularity of the M2.7 model. This measure targets high-concurrency automated tasks to ensure fair resource allocation. M2.7 recently launched with strong coding benchmarks rivaling top models.
Together AI compared DeepSeek-V4 Flash 0731 with GPT-5.6 Luna across 900 DeepSWE rollouts. GPT-5.6 Luna achieved a 14-point lead in pass@1, while DeepSeek produced 4.8 times more solves per dollar.
Article rigorously tests DeepSeek V4's long text, code, and reasoning abilities. DeepSeek positioned as 'actively admitting defeat' in this evaluation.
Introduces framework using augmented Item Response Theory (IRT) to predict success/failure on individual agentic coding tasks. Decomposes agent ability into LLM and scaffold components for cross-leaderboard aggregation. Enables predictions for unseen benchmarks and agent combinations, aiding benchmark calibration.
Elon Musk's xAI, launched in March 2023 with 12 co-founders, has lost six and is slashing staff while trailing in coding benchmarks. Musk's response is a second full rebuild from scratch to pursue the most powerful AI.
Reddit post debates if Qwen3.5-9B suffices for agentic coding. It beats Qwen3-30B-A3B on all coding items and rivals Qwen3-Next-80B/GPT-OSS-20B on many. Targets 8GB VRAM laptops with Q8 quant and long contexts.