LLM-WikiRace Reveals LLM Planning Limits

Post LinkedIn

📄Read original on ArXiv AI

#benchmark#planning#reasoning#knowledge-graphsllm-wikirace

💡New benchmark shows even Gemini-3/GPT-5 fail 77% on hard LLM planning tasks

⚡ 30-Second TL;DR

What Changed

Introduces Wikipedia hyperlink navigation task for LLM planning

Why It Matters

This benchmark exposes critical gaps in frontier LLMs' reasoning, pushing development toward better long-horizon planning. It provides a simple, reproducible testbed for improving agentic capabilities in real-world knowledge navigation.

What To Do Next

Visit https://llmwikirace.github.io to download code and benchmark your LLM on hard levels.

Who should care:Researchers & Academics

Key Points

•Introduces Wikipedia hyperlink navigation task for LLM planning
•Gemini-3 leads at 23% on hard levels, superhuman on easy
•World knowledge essential but long-horizon planning dominant
•Top models loop instead of replanning after failures
•Open code and leaderboard at llmwikirace.github.io

🧠 Deep Insight

Background and context from public sources — not the original article. 2 sources cited.

🔑 Enhanced Key Takeaways

•LLM-WikiRace benchmarks LLMs on navigating Wikipedia hyperlinks from source to target pages, stratifying tasks into easy, medium, and hard splits based on shortest-path length[1][2].
•Frontier models like Gemini-3, GPT-5, and Claude Opus 4.5 achieve over 90% success on easy levels (superhuman performance) but drop to 23% for Gemini-3 on hard levels[1][2].
•World knowledge is essential up to a threshold, after which long-horizon planning and reasoning dominate performance[1][2].
•Top models fail to replan after errors, often entering loops instead of recovering[1][2].
•Metrics include success rate, suboptimal steps (excess beyond shortest path), and average cost (tokens and monetary for closed models); step limit is 30, providing 3x budget over optimal paths up to 8 steps[1].

📊 Competitor Analysis▸ Show

Feature	LLM-WikiRace	Other Benchmarks
Task	Wikipedia hyperlink navigation for planning/reasoning	Varies (e.g., multi-modal in others)
Difficulty Splits	Easy (>90% success), Medium (50-70%), Hard (<25%)	Not specified
Metrics	Success rate, suboptimal steps, avg cost	Varies
Frontier Model Hard Success	Gemini-3: 23%	N/A (isolates textual planning)

🛠️ Technical Deep Dive

•Task requires step-by-step hyperlink navigation with 30-step limit (plateaus beyond; longest optimal path is 8 steps)[1].
•Algorithm detailed in Appendix B; evaluates full episodes including failures[1].
•Fine-tuning improves easy split (22.5% to 67.5% after 300 steps), modest on medium (1.3% to 4.6%), none on hard (0%)[1].
•Open code and leaderboard at llmwikirace.github.io[1][2].
•Subjects: Artificial Intelligence (cs.AI), Machine Learning (cs.LG)[2].

🔮 Future ImplicationsAI analysis grounded in cited sources

Highlights persistent gaps in long-horizon planning for frontier LLMs, emphasizing need for better replanning and recovery mechanisms; serves as open benchmark to drive progress in reasoning systems beyond world knowledge reliance.

⏳ Timeline

2026-02

LLM-WikiRace paper released on arXiv (2602.16902v1), introducing benchmark with results on Gemini-3, GPT-5, Claude Opus 4.5

📎 Sources (2)

Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.

📄Read original article on ArXiv AI

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #benchmark

Same product