🐯虎嗅•Stalecollected in 3h
ProgramBench Crushes Top AI Coders at 0%
💡New benchmark shows even top LLMs can't rebuild real software—must-read for AI coders
⚡ 30-Second TL;DR
What Changed
All leading models achieve 0% completion on rebuilding projects like FFmpeg, SQLite without source code.
Why It Matters
Highlights critical gap in AI's software engineering capabilities, pushing need for better long-term planning in agents. May shift focus from code gen to system architecture training. Benchmarks like this will drive future model improvements.
What To Do Next
Download ProgramBench dataset and test your fine-tuned coding model on FFmpeg rebuild task.
Who should care:Researchers & Academics
Key Points
- •All leading models achieve 0% completion on rebuilding projects like FFmpeg, SQLite without source code.
- •Claude Opus 4.7 leads with 3% 'almost complete' (95%+) tasks; models favor monolithic single-file code.
- •Better performance on C/C++ vs. Rust; simple CLI tools succeed more than complex systems.
- •Uses behavioral equivalence testing via agent-driven fuzzing, not code similarity.
🧠 Deep Insight
AI-generated analysis for this event.
🔑 Enhanced Key Takeaways
- •ProgramBench utilizes a 'sandbox-in-the-loop' architecture that dynamically recompiles and executes the target software in real-time to verify behavioral equivalence against the original project's test suites.
- •The benchmark specifically targets 'long-horizon planning' failures, identifying that while models can generate syntactically correct functions, they fail to maintain state consistency across thousands of lines of interdependent header files.
- •Initial analysis suggests that the 0% success rate is heavily influenced by the 'hallucination of dependencies'—models frequently invent non-existent APIs or libraries that they assume exist within the target project's ecosystem.
📊 Competitor Analysis▸ Show
| Feature | ProgramBench | SWE-Bench | HumanEval | MBPP |
|---|---|---|---|---|
| Focus | Full System Rebuild | Issue Resolution | Function Completion | Basic Python Tasks |
| Verification | Behavioral Equivalence | Unit Test Pass | Unit Test Pass | Unit Test Pass |
| Complexity | High (System Level) | Medium (Repo Level) | Low (Snippet Level) | Low (Snippet Level) |
🛠️ Technical Deep Dive
- •Evaluation Engine: Uses a containerized environment (Docker/Podman) that enforces strict network isolation to prevent models from accessing external package repositories during the build process.
- •Metric Calculation: Employs a multi-stage verification pipeline: 1) Compilation success, 2) Linker integrity, 3) Behavioral equivalence via differential fuzzing against the original binary's output.
- •Input Constraints: Models are provided with a 'Documentation-Only' context window, stripping all original source code and forcing the agent to rely on provided man-pages and API specifications.
- •Failure Mode Analysis: Models exhibit 'Context Drift' where the agent loses track of global variable definitions and macro expansions required for successful C-preprocessor execution.
🔮 Future ImplicationsAI analysis grounded in cited sources
AI development will shift from 'code generation' to 'architectural planning' benchmarks.
The failure of current SOTA models on ProgramBench proves that local code generation is a solved problem, while system-level structural planning remains a critical bottleneck.
Future LLM architectures will require persistent, externalized memory modules for large-scale software projects.
The inability of models to maintain state across complex, multi-file systems suggests that current context-window-based architectures are insufficient for full-system engineering.
⏳ Timeline
2025-11
Initial research paper on 'System-Level Behavioral Equivalence' published by the SWE-Bench team.
2026-03
Beta release of ProgramBench to select academic partners for stress-testing.
2026-05
Public release of ProgramBench results highlighting the 0% success rate for top-tier models.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: 虎嗅 ↗