Beihang's Code2Bench Ends Code Model Score Inflation

🔑 Enhanced Key Takeaways

• •Code2Bench, developed by Beihang University researchers, is an open-sourced dynamic benchmark for code LLMs that generates fresh coding problems on-the-fly to prevent data contamination, as detailed on its official GitHub site code2bench.github.io.
• •It employs dual scaling mechanisms: problem complexity scaling by expanding source problem variants and test rigor scaling with diverse edge cases, addressing weak evaluation rigor in existing benchmarks like HumanEval.
• •The benchmark features a fully automated end-to-end framework for problem generation, execution, and evaluation, eliminating human biases and the 'illusion of correctness' from memorized solutions.
• •Accepted to ICLR 2026, Code2Bench has a live leaderboard ranking top code models like GPT-4o, Claude 3.5 Sonnet, and DeepSeek-Coder-V2, revealing significant performance drops compared to contaminated benchmarks.
• •Early leaderboard results show top models scoring 20-40% lower than on static benchmarks, validating its effectiveness in measuring true generalization capabilities.

🛠️ Technical Deep Dive

• •Problem Generation: Uses a seed set of 236 core problems; applies dual scaling with 5 complexity levels (expanding inputs/outputs) and 4 test rigor levels (unit tests, edge cases, multi-step verification).
• •Automation Pipeline: LLM-driven generation via GPT-4, followed by execution in isolated Docker environments with Python 3.10, supporting libraries like numpy, pandas.
• •Evaluation Metrics: Pass@1, Pass@10 under temperature=0; strict equivalence checking with hidden test cases to prevent leakage.
• •Contamination Mitigation: Problems regenerated periodically; checks against training data corpora like The Stack v2.
• •Implementation: Built with Python, available at github.com/code2bench/code2bench; supports custom model integration via OpenAI/VLLM APIs.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline