ResearchGym: AI Agents Research Benchmark

• Benchmark Construction: Five oral and spotlight papers from ICML, ICLR, and ACL repurposed into containerized task environments with 39 total sub-tasks[1]
• Preserved Components: Original datasets, evaluation harnesses, and baseline implementations retained; proposed methods withheld to force novel hypothesis generation[1]
• Agent Evaluation Protocol: Agents must propose hypotheses, execute experiments, and attempt to surpass strong human baselines on paper metrics[1]
• Model Variants Tested: GPT-5 (primary), Claude Code (Opus-4.5), and Codex (GPT-5.2) agent scaffolds evaluated[1]
• Execution Environment: Closed-loop research infrastructure enabling systematic evaluation and analysis of autonomous agent behavior[1]
• Identified Failure Modes: Impatience, overconfidence, and poor parallel experiment coordination documented in agent behavior[1]

ResearchGym addresses a critical infrastructure gap in AI agent evaluation, establishing standardized benchmarks for research automation. The demonstrated capability-reliability gap across multiple frontier models (GPT-5, Claude Opus-4.5, DeepSeek) suggests that current language model agents require significant improvements in reasoning consistency, resource management, and hypothesis validation before autonomous research workflows become reliable. This benchmark will likely drive development of more robust agent architectures and training methodologies. The framework's success in identifying systematic failure modes provides a foundation for iterative improvements in agent design, potentially accelerating progress toward more autonomous scientific discovery systems. However, the low success rates indicate that near-term applications should focus on agent-assisted rather than fully autonomous research tasks.