Rebuttal Experiments Often Harm ML Papers

🔑 Enhanced Key Takeaways

• •The 'rebuttal-driven experimentation' phenomenon is exacerbated by the 'reviewer-author feedback loop' where reviewers feel compelled to justify their scores by requesting additional empirical evidence, leading to a 'reviewer-as-supervisor' dynamic rather than a 'reviewer-as-gatekeeper' role.
• •Major AI conferences like NeurIPS and ICLR have begun implementing 'rebuttal guidelines' that explicitly discourage reviewers from requesting new experiments that require significant computational resources or time, though enforcement remains inconsistent across sub-committees.
• •The rise of 'rebuttal-induced noise' has led to a measurable increase in 'rebuttal-induced performance degradation,' where authors rush training cycles or hyperparameter tuning, resulting in lower-quality results than those presented in the original submission.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline