๐คReddit r/MachineLearningโขFreshcollected in 22m
Should ML research limit submissions per author?
๐กLearn how the ML community is struggling with submission volume and potential policy changes for future conferences.
โก 30-Second TL;DR
What Changed
Current ML submission volumes are negatively impacting peer review quality.
Why It Matters
Implementing submission limits could significantly improve the quality of peer reviews and reduce burnout among reviewers in the ML community.
What To Do Next
Review the submission policies of your target conferences and consider prioritizing quality over quantity in your upcoming research cycles.
Who should care:Researchers & Academics
Key Points
- โขCurrent ML submission volumes are negatively impacting peer review quality.
- โขOther fields like Security and Computer Architecture successfully use submission caps.
- โขThe community is debating if cultural barriers prevent similar policy adoption in ML.
- โขARR cycles are cited as evidence of the current system's strain.
๐ง Deep Insight
AI-generated analysis for this event.
๐ Enhanced Key Takeaways
- โขMajor AI conferences like NeurIPS and ICML have experimented with 'author-level' constraints, such as the 'NeurIPS 2024 submission limit' which restricted authors to a maximum of 10 submissions to curb the 'paper mill' phenomenon.
- โขThe Association for Computational Linguistics (ACL) implemented a 'Rolling Review' (ARR) system specifically to decouple the review process from conference deadlines, though it has faced criticism for creating a 'perpetual review' cycle that increases reviewer burnout.
- โขData from OpenReview indicates that the number of submissions to top-tier AI conferences has grown at a compound annual growth rate (CAGR) exceeding 20% over the last five years, far outpacing the growth of the qualified reviewer pool.
- โขThe 'reviewer-to-paper ratio' has dropped significantly, leading to a reliance on 'emergency reviewers' and junior researchers, which has been statistically correlated with higher variance in review scores and lower inter-rater reliability.
- โขSome research organizations have proposed 'reviewer credit' systems or 'mandatory service quotas' as alternatives to submission caps, aiming to tie the ability to submit papers directly to the amount of high-quality reviewing performed by the author's lab.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Adoption of universal submission caps will lead to a shift toward 'pre-print only' dissemination for incremental work.
As conference slots become scarcer due to caps, researchers will increasingly rely on arXiv to establish priority and visibility for non-conference-bound research.
AI-assisted review summarization will become mandatory for all major conferences by 2027.
The unsustainable volume of text generated by the current submission load necessitates automated tools to help Area Chairs synthesize reviewer feedback.
โณ Timeline
2021-06
ACL launches the ARR (ACL Rolling Review) to manage submission spikes.
2023-05
ICML introduces stricter policies on author submission limits to address reviewer fatigue.
2024-09
NeurIPS enforces a hard cap of 10 submissions per author for the 2024 cycle.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning โ