Papers with Code adds SOTA badges and external evals
๐กDiscover how the revamped Papers with Code tracks SOTA models using new community-driven evaluation metrics.
โก 30-Second TL;DR
What Changed
Reintroduced SOTA badges for papers ranking in the top 3 of specific benchmarks.
Why It Matters
These features make it significantly easier for researchers to identify state-of-the-art models and verify performance claims through community-driven external benchmarks.
What To Do Next
Visit the updated tasks page on paperswithco.de to verify if your latest model benchmarks are correctly represented with external evals.
Key Points
- โขReintroduced SOTA badges for papers ranking in the top 3 of specific benchmarks.
- โขUpdated trending algorithm to include GitHub star velocity and Hugging Face artifact activity.
- โขAdded support for external, third-party evaluation results beyond the original paper's metrics.
- โขExpanded coverage of benchmarks including ImageNet, 3D semantic segmentation, and object counting.
๐ง Deep Insight
AI-generated analysis for this event โ not the original article.
๐ Enhanced Key Takeaways
- โขPapers with Code was acquired by Meta AI in 2021, which significantly accelerated its integration into the broader open-science ecosystem.
- โขThe platform utilizes a crowdsourced approach combined with automated extraction pipelines to link GitHub repositories to research papers.
- โขThe integration of Hugging Face artifacts marks a shift toward 'living' benchmarks where model weights and inference code are directly executable within the browser.
- โขThird-party evaluation support addresses the 'reproducibility crisis' by allowing community-verified metrics to supersede potentially biased self-reported results.
- โขThe platform's metadata is increasingly used by downstream AI agents and automated literature review tools to train and validate RAG (Retrieval-Augmented Generation) systems.
๐ Competitor Analysisโธ Show
| Feature | Papers with Code | Hugging Face Spaces | arXiv Sanity |
|---|---|---|---|
| Primary Focus | Benchmark Tracking | Model Hosting/Demos | Paper Discovery |
| Pricing | Free (Open) | Free/Enterprise | Free |
| Benchmarks | Extensive/Automated | Limited/Community | None |
๐ ๏ธ Technical Deep Dive
- The platform employs a combination of Natural Language Processing (NLP) models to parse PDF research papers and extract tables containing benchmark results.
- SOTA (State-of-the-Art) badges are calculated using a dynamic ranking system that normalizes metrics across different evaluation protocols.
- External evaluation support is implemented via a standardized API that allows researchers to submit JSON-formatted results verified by cryptographic hashes.
- The trending algorithm utilizes a weighted moving average of GitHub API events (stars, forks, commits) and Hugging Face Hub activity (downloads, likes, and inference requests).
๐ฎ Future ImplicationsAI analysis grounded in cited sources
โณ Timeline
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning โ
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.