IDP Leaderboard Benchmarks 16 VLMs

🔑 Enhanced Key Takeaways

• •IDP Leaderboard evaluates models across 16 datasets spanning 6 tasks: OCR, KIE, document classification, VQA, table extraction, and long document processing[1][2].
• •Developed in collaboration with Indian Institute of Technology Indore and sponsored by Nanonets, filling a gap left by benchmarks like OpenVLM, Chatbot Arena, and LiveBench that lack comprehensive IDP coverage[1][2].
• •Gemini 2.5 Flash is the top performer overall, though it trails Gemini-2.0-Flash slightly on OCR (1.84% lower) and classification (0.05% lower)[1].
• •Upcoming expansions include a confidence score calibration task and addition of more models to reflect evolving document AI capabilities[2].

🛠️ Technical Deep Dive

• •Evaluates 10 models across 16 datasets totaling 9,229 documents, using public, synthetic, and newly annotated data[1].
• •Task scores average performance across multiple datasets per task (e.g., OCR splits handwritten and digital text); overall score averages task scores[1].
• •Employs task-specific accuracy metrics with ground-truth answers for all datasets[1].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline