來源ArXiv AI•較早收集於 11h
VAMPS：視覺輔助數學問題解決基準測試
💡了解為何頂尖 LLM 在數學問題上無法有效使用視覺工具，以及如何評估您模型的推理能力。
⚡ 30 秒速覽
有什麼變化
引入 1,168 個基於伊朗大學入學考試問題的多模態雙語問答對。
為什麼重要
此基準測試突顯了代理型 AI 工作流程中的關鍵瓶頸，即模型在工具使用與推理之間的銜接能力不足。它為研究人員提供了一種標準化方法，用以衡量並改進未來多模態系統的「視覺基礎（visual grounding）」能力。
下一步行動
評估您目前的多模態代理在 VAMPS 基準測試上的表現，以確認其是否在視覺工具整合方面存在困難。
誰應關注：Researchers & Academics
關鍵要點
- •引入 1,168 個基於伊朗大學入學考試問題的多模態雙語問答對。
- •測試模型在構建視覺圖表（如交點、漸近線等）並據此得出答案的能力。
- •研究發現，目前大多數模型在直接分析推理上的表現優於工具輔助的視覺推理。
- •提供了一個診斷框架，用於評估模型如何處理外部工具的輸出結果。
🧠 深度解析
背景與延伸：來自公開資料，非原文內容。引用 13 個來源。
🔑 增強重點摘要
- •VAMPS uniquely focuses on evaluating LLMs' capability to actively construct and then reason over tool-generated visual plots, distinguishing it from prior multimodal benchmarks that primarily assess reasoning over pre-existing, fixed visual inputs.
- •The benchmark's problems are sourced from the Iranian University Entrance Exams, with 218 real problems translated into English and expanded with human-reviewed, LLM-generated synthetic variants, ensuring a diverse and challenging dataset.
- •A significant finding from VAMPS and related research (e.g., VisAidMath) is that current models frequently exhibit 'hallucination regarding the implicit visual reasoning process,' struggling to accurately interpret and integrate visual information, even when provided with explicit visual aids.
- •VAMPS serves as both a benchmarking tool and a diagnostic framework, designed to not only measure performance but also to identify and analyze the specific reasons why models fail to effectively utilize external tool outputs for visual reasoning.
📊 競品分析▸ Show
| Benchmark | Focus / Key Feature | Dataset Size | Problem Source/Level | Multilinguality | Key Findings (if applicable) |
|---|---|---|---|---|---|
| VAMPS | Tool-enabled visual reasoning (constructing & grounding plots) | 1,168 QA pairs | Iranian University Entrance Exam (Algebra & Calculus) | Bilingual (Persian/English) | Direct analytical solving outperforms tool-enabled visual solving. |
| MathVista | Comprehensive math reasoning in diverse visual contexts | 6,141 examples | 28 existing + 3 new datasets (IQTest, FunctionQA, PaperQA) | Not specified | Systematically studies math reasoning in visual contexts. |
| U-MATH | University-level mathematical thinking, includes a meta-benchmark (μ-MATH) for judging solutions | 1,100 problems (20% visual) | Curriculum from top US universities | Not specified | Challenging for current LLMs; Gemini 2.0 Flash Thinking achieved ~73.6% accuracy. |
| VisAidMath | Evaluating visual-aided mathematical reasoning, explicit and implicit visual contexts | 1,200 problems | Textbooks, exams, Olympiads | Not specified | GPT-4V achieved 45.33% accuracy, highlighting hallucination in visual reasoning. |
| VC-Bench | Explicit visual dependency in multimodal mathematical reasoning, multi-image tasks | 1,720 problems (6,697 images) | Six cognitive domains | Not specified | Top models unable to exceed 50% accuracy, highlighting challenges in visual-mathematical integration. |
| MATHNET | Olympiad-level math reasoning and retrieval | 30,676 problems | 47 countries, 2 decades of competitions | Multilingual (17 languages) | SOTA models (GPT-5, Gemini 2.5 Pro) challenged (72%, 66% accuracy respectively). |
🛠️ 技術深入
- Dataset Composition: VAMPS comprises 1,168 multimodal, bilingual (Persian and English) multiple-choice question-answer pairs.
- Problem Sourcing: The core of the benchmark consists of 218 real problems from Iranian University Entrance Exams, each provided in Persian and with a manually checked English translation, resulting in 436 original question instances.
- Data Augmentation: This core is extended with synthetic multimodal variants, generated with LLM assistance from the real question seeds and subsequently reviewed by humans to ensure quality and relevance.
- Problem Characteristics: Problems are specifically chosen where plotting offers a natural solution strategy, involving concepts like intersections, extrema, and asymptotes, to effectively test visual reasoning.
- Evaluation Focus: The benchmark is designed to assess whether models can successfully construct a useful graph from problem descriptions and then accurately ground their answers in the resulting visualization.
- Diagnostic Framework: VAMPS includes a diagnostic framework aimed at evaluating how well models process and integrate outputs from external visualization tools.
🔮 前景展望基於引用來源的 AI 分析
Future LLMs will require more robust visual tool integration capabilities.
Current models struggle to effectively use and reason over tool-generated plots, indicating a critical need for architectural and training improvements in this specific area.
New evaluation methodologies will emerge to diagnose specific failure modes in multimodal reasoning.
Benchmarks like VAMPS are designed for diagnosis, moving beyond simple accuracy to understand why models fail, which will drive more targeted research and development.
Multimodal LLMs will need to overcome 'hallucination regarding implicit visual reasoning.'
The identified deficiency in processing visual information implicitly suggests that models are not truly understanding the visual context, leading to errors and necessitating advancements in visual comprehension.
⏳ 時間線
2021
Introduction of foundational text-only math benchmarks like GSM8K, setting early baselines for LLM mathematical reasoning.
2024-10
VisAidMath benchmark introduced, specifically addressing the insufficient analysis of how LLMs process visual information during mathematical problem-solving.
2025-04
VC-Bench introduced, focusing on evaluating multimodal mathematical reasoning with explicit visual dependencies and multi-image tasks.
2025-07
A comprehensive survey on mathematical reasoning in the era of multimodal LLMs is published, reviewing over 200 studies and categorizing benchmarks, methodologies, and challenges.
2026-06
VAMPS benchmark introduced, specifically designed to test LLMs' ability to construct and ground answers in tool-generated visual plots for complex algebra and calculus problems.
📎 來源 (13)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： ArXiv AI ↗
每週電子報
每週一封，可隨時退訂。