⚡雷峰网•較早收集於 2h
CodePercept：利用代碼提升 STEM 視覺推理能力
💡了解為什麼「看得懂代碼」是解決多模態模型中複雜 STEM 視覺推理任務的關鍵。
⚡ 30-Second TL;DR
有什麼變化
指出視覺感知而非推理能力，是目前 STEM 多模態模型性能的主要瓶頸。
為什麼重要
這項研究將多模態模型開發的重點從單純優化推理鏈轉向提升視覺感知的精確度，可能為 STEM 領域的 AI 模型樹立新的標準。
下一步行動
探索 CodePercept 的 GitHub 儲存庫，將基於代碼的驗證機制整合到您自己的多模態視覺語言模型訓練流程中。
誰應關注：Researchers & Academics
關鍵要點
- •指出視覺感知而非推理能力，是目前 STEM 多模態模型性能的主要瓶頸。
- •引入「代碼驅動描述」與「圖像轉代碼」技術，以精確且可驗證的代碼取代模糊的自然語言。
- •構建了包含 100 萬個高品質圖像-描述-代碼三元組的 ICC-1M 數據集。
- •實施了兩階段漸進式訓練策略，以提升模型的描述與生成式視覺感知能力。
🧠 深度解析
Web-grounded analysis with 7 cited sources.
🔑 增強重點摘要
- •A systematic scaling analysis conducted by the CodePercept team revealed that scaling visual perception consistently yields greater performance gains than scaling reasoning, confirming perception as the primary bottleneck in STEM visual reasoning for Multimodal Large Language Models (MLLMs).
- •The ICC-1M dataset, comprising over 1 million high-quality STEM Image-Caption-Code triplets, was constructed using three distinct pipelines: Image Reproduction (decomposing existing images into code), Image Diversity (procedurally generating variations from scientific principles), and Solid Geometry Synthesis (programmatically creating 3D scenes using parametric templates to overcome MLLM limitations in 3D spatial thinking).
- •CodePercept introduced STEM2Code-Eval, a novel, manually annotated benchmark of 1,000 images designed to directly evaluate visual perception in STEM domains by requiring models to generate executable Python code for image reconstruction, offering a deterministic and verifiable assessment beyond traditional problem-solving accuracy proxies.
- •The model employs a two-stage progressive training strategy: Supervised Finetuning (CodePercept-S1) for joint optimization of image captioning and image-to-code translation, followed by Reinforcement Learning (CodePercept-R1) using Group Relative Policy Optimization (GRPO) with a reward mechanism that enforces valid Python syntax, execution success, code-level semantic equivalence (evaluated by GPT-4o), and image-level visual similarity.
- •CodePercept-8B-S1 demonstrated state-of-the-art performance on STEM reasoning benchmarks, outperforming significantly larger models such as Qwen2.5-VL-72B and approaching the capabilities of frontier models when integrated with a powerful Large Language Model (LLM) solver.
🛠️ 技術深入
- Base Architecture: CodePercept is built upon the Qwen3-VL architecture.
- Code as Perceptual Medium: The system utilizes executable Python code to represent and ground visual perception in STEM tasks.
- Dataset (ICC-1M): A large-scale dataset of 1 million Image-Caption-Code triplets.
- Construction Pipelines:
- Image Reproduction: Decomposing existing images into code using high-end MLLMs.
- Image Diversity: Extracting scientific principles and procedurally generating new variations.
- Solid Geometry Synthesis: Programmatically creating 3D scenes with parametric templates to ensure accuracy.
- Construction Pipelines:
- Training Tasks:
- Code-Grounded Caption Generation: Treats executable code as ground truth for image captions to eliminate hallucinations.
- STEM Image-to-Code Translation: Prompts models to generate reconstruction code to mitigate natural language ambiguity.
- Training Strategy: Two-stage progressive training.
- Stage 1 (CodePercept-S1): Supervised Finetuning, jointly optimizing image captioning and image-to-code translation using ICC-1M.
- Stage 2 (CodePercept-R1): Reinforcement Learning using Group Relative Policy Optimization (GRPO).
- Reinforcement Learning Reward Mechanism: Enforces:
- Format Reward: Ensures valid Python syntax.
- Content Reward: Combines execution success, code-level semantic equivalence (evaluated via GPT-4o), and image-level visual similarity.
- Evaluation Benchmark (STEM2Code-Eval): A manually annotated benchmark of 1,000 images that evaluates visual perception through deterministic, executable Python code generation for image reconstruction.
- Code Generation Libraries: Primarily uses Matplotlib for code generation.
🔮 前景展望AI analysis grounded in cited sources
The 'code-as-perception' paradigm will significantly influence the development of future multimodal AI systems beyond STEM.
By demonstrating the superior precision and verifiability of code over natural language for visual grounding, this approach offers a robust method to reduce hallucinations and improve factual accuracy in diverse visual understanding tasks.
CodePercept's methodology could lead to more robust and interpretable AI systems for scientific discovery and engineering.
The use of verifiable, executable code for visual perception provides a clear, structured representation that can be directly tested and debugged, fostering greater trust and utility in scientific applications.
Future iterations of CodePercept could expand beyond Matplotlib to include more complex engineering and design formats.
The current limitation to Matplotlib suggests a clear path for extending the framework to CAD formats or specialized physics engine code, enabling broader applications in AI-driven engineering.
⏳ 時間線
2024-08-30
Qwen2-VL series released by the Qwen team, providing foundational multimodal capabilities.
2024-12-25
QvQ-72B-Preview, an experimental model focusing on enhancing visual reasoning, released by the Qwen team.
2026-02-15
Qwen3.5, a native vision-language model with improved reasoning, coding, and multimodal understanding, released by the Qwen team.
2026-03-11
CodePercept paper, 'Code-Grounded Visual STEM Perception for MLLMs,' submitted to arXiv by Shanghai Jiao Tong University and the Qwen team.
2026-03-11
The STEM2Code-Eval benchmark and ICC-1M dataset are introduced as part of the CodePercept paper.
📎 來源 (7)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： 雷峰网 ↗