⚡雷峰网•Stalecollected in 2h
CodePercept: Using Code to Enhance STEM Visual Reasoning
💡Discover why 'understanding code' is the key to solving complex STEM visual reasoning tasks in multimodal models.
⚡ 30-Second TL;DR
What Changed
Identified visual perception, rather than reasoning, as the primary bottleneck for STEM-related multimodal model performance.
Why It Matters
This research shifts the focus of multimodal model development from purely optimizing reasoning chains to improving the precision of visual perception, potentially setting a new standard for STEM-focused AI models.
What To Do Next
Explore the CodePercept GitHub repository to integrate code-based verification into your own multimodal vision-language model training pipelines.
Who should care:Researchers & Academics
Key Points
- •Identified visual perception, rather than reasoning, as the primary bottleneck for STEM-related multimodal model performance.
- •Introduced 'Code-Grounded Captioning' and 'Image-to-Code Translation' to replace ambiguous natural language with precise, verifiable code.
- •Constructed the ICC-1M dataset, containing 1 million high-quality image-caption-code triplets for training.
- •Implemented a two-stage progressive training strategy to enhance both descriptive and generative visual capabilities.
🧠 Deep Insight
Web-grounded analysis with 7 cited sources.
🔑 Enhanced Key Takeaways
- •A systematic scaling analysis conducted by the CodePercept team revealed that scaling visual perception consistently yields greater performance gains than scaling reasoning, confirming perception as the primary bottleneck in STEM visual reasoning for Multimodal Large Language Models (MLLMs).
- •The ICC-1M dataset, comprising over 1 million high-quality STEM Image-Caption-Code triplets, was constructed using three distinct pipelines: Image Reproduction (decomposing existing images into code), Image Diversity (procedurally generating variations from scientific principles), and Solid Geometry Synthesis (programmatically creating 3D scenes using parametric templates to overcome MLLM limitations in 3D spatial thinking).
- •CodePercept introduced STEM2Code-Eval, a novel, manually annotated benchmark of 1,000 images designed to directly evaluate visual perception in STEM domains by requiring models to generate executable Python code for image reconstruction, offering a deterministic and verifiable assessment beyond traditional problem-solving accuracy proxies.
- •The model employs a two-stage progressive training strategy: Supervised Finetuning (CodePercept-S1) for joint optimization of image captioning and image-to-code translation, followed by Reinforcement Learning (CodePercept-R1) using Group Relative Policy Optimization (GRPO) with a reward mechanism that enforces valid Python syntax, execution success, code-level semantic equivalence (evaluated by GPT-4o), and image-level visual similarity.
- •CodePercept-8B-S1 demonstrated state-of-the-art performance on STEM reasoning benchmarks, outperforming significantly larger models such as Qwen2.5-VL-72B and approaching the capabilities of frontier models when integrated with a powerful Large Language Model (LLM) solver.
🛠️ Technical Deep Dive
- Base Architecture: CodePercept is built upon the Qwen3-VL architecture.
- Code as Perceptual Medium: The system utilizes executable Python code to represent and ground visual perception in STEM tasks.
- Dataset (ICC-1M): A large-scale dataset of 1 million Image-Caption-Code triplets.
- Construction Pipelines:
- Image Reproduction: Decomposing existing images into code using high-end MLLMs.
- Image Diversity: Extracting scientific principles and procedurally generating new variations.
- Solid Geometry Synthesis: Programmatically creating 3D scenes with parametric templates to ensure accuracy.
- Construction Pipelines:
- Training Tasks:
- Code-Grounded Caption Generation: Treats executable code as ground truth for image captions to eliminate hallucinations.
- STEM Image-to-Code Translation: Prompts models to generate reconstruction code to mitigate natural language ambiguity.
- Training Strategy: Two-stage progressive training.
- Stage 1 (CodePercept-S1): Supervised Finetuning, jointly optimizing image captioning and image-to-code translation using ICC-1M.
- Stage 2 (CodePercept-R1): Reinforcement Learning using Group Relative Policy Optimization (GRPO).
- Reinforcement Learning Reward Mechanism: Enforces:
- Format Reward: Ensures valid Python syntax.
- Content Reward: Combines execution success, code-level semantic equivalence (evaluated via GPT-4o), and image-level visual similarity.
- Evaluation Benchmark (STEM2Code-Eval): A manually annotated benchmark of 1,000 images that evaluates visual perception through deterministic, executable Python code generation for image reconstruction.
- Code Generation Libraries: Primarily uses Matplotlib for code generation.
🔮 Future ImplicationsAI analysis grounded in cited sources
The 'code-as-perception' paradigm will significantly influence the development of future multimodal AI systems beyond STEM.
By demonstrating the superior precision and verifiability of code over natural language for visual grounding, this approach offers a robust method to reduce hallucinations and improve factual accuracy in diverse visual understanding tasks.
CodePercept's methodology could lead to more robust and interpretable AI systems for scientific discovery and engineering.
The use of verifiable, executable code for visual perception provides a clear, structured representation that can be directly tested and debugged, fostering greater trust and utility in scientific applications.
Future iterations of CodePercept could expand beyond Matplotlib to include more complex engineering and design formats.
The current limitation to Matplotlib suggests a clear path for extending the framework to CAD formats or specialized physics engine code, enabling broader applications in AI-driven engineering.
⏳ Timeline
2024-08-30
Qwen2-VL series released by the Qwen team, providing foundational multimodal capabilities.
2024-12-25
QvQ-72B-Preview, an experimental model focusing on enhancing visual reasoning, released by the Qwen team.
2026-02-15
Qwen3.5, a native vision-language model with improved reasoning, coding, and multimodal understanding, released by the Qwen team.
2026-03-11
CodePercept paper, 'Code-Grounded Visual STEM Perception for MLLMs,' submitted to arXiv by Shanghai Jiao Tong University and the Qwen team.
2026-03-11
The STEM2Code-Eval benchmark and ICC-1M dataset are introduced as part of the CodePercept paper.
📎 Sources (7)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: 雷峰网 ↗