Hugging Face Tops DABStep with Data Scientist Agent

🔑 Enhanced Key Takeaways

• •Jupyter Agent is an open-source project hosted on GitHub that integrates directly into Jupyter notebooks, reading context, executing Python code with libraries like pandas and numpy, and generating step-by-step reasoning traces.[1]
• •The agent was trained using a custom dataset derived from the 2TB Meta Kaggle Notebooks, processed through deduplication, dataset fetching, quality scoring, and generation of ~2B tokens of reasoning and execution traces.[1]
• •Fine-tuning a 4B Qwen3-4B-Instruct model on this dataset boosted DABStep performance from 38.7% (base) to 75%, establishing SOTA for small models on realistic data science tasks.[1]

🛠️ Technical Deep Dive

• •Dataset pipeline: Processes Meta Kaggle Notebooks by deduplicating ~90% of content, fetching linked datasets, scoring for educational quality, filtering irrelevancies, and generating QA pairs with reasoning traces (~2B tokens total).[1]
• •Training results on DABStep easy split: Base Qwen3-4B-Instruct at 38.7%, with scaffolding at 52.8%, and post-fine-tuning at 75%.[1]
• •Designed for Jupyter environment: Reads notebook and dataset context, executes Python (pandas, numpy, matplotlib), produces intermediate computation traces; compared to Cursor but native for data analysis.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline