SourceTestingCatalog•Stalecollected in 25m
TinyFish launches Bigset for automated web dataset generation
💡Automate your RAG data pipeline with this new open-source multi-agent tool for live web data extraction.
⚡ 30-Second TL;DR
What Changed
Utilizes a multi-agent architecture to automate web data extraction.
Why It Matters
Bigset significantly reduces the engineering overhead required for building custom datasets for RAG or fine-tuning. It enables faster iteration cycles for AI applications that rely on real-time web information.
What To Do Next
Clone the Bigset repository and test it by generating a live dataset for your next RAG project to evaluate its extraction accuracy.
Who should care:Developers & AI Engineers
Key Points
- •Utilizes a multi-agent architecture to automate web data extraction.
- •Converts natural language prompts directly into structured, live datasets.
- •Features self-refreshing capabilities to keep data current without manual intervention.
🧠 Deep Insight
Background and context from public sources — not the original article. 8 sources cited.
🔑 Enhanced Key Takeaways
- •Bigset is an open-source project, licensed under AGPL-3.0, with its full codebase available on GitHub, allowing developers to self-host it via Docker and integrate with external LLM services like OpenRouter.
- •The system employs a two-tier multi-agent architecture where an orchestrator agent performs breadth-first discovery and dispatches parallel sub-agents, each capped at 6 tool calls, to research and extract data for individual entities.
- •Bigset integrates with large language models, specifically using Claude Sonnet for schema inference and Qwen (qwen/qwen3.7-max) for the orchestrator agent, with these LLM calls routed through OpenRouter.
- •TinyFish, the company behind Bigset, was founded in 2024 and secured $47 million in Series A funding in August 2025, led by ICONIQ Capital, with a vision to build AI agents that browse the web like humans at scale.
- •Beyond Bigset, TinyFish offers an enterprise platform for AI web agents that automates complex workflows across authenticated systems, dynamic applications, and sites that block traditional scrapers, achieving 95% task success and 99.99% uptime in production.
🛠️ Technical Deep Dive
- Architecture: Two-tier multi-agent system comprising an orchestrator agent and parallel sub-agents.
- Orchestrator Role: Responsible for breadth-first discovery, identifying relevant entities/rows, and dispatching sub-agents.
- Sub-Agent Role: Each sub-agent handles one entity (one row), operates under a budget of 6 tool calls, uses TinyFish Fetch to retrieve real page content, extracts relevant fields, and inserts a verified row with source URLs.
- LLM Integration: Utilizes Claude Sonnet for schema inference and Qwen (qwen/qwen3.7-max) for the orchestrator agent, with all LLM calls routed through OpenRouter.
- Data Handling: Infers schema from natural language prompts, performs primary key deduplication, and produces downloadable datasets in CSV or XLSX formats. SQL query support and an agent-native API are planned for the roadmap.
- Security and Reliability: Sub-agents are instructed to avoid fabricating values, leave fields blank if unconfirmed, and automatically reject duplicate primary keys. The dataset ID is secured within a JavaScript closure, invisible to the LLM, to prevent prompt injection redirects.
- Self-Hosting: Bigset is licensed under AGPL-3.0 and designed for self-hosting via Docker. It requires an OpenRouter API key for LLM access and Clerk for user authentication.
- Core Technology (TinyFish platform): Employs semantic element finding instead of traditional CSS selectors, enhancing resilience against website structural changes.
🔮 Future ImplicationsAI analysis grounded in cited sources
Bigset will significantly lower the barrier to entry for data-driven applications.
By automating complex web data extraction from natural language prompts, it allows developers and non-technical users to quickly acquire structured datasets without needing specialized scraping knowledge or extensive infrastructure.
The open-source nature of Bigset will foster a community-driven ecosystem for web data collection.
Its AGPL-3.0 license and GitHub availability encourage contributions, extensions, and wider adoption, potentially leading to a richer set of tools and use cases built on its foundation.
Bigset's multi-agent architecture will drive innovation in resilient and adaptive web automation.
The system's ability to infer schemas, dispatch specialized agents, and handle dynamic web content through LLM-powered reasoning offers a more robust approach than traditional brittle scraping methods, pushing the boundaries of what automated web interaction can achieve.
⏳ Timeline
2024
TinyFish founded by Sudheesh Nair, Shuhao Zhang, and Keith Zhai
2025-08
TinyFish announced $47 million in Series A funding led by ICONIQ Capital
2026-06-02
TinyFish launches Bigset, an open-source multi-agent system for automated web dataset generation
📎 Sources (8)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: TestingCatalog ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.