๐TestingCatalogโขStalecollected in 25m
TinyFish launches Bigset for automated web dataset generation
๐กAutomate your RAG data pipeline with this new open-source multi-agent tool for live web data extraction.
โก 30-Second TL;DR
What Changed
Utilizes a multi-agent architecture to automate web data extraction.
Why It Matters
Bigset significantly reduces the engineering overhead required for building custom datasets for RAG or fine-tuning. It enables faster iteration cycles for AI applications that rely on real-time web information.
What To Do Next
Clone the Bigset repository and test it by generating a live dataset for your next RAG project to evaluate its extraction accuracy.
Who should care:Developers & AI Engineers
Key Points
- โขUtilizes a multi-agent architecture to automate web data extraction.
- โขConverts natural language prompts directly into structured, live datasets.
- โขFeatures self-refreshing capabilities to keep data current without manual intervention.
๐ง Deep Insight
Web-grounded analysis with 8 cited sources.
๐ Enhanced Key Takeaways
- โขBigset is an open-source project, licensed under AGPL-3.0, with its full codebase available on GitHub, allowing developers to self-host it via Docker and integrate with external LLM services like OpenRouter.
- โขThe system employs a two-tier multi-agent architecture where an orchestrator agent performs breadth-first discovery and dispatches parallel sub-agents, each capped at 6 tool calls, to research and extract data for individual entities.
- โขBigset integrates with large language models, specifically using Claude Sonnet for schema inference and Qwen (qwen/qwen3.7-max) for the orchestrator agent, with these LLM calls routed through OpenRouter.
- โขTinyFish, the company behind Bigset, was founded in 2024 and secured $47 million in Series A funding in August 2025, led by ICONIQ Capital, with a vision to build AI agents that browse the web like humans at scale.
- โขBeyond Bigset, TinyFish offers an enterprise platform for AI web agents that automates complex workflows across authenticated systems, dynamic applications, and sites that block traditional scrapers, achieving 95% task success and 99.99% uptime in production.
๐ ๏ธ Technical Deep Dive
- Architecture: Two-tier multi-agent system comprising an orchestrator agent and parallel sub-agents.
- Orchestrator Role: Responsible for breadth-first discovery, identifying relevant entities/rows, and dispatching sub-agents.
- Sub-Agent Role: Each sub-agent handles one entity (one row), operates under a budget of 6 tool calls, uses TinyFish Fetch to retrieve real page content, extracts relevant fields, and inserts a verified row with source URLs.
- LLM Integration: Utilizes Claude Sonnet for schema inference and Qwen (qwen/qwen3.7-max) for the orchestrator agent, with all LLM calls routed through OpenRouter.
- Data Handling: Infers schema from natural language prompts, performs primary key deduplication, and produces downloadable datasets in CSV or XLSX formats. SQL query support and an agent-native API are planned for the roadmap.
- Security and Reliability: Sub-agents are instructed to avoid fabricating values, leave fields blank if unconfirmed, and automatically reject duplicate primary keys. The dataset ID is secured within a JavaScript closure, invisible to the LLM, to prevent prompt injection redirects.
- Self-Hosting: Bigset is licensed under AGPL-3.0 and designed for self-hosting via Docker. It requires an OpenRouter API key for LLM access and Clerk for user authentication.
- Core Technology (TinyFish platform): Employs semantic element finding instead of traditional CSS selectors, enhancing resilience against website structural changes.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Bigset will significantly lower the barrier to entry for data-driven applications.
By automating complex web data extraction from natural language prompts, it allows developers and non-technical users to quickly acquire structured datasets without needing specialized scraping knowledge or extensive infrastructure.
The open-source nature of Bigset will foster a community-driven ecosystem for web data collection.
Its AGPL-3.0 license and GitHub availability encourage contributions, extensions, and wider adoption, potentially leading to a richer set of tools and use cases built on its foundation.
Bigset's multi-agent architecture will drive innovation in resilient and adaptive web automation.
The system's ability to infer schemas, dispatch specialized agents, and handle dynamic web content through LLM-powered reasoning offers a more robust approach than traditional brittle scraping methods, pushing the boundaries of what automated web interaction can achieve.
โณ Timeline
2024
TinyFish founded by Sudheesh Nair, Shuhao Zhang, and Keith Zhai
2025-08
TinyFish announced $47 million in Series A funding led by ICONIQ Capital
2026-06-02
TinyFish launches Bigset, an open-source multi-agent system for automated web dataset generation
๐ Sources (8)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: TestingCatalog โ