Deterministic Analytics Beat Runtime Agents
π‘See why pre-approved programs achieved 110/110 complete analytics contracts while runtime planners achieved 0/330.
β‘ 30-Second TL;DR
What Changed
The architecture separates natural-language intent interpretation from analytical program selection and execution.
Why It Matters
The work suggests that enterprise analytics may gain reliability and auditability by constraining LLMs to interpretation rather than execution planning. However, the result is configuration-specific and does not establish that runtime agents cannot succeed with different designs.
What To Do Next
Prototype a governed analytics pipeline that maps LLM intent into a versioned allowlist of SQL and analysis programs, then replay it against a fixed evaluation set.
Key Points
- β’The architecture separates natural-language intent interpretation from analytical program selection and execution.
- β’The supported analytical class includes relational operations, aggregation, comparison, window functions, ranking, and similarity.
- β’Fixed semantics, policies, data, and execution rules make analytical results replayable.
- β’Three 8B models generated SQL and selected tools at runtime, but none of 330 planning episodes met the full contract.
- β’Qwen3-8B interpreted intent while policy executed approved programs, achieving 110 of 110 successful runs.
Weekly AI Recap
Read this week's curated digest of top AI events β
πRelated Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI β
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.