ESQ-Bench Exposes Enterprise NL2SQL’s Silent Failures

Post LinkedIn

📄Read original on ArXiv AI

#nl2sql#benchmarking#silent-divergence#database-agentsesq-benchesq-benchoraclegpt-4oclaude sonnet 4.6llama 3.2

💡Enterprise NL2SQL can pass execution while returning wrong results—ESQ-Bench measures the hidden risk.

⚡ 30-Second TL;DR

What Changed

The benchmark contains six populated schemas spanning 465 tables and 164,682 rows, with identical seed data across Oracle, PostgreSQL, MySQL, and SQL Server.

Why It Matters

The benchmark challenges the common practice of evaluating NL2SQL primarily through execution accuracy, especially in enterprise settings where wrong-but-executable queries can silently affect decisions. AI teams building database agents should measure semantic correctness and dialect robustness separately rather than relying on standard academic benchmarks alone.

What To Do Next

Run your NL2SQL pipeline against ESQ-Bench across all three tiers and track silent divergence alongside execution accuracy before deploying it on enterprise databases.

Who should care:Enterprise & Security Teams

Key Points

•The benchmark contains six populated schemas spanning 465 tables and 164,682 rows, with identical seed data across Oracle, PostgreSQL, MySQL, and SQL Server.
•Its 550 gold-validated question-query pairs are divided across Tier 1, Tier 2, and Tier 3 complexity levels, with EM, EX, SR, and SD evaluation metrics.
•GPT-4o schema-linked execution match fell to 79.8%, 60.3%, and 57.2% across the three tiers.
•Operational silent divergence affected 73% to 99% of queries that passed execution, showing that executable SQL can still return semantically wrong results.
•Claude Sonnet 4.6 achieved 87.4%, 74.9%, and 68.7% EX, while local Llama 3.2 reached only 73 of 550 bank-wide.

🧠 Deep Insight

Background and context from public sources — not the original article. 1 sources cited.

🔑 Enhanced Key Takeaways

•ESQ-Bench introduces a specific focus on 'dialect generalization,' measuring how effectively models translate natural language across different SQL flavors (Oracle, PostgreSQL, MySQL, SQL Server) using a unified schema.
•The benchmark utilizes a proprietary 'Silent Semantic Divergence' (SSD) metric to quantify the gap between executable SQL and business-logic accuracy, a metric absent in standard benchmarks like Spider or BIRD.
•The dataset architecture is specifically designed to simulate 'Enterprise Oracle' environments, which are characterized by high-density schemas and complex relational constraints that often cause hallucinations in general-purpose LLMs.
•The research highlights that current industry-standard benchmarks rely too heavily on Execution Match (EX), which ESQ-Bench proves is an insufficient proxy for reliability in production-grade database systems.
•The benchmark includes a 'schema-linked' evaluation protocol, forcing models to navigate 465 tables, which tests the model's ability to perform accurate entity resolution in large-scale enterprise data warehouses.

📊 Competitor Analysis▸ Show

Feature	ESQ-Bench	Spider 2.0	BIRD-SQL
Primary Focus	Enterprise Oracle/Dialect Generalization	Academic/General SQL	Large-scale/Efficiency
Schema Size	465 Tables (Enterprise)	Moderate	Large (Public Data)
Silent Failure Detection	Yes (SSD Metric)	No	Limited
Dialect Support	Multi-dialect (4)	Single (SQLite)	Multi-dialect

🛠️ Technical Deep Dive

The benchmark employs a multi-tier complexity framework (Tiers 1-3) that scales from simple CRUD operations to complex multi-join analytical queries involving window functions and common table expressions (CTEs).
Evaluation metrics include EM (Exact Match), EX (Execution Match), SR (Schema Recall), and the novel SD (Semantic Divergence) score.
The dataset is normalized across four major SQL dialects to ensure that the ground truth remains consistent regardless of the underlying database engine's syntax requirements.
The benchmark uses a gold-validated query set consisting of 550 pairs, specifically curated to trigger common edge cases in SQL dialect translation.

🔮 Future ImplicationsAI analysis grounded in cited sources

Enterprise adoption of NL2SQL will shift from EX-based metrics to semantic-divergence-aware validation.

The high rate of silent failures (up to 99%) makes current execution-based testing insufficient for mission-critical business intelligence.

Model fine-tuning for SQL will increasingly prioritize dialect-specific constraints over general-purpose reasoning.

The performance gap between Claude Sonnet 4.6 and Llama 3.2 suggests that enterprise-grade SQL accuracy requires specialized training on complex, multi-table schema architectures.