Proxy State Eval Scales LLM Agent Benchmarks

• Scenario Schema: Each scenario specifies user goal, user/system facts, expected final state, and expected agent behavior, enabling structured evaluation without deterministic databases • LLM State Tracker Component: Infers structured proxy state from full interaction trace, preserving final state-based evaluation semantics • LLM Judge Verification: Verifies goal completion and detects tool/user hallucinations against scenario constraints with >90% agreement with human judges • Ablation Study Results: Confirms robustness of proxy state tracker and sensitivity to scenario completeness; user persona variability captured while maintaining low user-induced error • Model-Differentiating Rankings: Produces stable, interpretable metrics across model families and reasoning-effort settings (SFT, RFT training approaches) • Complementary Evaluation Patterns: Integrates with trajectory-aware protocols for multi-dimensional analysis and state-diff methodologies for outcome verification

Proxy State-Based Evaluation addresses a critical scalability bottleneck in LLM agent benchmarking by decoupling evaluation from deterministic backend maintenance. This framework enables rapid iteration on agent benchmarks without proportional infrastructure costs, likely accelerating the pace of agent capability assessment across industry. The >90% human-LLM judge agreement validates automated evaluation at scale, reducing evaluation costs while maintaining reliability. The transferability of training data to unseen scenarios suggests this approach could become a standard pattern for industrial LLM agent development, particularly as multi-agent systems and tool-calling complexity increase. Integration with complementary approaches like robustness testing under noise and privacy leakage detection indicates a maturing ecosystem of specialized benchmarking frameworks tailored to different agent deployment contexts.