SetupAI
繁
SourceStalecollected in 17h

ClawForge: Benchmarking Command-Line Agents Under State Conflict

Read original on ArXiv AI
#agentic-workflow#benchmarking

Discover why top AI models fail at command-line tasks when faced with persistent state conflicts.

30-Second TL;DR

What Changed

Introduces ClawForge-Bench with 17 scenarios across 6 ability categories.

Why It Matters

This research highlights a critical gap in current agentic workflows: the inability to manage persistent environments. It suggests that future agent development must prioritize state awareness to improve reliability in real-world software engineering tasks.

What To Do Next

Incorporate state-inspection steps into your agent's tool-use loop to verify the environment before executing destructive commands.

Who should care:Researchers & Academics

Key Points

  • •Introduces ClawForge-Bench with 17 scenarios across 6 ability categories.
  • •Evaluates agents based on persistent workflow surfaces rather than static prompt completion.
  • •Identifies that agent performance is heavily dependent on the ability to inspect existing state.
  • •Reveals that top frontier models achieve only 45.3% strict accuracy in complex command-line tasks.
Key numbers17%45.3%

Deep Insight

Background and context from public sources — not the original article. 13 sources cited.

Enhanced Key Takeaways

  • •ClawForge is a generator-backed benchmark system that automatically constructs tasks from scenario templates, grounded slots, initialized state, reference trajectories, and validators, ensuring reproducibility and scalability.
  • •The framework employs a 'result-first evaluation' protocol, scoring agents based on normalized end states and observable side effects rather than requiring exact command sequence matching, which allows for diverse valid solution paths.
  • •ClawForge-Bench includes 17 scenarios categorized into six core abilities: Duplicate Avoidance, Gap Completion, Information Transfer, Multi-Source Reasoning, State Repair, and Workflow Completion.
  • •The evaluation revealed that 'wrong-state replacement' tasks achieved below 17% strict accuracy across all seven frontier models tested, indicating a significant and widespread failure mode for current AI agents.
  • •Analysis of model performance showed that while the best model (Claude Opus 4.6) achieved 45.3% strict accuracy, another model (GPT-5.3-Codex) had the highest partial credit, suggesting qualitatively different failure styles among frontier models.

Competitor Analysis

ClawForge
Focus
Command-line agents under state conflict
Key Features
Generator-backed task creation, persistent workflow surfaces, result-first evaluation, 17 scenarios across 6 categories.
Performance Metrics
Strict accuracy, partial credit, step efficiency, wrong-state replacement.
Best Model Accuracy (if applicable)
45.3% (Claude Opus 4.6) strict accuracy
Terminal-Bench 2.0
Focus
Command-line interfaces
Key Features
89 hard, realistic tasks inspired by real workflows, sandboxed environment, human-written solutions, comprehensive tests.
Performance Metrics
Final outcome verification.
Best Model Accuracy (if applicable)
Less than 65% (frontier models)
ClawWork
Focus
Economic performance of AI agents
Key Features
Agents start with $10, complete 220 professional tasks across 44 job categories in a simulated economy, can spend money on tools.
Performance Metrics
Money left at the end of 8 hours.
Best Model Accuracy (if applicable)
$19,915 (Claude Opus 4)
ClawBench
Focus
AI browser agents on real-world online tasks
Key Features
153 tasks across 144 live websites, 5 layers of behavioral data, request interceptor for safe evaluation, human ground-truth.
Performance Metrics
Success rate.
Best Model Accuracy (if applicable)
33.3% (Claude Sonnet 4.6)
WildClawBench
Focus
Real-world, long-horizon CLI agent evaluation
Key Features
60 human-authored, bilingual, multimodal tasks, native-runtime in Docker containers, real tools, hybrid grading (rule-based, environment-state, LLM/VLM judge).
Performance Metrics
Overall accuracy.
Best Model Accuracy (if applicable)
62.2% (Claude Opus 4.7 under OpenClaw)

Technical Deep Dive

  • ClawForge is a generator-backed benchmark system that compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications.
  • It evaluates agents step-by-step over persistent workflow surfaces using a CLI-style interface.
  • Correctness is determined by normalized end states and observable side effects, rather than requiring an exact sequence of actions, allowing for multiple valid solution paths.
  • The framework supports various state modes (mock or real) and prompt policies for task generation.
  • ClawForge-Bench's 17 scenarios are organized into six ability categories: Duplicate Avoidance, Gap Completion, Information Transfer, Multi-Source Reasoning, State Repair, and Workflow Completion.
  • Evaluation includes partial-credit and step-efficiency analyses to provide deeper insights into agent failure modes and styles.

Future ImplicationsAI analysis grounded in cited sources

Future AI agents will require more sophisticated state inspection and management capabilities.
ClawForge highlights that current frontier models struggle significantly with inspecting existing state before acting, leading to low accuracy in state-conflict scenarios, necessitating advancements in this area.
Benchmarking frameworks will increasingly focus on persistent, real-world state and complex workflows rather than static or clean-state evaluations.
ClawForge, along with other recent benchmarks, emphasizes the critical need for evaluating agents in dynamic, stateful, and realistic environments to accurately assess their practical capabilities.
The development of AI agents will shift towards architectures that explicitly handle state consistency, conflict resolution, and robust memory management.
The challenges identified by ClawForge and broader research on AI agents underscore the necessity of robust state persistence strategies and advanced memory management for reliable and trustworthy agent performance.

Timeline

2026-01-17
Terminal-Bench 2.0 paper released, benchmarking agents on hard, realistic CLI tasks.
2026-02-04
Research highlights persistent challenges in AI agent development, including retrieval, embeddings, and agent memory.
2026-03-17
Discussion of seven state persistence strategies for long-running AI agents, addressing focus degradation and memory bloat.
2026-03-25
ClawWork benchmark released, evaluating AI agents on economic performance in professional tasks within a simulated economy.
2026-04-14
ClawBench introduced, evaluating AI browser agents on real-world tasks across live websites.
2026-05-13
ClawForge paper released, introducing a generator-backed benchmark for command-line agents under state conflict.
Read original article on ArXiv AI

Weekly AI Recap

Read this week's curated digest of top AI events →

PostLinkedIn

AI-curated news aggregator. All content rights belong to original publishers.
Original source: ArXiv AI ↗

This is a summary, not the original. Read the source, or get the weekly briefing.

The weekly digest

One email a week. Unsubscribe anytime.