來源NVIDIA Developer Blog•較早收集於 18m
NVIDIA 在全新 Agentic AI 編碼基準測試中取得領先
💡首個 AI Agent 編碼效能標準化基準測試；看看 NVIDIA 在真實任務中的表現如何。
⚡ 30 秒速覽
有什麼變化
AA-AgentPerf 是首個針對 AI Agent 編碼軌跡的多供應商開放基準測試。
為什麼重要
此基準測試為評估 Agentic 系統設定了新標準，超越了簡單的靜態模型測試。這可能會迫使其他 AI 提供商優化其 Agentic 工作流程，以在標準化效能排名中保持競爭力。
下一步行動
檢視 AA-AgentPerf 的評測方法，以了解您的 Agentic 工作流程與業界標準編碼基準測試的比較結果。
誰應關注：Researchers & Academics
關鍵要點
- •AA-AgentPerf 是首個針對 AI Agent 編碼軌跡的多供應商開放基準測試。
- •NVIDIA 在此新標準下展示了處理複雜推理工作負載的領先效能。
- •該基準測試解決了業界對於 Agentic AI 效能評估標準化的需求。
🧠 深度解析
背景與延伸：來自公開資料，非原文內容。引用 17 個來源。
🔑 增強重點摘要
- •AA-AgentPerf is a hardware benchmark designed to measure how many active users an inference deployment can support under realistic agentic workloads, focusing on user experience performance targets like time to first token and output speed.
- •The benchmark utilizes real agentic trajectories derived from public code repositories, encompassing multi-turn coding sessions with interleaved reasoning, tool calls, and variable context lengths across more than 12 programming languages.
- •Performance evaluation in AA-AgentPerf is based on market-derived Service-Level Objectives (SLOs), reflecting quality-of-service levels observed across providers, and uses P25 output speed due to the prevalence of small output sequence length requests in agentic workloads.
- •NVIDIA's Blackwell architecture has demonstrated a significant efficiency improvement on the AgentPerf benchmark, capable of running 20 times more AI agents per megawatt compared to the previous Hopper generation.
- •The benchmark measures the performance of the full agent stack, including both the underlying model and the 'harness' (the software layer for orchestration, context, memory, tool use, and security), indicating that the harness significantly influences cost and performance.
📊 競品分析▸ Show
Competitor Analysis: AI Agent Coding Benchmarks and Platforms
| Feature/Category | NVIDIA (AA-AgentPerf Context) | Other Leading AI Coding Agents/Benchmarks |
|---|---|---|
| Benchmark Focus | Hardware efficiency, concurrent users, real-world agentic coding trajectories, full stack (model + harness) performance. | SWE-bench Verified: Real GitHub issues, widely used. Terminal-Bench v2: Agentic terminal use. SWE-Atlas-QnA: Technical Q&A requiring codebase exploration. τ-bench: General-purpose service tasks, multi-turn. WebArena: Web tasks (e-commerce, forums, code development). GAIA: General AI assistants, reasoning, multimodality, tool-use. |
| Key Offerings | NVIDIA Agent Toolkit (Nemotron models, OpenShell runtime, AI-Q blueprint), NeMo for agent lifecycle, NIM for deployment, Blueprints for reference workflows, Blackwell architecture for inference. | Claude Code: Strong 'coding brain,' direct access to Claude models. Cursor: AI coding IDE, multi-file changes, parallel agents. GitHub Copilot: Widely adopted, integrates with GitHub ecosystem, Agent Mode. Devin: First AI software engineer. Augment Code: Enterprise-focused, multi-agent orchestration. Amazon Q Developer: AWS-heavy projects, cloud modernization. JetBrains AI Assistant: Integrates across IDEs. Tabnine: Fast completions, local/on-prem options. Grok Build: Agentic engineering, large context. |
| Pricing Model | Nemotron 3 Ultra currently free in Kilo. NVIDIA's focus is on hardware and enterprise software platforms. | Claude Code: Starts at $20/month, heavy usage $150-$200/month. Cursor: Credit-based pricing, potential for overages. GitHub Copilot: Free tier, $10/month Pro, Enterprise $21/user/month additional. Qodo: Enterprise pricing starting at $50K/year for on-prem/private cloud. |
| Benchmark Scores (SWE-bench Verified) | Nemotron 3 Ultra: 91% (PinchBench). | Claude Opus 4.8: 87.6%. GPT 5.5: 88.7%. |
| Technical Differentiators | Blackwell's FP4 precision, 2nd-gen Transformer Engine, advanced NVLink for efficiency. Emphasis on co-designed hardware/software stack. | Agent scaffolding/harness significantly impacts performance and cost, sometimes more than the underlying model. Context engineering for repository understanding. |
🛠️ 技術深入
- AA-AgentPerf Measurement Focus: Measures the maximum number of active users an inference deployment can support while meeting user experience targets for time to first token and output speed.
- Workload Characteristics: Uses real agentic trajectories, which are multi-turn coding sessions involving interleaved reasoning, tool calls, and variable context lengths, rather than synthetic uniform prompts.
- Dataset Composition: The dataset includes real agentic trajectories from public code repositories, covering over 12 programming languages. Input sequence lengths range from ~1K to ~131K tokens (mean ~27K), and output sequence lengths have a median of ~150 tokens.
- Metrics: Key metrics calculated include Time to First Token (TTFT), Output Speed (tokens per second), and System Output Throughput (aggregate tokens per second).
- Service-Level Objectives (SLOs): Performance SLOs are derived from Artificial Analysis's serverless API benchmarking data, reflecting market-available service tiers. P25 output speed is used for evaluation due to the nature of agentic workloads.
- NVIDIA Blackwell Architecture: Achieves efficiency gains through FP4 precision, a second-generation Transformer Engine, and advanced NVLink designs that enhance GPU communication within a system.
- NVIDIA Agent Toolkit: Includes open models like NVIDIA Nemotron, the NVIDIA OpenShell open-source runtime for policy-based security and guardrails, and the NVIDIA AI-Q open agent blueprint for creating custom AI agents.
- NVIDIA AI Agent Building Blocks: Utilizes NVIDIA NeMo for managing the AI agent lifecycle, NVIDIA NIM for fast, enterprise-ready deployment, and NVIDIA Blueprints for accelerating development with customizable reference workflows.
🔮 前景展望基於引用來源的 AI 分析
The emphasis on hardware-aware agent benchmarks will accelerate the development of highly optimized AI inference hardware.
Benchmarks like AA-AgentPerf directly quantify hardware efficiency under complex agentic workloads, incentivizing chip manufacturers to design specialized architectures for these demands.
The 'harness' or orchestration layer will become a critical differentiator for AI agent performance and cost-effectiveness.
The benchmark highlights that the software framework wrapping an AI model significantly impacts real-world performance and operational costs, shifting innovation focus beyond just the underlying large language model.
Standardized multi-vendor benchmarks will foster greater transparency and competition, leading to more robust and cost-efficient AI agent solutions.
By providing a common, open framework for evaluating AI agents across different providers, these benchmarks enable clearer comparisons and drive continuous improvement in the industry.
⏳ 時間線
2026-03
Artificial Analysis introduces the AA-AgentPerf benchmark.
2026-03-16
NVIDIA announces its Open Agent Development Platform, including the Agent Toolkit, Nemotron models, and OpenShell.
2026-05-05
NVIDIA discusses 'Building for the Rising Complexity of Agentic Systems with Extreme Co-Design,' highlighting hardware-software optimization for agents.
2026-05-12
Artificial Analysis introduces the Coding Agent Index, a composite benchmark for coding agents.
2026-05-31
NVIDIA unveils new software, open-source models (Nemotron 3 Ultra, NemoClaw), and partnerships for enterprise AI agents.
2026-06-12
NVIDIA's Blackwell architecture demonstrates 20x more AI agents per megawatt than Hopper on the AgentPerf benchmark.
📎 來源 (17)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： NVIDIA Developer Blog ↗
每週電子報
每週一封，可隨時退訂。