๐คTogether AI BlogโขStalecollected in 19h
Benchmarking inference performance for coding agents
๐กSee how Together AI outperforms Claude Opus and TensorRT-LLM in coding agent inference speed and cost.
โก 30-Second TL;DR
What Changed
Achieved 31% higher throughput (TPS) compared to TensorRT-LLM.
Why It Matters
These benchmarks suggest that developers can significantly reduce operational costs and latency for agentic workflows by switching infrastructure providers. It sets a new performance bar for high-demand coding AI applications.
What To Do Next
Evaluate your current inference costs and latency by running a pilot workload on Together AI's infrastructure to compare against your existing setup.
Who should care:Developers & AI Engineers
Key Points
- โขAchieved 31% higher throughput (TPS) compared to TensorRT-LLM.
- โขDelivered 2x better Time-To-First-Token (TTFT) at saturation.
- โขReduced inference costs by 76% compared to Claude Opus 4.6.
- โขOptimized specifically for real-world coding agent workloads.
๐ง Deep Insight
Web-grounded analysis with 19 cited sources.
๐ Enhanced Key Takeaways
- โขTogether AI operates an AI-inference-as-a-service platform leveraging a decentralized network of GPUs, offering a cost-effective alternative to traditional cloud services for open-source AI models.
- โขThe Together AI platform supports custom Dockerized workloads on managed GPU infrastructure, providing capabilities for compute provisioning, autoscaling, networking, and observability.
- โขTogether AI integrates with leading AI agent frameworks such as LangGraph, CrewAI, AutoGen, and DSPy, and provides 'skills' to enable coding agents to interact with its platform for various AI tasks including inference, training, embeddings, and function calling.
- โขBeyond the specific coding agent benchmarks, Together AI's inference engine is generally cited as being up to 3 times faster than other inference APIs like TGI, vLLM, Perplexity, Anyscale, or Mosaic ML, with costs potentially 6 times lower than GPT 3.5 Turbo when using Llama2-13B.
- โขTogether AI has collaborated with Agentica Project to develop DeepCoder, an entirely open-source model for code reasoning and generation, which achieved a notable accuracy of 60.6% on LiveCodeBench with only 14 billion parameters.
๐ Competitor Analysisโธ Show
| Feature/Aspect | Together AI | NVIDIA TensorRT-LLM | Anthropic Claude Opus 4.6/4.7 |
|---|---|---|---|
| Type | AI Inference-as-a-Service Platform | Open-source LLM Inference Optimization Library | Proprietary LLM API |
| Core Offering | Decentralized GPU network for open-source model inference, training, fine-tuning, custom deployments | High-performance inference optimization for LLMs on NVIDIA GPUs | Flagship model for complex reasoning, coding, agentic workflows |
| Optimization Focus | Throughput, Time-To-First-Token, Cost Efficiency for coding agents and general LLMs | Maximizing inference performance on NVIDIA GPUs via compilation, quantization, batching, parallelism | Advanced reasoning, instruction following, long context handling for agentic tasks |
| Key Technologies | Kubernetes-based GPU clusters, custom Docker support, Jig CLI, Sprocket SDK, various inference optimization techniques | TensorRT Engines, FP8/FP4 quantization, in-flight batching, FlashAttention, various parallelization strategies | Advanced transformer architecture, large context windows (up to 1M tokens), improved tokenizer (Opus 4.7) |
| Pricing Model | Per-token API usage, GPU server rentals | Library is free; costs associated with NVIDIA GPU hardware and development effort | $5/M input tokens, $25/M output tokens (Opus 4.7, effective cost potentially higher due to tokenizer) |
| Coding Agent Benchmarks | 31% higher throughput, 2x better TTFT vs TensorRT-LLM; 76% lower cost vs Claude Opus 4.6 (from article) | Benchmarks are typically for optimized models running on NVIDIA hardware; article states Together AI outperforms it. | Claude Code (using Opus 4.5) achieved 80.9% on SWE-bench. Opus 4.7 shows 10-15% lift over 4.6 in agentic reasoning. |
๐ ๏ธ Technical Deep Dive
- Together AI's GPU clusters are built on Kubernetes, featuring a control plane, GPU-equipped worker nodes, high-speed InfiniBand networking for multi-node communication, and a robust storage layer.
- The platform supports custom Dockerized workloads through 'Dedicated Containers,' where users provide container images, and Together AI manages compute provisioning, autoscaling, networking, and observability.
- It includes a deployment CLI called Jig for Dockerfile generation, image building, deployment creation, and secrets management, and a Python SDK named Sprocket for building inference workers that integrate with Together AI's job queue.
- Together AI's internal data architecture routes raw events from Kinesis and Kafka, stores them in S3 as Iceberg tables, processes them through a data warehouse using dbt and Airflow, and then moves them into MotherDuck for analytics and AI agent queries.
- NVIDIA TensorRT-LLM, a competitor in inference optimization, compiles large language models into highly optimized TensorRT Engines.
- TensorRT-LLM leverages techniques such as quantization (including FP8, NVFP4, SmoothQuant, AWQ, GPTQ), in-flight batching (also known as continuous batching), and advanced attention mechanisms like multi-query attention, grouped-query attention, and FlashAttention to enhance LLM inference efficiency.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Together AI's cost and performance advantages will accelerate the adoption of open-source coding agents in enterprise environments.
By significantly reducing inference costs and improving speed, Together AI lowers the barrier for companies to deploy and scale AI coding solutions based on open models, making them more competitive against proprietary alternatives.
The focus on coding agent benchmarks will drive further specialization and optimization in LLM inference for specific vertical applications.
As benchmarks highlight performance differences for tasks like coding, AI infrastructure providers will increasingly tailor their offerings to excel in particular domains, moving beyond general-purpose LLM optimization.
Together AI's decentralized GPU network model will gain traction as enterprises seek alternatives to traditional cloud vendors for AI infrastructure.
The platform's ability to run open-source models cost-effectively on a decentralized network addresses concerns about vendor lock-in and high costs associated with major cloud providers.
โณ Timeline
2022-06
Together AI founded by Chris Re, Ce Zhang, Percy Liang, Vipul Ved Prakash, and Tri Dao.
2023-05-16
Together AI raised $20 million in Seed funding.
2023-11-29
Together AI raised $102 million in Series A funding.
2024-03
Together AI raised an additional $106 million in a Series A extension, valuing the company at $1.25 billion.
2025-02-20
Together AI raised $305 million in Series B funding, achieving a $3.3 billion valuation.
2025-05
Together AI acquired Refuel.ai, expanding into data transformation and structuring.
๐ Sources (19)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Together AI Blog โ