๐คTogether AI BlogโขRecentcollected in 28h
A/B Test Models Directly in Production
๐กLearn how to test whether users actually prefer a new modelโnot just whether it runs reliably.
โก 30-Second TL;DR
What Changed
Shadow traffic validates operational readiness but does not measure user preference.
Why It Matters
This makes model evaluation more closely tied to real user outcomes, such as engagement or task success. It can also reduce deployment complexity by moving experiment routing out of individual applications.
What To Do Next
Configure a Together AI endpoint experiment that splits a small percentage of production traffic between your current and candidate models, then track user-level success metrics.
Who should care:Developers & AI Engineers
Key Points
- โขShadow traffic validates operational readiness but does not measure user preference.
- โขModel traffic splitting can be performed at the endpoint level.
- โขEndpoint-level testing avoids implementing routing logic in application code.
๐ง Deep Insight
AI-generated analysis for this event.
๐ Enhanced Key Takeaways
- โขTogether AI's implementation utilizes weighted traffic distribution, allowing developers to assign specific percentages of requests to different model versions without modifying client-side code.
- โขThe platform integrates with observability tools to provide real-time latency and error rate comparisons between the control and candidate models during the A/B test.
- โขThis feature supports 'sticky' sessions or request-based routing, ensuring that a single user consistently interacts with the same model version throughout a conversation thread.
- โขThe infrastructure leverages Together AI's serverless inference engine, which dynamically scales resources to accommodate the concurrent execution of multiple model versions.
- โขBy moving routing logic to the API gateway layer, the system reduces the overhead of managing complex load balancing configurations within the application's backend services.
๐ Competitor Analysisโธ Show
| Feature | Together AI (A/B Testing) | AWS Bedrock (Model Evaluation) | LangSmith (LangChain) |
|---|---|---|---|
| Routing Logic | Endpoint-level (Gateway) | Managed via Aliases/Provisioned Throughput | Application-level (SDK) |
| Traffic Splitting | Native weighted distribution | Manual alias updates | Custom implementation required |
| Primary Focus | Production inference optimization | Enterprise governance & security | LLM observability & debugging |
๐ ๏ธ Technical Deep Dive
- The traffic splitting mechanism operates at the API Gateway level, intercepting incoming HTTP requests before they reach the inference engine.
- It utilizes a deterministic hashing algorithm based on request headers or user IDs to maintain session consistency across multiple API calls.
- The system supports dynamic weight adjustment, allowing engineers to shift traffic percentages in real-time without restarting the inference instances.
- Integration with the inference backend allows for shared KV-cache management, minimizing the memory footprint when running multiple model variants simultaneously.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Automated 'winner' promotion will become a standard feature in LLM deployment pipelines.
Integrating A/B testing directly into the inference layer enables closed-loop systems that can automatically route 100% of traffic to the model with higher user engagement metrics.
Application-level routing logic will be deprecated in favor of infrastructure-managed model versioning.
As model deployment becomes more complex, offloading traffic management to the inference provider reduces technical debt and improves deployment velocity.
โณ Timeline
2023-06
Together AI launches its platform for training and inference of open-source models.
2024-02
Introduction of the Together Inference Engine, optimized for high-throughput model serving.
2025-05
Expansion of API capabilities to include advanced model management and versioning tools.
2026-08
Official release of endpoint-level A/B testing for production traffic.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Together AI Blog โ