50% Faster RL Rollouts with DAS

💡50% faster RL rollouts with no reward loss – key for efficient post-training.

⚡ 30-Second TL;DR

What Changed

Up to 50% faster RL rollouts

Why It Matters

Speeds up RLHF and alignment workflows, cutting compute costs and iteration time for practitioners training large models.

What To Do Next

Test DAS on Together AI platform for your next RL post-training experiment.

Who should care:Researchers & Academics

AI-generated analysis for this event — not the original article.

•DAS leverages the observation that RL rollout policies often exhibit high confidence in token generation, allowing the draft model to achieve higher acceptance rates compared to standard speculative decoding.
•The implementation utilizes a specialized kernel optimization that minimizes the overhead of the draft-verify loop, specifically targeting the latency-sensitive nature of PPO (Proximal Policy Optimization) rollout phases.
•By dynamically adjusting the draft model's temperature and sampling strategy based on the target policy's distribution, DAS maintains reward alignment without requiring additional fine-tuning or reward model re-evaluation.

📊 Competitor Analysis▸ Show

Feature	Together AI (DAS)	vLLM (Speculative Decoding)	NVIDIA TensorRT-LLM
Primary Focus	RL Rollout Acceleration	General Inference	Hardware-Specific Optimization
Adaptive Logic	Distribution-Aware	Static/Heuristic	Static/Heuristic
Reward Integrity	Guaranteed (Zero Degradation)	N/A (General Purpose)	N/A (General Purpose)
Deployment	Together AI Cloud/API	Open Source / Self-Hosted	Open Source / Self-Hosted

Mechanism: DAS employs a lightweight draft model that predicts the next N tokens, which are then verified in parallel by the larger target policy model.
Distribution Awareness: Unlike standard speculative decoding, DAS incorporates a KL-divergence-based filter that rejects draft tokens if the draft distribution deviates significantly from the target policy's expected distribution.
Kernel Optimization: Uses custom CUDA kernels to fuse the verification step with the target model's forward pass, reducing memory bandwidth bottlenecks during the rollout phase.
Integration: Designed to be drop-in compatible with standard RLHF/RLAIF pipelines, requiring minimal changes to existing rollout infrastructure.

RLHF training costs will decrease by at least 30% for large-scale models.

Reducing rollout time directly correlates to lower compute hours required for the most expensive phase of the RLHF training loop.

On-device RL fine-tuning will become viable for medium-sized models.

Accelerating the rollout bottleneck lowers the hardware threshold required to perform iterative policy updates in resource-constrained environments.

2023-06

Together AI launches with a focus on decentralized cloud infrastructure for LLMs.

2024-02

Together AI introduces support for speculative decoding across its inference API.

2025-09

Together AI releases optimized kernels for RLHF training workflows.

2026-04

Together AI announces Distribution-Aware Speculative Decoding (DAS) for RL rollouts.

Weekly AI Recap

Read this week's curated digest of top AI events →

Same topic

Explore #rl

Same product