TorchSpec Enables Scaled Speculative Decoding Training

🔑 Enhanced Key Takeaways

• •TorchSpec employs a two-phase training strategy: phase 1 uses small batches with 4k token sequences via standard causal LM training, while phase 2 uses large batches with 256 token sequences from the base model, with a 5:2 step ratio.[1]
• •Speculator heads attached to the base model outperform smaller draft models like Llama 7B for Llama 70B in both quality and latency gains during speculative decoding training.[1]
• •TorchSpec leverages PyTorch FSDP and IBM FMS for distributed training of speculators at scale.[1]

🛠️ Technical Deep Dive

• •Speculator training avoids KV-cache replication during prediction and modifies attention masks for verifying the N+1th token to ensure outputs match the original model without deviation.[1]
• •Two-phase training: Phase 1 focuses on long sequences (4k tokens) with small batches; Phase 2 shifts to short sequences (256 tokens) with large batches to tune heads to base model outputs, using a 5:2 phase 1 to phase 2 step ratio.[1]
• •Uses PyTorch FSDP for distributed training and IBM FMS, prioritizing attached speculator heads over smaller independent draft models for better efficiency and quality.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline