StepFun Fully Open-Sources Step 3.5 Flash

🔑 Enhanced Key Takeaways

• •Step 3.5 Flash uses a 45-layer sparse-MoE Transformer backbone with 3 dense layers and 42 MoE layers, each containing 288 routed experts plus one shared expert, activating top-8 experts per token for 11B active parameters out of 196B total.[1]
• •Steptron is a lightweight PyTorch and Megatron-LM based framework that unifies pre-training, post-training, and RL workflows for efficient model development.[1]
• •The model incorporates hybrid attention layouts, SWA, head scaling, head-wise gated attention, and activation clipping in MoE experts to optimize performance and efficiency.[1]

🛠️ Technical Deep Dive

• •Architecture: 45-layer sparse-MoE Transformer (3 dense + 42 MoE layers) with hybrid attention; each MoE layer has 288 routed experts + 1 shared expert; top-k=8 routing activates 11B parameters per token from 196B total.[1]
• •Optimizations: Includes SWA (Sliding Window Attention), head scaling, head-wise gated attention vs. sink tokens; activation clipping in MoE FFN intermediates before output projection.[1]
• •Ablations: Tested end-to-end pipelines (pre-training, 32k long-context extension, 64k SFT) and scaled to 100B parameters to validate efficiency without performance loss.[1]
• •Training: Powered by Steptron framework on PyTorch/Megatron-LM, supporting large-scale pre-training, post-training, and RL under a unified stack.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline