Nemotron 3 Super Launches on Together AI

🔑 Enhanced Key Takeaways

• •Nemotron 3 Super features approximately 100B total parameters with 10B active per token, positioning it between the smaller Nano (30B/3B) and larger Ultra (500B/50B) variants.
• •It employs a hybrid Mamba-Transformer MoE architecture combined with Latent MoE for enhanced expert specialization and 4x more experts at the same inference cost.
• •Trained using NVIDIA's 4-bit NVFP4 precision on Blackwell architecture, enabling reduced memory usage and faster training without accuracy loss.
• •Outperforms models like GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507 on benchmarks, with the Nano variant showing 3.3x higher throughput than Qwen3-30B-A3B.

🛠️ Technical Deep Dive

• •Hybrid Mamba-Transformer MoE architecture: Integrates Mamba layers for efficient long-range sequence modeling, Transformer layers for precise reasoning, and MoE routing for scalable compute.
• •Latent MoE: Experts process shared latent representations before token projection, supporting 4x more experts for better specialization in multi-hop reasoning.
• •Multi-Token Prediction (MTP) layers: Improve long-form text generation efficiency and model quality.
• •NVFP4 4-bit floating-point format: Used for pretraining on 25T token dataset, optimizing cost-accuracy for training and inference on Blackwell GPUs.
• •Positioned for multi-agent workloads like IT ticket automation, with up to 4x higher token throughput and reduced reasoning tokens compared to Nemotron 2.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline