Nemotron 3 Super Loops on Self-Reasoning

🔑 Enhanced Key Takeaways

• •Nemotron 3 Super employs a hybrid Mamba-Transformer MoE architecture with 120B total parameters but only 12B active per forward pass, enabling 5x higher throughput than its predecessor.[1][2]
• •The model supports a native 1M-token context window and features three controllable reasoning modes (reasoning-off, low-effort, regular) that stream thinking tokens before final responses.[1][2]
• •It uses Multi-Token Prediction (MTP) to generate multiple tokens per forward pass, supporting speculative decoding for faster inference and improved reasoning quality.[2][4]

🛠️ Technical Deep Dive

• •Hybrid backbone: 6-layer repeating block with three Mamba-2 layers (50% state-space for linear cost efficiency), two LatentMoE layers (compress tokens to lower dimension before routing to 512 experts), and one attention layer (17% for precision).[2][6]
• •Pre-trained on 25 trillion tokens in NVFP4 precision (optimized for NVIDIA Blackwell, 4x inference speedup on B200 vs FP8 on H100), followed by SFT on 7M samples and RL with 1.2M rollouts across 21 environments.[1][4]
• •Post-training in BF16; supports PinchBench score of 85.6% as top open model for agentic tasks; incorporates MTP for multi-step dependency capture and native speculative decoding.[1][3][4]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline