Uncensored Nemotron-3-Super-120B for MLX

🔑 Enhanced Key Takeaways

• •Nemotron 3 Super was officially released by NVIDIA with fully public weights, training data, recipes, and checkpoints available for download.
• •The model was pre-trained on over 10 trillion tokens and supports up to 1M token context length across English, French, German, Italian, Japanese, Spanish, and Chinese.
• •It achieves 85.6% on PinchBench, the highest score among open models for agentic performance in OpenClaw environments.
• •On NVIDIA Blackwell GPUs like B200, it delivers up to 5x higher throughput than prior Nemotron models using NVFP4 precision.

🛠️ Technical Deep Dive

• •Hybrid architecture: LatentMoE with interleaved Mamba-2 layers (linear-time sequence efficiency), MoE layers (top-22 routing, 120.6B total / 12.7B active parameters), and select Transformer Attention layers for precision reasoning.
• •Multi-Token Prediction (MTP): Predicts multiple future tokens per forward pass, enabling speculative decoding and up to 7.5x higher throughput on long outputs (e.g., 64k tokens) vs. Qwen3.5-122B.
• •NVFP4 pre-training: First in Nemotron 3 family; uses NVFP4 for most linear layers (weights, activations, gradients) with BF16/MXFP8 for stability in key layers like QKV/attention and embeddings.
• •Post-training: Multi-environment RL across 21 configurations with 1.2M rollouts using NeMo Gym/RL; optimized for Blackwell GPUs (4x faster inference vs. FP8 on H100).

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline