CDLM: 14x Faster Diffusion LM Inference

• Trajectory Distillation (T3D): Generalizes rectification processes to intermediate states along diffusion trajectories, reducing Conditional Total Correlation and enabling more accurate few-step generation[1] • Conditional Total Correlation Reduction: Theoretical analysis demonstrates that trajectory-level supervision induces lower conditional dependencies, providing stronger inductive bias toward factorized decoding[1] • Block-Wise Attention Optimization: Attention scores computed at the first denoising step (All-[MASK] block) contain sufficient signal to guide sparse attention throughout subsequent denoising steps[3] • Attention Score Skewness: Layers exhibit varying levels of attention-score skewness that remains stable across denoising steps; optimal KV entry selection varies by layer under fixed computation budgets[3] • Dual-Loss Training Objective: Combines cross-entropy loss with KL divergence loss to encourage sparse-constrained models to mimic exact teacher outputs, addressing insufficient signal from cross-entropy alone[3] • Distribution Shift Elimination: Teacher-trajectory-based distillation eliminates distribution shift and stabilizes training without requiring additional ground-truth supervision[1]

The convergence of trajectory-level distillation and block-wise sparse attention techniques suggests a shift toward practical, production-ready diffusion language models. By achieving 14.5x latency improvements without quality degradation, these methods address the primary barrier to DLLM adoption in real-world applications. The post-training recipe approach—requiring no model retraining—lowers deployment friction for existing models. As sparse attention patterns become more sophisticated and theoretically grounded, diffusion LMs may become competitive with autoregressive models for latency-sensitive applications, particularly in scenarios requiring iterative refinement or non-causal generation. The focus on eliminating distribution shift through teacher supervision rather than ground-truth data suggests scalability advantages for future model sizes.