SetupAI
繁
SourceFreshcollected in 4h

FlashAttention-4 Adds Low-Precision Blackwell Support

Read original on PyTorch Blog
#attention-kernels#mxfp8#gpu-optimization

A kernel upgrade could unlock multi-PFLOPS attention performance on Blackwell GPUs.

30-Second TL;DR

What Changed

MXFP8 is supported for both forward and backward attention passes.

Why It Matters

Lower-precision attention can improve training and inference throughput while reducing memory and bandwidth pressure. Teams using Blackwell GPUs may gain substantial performance by upgrading kernels rather than changing model architecture.

What To Do Next

Run your training kernels with FlashAttention-4 MXFP8 on a Blackwell test node and compare throughput, loss stability, and memory use.

Who should care:Developers & AI Engineers

Key Points

  • •MXFP8 is supported for both forward and backward attention passes.
  • •Reported performance reaches 2.85 PFLOPS forward and 2 PFLOPS backward.
  • •Internal-shape testing reports 2.54 PFLOPS with FA4 MX8.
Key numbers25%60%

Deep Insight

Background and context from public sources — not the original article. 13 sources cited.

Enhanced Key Takeaways

  • •FlashAttention-4 re-architects the attention pipeline around Blackwell SM100's new TCGEN05 instructions and 256 KB per-SM Tensor Memory (TMEM), abandoning Hopper's WGMMA paradigm.
  • •The implementation is authored via NVIDIA's CuTeDSL in Python rather than monolithic CUDA C++, cutting JIT compilation time from roughly 55 seconds to 2.5 seconds.
  • •FA4 addresses Blackwell's 'asymmetric hardware scaling' bottleneck, where non-MMA operations like softmax threatened to consume 25% to 60% more runtime due to unscaled shared memory and exponential units.
  • •Hardware support is strictly limited to datacenter Blackwell chips (SM100/SM103 like B100, B200, GB200) and Hopper, completely excluding consumer Blackwell GPUs (SM120) which lack TMEM hardware.
  • •On Blackwell HGX B200 systems, FA4 delivers a 1.2× to 1.3× speedup over NVIDIA cuDNN v9.13 and a 2.1× to 2.7× speedup over Triton implementations.

Competitor Analysis

FlashAttention-4 (FA4 MX8)
Relative Blackwell Performance
Reference standard (up to 2.85 PF/s)
Core Architecture / Language
Python via CuTeDSL; TMEM & TCGEN05
Integration & Accessibility
Native FlexAttention, vLLM (v0.17+), SGLang
NVIDIA cuDNN (v9.13)
Relative Blackwell Performance
1.2× – 1.3× slower than FA4
Core Architecture / Language
Closed-source C++/CUDA vendor binaries
Integration & Accessibility
Standard NVIDIA runtime library
OpenAI / PyTorch Triton
Relative Blackwell Performance
2.1× – 2.7× slower than FA4
Core Architecture / Language
Python-based Triton compiler IR
Integration & Accessibility
PyTorch Inductor default backend

Technical Deep Dive

  • TMEM & TCGEN05 Co-Design: Blackwell SM100 removed Hopper's WGMMA instruction set; FlashAttention-4 restructures data flow around the 256 KB Tensor Memory (TMEM) subsystem and hardware-managed Tensor Memory Accelerator (TMA).
  • Asymmetric Scaling Mitigation: While raw Blackwell Tensor Core math throughput doubled over Hopper, exponential and shared memory bandwidth did not scale proportionally; FA4 uses selective rescaling and pipelining to keep non-MMA ops from bottlenecking execution.
  • CuTeDSL Python Kernel Implementation: Built using NVIDIA's CuTeDSL Python dialect instead of C++/CUDA, reducing JIT compilation latency from ~55 seconds down to ~2.5 seconds while sustaining peak compute efficiency.
  • End-to-End Block Scaling (MXFP8): Uses microscopic scaling factor formats across both forward and backward attention loops to maximize Tensor Core throughput without accuracy degradation.
  • Hardware Portability Gate: Requires datacenter compute capabilities (SM100/SM103) with dedicated TMEM blocks, rendering the kernel unsupported on consumer chips like the RTX 5090 (SM120).

Future ImplicationsAI analysis grounded in cited sources

Consumer-grade Blackwell GPUs will remain excluded from state-of-the-art low-precision attention kernels.
Consumer Blackwell SM120 silicon omits the physical Tensor Memory (TMEM) hardware architecture fundamentally required by the FlashAttention-4 execution pipeline.
CuTeDSL will rapidly replace monolithic CUDA C++ as the standard for frontier attention kernel development.
Dropping kernel compilation times from 55 seconds to 2.5 seconds without sacrificing low-level peak hardware utilization significantly accelerates research and deployment iteration loops.

Timeline

2026-03
FlashAttention-4 architecture paper published and FlexAttention backend integration launched
2026-09
PyTorch team adds native end-to-end MXFP8 forward and backward support for NVIDIA Blackwell in FlashAttention-4
Read original article on PyTorch Blog

Weekly AI Recap

Read this week's curated digest of top AI events →

PostLinkedIn

AI-curated news aggregator. All content rights belong to original publishers.
Original source: PyTorch Blog ↗

This is a summary, not the original. Read the source, or get the weekly briefing.

The weekly digest

One email a week. Unsubscribe anytime.