Attention Proven d² Not n² Problem

🔑 Enhanced Key Takeaways

• •Quadratic attention kernels, implemented via squaring dot products (QK²), have been available in libraries like Fast-Transformers since at least 2020 as custom attention modules with tunable temperature parameters[2].
• •Research on linear attention mechanisms, including optimized CUDA kernels, achieves up to 3.3x speedups and 3.6x memory reductions compared to standard attention while matching expressivity on reasoning benchmarks[5].
• •Subquadratic attention methods, such as those in Reformer using locality-sensitive hashing, have been proposed since 2020 to approximate full attention within token chunks, reducing complexity to near-linear[6].