Transformers Collapse to Low-Dim Manifolds
Transformer training on modular arithmetic tasks collapses high-dimensional parameters to 3-4D execution manifolds. This structure explains attention concentration, SGD integrability, and sparse autoencoder limits. Core computation occurs in reduced subspaces amid overparameterization.








