Search

Few direct matches — filled in with the latest updates.

Tag: #learning-dynamics1 results

Transformers Collapse to Low-Dim Manifolds

Transformers Collapse to Low-Dim Manifolds

Transformer training on modular arithmetic tasks collapses high-dimensional parameters to 3-4D execution manifolds. This structure explains attention concentration, SGD integrability, and sparse autoencoder limits. Core computation occurs in reduced subspaces amid overparameterization.

ArXiv AIResearchFeb 12#research#arxiv-ai#v1