Analysis of Metastable States in the Transformer Activation Space
By Zach Baker
The first in a planned series testing a mathematical theory that models transformer attention as a dynamical system on a sphere, where tokens cluster and drift toward consensus with metastable structure. The author empirically checks how much of the theory survives in trained models, finding most predictions hold but the energy-monotonicity claim fails universally, traced to the value matrix.