It's my favorite kind of work: linear algebra insight + fast kernels. When playing w Muon a while ag...
By @tri_dao
Tri Dao describes work on speeding up Muon optimizer by operating on the small square matrix X X^T instead of the large rectangular X, requiring months of understanding eigenvalues/eigenvectors to develop a simple, elegant algorithm.