Social Twitter Mar 30
It's my favorite kind of work: linear algebra insight + fast kernels. When playing w Muon a while ag...
By @tri_dao
82 score
AI Analysis
Tri Dao describes work on speeding up Muon optimizer by operating on the small square matrix X X^T instead of the large rectangular X, requiring months of understanding eigenvalues/eigenvectors to develop a simple, elegant algorithm.
It's my favorite kind of work: linear algebra insight + fast kernels.
When playing w Muon a while ago, we were thinking why not speed it up by operating on the small square matrix X X^T instead of the large rectangular matrix X. Jack, Noah, and Berlin spent many months understanding eigenvalues/vectors of the intermediate matrices in Muon, and finally came up with a simple and elegant algo to make this work.
Muon optimizertraining efficiencylinear algebraGPU kernelsoptimization
Source details
It's my favorite kind of work: linear algebra insight + fast kernels.
When playing w Muon a while ago, we were thinking why not speed it up by operating on the small square matrix X X^T instead of the large rectangular matrix X. Jack, Noah, and Berlin spent many months understanding eigenvalues/vectors of the intermediate matrices in Muon, and finally came up with a simple and elegant algo to make this work.
Muon optimizertraining efficiencylinear algebraGPU kernelsoptimization