https://arxiv.org/pdf/2605.17109

Motivation

“How can we evolve the optimization algorithm itself during training?”

Notable Methods

For \(W_{t}\) weight matrix and \(M_{t}\) update matrix, we weight the singular directions:

\begin{equation} W_{t+1} = W_{t} - \eta U^{(t)} \Sigma^{\rho^{(t)}} V^{(t)} \end{equation}

and in particular we can dial up and down your choice of \(\rho\) and do stuff. some choices of \(p\):

• \(p=0\), muon OG, the idea of muon is to set all singular values to 1 so that we only prioritize direction
• \(p=1\), some aggressive weight-based updates, proposed idea,
• \(p < 0\), then we prioritize adjusting small directions

generally key insight: “in the beginning, we should prioritize large directions; then, we should prioritize small directions more.”