https://arxiv.org/pdf/2605.17109

Motivation

“How can we evolve the optimization algorithm itself during training?”

Notable Methods

For \(W_{t}\) weight matrix and \(M_{t}\) update matrix, we weight the singular directions:

\begin{equation} W_{t+1} = W_{t} - \eta U^{(t)} \Sigma^{\rho^{(t)}} V^{(t)} \end{equation}

and in particular we can dial up and down your choice of \(\rho\) and do stuff. some choices of \(p\):

• \(p=0\), muon OG, the idea of muon is to set all singular values to 1 so that we only prioritize direction
• \(p=1\), some aggressive weight-based updates, proposed idea,
• \(p < 0\), then we prioritize adjusting small directions

generally key insight: “in the beginning, we should prioritize large directions; then, we should prioritize small directions more.”

Sessions

• ICML2026 Kakade: Pretraining

Posters

• ICML2026 Poster Day 1-2
• ICML2026 Poster Day 3

Workshops

• ICLR2026 Workshop: Learning Dynamics
• ICLR2026 Rishi: Public Impact

Pretraining Hard

• base model quality is hard
• continual learning is hard
• shapes may change

Pretraining’s solutions

• When do we stop?: cosine bakes in an end time
• How big a batch?: more parallelism (fewer steps)… what’s the limits such that batches are too big?
• does momentum help?

quadratic models

ICML2026 Svete: revisiting padded transformer expressivity

How to transformer give TC0 full coverage, Luke transformer breaks out of T0 into TCD

(Ryan Kattwral and Will Merrill)

ICML2026 Su: rewiring experts on the fly

Take the prefill prefix at runtime and compute LM loss; use to build additive weight on moe routing

ICML2026 Kreitner: efficient numeracy in language models

Just encode numerical tokens in IEEE floats as the embedding + padding. So each <num> token is one token and Fourier math just works.

ICML2026 Zhang: generalist value model

Use embedding similarity model with respect to previous group performance as the value model, such the value model can take previous performance as well as rollout as input, such hard to predict values more generally

ICML2026 Jayalath: compute as teacher

Consensus voting among multiple rollout used as supervisory target for reward and gold derivation

ICML2026 Mirvakabova: dirchlet prior sampling

For expert upcycling, initialize router based on weighted dirchlet priors instead of uniform