ICML2026 Svete: revisiting padded transformer expressivity

How to transformer give TC0 full coverage, Luke transformer breaks out of T0 into TCD

(Ryan Kattwral and Will Merrill)

ICML2026 Su: rewiring experts on the fly

Take the prefill prefix at runtime and compute LM loss; use to build additive weight on moe routing

ICML2026 Kreitner: efficient numeracy in language models

Just encode numerical tokens in IEEE floats as the embedding + padding. So each <num> token is one token and Fourier math just works.

ICML2026 Yang: inner layer self modulation

Per layer and embeddingsgs added to the visual induces of scaling law that is nearly FLOPs feee

ICML2026 Valarin: KSVD

Old school algorithm after speed up performs par with sae

ICML2026 Roulet: for example gradient

Per example transient allows more efferent batch size affect scaling

ICML2026 Kaissis: step resolved data attribution

Unroll, sketch, and check for alignment in looped transformers

ICML2026 WildCAT: near linear attention

Do not understand, cool, linear attention and kernel

ICML2026 Huang: gradient stabilizer fix the norm not the gradient

When you do gradient and clip the magnitude not the direction

ICML2026 Bick: retrieval aware SSM hybrid

Figure out which heads does the retrieval, distill the rest

ICML2026 Du: scaling mode collapse in LLM generation

Identify low rank spectral signature and prune it out to reduce mode collapse

ICML2026 Fu: selective latent iterations

Have a decider to think about how much to think, but importantly keep around all of the residuals throughout thinking, and then cross attend to them

ICML2026 Liu: less data faster training

At the beginning of training repeating a small subset of data may work better than training on the whole data set, hypothesis is that it creates better inductive bias with stronger gradient signal

ICML2026 Cao: search or accelerate

Based on confidence, at high confidence, just do parallel decoding, otherwise, do beam search, diffusion models

ICML2026 Wan: the shadow price of reasoning

One way to allocate computation is to think about a global utility and a global budget for tokens, allocating tokens based on shadow prices re duals variables

ICML2026 Moose: understanding dynamic allocation recurring transformers

A nice set of synthetic tasks that show shows that block wise UT’s do indeed allocation and tokens of high uncertainty based on authors definition of uncertainty, stuff like physics and graph tasks

ICML2026 Du: RL guided KV cash compression

rl a on policy parity mask which tells you which kv pairs to keep for kv cache compression

ICML2026 Shao: rethinking training signals in RLVR

Spurious rewards actually works fine on Qwen 2.5; this is because the clipping range on GRPO induces a bias that upweights prior tokens; so if you have a good prior literally anything works because you’re essentially doing self distillation

ICML2026 Jing: information adaptive framework for discreet diffusion LM PT

use information as a rough reward to parameterize a small surrogate network to decide where to mask next

ICML2026 Li: delinerizing agent traces with Bayesian partial order

Agent traces induce a linearization observation on actually a more complex graph, so, using partial ordering, learn structure of this graph in terms of 01 transition matrix

ICML2026 Hermann: interestingness is an inductive heuristic for future progress

Position paper: something is relatively more interesting if there are more future compression opportunities, empirical results demonstrate that longer compression stagnation of given algo implies less future progress

ICML2026 Gopalakirshnan: POPE decoupling what and how

ROPE position embeddings because of the fact that it’s a couple imaginary numbers between two channels simultaneously encode position information as well as a little bit of semantic information with respective magnitude and activation on the imaginary channel, this is bad because the normal attention already includes magnitude information, which means that is confounds magnitude and position information in one dot product operation. If you parameterize into polar coordinates, there is a way to decouple so that you only track angles by rotating each channel separately

ICML2026 Hu: residual context diffusion language model

Instead of complete remasking of low entropy tokens, take the residuals from the low entry tokens and add them back after mask, the idea is to use these low entropy tokens as a weak prior for next round generation 

ICML2026 Chen: mind the gap

transition locality and lack of surprise is too good heuristics for determining non-hallucination content tend to have surprise jumps in entropy wrt oracle ground truth