ICLR2025 Tokenizer-Free Approaches

Talks

• ICLR2025 Kilani: MrT5 Tokenizer-Free
• ICLR2025 Neitemeier: Hierachical Autoregressive Transformers

Downsides of Subword Tokenization

1. not learned end to end: vocab is fixed, can’t adapt to difficulty
2. non-smoothness: similar inputs get mapped to very different token sequences
   • [token][ization]
   • typo: [token][zi][ation] <- suddenly bad despite small typo
3. huge vocabs: yes
4. non-adaptive compression ratio: you can’t decide how much to compress (affects FLOPs/document)