To determine

Question: can we find a good context permutation to improve reasoning capabilities.

One-Liner

Notable Methods

Two key evaluations:

• evalutanig relationships between gold documents; notice that performance relates to distance between documents (but FTing helps)
• investigate the effects between different attention masks (i.e., the use of prefix vs continuation masks)

IC Score

attention-based context attribution method

New Concepts

Key insight: correct answers will have single peak of IC scores at gold results; incorrect answers will have more dispersed IC scores.

Talks

• ACL2025 Keynote: Luke Zettemoyer
• ACL2025 Orals: Language Modeling 1
• ACL2025 Orals: QA

Posters

• ACL2025 Monday Morning Posters
• ACL2025 Tuesday Morning Posters
• ACL2025 Tuesday Afternoon Posters

Takes

• mayhaps we can apply thoughtbubbbles intutiton to BLT token pruning?

Naively: “almost everything comes from pretraining.” How much simple supervision will it radically change the behavior of our language model.

Key Directions

1. data long-tail: tokenizer free LLMs
2. data modules: how to we specialize quickly?

Tokenizer-Free LM

Byte-Level LMs are just more expensive (i.e., there is just a bunch more residual streams! and that’s pretty bad). High level intution: takes the input bytes, create some “strides”/“patches”, and then send the patches through a transformer, and then unpatch

Method to adapt tokenization across models.

Notable Methods

1. use pairwise cosine similarity between token embeddings to create a grid of alignment
2. initialize new adapted embeddings for each id’s most similar tokens
3. tune