Talks

• ICLR2025 Kilani: MrT5 Tokenizer-Free
• ICLR2025 Neitemeier: Hierachical Autoregressive Transformers

Downsides of Subword Tokenization

1. not learned end to end: vocab is fixed, can’t adapt to difficulty
2. non-smoothness: similar inputs get mapped to very different token sequences
   • [token][ization]
   • typo: [token][zi][ation] <- suddenly bad despite small typo
3. huge vocabs: yes
4. non-adaptive compression ratio: you can’t decide how much to compress (affects FLOPs/document)

Motivation

Previous works contain “heads” that perform some specific mechanism from context retrieval.

Retrieval Head

Authors shows that Retrieval Heads exist in transformers: using Needle in a Haystack framework.

Key Insight

There exists certain heads which performs retrieval, as measured by the retrieval score.

Methods

Measuring Retrieval Behavior

“retrieval score”: how often does a head engage in copy-paste behavior.

1. token inclusion: current generated token \(w\) is in the edle
2. maximal attention: same token gives the maximum attenion score

“RAG performance can scale almost linearly w.r.t. log inference FLOPs”

Demonstration Based RAG (DRAG)

Method

Adding demonstrations as k in-context examples.

Prompt: documents, input query, final answer.

Parameters: number of documents, number of in context samples, number of iterations upper bound.

Iterative Demonstration Based RAG (IterDRAG)

Method

DRAG above, and then the model can generate a new sub-query. The model decides

Parameters: number of documents, number of in context samples, number of iterations upper bound.

identities allows another number to retain its identity after an operation.

What identities are applicable is group dependent. Identities are almost always object dependent.

<> NUS-HIST301 American History

The idea of identity politics is proposed, that politics became associated with sub-population of identities:

• Black Pride Movement
• Chicano Activism
• The American Indian movement
  • Termination of reservation system
  • Pan-Indian Rights
  • Alcatraz and Wounded Knee Occupations
• LGBT movement
  • Stonewall
  • GLF starts marching
• Asian American
  • Yellow Peril
  • Model minority movement
• NOW Femanism Acts
  • The Equal Rights Act almost possible, and then Phyllis Schlafly happened
• Environmental Movement
  • Silent Spring
  • Cuyahoga River on fire
  • Richard Nixon creates the EPA
  • Earth Day