Consider in Kaplan et al., 2020 scaling laws, at scale, we can get reasonably smooth trends for MMLU / ARC / etc.

We would like to predict these IN ADVANCE at smaller scale.

• Small scale: DCLM Baseline Data
• Legally friendly data: CommonPile
• Web scraped data with quality group: NemoTron

people measure isoFLOPS

Problems of pre-training data

1. pre-training influence downstream capabilities
2. …and therefore can escape into model generation
3. real world users expect novelty

Changes in Distribution

Big Pretraining Data

GPT2

• deduplicated data
• Removed Wikipedia (to prevent data leak)
• Heuristic based cleaning

GPT3

• Deduplicated
• based on leaked data

Llama

the usual spheal

• removed high perplexity data using wiki n-gram model
• removed non-English
• deduplicated

Llama 2

• removed high volue of PII
• Removed non-english

Pretraining Curation Decisions

• what to include
• what is the timestamp being scraped
• heuristic based cleaning? data cleaning? etc.
• language filtering (only take English?)
• PII removal
• dedup
• Toxicity + SafeURL filtering
• “quality filtering”
• sampling distributions

Change in Model Age

Good alignment shown between validation year and pre-training year, even mixing in older data.

Key Sequence

Notation

New Concepts

• unsupervised learning
  • k-means clustering
    • k-means++
  • Gaussian mixture model
  • expectation maximization
    • Jensen’s Inequality

Important Results / Claims

Questions

Interesting Factoids

Name: bodwin.jemoka.com

Technology: Airpods Pro A3063

Description: Airpods Pro 3

Lost

If you have stumbled upon this page due to finding this device in the wild, thank you so much! Reach out to [email protected] or +1 650 209-0966 to get in touch. Please do the right thing.

Thank you.