consider something like a polynomial interpolation:

Interpolating polynomial (or most ML models in general) are smooth, and so interpolating between points will result in “overshooting” regional points and “bouncing around”

…as a function of parameters

At a fixed dataset, just increasing the number of parameters will increase

Key Sequence

Notation

New Concepts

• bias variance tradeoff
• Model Evaluation
  • Hold-out cross-validation
  • k-fold cross validation
• Model Selection
• Regularization

Important Results / Claims

Questions

Interesting Factoids

A general formulation of decision networks; track:

1. track how distributions affect each other (i.e. generalize your problem into a Bayes Net)
2. apply transformation of the edges to your new problem type structure
3. transform back into your new formulation’s solver
4. solve

One-Liner

“If you have infinite compute but limited samples, how do you pretrain?”

Novelty

• closed-form best approach which reduces loss given the current data budget

Notable Methods

Outline

• Takes 200 million tokens, 300 million parameters, from a corpus
• Measure validation loss
• Vary training recipes

regularized parameter scaling

Stick in some more weight decay: more parameters, more weight decay

ensembling

Train a bunch of seperate models (i.e. with random shuffles?) and then parameter merge; different initialization, etc.