Model selection:

• choose some parameters
• evaluate every model using a Model Evaluation method of some kind
• and in production…
  • k-fold or LOOCV: retrain best model on all data
  • Hold-out cross-validation: optionally retrain, if you have time

A special case of model selection is feature selection:

• choose a subset of the most relevant features to train on
• note that power set is \(2^{m}\) in size; so instead of doing this we train \(O\qty(n)\) by starting out with an empty set, and then adding features sequentially that would give us the best performance

consider something like a polynomial interpolation:

Interpolating polynomial (or most ML models in general) are smooth, and so interpolating between points will result in “overshooting” regional points and “bouncing around”

…as a function of parameters

At a fixed dataset, just increasing the number of parameters will increase

Key Sequence

Notation

New Concepts

• bias variance tradeoff
• Model Evaluation
  • Hold-out cross-validation
  • k-fold cross validation
• Model Selection
• Regularization

Important Results / Claims

Questions

Interesting Factoids

A general formulation of decision networks; track:

1. track how distributions affect each other (i.e. generalize your problem into a Bayes Net)
2. apply transformation of the edges to your new problem type structure
3. transform back into your new formulation’s solver
4. solve

One-Liner

“If you have infinite compute but limited samples, how do you pretrain?”

Novelty

• closed-form best approach which reduces loss given the current data budget

Notable Methods

Outline

• Takes 200 million tokens, 300 million parameters, from a corpus
• Measure validation loss
• Vary training recipes

regularized parameter scaling

Stick in some more weight decay: more parameters, more weight decay

ensembling

Train a bunch of seperate models (i.e. with random shuffles?) and then parameter merge; different initialization, etc.