Model selection:

• choose some parameters
• evaluate every model using a Model Evaluation method of some kind
• and in production…
  • k-fold or LOOCV: retrain best model on all data
  • Hold-out cross-validation: optionally retrain, if you have time

A special case of model selection is feature selection:

• choose a subset of the most relevant features to train on
• note that power set is \(2^{m}\) in size; so instead of doing this we train \(O\qty(n)\) by starting out with an empty set, and then adding features sequentially that would give us the best performance