Pretraining Hard
- base model quality is hard
- continual learning is hard
- shapes may change
Pretraining’s solutions
- When do we stop?: cosine bakes in an end time
- How big a batch?: more parallelism (fewer steps)… what’s the limits such that batches are too big?
- does momentum help?