Pretraining data, dynamics and optimizers.
PreTraining Basics
Pre-training math basics (autoregressive token modeling, MLE / NLL training objectives, training gradients, KL divergence)
Pre-Training
Pretraining data, dynamics and optimizers.
PreTraining Data Selection
Pre-training data recipes & paper list (data sampling selection, data filtering / de-duplication, model-based data scoring)
Distributed Parallel Systems for LLM Pre-Training
LLM Pretraining Parallelism (GPU Communication, DP/DDP, ZeRO, FSDP, TP, SP, CP, PP, EP, Mixed Precision)
Optimizers
Classical optimizers for LLM pretraining. (SGD, Adam, MUON)