Above All
Advanced LLM Notes (Model-Arch, Infra, Data, Pre/Mid/Post-Training), always updating.
Posts
Model Architecture
Base Model Architecture
Pre-Training
Pretraining data, dynamics and optimizers.
Mid-Training
Mid-training: papers and tech reports. (Continual pretraining, data recipe, retention)
Post Training
Frontier LLM post-training: data mixture & post-training algorithms.
LLM Infra
LLM Codebase & Infra