Pre-Training on Frontier LLM Notes

Pre-Training on Frontier LLM Notes https://xiyuanyang-code.github.io/llm/posts/pre-training/ Recent content in Pre-Training on Frontier LLM Notes Hugo zh-cn Mon, 02 Nov 2026 20:39:37 +0800 PreTraining Basics https://xiyuanyang-code.github.io/llm/posts/pre-training/pretraining-basics/ Mon, 02 Nov 2026 20:39:37 +0800 https://xiyuanyang-code.github.io/llm/posts/pre-training/pretraining-basics/ Pre-training math basics (autoregressive token modeling, MLE / NLL training objectives, training gradients, KL divergence) PreTraining Data Selection https://xiyuanyang-code.github.io/llm/posts/pre-training/pretraining-data-selection/ Mon, 02 Nov 2026 20:39:37 +0800 https://xiyuanyang-code.github.io/llm/posts/pre-training/pretraining-data-selection/ Pre-training data recipes & paper list (data sampling selection, data filtering / de-duplication, model-based data scoring) Distributed Parallel Systems for LLM Pre-Training https://xiyuanyang-code.github.io/llm/posts/pre-training/distributed-parallel-systems/ Wed, 14 Oct 2026 17:00:13 +0800 https://xiyuanyang-code.github.io/llm/posts/pre-training/distributed-parallel-systems/ LLM Pretraining Parallelism (GPU Communication, DP/DDP, ZeRO, FSDP, TP, SP, CP, PP, EP, Mixed Precision) Optimizers https://xiyuanyang-code.github.io/llm/posts/pre-training/optimizers/ Wed, 30 Sep 2026 23:28:50 +0800 https://xiyuanyang-code.github.io/llm/posts/pre-training/optimizers/ Classical optimizers for LLM pretraining. (SGD, Adam, MUON)