Advanced LLM Notes (Model-Arch, Infra, Data, Pre/Mid/Post-Training), always updating.
Above All
Advanced LLM Notes (Model-Arch, Infra, Data, Pre/Mid/Post-Training), always updating.
Model Architecture
Base Model Architecture
Pre-Training
Pretraining data, dynamics and optimizers.
Mid-Training
Mid-training: papers and tech reports. (Continual pretraining, data recipe, retention)
Post Training
Frontier LLM post-training: data mixture & post-training algorithms.
LLM Infra
LLM Codebase & Infra
Agentic RL
From PPO, GRPO to frontier agentic reinforcement learning
PreTraining Basics
Pre-training math basics (autoregressive token modeling, MLE / NLL training objectives, training gradients, KL divergence)
PreTraining Data Selection
Pre-training data recipes & paper list (data sampling selection, data filtering / de-duplication, model-based data scoring)
Advanced LLMs 4: Mixture of Experts
MoE: Routing, balancing and training.
Agentic RL 2: RL for LLMs
RL Methods for LLM PostTraining (vanilla-ppo, RLHF-PPO, DPO, RLVR, DeepSeek-Math-GRPO)
Advanced LLMs 3: FLOPs and Decoding
FLOPs Analysis, Speculative Decoding, Multi-Token Prediction.
Agentic RL 3: From Tokens to Trajectories, Long-Horizon Reinforcement Learning
What is after GRPO? Advanced long-horizon agentic reinforcement learning. (DAPO, GSPO, SAO, FlashREINFORCE, more advanced training recipes.)
Advanced LLMs 2: Normalizations, FFNs & RoPE
Pre-Norm / Post-Norm; RMSNorm; Activation function(SwiGLU FFN), Rope rotation and length-extrapolation (PI / NTK-aware / YaRN).
Distributed Parallel Systems for LLM Pre-Training
LLM Pretraining Parallelism (GPU Communication, DP/DDP, ZeRO, FSDP, TP, SP, CP, PP, EP, Mixed Precision)
MidTraining Overview
Mid-training paper list & tech reports. (Continual pretraining, retention, RL scaling)
Agentic RL 1: From Policy Gradient to PPO
From RL Basis to Policy Gradient (reward-to-go, advantage estimations), REINFORCE, Actor-Critic, PPO (Importance Sampling and Clipping)
Slime 101
THUDM Slime infra: bridging Megatron and SGLang for RL-scaling LLM post-training.
Slime101: Slime Core Training Pipeline
Core Slime training pipeline analysis. (Ray, Megatron, SGLang)
Frontier Post Training Pipelines
Post-training analysis & summaries from frontier base model tech reports (continuously updated).
Advanced LLMs 1: Attentions
Frontier LLM's Attention (MQA, GQA, MLA, Linear Attention Family, Flash Attention Family, Sparse Attention Family, Advanced Attentions)
Optimizers
Classical optimizers for LLM pretraining. (SGD, Adam, MUON)