Base Model Architecture
Advanced LLMs 4: Mixture of Experts
MoE: Routing, balancing and training.
Model Architecture
Base Model Architecture
Advanced LLMs 3: FLOPs and Decoding
FLOPs Analysis, Speculative Decoding, Multi-Token Prediction.
Advanced LLMs 2: Normalizations, FFNs & RoPE
Pre-Norm / Post-Norm; RMSNorm; Activation function(SwiGLU FFN), Rope rotation and length-extrapolation (PI / NTK-aware / YaRN).
Advanced LLMs 1: Attentions
Frontier LLM's Attention (MQA, GQA, MLA, Linear Attention Family, Flash Attention Family, Sparse Attention Family, Advanced Attentions)