Advanced LLMs 2: Normalizations, FFNs & RoPE
Pre-Norm / Post-Norm; RMSNorm; Activation function(SwiGLU FFN), Rope rotation and length-extrapolation (PI / NTK-aware / YaRN).
Transformer
Advanced LLMs 1: Attentions
Frontier LLM's Attention (MQA, GQA, MLA, Linear Attention Family, Flash Attention Family, Sparse Attention Family, Advanced Attentions)