Model Architecture on LLM Notes

Model Architecture on LLM Notes https://xiyuanyang-code.github.io/llm/posts/model-arch/ Recent content in Model Architecture on LLM Notes Hugo en Tue, 22 Sep 2026 00:00:00 +0000 Advanced LLMs 1: Attentions https://xiyuanyang-code.github.io/llm/posts/model-arch/attentions/ Tue, 22 Sep 2026 00:00:00 +0000 https://xiyuanyang-code.github.io/llm/posts/model-arch/attentions/ 从 MHA、MQA/GQA、MLA 到稀疏注意力（LongFormer / StreamingLLM / NSA / DSA / MSA）、线性注意力（Mamba / Gated DeltaNet）与 Flash Attention。 Advanced LLMs 2: Normalizations, FFNs & RoPE https://xiyuanyang-code.github.io/llm/posts/model-arch/normalizations-ffn-rope/ Tue, 22 Sep 2026 00:00:00 +0000 https://xiyuanyang-code.github.io/llm/posts/model-arch/normalizations-ffn-rope/ Pre-Norm / Post-Norm 与 RMSNorm 的推导、激活函数与 SwiGLU FFN，以及 RoPE 的旋转矩阵与长度外推（PI / NTK-aware / YaRN）。 Advanced LLMs 3: Decoding https://xiyuanyang-code.github.io/llm/posts/model-arch/decoding/ Tue, 22 Sep 2026 00:00:00 +0000 https://xiyuanyang-code.github.io/llm/posts/model-arch/decoding/ Prefill 与 Decode 两个阶段的 FLOPs 与 Memory-Bound 分析，以及 Speculative Decoding、Multi-Token Prediction。 Advanced LLMs 4: Optimizers https://xiyuanyang-code.github.io/llm/posts/model-arch/optimizers/ Tue, 22 Sep 2026 00:00:00 +0000 https://xiyuanyang-code.github.io/llm/posts/model-arch/optimizers/ 预训练优化器与训练稳定性。 Advanced LLMs 5: Mixture of Experts https://xiyuanyang-code.github.io/llm/posts/model-arch/mixture-of-experts/ Tue, 22 Sep 2026 00:00:00 +0000 https://xiyuanyang-code.github.io/llm/posts/model-arch/mixture-of-experts/ MoE 的路由、负载均衡与训练。