Distributed Parallel Systems for LLM Pre-Training
LLM Pretraining Parallelism (GPU Communication, DP/DDP, ZeRO, FSDP, TP, SP, CP, PP, EP, Mixed Precision)
Distributed-Training
Slime101: Slime Core Training Pipeline
Core Slime training pipeline analysis. (Ray, Megatron, SGLang)