PreTraining Data Selection

在 PreTrain/MidTrain-Data 的数据配比系列研究中，往往会关心如下的问题：

• Data Sampling Selection: 如何确定 domain-wise/sample-wise 的数据配比，进而控制不同来源数据的上采样/下采样比例？
• Data Filtering: 如何清洗脏数据? 如何针对数据做数据去重等等策略?
• Data Scoring: 通过打分的方式获得对训练数据质量的反馈，如何训练一个好的打分模型?

Paper List:

• 【Data Sampling Selection】Data Selection via Importance Sampling: https://arxiv.org/pdf/2302.03169
• 【Data Sampling Selection】DoReMi (Data Mixtures for LLM PreTraining): https://arxiv.org/pdf/2305.10429
• 【Data Sampling Selection】Doge (Data Mixtures by Proxy Model Sampling): https://arxiv.org/pdf/2310.15393
• 【Data filtering】D4 (Document De-Duplication and Diversification): https://arxiv.org/pdf/2308.12284
• 【Data Scoring】QuRating (Model Based Data Selection): https://arxiv.org/pdf/2402.09739
• 【Data Scoring】FineWebEdu (Model Based Data Selection): https://arxiv.org/pdf/2406.17557
• Rho-1 Not All Tokens Are What You Need: https://arxiv.org/pdf/2404.07965
• 【Data Sampling Selection】RegMix (Data Mixture as regression): https://arxiv.org/pdf/2407.01492
• Grape (Dynamic Data Mixture weighting): https://arxiv.org/pdf/2505.20380
• 【Data Sampling Selection】Data Selection co-evolve during optimizers: https://arxiv.org/pdf/2610.03432
• 【Data Scoring】DataMan: https://arxiv.org/pdf/2502.19363
• 【Data Ordering】DELT: https://arxiv.org/pdf/2506.21545
• Nemotron-CC: https://arxiv.org/pdf/2412.02595v2
• 【Data Sampling Selection】OPUS: https://arxiv.org/pdf/2602.05400