DataFlex

Data Select · Mix · Reweight — Right in the LLM Training Loop

📰 1. News

• [2026-09-25] 🎉 DataFlex has been accepted to the NeurIPS 2026 Evaluation & Dataset Track! Our paper, DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models, will appear at NeurIPS 2026.
• [2026-04-04] 🎉 Our technical report ranked #1 on the Hugging Face Daily Papers leaderboard for that day.
• [2026-03-17] We now support gradient computation under DeepSpeed ZeRO-3, enabling training and analysis of larger-scale models.
• [2025-12-23] 🎉 We’re excited to announce the first Data-Centric Training System DataFlex, is now released! Stay tuned for future updates.

🔍 2. Overview

DataFlex is an advanced dynamic training framework built on top of LLaMA-Factory.
It intelligently schedules training data during optimization and integrates several difficult-to-reproduce repositories into a unified framework. The system provides reproducible implementations of Data Selection, Data Mixture, and Data Reweighting, thereby improving both experimental reproducibility and final model performance.

DataFlex integrates seamlessly with LLaMA-Factory, offering researchers and developers more flexible and powerful training control. For goals and design philosophy, please refer to DataFlex-Doc. We summarize repositories related to Data Selection, Data Mixture, and Data Reweighting. ❌ indicates that no official repository is available; ✅ indicates that an official repository is available; ⚠️ indicates that an official repository exists but contains issues.

• Data Selection: Dynamically selects training samples according to a given strategy (e.g., focus on “hard” samples). The data selection algorithms are summarized as follows:

• Data Mixture: Dynamically adjusts the ratio of data from different domains during training. The data mixture algorithms are summarized as follows:

• Data Reweighting: Dynamically adjusts sample weights during backpropagation to emphasize data preferred by the model. The data reweighting algorithms are summarized as follows:

• Full compatibility with LLaMA-Factory, drop-in replacement.

📌 3. Quick Start

Please use the following commands for environment setup and installation👇

Install torch first, so pip does not resolve a different build and then have to replace it:

pip install --index-url https://download.pytorch.org/whl/cu124 \
    torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Then install DataFlex:

pip install dataflex

Or install from source for development:

git clone https://github.com/OpenDCAI/DataFlex.git
cd DataFlex
pip install -e .

Every config under examples/deepspeed needs DeepSpeed, which ships as an optional extra:

pip install "dataflex[deepspeed]"        # from source: pip install -e ".[deepspeed]"

Note: Requires Python 3.11+ and LlamaFactory 0.9.5+, installed automatically along with the other core dependencies. Works on transformers 4.55 through 5.6; the newer model families (Qwen3.5, Gemma 4) need transformers 5.5+. We recommend transformers 5.3+ if you need train_from_scratch under DeepSpeed ZeRO-3.

The deepspeed extra stays below 0.17 on purpose: 0.17+ fails to import on the torch pinned above. On a newer torch you are free to lift that cap.

The LESS selector needs TRAK, which is another optional extra: pip install dataflex[less].

The launch command is similar to LLaMA-Factory. Below is an example using LESS :

dataflex-cli train examples/train_lora/selectors/less.yaml

Unlike vanilla LLaMA-Factory, your .yaml config file must also include DataFlex-specific parameters. For details, please refer to DataFlex-Doc.

📖 Skills

• How to Use DataFlex — Installation, CLI commands, YAML configuration, training modes, and supported algorithms.
• How to Add a New Algorithm — Architecture overview, registry system, base class interfaces, and step-by-step guide for adding selectors/mixers/weighters.

📚 4. Experimental Results

Using DataFlex can improve performance over the default LLaMA-Factory training.

Data Selector & Reweightor Results

We use a subset of Open-Hermes-2.5 as the training dataset. The data selection algorithms and data reweighting algorithm outperform the random selector baseline on the MMLU benchmark subset relevant to the training dataset. For the Less and Nice algorithm, we set the validation set as the MMLU-Validation-Set, using a GPT-5-generated trajectory.

Data Mixture Results

We use subsets of SlimPajama-627B for data mixture. The data mixture algorithms outperform the baseline (default data mixture) on MMLU accuracy while also achieving lower perplexity across different data domains.

🧩 5. Ecosystem

DataFlex focuses on data scheduling during training. For a complete pipeline starting from raw data, it pairs well with DataFlow:

DataFlow converts raw files into LLM training data through composable operator pipelines — document parsing, knowledge cleaning, QA / CoT synthesis, and training format conversion. The output JSON can be fed directly into DataFlex. The two projects are independent with no code dependency, connected only by standard data formats. DataFlex accepts training data from any source — DataFlow, manual annotation, HuggingFace datasets, or custom processing scripts.

🤝 6. Acknowledgements

We thank LLaMA-Factory for offering an efficient and user-friendly framework for large model fine-tuning, which greatly facilitated rapid iteration in our training and experimentation workflows.
We thank Zhongguancun Academy for their API and GPU support. Our gratitude extends to all contributors in the open-source community—their efforts collectively drive the development of DataFlex.

📜 7. Citation

The DataFlex paper has been accepted to the NeurIPS 2026 Evaluation & Dataset Track. Until the official proceedings citation is available, please cite the arXiv version:

@article{liang2026dataflex,
  title={DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models},
  author={Liang, Hao and Zhao, Zhengyang and Qiang, Meiyi and Chen, Mingrui and Ma, Lu and Yu, Rongyi and Feng, Hengyi and Sun, Shixuan and Meng, Zimo and Ma, Xiaochen and others},
  journal={arXiv preprint arXiv:2603.26164},
  year={2026}
}

@article{liang2026towards,
  title={Towards Next-Generation LLM Training: From the Data-Centric Perspective},
  author={Liang, Hao and Zhao, Zhengyang and Han, Zhaoyang and Qiang, Meiyi and Ma, Xiaochen and Zeng, Bohan and Cai, Qifeng and Li, Zhiyu and Tang, Linpeng and Zhang, Wentao and others},
  journal={arXiv preprint arXiv:2603.14712},
  year={2026}
}

🤝 8. Community & Support

We welcome contributions of new trainers and selectors! Please ensure code formatting is consistent with the existing style before submitting a PR.

We also welcome you to join the DataFlex and DataFlow open-source community to ask questions, share ideas, and collaborate with other developers!

• 📮 GitHub Issues: Report bugs or suggest features

• 🔧 GitHub Pull Requests: Contribute code improvements

• 💬 Join our community groups to connect with us and other contributors!