---

Why this repository?

LLM development is often organized by training stage (pretraining, SFT, RL) or by isolated techniques (synthetic data, selection, RAG, agents). This repository instead organizes the literature around the operation performed on information.

The accompanying survey asks three operational questions:

1. What information-bearing artifacts or environments are available?
2. How is supervision or data content created, transformed, filtered, or selected?
3. How, when, and under what contextual conditions is available information consumed by the model?

This perspective makes it easier to place methods that cross conventional boundaries—for example, model-aware selection, data mixing, rejection-sampling loops, executable agent environments, retrieval, test-time search, and verification.

What you will find here

If this repository is useful for your research, consider starring it. It helps other researchers discover the resource.

---

Taxonomy at a Glance

flowchart LR
    A["Tier 1 · Data Substrates<br/>What information exists?"]
    B["Tier 2 · Data Creation & Selection<br/>What supervision is created or changed?"]
    C["Tier 3 · Data Ingestion Strategies<br/>How and when is information consumed?"]
    M["Target Model<br/>Capability State"]

    A --> B
    A --> C
    B --> M
    C --> M
    M -. "errors · loss · confidence · interaction feedback" .-> B
    M -. "adaptive scheduling / retrieval" .-> C

Placement rule.
Tier 2 changes the supervision or data artifacts that exist; Tier 3 changes how available supervision or information is delivered to and consumed by the model.

The unit of analysis is the operation rather than the paper as a whole. A method can therefore span multiple tiers. For example, rejection-sampling fine-tuning may generate and filter trajectories in Tier 2, then immediately consume the retained trajectories during optimization in Tier 3.

---

How to Use This Repository

---

Automated Literature Tracking

To keep this repository useful beyond the survey's fixed literature cut-off, we provide an automated literature-tracking workflow built with GitHub Actions. The bot continuously discovers recent work related to the LLM data lifecycle, filters duplicates, assigns candidates to the operation-centered taxonomy, and proposes updates through pull requests for human review.

The automation is designed to assist curation rather than replace it: newly discovered papers are never auto-merged into the curated list.

How it works

Recent arXiv papers
        ↓
Topic-specific query families
        ↓
Date / category / LLM-data relevance filtering
        ↓
README + historical-database deduplication
        ↓
Operation-centered taxonomy classification
        ↓
Primary tier + optional cross-tier role
        ↓
Generated Markdown update
        ↓
GitHub Pull Request
        ↓
Human review → merge

The classifier follows the same placement principle as the survey:

• Data Substrates — data-bearing artifacts, benchmarks, trajectories, and executable environments.
• Data Creation and Selection — annotation, synthesis, transformation, filtering, and model-aware selection.
• Data Ingestion Strategies — organization, mixing, curriculum, replay, retrieval, search, decoding, and verification.

Because the unit of analysis is the operation rather than the paper as a whole, the bot can also flag cross-tier papers. For example, a method may generate or filter supervision in Tier 2 and immediately consume that supervision during optimization in Tier 3.

Two update modes

The historical mode is useful when the repository has not been updated for a period of time or when a new query family is introduced and earlier papers need to be recovered.

Repository automation

The implementation lives in:

.github/workflows/
├── update-papers.yml       # scheduled / manual weekly update
└── backfill-papers.yml     # exact-date historical scan

scripts/
└── paper_bot.py            # fetch → filter → deduplicate → classify → render

config/
└── paper_bot.yaml          # search queries, taxonomy keywords, thresholds

data/
└── papers.json             # persistent bot-curated paper history

The automatically maintained README block is bounded by:

<!-- AUTO-LITERATURE:START -->
## Recent Automatically Discovered Papers

> This section is generated by the weekly literature bot and reviewed through pull requests before merge.
> Last generated: **2026-08-24** · Showing up to **30** recent papers.

### Data Substrates

- **[Benchmarking Patent Drafting from Inventor-Style Disclosures](http://arxiv.org/abs/2608.21249v1)** — Lekang Jiang et al. (2026). *Other*
- **[Enhancing LLMs in Predictive Political QA with Semi-Structured Data](http://arxiv.org/abs/2608.21218v1)** — Yinan Liu et al. (2026). *Other*
- **[Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment](http://arxiv.org/abs/2608.21057v1)** — Emma Granqvist et al. (2026). *Other*
- **[MigrationNarrate: A Dataset for Detection of Migration Narratives in YouTube Videos](http://arxiv.org/abs/2608.20984v1)** — Fatima Haouari et al. (2026). *Other*
- **[VortexChat: An agentic framework for autonomous multi-objective integrated photonic design](http://arxiv.org/abs/2608.20688v1)** — Faqian Chong et al. (2026). *Other*
- **[ARQ: Agentic CodeQL Query Refinement for C/C++ Vulnerability Detection](http://arxiv.org/abs/2608.20637v1)** — Chunyi Wang et al. (2026). *Other*
- **[Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents](http://arxiv.org/abs/2608.20631v1)** — Quang Dao et al. (2026). *Other*
- **[ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models](http://arxiv.org/abs/2608.20338v1)** — Sahil Kale et al. (2026). *Other*
- **[From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation](http://arxiv.org/abs/2608.20195v1)** — Zhijun Gao et al. (2026). *Other*
- **[OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models](http://arxiv.org/abs/2608.20106v1)** — Nikita Khudov (2026). *Other*

### Data Creation and Selection

- **[Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models](http://arxiv.org/abs/2608.21019v1)** — Zhen Yang et al. (2026). *Data Selection*
- **[An Agentic Approach for Active Data Collection, Travel Behavior Modeling, and Weather-Sensitive Demand Prediction](http://arxiv.org/abs/2608.20320v1)** — Narges Ahmadi et al. (2026). *Annotation / Processing*

### Data Ingestion Strategies

- **[Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis](http://arxiv.org/abs/2608.21310v1)** — Qisheng Lu et al. (2026). *Runtime Retrieval / Search / Verification*
- **[EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering](http://arxiv.org/abs/2608.21252v1)** — Xuanyu Meng et al. (2026). *Runtime Retrieval / Search / Verification*
- **[Specification Portability Across LLM Development Agents: Cross-Agent Compatibility in Specification-Driven Software Migration](http://arxiv.org/abs/2608.21208v1)** — Oleg Grynets et al. (2026). *Runtime Retrieval / Search / Verification*
- **[ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models](http://arxiv.org/abs/2608.21100v1)** — Wenzheng Jiang et al. (2026). *Runtime Retrieval / Search / Verification* · _Cross-tier: Data Substrates_
- **[Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems](http://arxiv.org/abs/2608.21095v1)** — Balkrishna Giri et al. (2026). *Runtime Retrieval / Search / Verification*
- **[$Z^2$-ACT: End-to-End Verifiable Agentic Intent Control for Open 6G RAN](http://arxiv.org/abs/2608.21049v1)** — Sunder Ali Khowaja et al. (2026). *Runtime Retrieval / Search / Verification*
- **[UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists](http://arxiv.org/abs/2608.20918v1)** — Ye Chen et al. (2026). *Curriculum / Mid-Training / Replay*
- **[TRACE: Agentic Catalog Enrichment with Multi-source Evidence Grounding](http://arxiv.org/abs/2608.20844v1)** — Rohan Kumar et al. (2026). *Runtime Retrieval / Search / Verification*
- **[Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation](http://arxiv.org/abs/2608.20794v1)** — Haodong Chen et al. (2026). *Curriculum / Mid-Training / Replay*
- **[Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation](http://arxiv.org/abs/2608.20756v1)** — Rujin Liang et al. (2026). *Runtime Retrieval / Search / Verification*
- **[AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification](http://arxiv.org/abs/2608.20711v1)** — Ji Liu et al. (2026). *Runtime Retrieval / Search / Verification*
- **[Towards Faithful Simulation of Human Shopping Behavior](http://arxiv.org/abs/2608.20707v1)** — Jiakai Tang et al. (2026). *Runtime Retrieval / Search / Verification*
- **[JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification](http://arxiv.org/abs/2608.20607v1)** — Tianxin Zhou et al. (2026). *Runtime Retrieval / Search / Verification*
- **[Terminal Agents: A Survey of AI Agents in Command-Line Environments](http://arxiv.org/abs/2608.20485v1)** — Yi Bin et al. (2026). *Runtime Retrieval / Search / Verification*
- **[MidTool: Mid-training Data Synthesis for Agentic Tool Use](http://arxiv.org/abs/2608.20314v1)** — Fengqing Jiang et al. (2026). *Curriculum / Mid-Training / Replay* · _Cross-tier: Data Creation and Selection_
- **[Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization](http://arxiv.org/abs/2608.20281v1)** — Qian Kou et al. (2026). *Runtime Retrieval / Search / Verification*
- **[Decoding silent reading from non-invasive EEG](http://arxiv.org/abs/2608.20186v1)** — Ingo Marquardt et al. (2026). *Runtime Retrieval / Search / Verification*
- **[FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models](http://arxiv.org/abs/2608.20153v1)** — Dingzirui Wang et al. (2026). *Runtime Retrieval / Search / Verification*

<!-- AUTO-LITERATURE:END -->

so the workflow never rewrites the manually curated core of this repository.

Curation policy

Automation is intentionally conservative:

1. Discovery is automatic; acceptance is human-reviewed.
2. Existing README entries and previously accepted bot records are deduplicated before proposal.
3. The workflow prefers high-recall discovery followed by stricter relevance and taxonomy filtering.
4. Ambiguous methods may receive a primary tier plus a cross-tier label.
5. Important accepted papers can later be moved from the automatically maintained recent-paper block into the main curated sections.

This makes the repository a living companion to the survey while preserving the quality and interpretability of the manually curated taxonomy.

---

Scope and Update Policy

• Survey literature cut-off: June 2026.
• Repository status: living resource; a GitHub Actions literature bot tracks new candidate papers beyond the paper cut-off, with all additions reviewed through pull requests.
• In scope: work where data resources, data construction/selection, information scheduling, retrieval, verification, trajectories, or executable environments play a substantive role in LLM development or evaluation.
• Out of scope by default: architecture-only or optimization-only papers where the data lifecycle is incidental.
• Source preference: primary papers, official project pages, ACL Anthology, OpenReview, arXiv, or official dataset/model repositories whenever available.

2026 coverage highlights

Recent additions reflected in the survey/repository include:

• Research-level reasoning data: ResearchMath-14K.
• Real and executable coding-agent data: Multi-SWE-bench, SWE-Chat, SWE-RL, SWE-Playground.
• Tool-agent trajectories and environments: TOUCAN, FunReason-MT, UniToolCall, WebWorld, ClawGym.
• Environment/data co-evolution: Agent-World, Nex-N1, ClawEnvKit, AgentFrontier.
• Recent survey context: Agentic Tool Use in LLMs, Agentic Environment Engineering, and The LLM Data Auditor.

---

Contents

• 1. Surveys
• 2. Taxonomy
• 3. Data Substrates
  • 3.1 General Domain
    • 3.1.1 Pretrain
    • 3.1.2 SFT
    • 3.1.3 RL
  • 3.2 Specific Domain
    • 3.2.1 Reasoning and Code
    • 3.2.2 Safety and Alignment
    • 3.2.3 Agent and Tool Use
  • 3.3 Summary
• 4. Data Creation and Selection
  • 4.1 Annotation
  • 4.2 Synthesis
  • 4.3 Selection
  • 4.4 Summary
• 5. Data Ingestion Strategies
  • 5.1 Efficiency & Effectiveness
  • 5.2 Training Pipeline Optimization
  • 5.3 Capability Alignment Strategies
  • 5.4 Truthfulness & Consistency
  • 5.5 Summary
• 6. Cross-Tier Synthesis
• 7. Future Trends
  • 7.1 Data Resources and Governance
  • 7.2 Data Creation and Selection
  • 7.3 Data Ingestion Strategies
• Automated Literature Tracking
• Citation
• Contributing
• Acknowledgements

1. Surveys

Data Selection

• A survey on data selection for language models Transactions on Machine Learning Research 2024.
• A survey on Data selection for llm instruction tuning Journal of Artificial Intelligence Research 2025.

Generation

• Data-centric Artificial Intelligence: A Survey ACM Computing Surveys 2023

• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP TACL 2023.

• Best Practices and Lessons Learned on Synthetic Data for Language Models Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai. COLM 2024.

• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, Haobo Wang. arXiv 2024.

• Large Language Models for Data Annotation: A Survey Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu. arXiv 2024.

• A Survey on Data Synthesis and Augmentation for Large Language Models arXiv 2024.

• Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application ACM Transactions on Intelligent Systems and Technology 2024.

• Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges arXiv 2024.

• Quality, Diversity, and Complexity in Synthetic Data SURVEYING THE EFFECTS OF QUALITY, DIVERSITY, AND COMPLEXITY IN SYNTHETIC DATA FROM LARGE LANGUAGE MODELS arXiv 2024.

• A Survey of Multimodal Large Language Model from A Data-centric Perspective arXiv 2024.

• Automatically Correcting Large Language Models : Surveying the Landscape of Diverse Automated Correction Strategies TACL 2024.

• A Survey of LLM × DATA arXiv 2025.

• Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future DirectionsarXiv 2025.

• AI Alignment: A Comprehensive Survey arXiv 2025.

• A Survey of Post-Training Scaling in Large Language Models ACL 2025.

• Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives arXiv 2025.

• Agentic Tool Use in Large Language Models. arXiv 2026.

• Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application. arXiv 2026.

• The LLM Data Auditor: A Survey on Data-Centric Metrics for Cross-Modal Large Language Models. arXiv 2026.

Blogs

• Synthetic dataset generation techniques: Self-Instruct Daniel van Strien. 2024
• LLM-Driven Synthetic Data Generation, Curation & Evaluation Cobus Greyling. 2024
• The Rise of Agentic Data Generation Maxime Labonne. 2024
• https://blog.csdn.net/qq_43688587/article/details/148533722

2. Taxonomy

We organize the LLM data lifecycle through an operation-centered, three-tier taxonomy. The goal is not merely to separate training stages, but to distinguish what information-bearing objects exist, how their content or membership is changed, and how available information is presented to and consumed by the model.

Placement rule. Tier 2 changes the supervision or data artifacts that exist; Tier 3 changes how available supervision or information is delivered to and consumed by the model. For example, offline data selection is primarily Tier 2, whereas dynamic data mixing is Tier 3. An agent trajectory is a Tier-1 substrate; generating or filtering trajectories is Tier 2; using them in iterative optimization or test-time interaction is Tier 3.

Cross-tier methods. The unit of analysis is the operation rather than the paper as a whole. Methods such as rejection-sampling fine-tuning or error-driven preference optimization can span multiple tiers: they may generate/filter supervision in Tier 2 and immediately consume the retained supervision during optimization in Tier 3.

3. Data Substrates

3.1 General Domain

3.1.1 Pretrain

• SlimPajama: A 627B Token Cleaned and Deduplicated Version of RedPajama. NeurIPS 2024 Datasets & Benchmarks.
• Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. ACL 2024.
• GneissWeb: Preparing High Quality Data for LLMs at Scale. arXiv 2025.
• Essential-Web v1.0: 24T Tokens of Organized Web Data. arXiv 2025.

3.1.2 SFT

• FLAN: Scaling Instruction-Finetuned Language Models. JMLR 2024.
• Dolly: Free Dolly—Introducing the World's First Truly Open Instruction-Tuned LLM. Databricks Blog 2023.
• Alpaca: A Strong, Replicable Instruction-Following Model. Stanford 2023.
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4. ICML 2023.
• Tülu: How Far Can Camels Go?. NeurIPS 2023.
• Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data. EMNLP 2023.

3.1.3 RL

• InstructGPT: Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022.
• RLAIF: Constitutional AI: Harmlessness from AI Feedback. NeurIPS 2022.
• UltraFeedback: Boosting Language Models with High-Quality Feedback. ICML 2024.
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2024.
• ORM/PRM: Let's Verify Step by Step. ICLR 2024.

3.2 Specific Domain

3.2.1 Reasoning and Code

• GSM8K: Training Verifiers to Solve Math Word Problems. NeurIPS 2021.
• MATH: Measuring Mathematical Problem Solving with the MATH Dataset. NeurIPS 2021 Datasets & Benchmarks.
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. ACL 2024.
• AMC 23: AIMO Validation AMC Dataset on Hugging Face. 2023.
• AIME 24: AIMO Validation AIME Dataset on Hugging Face. 2024.
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. ICLR 2024.
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning. ICLR 2024.
• LILA: A Unified Benchmark for Mathematical Reasoning. EMNLP 2022.
• Arithmo2: Arithmo-Mistral-7B Mathematical Reasoning Model. Hugging Face 2023.
• MathGenie: Generating Synthetic Data with Question Back-Translation for Enhancing Mathematical Reasoning of LLMs. ACL 2024.
• OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset. NeurIPS 2024 Datasets & Benchmarks.
• LIMO: Less Is More for Reasoning. CoLM 2025.
• S1: Simple Test-Time Scaling. ArXiv 2025.
• ResearchMath-14K: Scaling Research-Level Mathematics via Agents. arXiv 2026.
• HumanEval: Evaluating Large Language Models Trained on Code. ICML 2021.
• MBPP: Program Synthesis with Large Language Models. arXiv 2021.
• MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. ACL 2022.
• MdEval: Massively Multilingual Code Debugging. ICLR 2025.
• CodeAlpaca: An Instruction-Following LLaMA Model for Code Generation. GitHub 2023.
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct. ICLR 2024.
• WaveCoder: Widespread and Versatile Enhancement for Code Large Language Models by Instruction Tuning. ACL 2024.
• Magicoder: Empowering Code Generation with OSS-Instruct. ICML 2024.
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. ICLR 2025.
• McEval: Massively Multilingual Code Evaluation. arXiv 2025.
• CodeEditorBench: Evaluating Code Editing Capability of Large Language Models. ICLR Workshop 2025.
• Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. NeurIPS 2025.
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. arXiv 2025.
• SWE-Chat: Coding Agent Interactions from Real Users in the Wild. arXiv 2026.
• SWE-Playground: Training Versatile Coding Agents in Synthetic Environments. arXiv 2025.

3.2.2 Safety and Alignment

• TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.
• AlpacaEval: An Automatic Evaluator of Instruction-Following Models. arXiv 2023.
• MT-Bench: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
• XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024.
• WildGuardTest: WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. NeurIPS 2024.
• JailbreakTrigger: JailbreakTrigger: A Standardized Evaluation Framework for Automated Red Teaming. ICML 2024.
• DoAnythingNow: "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. CCS 2024.
• WildJailbreak: WildTeaming at Scale: From In-The-Wild Jailbreaks to (Adversarially) Safer Language Models. NeurIPS 2024.
• ArenaHard: From Live Data to High-Quality Benchmarks — The Arena-Hard Pipeline. arXiv 2024.
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming. ICML 2024.
• HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection. NeurIPS 2024.
• Enhancing Leakage Attacks on Searchable Symmetric Encryption Using LLM-Based Synthetic Data Generation. arXiv 2025.
• Model Prompts: Private Text Generation by Seeding Large Language Model Prompts. arXiv 2025.
• SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy. arXiv 2024.

3.2.3 Agent and Tool Use

• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. EMNLP 2023.
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. ICLR 2024.
• AgentBench: Evaluating LLMs as Agents. ICLR 2024.
• BFCL: The Berkeley Function Calling Leaderboard. ICML 2025.
• MetaTool: MetaTool Benchmark for Large Language Models — Deciding Whether to Use Tools and Which to Use. ICLR 2024.
• GAIA: A Benchmark for General AI Assistants. ICLR 2024.
• ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. NAACL 2025 Findings.
• OpenFunctions: The Berkeley Function Calling Leaderboard (BFCL) V2. NeurIPS 2024.
• Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023.
• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. arXiv 2023.
• AgentTuning: Enabling Generalized Agent Abilities for LLMs. ACL 2024 Findings.
• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. NeurIPS 2024 Datasets & Benchmarks.
• AgentInstruct: Toward Generative Teaching with Agentic Flows. arXiv 2024.
• Magnet: Multi-Turn Tool-Use Data Synthesis and Distillation via Graph Translation. ACL 2025.
• T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning. ACL 2025.
• The Behavior Gap: Evaluating Zero-Shot LLM Agents in Complex Task-Oriented Dialogs. ACL 2025 Findings.
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv 2024.
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. NeurIPS 2025 Datasets & Benchmarks.
• BUTTON: Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning. arXiv 2024.
• TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments. arXiv 2025.
• FunReason-MT: Overcoming the Complexity Barrier in Multi-Turn Function Calling. arXiv 2025.
• UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents. arXiv 2026.
• WebWorld: A Large-Scale World Model for Web Agent Training. arXiv 2026.
• ClawGym: A Scalable Framework for Building Effective Claw Agents. arXiv 2026.
• Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence. arXiv 2026.
• Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction. arXiv 2025.
• ClawEnvKit: Automatic Environment Generation for Claw-Like Agents. arXiv 2026.
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis. arXiv 2025.

3.3 Summary: Substrate Dynamics and Trade-offs

The substrate view highlights a shift from raw scale toward curated information density, and—especially for agents—from stored text/trajectories toward executable environments that can generate tasks, observations, failures, recovery paths, and verification signals on demand.

4. Data Creation and Selection

4.1 Annotation

4.1.1 Data Processing

• Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. ACL 2024.
• DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. NeurIPS 2024 Datasets & Benchmarks.
• AutoClean: LLMs Can Prepare Their Training Corpus. NAACL 2025.
• Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.
• Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data. arXiv 2025.
• Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning. ICML 2025.
• AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark. EMNLP 2025 Findings.

4.1.2 Prompt Engineering

• Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding. ICLR 2024.
• Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
• Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs. EMNLP 2024.
• PE2: Prompt Engineering a Prompt Engineer. ACL 2024 Findings.
• OPRO: Large Language Models as Optimizers. ICLR 2024.

4.2 Synthesis

4.2.1 Sampling-Based

• MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.
• GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models. ICLR 2024.
• OKD: Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models. EMNLP 2025 Findings.
• Sequence-Level Knowledge Distillation. EMNLP 2016.
• Distilling Step-by-Step: Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. ACL 2023 Findings.
• Fine-tune-CoT: Large Language Models Are Reasoning Teachers. ACL 2023.
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. ACL 2023.
• Bonito: Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation. ACL 2024 Findings.
• Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023.
• WizardLM: Empowering Large Pre-trained Language Models to Follow Complex Instructions. ICLR 2024.
• AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs. EMNLP 2025.
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data. ICLR 2025.
• Synthetic Data RL: Task Definition Is All You Need. arXiv 2025.
• Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement. ACL 2025.
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. ICLR 2025.

4.2.2 Data Transformation

• SODA: Million-Scale Dialogue Distillation with Social Commonsense Contextualization. EMNLP 2023.
• MAmmoTH2: Scaling Instructions from the Web. NeurIPS 2024.
• Tree-KG: An Expandable Knowledge Graph Construction Framework for Knowledge-Intensive Domains. ACL 2025.
• Personas: Scaling Synthetic Data Creation with 1,000,000,000 Personas. arXiv 2024.
• Reformatted Alignment. EMNLP 2024 Findings.

4.2.3 Back-Translation

• Self-Alignment with Instruction Backtranslation. ICLR 2024.
• LongForm: Effective Instruction Tuning with Reverse Instructions. EMNLP 2024 Findings.
• Back-and-Forth Translation: Better Alignment with Instruction Back-and-Forth Translation. EMNLP 2024 Findings.
• Safer-Instruct: Aligning Language Models with Automated Preference Data. NAACL 2024.
• Cycle-Instruct: Fully Seed-Free Instruction Tuning via Dual Self-Training and Cycle Consistency. EMNLP 2025.

4.2.4 Human-AI Collaboration

• SELF-ALIGN: Principle-Driven Self-Alignment of Language Models from Scratch. NeurIPS 2023.
• Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
• SPIN: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024.
• Self-Critiquing Models for Assisting Human Evaluators. arXiv 2022.
• AutoIF: Self-Play with Execution Feedback Improves Instruction-Following. ICLR 2025.
• APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training. ACL 2025 Findings.
• SwS: Self-Aware Weakness-Driven Problem Synthesis in Reinforcement Learning for LLM Reasoning. ICLR 2025.
• Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoning. ACL 2025.
• LEMMA: Learning from Errors for Mathematical Advancement in LLMs. ACL 2025 Findings.
• RISE: Subtle Errors in Reasoning — Preference Learning via Error-Injected Self-Editing. ACL 2025.

4.2.5 Symbolic Generation

• Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. EMNLP 2023 Findings.
• NSR: Large Language Models are Neurosymbolic Reasoners. AAAI 2024.
• SR-LCL: Symbolic Regression with a Learned Concept Library. NeurIPS 2024.
• SymbCoT: Faithful Logical Reasoning via Symbolic Chain-of-Thought. ACL 2024.
• ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs. arXiv 2025.
• Aristotle: Mastering Logical Reasoning with a Logic-Complete Decompose-Search-Resolve Framework. ACL 2025.
• Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models. ACL 2025.
• SC-NeuroSymbolic: Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretations. PMLR 2025.
• SymbolicThought: Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding. arXiv 2025.
• MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions. EMNLP 2024 Findings.
• CSDG: Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars. EMNLP 2024.
• NSAR: Enhancing Large Language Models with Neurosymbolic Reasoning for Multilingual Tasks. arXiv 2025.
• AdaCoT: Adaptive Multilingual Chain-of-Thought for Cross-Lingual Factual Reasoning. ACL 2025 Findings.

4.3 Selection

4.3.1 Diversity

• Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model. NAACL 2024.
• QDIT: Data Diversity Matters for Robust Instruction Tuning. EMNLP 2024 Findings.
• DEITA: Data-Efficient Instruction Tuning for Alignment. ICLR 2024.
• Instag: Instruction Tagging for Analyzing Supervised Fine-Tuning of Large Language Models. ICLR 2024.
• LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.
• Prismatic synthesis: Gradient-based data diversification boosts generalization in llm reasoning. ICLR 2025.

4.3.2 Quality

• LIFT: Rethinking the Instruction Quality — LIFT Is What You Need. EMNLP 2023 Findings.
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. EMNLP 2024.
• AlpaGasus: Training a Better Alpaca Model with Fewer Data. ICLR 2024.
• IFD: Instruction-Following Difficulty Score for Data Selection. NAACL 2024.
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2025.

4.3.3 Composite Strategy

• LIMA: Less Is More for Alignment. NeurIPS 2023.
• MoDS: Model-Oriented Data Selection for Instruction Tuning. arXiv 2023.
• SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection. NeurIPS 2024.
• Dataman: Data Manager for Pre-Training Large Language Models. ICLR 2025.

4.4 Summary: Creation and Selection Trade-offs

Across creation and selection methods, the survey highlights three recurring transitions: teacher distillation → autonomous/self-play generation, static quality filtering → model-conditional utility, and stored examples → verifier-grounded or environment-generated supervision. Selection and synthesis increasingly interact through closed loops rather than remaining independent preprocessing stages.

5. Data Ingestion Strategies

5.1 Efficiency & Effectiveness

5.1.1 Sample Construction

Cross-tier note: this section focuses on the ingestion/optimization-time role of sample construction. Methods that generate persistent new supervision (e.g., rejection sampling or targeted error injection) also have a Tier-2 creation component.

• Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-Tuning. ACL 2025 Findings.
• Aligning to Thousands of Preferences via System Message Generalization. NeurIPS 2024.
• CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions. EMNLP 2024.
• Persona-Plug: LLMs + Persona-Plug = Personalized LLMs. ACL 2025.
• RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv 2023.
• SeaPO: Strategic Error Amplification for Robust Preference Optimization. EMNLP 2025 Findings.

5.1.2 Sample Mixing

• D4: Improving LLM Pretraining via Document De-Duplication and Diversification. NeurIPS 2023 Datasets & Benchmarks.
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023.
• QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining. ICLR 2025.
• SampleMix: A Sample-Wise Pre-Training Data Mixing Strategy by Coordinating Data Quality and Diversity. ICLR 2025.
• Predictive Data Selection: The Data That Predicts Is the Data That Teaches. ICML 2025.
• Qwen2.5-Math Technical Report. arXiv 2024.
• Qwen2.5-Coder Technical Report. arXiv 2024.
• Baichuan-M2: Scaling Medical Capability with Large Verifier System. arXiv 2025.

5.2 Training Pipeline Optimization

5.2.1 Mid Training

• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. arXiv 2024.
• Phi-4 Technical Report. arXiv 2024.
• Instruction Pre-Training: Language Models Are Supervised Multitask Learners. EMNLP 2024.
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv 2024.
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. arXiv 2025.
• OctoThinker: Mid-Training Incentivizes Reinforcement Learning Scaling. ICLR 2025.
• RA3: Learning to Reason as Action Abstractions with Scalable Mid-Training RL. ICLR 2025.

5.2.2 Multi-Stage Training

• Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv 2025.
• Does the Order Matter? Curriculum Learning Over Languages. LREC-COLING 2024.
• Dump: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-Training. ICLR 2025.
• InsCL: A Data-Efficient Continual Learning Paradigm for Fine-Tuning Large Language Models with Instructions. NAACL 2024.
• How abilities in large language models are affected by supervised fine-tuning data composition. ACL 2024.
• SDFT: Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning. ACL 2024.
• Exploring Forgetting in Large Language Model Pre-Training. ACL 2025.

5.3 Capability Alignment Strategies

5.3.1 Constraint

• SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024.
• KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024.
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2025.
• Function Vectors: Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting. ICLR 2025.
• Spurious Forgetting in Continual Learning of Language Models. ICLR 2025.

5.3.2 Cognitive Alignment

• CDS: Data Synthesis Method Guided by Cognitive Diagnosis Theory. ACL 2025 Findings.
• CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task. EMNLP 2025 Findings.
• Cognitive Behaviors That Enable Self-Improving Reasoners. CoLM 2025.

5.4 Truthfulness & Consistency

5.4.1 Consistency

• Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
• Universal Self-Consistency for Large Language Model Generation. ICLR 2024.
• Multiagent Debate: Improving Factuality and Reasoning in Language Models. ICML 2024.
• Generative Verifiers: Reward Modeling as Next-Token Prediction. ICLR 2025.
• MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration. ACL 2025.
• Self-Certainty: Scalable Best-of-N Selection for Large Language Models via Self-Certainty. NeurIPS 2025.

5.4.2 RAG

• RAG-HAT: A Hallucination-Aware Tuning Pipeline for LLM in Retrieval-Augmented Generation. EMNLP 2024 Industry Track.
• TRAQ: Trustworthy Retrieval-Augmented Question Answering via Conformal Prediction. NAACL 2024.
• SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generation. EMNLP 2024.
• Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMs. SIGIR 2025.

5.4.3 Decoding

• DOLA: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR 2024.
• CAD: Context-Aware Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models. EACL 2024.
• ROSE: Reverse Prompt Contrastive Decoding Boosts Safety of Instruction-Tuned LLMs. ACL 2024 Findings.

5.5 Summary

Data ingestion spans both training-time scheduling and runtime information use. Training-time methods determine how supervision is organized, weighted, ordered, replayed, or conditionally presented; runtime methods determine when self-generated or external evidence is retrieved, searched, decoded, or verified.

6. Cross-Tier Synthesis

The lifecycle view exposes several recurring transitions that are less visible when datasets, synthesis, selection, training schedules, and agent systems are studied separately:

1. Static data quality → model-conditional utility. Data usefulness increasingly depends on the target model's current loss, confidence, errors, capability state, and learnability frontier.
2. Instance-level annotation → system-level specification and verification. As synthesis becomes more autonomous, human effort shifts toward task specification, verifier design, environment construction, governance, and exception handling.
3. Stored artifacts → executable data-producing environments. Agentic data increasingly includes environments that generate tasks, actions, observations, failures, recovery trajectories, and reward signals on demand.
4. Offline data preparation → adaptive runtime information acquisition. Retrieval, search, verification, and test-time compute extend data optimization beyond parameter training.

These trends motivate treating data engineering as the joint design of artifacts, transformations, and adaptive information-use policies.

7. Future Trends

7.1 Data Resources and Governance

7.1.1 Domain-Specific Data Scarcity

• CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation. TACL 2025.
• ResearchMath-14K: Scaling Research-Level Mathematics via Agents. arXiv 2026.

7.1.2 Cross-Domain Data Reutilization

• Synthetic Continued Pretraining. ICLR 2025.

7.1.3 Data Sensitivity and Privacy

• Differentially Private Synthetic Data via Foundation Model APIs 2: Text. ICML 2024.
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. NeurIPS 2025 Datasets & Benchmarks.

7.2 Data Creation and Selection

7.2.1 Agentic Data Pipelines and Closed-Loop Flywheels

• DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows. ACL 2024.
• Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis. arXiv 2025.
• Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction. arXiv 2025.

7.2.2 Multi-source Synthesis

• Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement. ACL 2025.
• CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation. TACL 2025.

7.2.3 User Collaboration

• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. ICLR 2025.

7.2.4 Cross-modal Knowledge Utilization

• TextHarmony: A Unified Architecture for Multimodal LLMs Integrating Vision, Language and Diffusion. NeurIPS 2024.

7.2.5 Data Efficiency

• LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

7.2.6 Length-Aware Data Synthesis and Selection

• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. TMLR 2025.

7.2.7 Model-Aware Filtering

• SeaPO: Strategic Error Amplification for Robust Preference Optimization. EMNLP 2025 Findings.
• AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis. arXiv 2025.

7.3 Data Ingestion Strategies

7.3.1 Unified Post-Training

• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning. arXiv 2025.

7.3.2 Cross-Stage Data Reutilization

• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. arXiv 2024.

7.3.3 Data Unlearning

• Large Language Model Unlearning. NeurIPS 2024.

7.3.4 Test-Time Strategy

• CarBoN: Calibrated Best-of-N Sampling Improves Test-Time Reasoning. arXiv 2025.
• Self-Certainty: Scalable Best-of-N Selection for Large Language Models via Self-Certainty. NeurIPS 2025.

Citation

If you use this repository or the taxonomy in your research, please cite the accompanying survey:

@article{rao2025datacentric,
  title   = {A Data-Centric Perspective on the Lifecycle of Large Language Models},
  author  = {Rao, Jun and Liu, Xuebo and Yan, Haotian and Shen, Junjie and Mo, Haosi and
             Dong, Yanghaopeng and Yan, Zihao and Wang, Ziyi and Lin, Zepeng and Meng, Xiaojun and
             Yu, Zixiong and Deng, Liqun and Wei, Jiansheng and Wang, Yunhe and Zhang, Min},
  journal = {TechRxiv},
  year    = {2025},
  doi     = {10.36227/techrxiv.176620610.03288677/v1},
  url     = {https://doi.org/10.36227/techrxiv.176620610.03288677/v1}
}

For GitHub-native citation support, we also recommend adding a CITATION.cff file to the repository root.

Contributing

Contributions are welcome. Please open an issue or pull request to add a paper, dataset, benchmark, correction, or missing link.

To keep the list consistent:

1. Use primary sources whenever possible — official paper, ACL Anthology, OpenReview, arXiv, or official project page.
2. Place resources by operation, not only by training stage or application domain.
3. Use cross-tier notes when needed. A method may appear under the tier corresponding to the operation being discussed while retaining a secondary role elsewhere.
4. Avoid duplicates. Prefer the formally published version over an arXiv duplicate when both are available.
5. Keep entries concise: Title — venue/year — link.
6. For new 2026+ work, briefly explain why it changes or extends the taxonomy when the placement is non-obvious.

Suggested pull-request title:

[Add] <Paper / Dataset / Benchmark Name> → <Section>

Acknowledgements

We thank the authors of the papers, datasets, benchmarks, and open-source resources collected here, as well as community contributors who help keep the list accurate and up to date.

---