A curated and continuously updated collection of papers, tools, and datasets on synthetic data generation using LLMs and agentic workflows.
16
28 commits
updated Aug 12, 2026
README
A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows
📄Paper •
As AI systems become increasingly data-hungry, the need for high-quality datasets has never been greater. However, real-world data collection faces major challenges: scarcity, privacy constraints, and high acquisition costs. Synthetic Data Generation (SDG) offers a compelling alternative—creating artificial data that mimics real-world patterns without the associated drawbacks. Beyond training, synthetic data also plays a crucial role in evaluation and benchmark creation, enabling controlled, diverse, and scalable testing environments that are often impractical to build with real data.
📌 This repository complements the paper A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows. It serves as a continuously updated resource, curating references on LLM-based synthetic data generation to support ongoing learning, exploration, and collaboration within the research community.
💡 If you’d like to contribute or suggest additions, feel free to open a pull request or issue!
Table of Contents
📚 Surveys
| Title | Publication Date | Overview |
|---|---|---|
| Self-Improvement of Large Language Models: A Technical Overview and Future Outlook | 03-2026 | Reviews self-improving LLMs as a closed loop of data acquisition, data selection, model optimization, and inference refinement, where models autonomously generate, evaluate, and refine their own training data. |
| A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data | 01-2026 | Surveys evaluation of LLM-generated data across six modalities, proposing an LLM Data Auditor framework that categorizes intrinsic quality and trustworthiness metrics and exposes gaps in current evaluation practice. |
| A Survey on Efficient Large Language Model Training: From Data-centric Perspectives | 10-2025 | Surveys data-efficient LLM post-training, taxonomizing data selection, quality enhancement, synthetic data generation, distillation, and self-evolving data ecosystems, and outlining open research directions. |
| Synthetic Data Generation Using Large Language Models: Advances in Text and Code | 03-2025 | Reviews recent progress in using LLMs to generate synthetic text and code, covering methods, evaluation, and challenges. |
| Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation | 02-2025 | Surveys the evolution of LLM benchmarks from static datasets to dynamic systems to better handle data contamination. |
| A Survey on Data Synthesis and Augmentation for Large Language Models | 10-2024 | Provides a broad overview of techniques and strategies for creating and augmenting data specifically for training large language models. |
| On LLMs- Driven Synthetic Data Generation, Curation, and Evaluation: A Survey | 06-2024 | Offers a comprehensive survey of the entire pipeline for using LLMs to create synthetic data, from generation to evaluation. |
| Best Practices and Lessons Learned on Synthetic Data | 04-2024 | Summarizes key strategies, best practices, and lessons learned from enterprise applications of synthetic data generation. |
| Data Augmentation Using LLMs: Data Perspectives, Learning Paradigms, and Challenges | 03-2024 | Explores how LLMs can be used to augment data, discussing various data types, learning methods, and associated difficulties. |
| Large Language Models for Data Annotation and Synthesis: A Survey | 02-2024 | Surveys the use of LLMs for data annotation and synthesis, organizing methods by annotation generation, assessment, and utilization, and analyzing how models learn from LLM-produced labels. |
| A Survey on Data Augmentation in the Large Model Era | 01-2024 | Reviews the landscape of data augmentation techniques, focusing on methods that are particularly effective for large-scale models. |
| Comprehensive Exploration of Synthetic Data Generation: A Survey | 01-2024 | Provides a thorough survey of synthetic data generation, covering its history, methods, applications, and future challenges. |
🧪 SDG Methods
🧠 Traditional Architectures: Single LLM without External Tools
| Title | Publication Date | Overview |
|---|---|---|
| Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics | 08-2026 | Introduces Question-begets-Question, a self-evolving curriculum where a teacher LLM generates variants of problems the current checkpoint can mostly solve, breaking apparent performance ceilings on competition math. |
| Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning | 07-2026 | Shows collapse in iterative instruction tuning appears as polarization of competence, then proposes KITE, combining failure-guided synthetic data generation with knowledge-boundary-aware uncertainty curation. |
| CodeAlchemy: Synthetic Code Rewriting at Scale | 06-2026 | Presents CodeAlchemy, which rewrites publicly sourced code into 500B+ synthetic tokens via five strategies spanning quality rewriting, QA, developer tasks, dialogues, and execution traces. |
| Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation | 06-2026 | Shows that activation steering of open LLMs with language-identity and quality directions improves diversity and downstream utility of synthetic classification data for 11 low-resource languages over few-shot prompting. |
| Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning | 05-2026 | Introduces DOMINO, which synthesizes domain-specific data from reference examples by learning minimal domain representations via prompt tuning with a contrastive disentanglement objective separating domain patterns from sample noise. |
| Make LLM Learn to Synthesize from Streaming Experiences through Feedback | 05-2026 | Introduces StreamSynth, a setting where synthesis tasks arrive sequentially, and SynLearner, a framework letting a model accumulate reusable experience and feedback across tasks to improve later synthetic data generation. |
| Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection | 05-2026 | Proposes MSIFR, a training-free framework that terminates low-quality generation trajectories at intermediate checkpoints using fast rule-based validators, cutting token consumption while preserving accuracy in LLM synthetic data generation. |
| WRAP++: Web discoveRy Amplified Pretraining | 04-2026 | Proposes WRAP++, which discovers cross-document relationships from web hyperlinks and synthesizes joint QA over document pairs, amplifying ~8.4B Wikipedia tokens into 80B relational pretraining tokens. |
| How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data | 04-2026 | Systematically studies prompt design, generator model, and source data across ~90 experiments, deriving a recipe for generating trillions of synthetic pretraining tokens by rewriting web documents with a single small LLM (companion FinePhrase Space). |
| Data-efficient pre-training by scaling synthetic megadocs | 03-2026 | Shows that stitching or stretching multiple synthetic rephrases of a web document into long "megadocs" improves loss scaling and data efficiency by up to 1.8x in data-constrained pretraining. |
| Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts | 02-2026 | Introduces Persona Generators, which evolve persona-generating code with LLM mutation operators to produce diverse synthetic populations maximizing coverage of opinions and rare trait combinations. |
| FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale | 01-2026 | Proposes FineInstructions, which transforms internet-scale pretraining documents into billions of synthetic instruction–answer pairs via ~18M instruction templates, enabling pretraining from scratch with an instruction-tuning objective. |
| MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy | 08-2025 | Synthesizes hard mathematical problems from scratch using PlanetMath concept-explanation pairs and a reinforcement-learning policy that optimizes problem validity, complexity, and reasoning consistency. |
| CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks | 07-2025 | Proposes CoT-Self-Instruct, which prompts an LLM to reason via Chain-of-Thought over seed tasks before generating new synthetic examples, then filters them with automatic quality metrics for training. |
| Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models | 06-2025 | Proposes REWIRE, a guided-rewriting method that transforms low-quality discarded web documents into useful synthetic pre-training data, improving downstream performance over training on filtered web data alone. |
| NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities | 05-2025 | Proposes a methodology to generate synthetic and retrieval-based pre-training data tailored to a community's language, cultural heritage, and values, demonstrated by building NileChat for Egyptian and Moroccan Arabic. |
| A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment | 05-2025 | Introduces a modular framework adapting small language models into clinical experts via pre-instruction tuning, model merging, and alignment, producing the MediFlow synthetic dataset of 2.5 million medical instructions. |
| O1 Replication Journey – Part 2: Surpassing O1-preview through Simple Distillation Big Progress or Bitter Lesson? | 11-2024 | A base model can outperform O1-preview on mathematical reasoning through simple knowledge distillation from O1's API. |
| Self-Judge: Selective Instruction Following with Alignment Self-Evaluation | 09-2024 | Introduces a framework for a large language model to self-evaluate and decide whether to respond to a given prompt. |
| Automated test generation to evaluate tool-augmented LLMs as conversational AI agents | 09-2024 | Presents a method for automatically generating tests to evaluate how well tool-augmented LLMs perform as conversational agents. |
| Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources | 09-2024 | Introduces a scalable pipeline that generates synthetic examples with intermediate reasoning grounded in real data sources, then curates them by discarding low-quality generations to raise dataset quality. |
| Is Child-Directed Speech Effective Training Data for Language Models? | 08-2024 | Investigates the effectiveness of training language models on child-directed speech for better language acquisition. |
| Case2Code: Learning Inductive Reasoning with Synthetic Data | 07-2024 | Teaches large language models inductive reasoning by training them on synthetic input-output examples and their corresponding code. |
| Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability | 07-2024 | Improves the cross-lingual abilities of large language models by using their own translation capabilities to generate training data. |
| Scaling Synthetic Data Creation with 1,000,000,000 Personas | 06-2024 | Presents a method for generating vast amounts of diverse synthetic data by prompting a large language model with a billion different personas. |
| Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing | 06-2024 | Proposes generating large-scale alignment data by feeding simple pre-query templates to an already aligned model. |
| ToolCoder: Teach Code Generation Models to use API search tools | 05-2024 | Teaches code-generating models to use API search tools for finding relevant APIs for specific tasks. |
| DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data | 05-2024 | Proposes a framework and a large synthetic dataset to improve theorem-proving abilities of large language models in natural language. |
| Phi-3 Technical Report | 04-2024 | Details the architecture, training, and capabilities of the Phi-3 family of small, powerful, open-source multimodal models. |
| Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks | 04-2024 | Investigates how well ChatGPT can replicate human-generated data labels for various social computing tasks. |
| OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset | 02-2024 | Introduces a dataset of 1.8 million math problems and solutions to improve the mathematical reasoning of open-source language models. |
| Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling | 01-2024 | Presents a method for more efficient language model training by rephrasing web documents into different styles like Wikipedia articles. |
| Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models | 01-2024 | Shows that a language model can improve its reasoning and problem-solving skills by "playing" against itself to solve problems. |
| Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models | 12-2023 | Explores a self-training method where a model generates and filters its own training data to improve problem-solving skills. |
| Orca 2: Teaching Small Language Models How to Reason | 11-2023 | Presents a method to teach smaller language models different reasoning strategies for various tasks, enhancing their overall capabilities. |
| Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models | 11-2023 | Examines using large language models and techniques like retrieval-augmented generation to create structured medical text data. |
| HELPSTEER: Multi-attribute Helpfulness Dataset for STEERLM | 11-2023 | Introduces a dataset annotating model response helpfulness across multiple attributes (e.g., correctness, coherence) to improve training. |
| CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation | 10-2023 | Uses a large language model's uncertainty to decide whether a human or the model should handle data annotation for greater efficiency. |
| ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback | 10-2023 | Explores using large-scale AI-generated feedback to align language models with human preferences and presents a corresponding dataset. |
| Textbooks Are All You Need II: phi-1.5 technical report | 09-2023 | Describes phi-1.5, a 1.3B parameter model trained on high-quality "textbook" data for common sense reasoning. |
| MAMMOTH: BUILDING MATH GENERALIST MODELS THROUGH HYBRID INSTRUCTION TUNING | 09-2023 | Introduces open-source language models trained on a diverse set of math problems to improve general math problem-solving skills. |
| METAMATH: BOOTSTRAP YOUR OWN MATHEMATICAL QUESTIONS FOR LARGE LANGUAGE | 09-2023 | Improves mathematical reasoning in language models by rewriting existing math questions from various perspectives for a more diverse training set. |
| DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation | 08-2023 | Presents a language model trained on high-quality medical data for more accurate and reliable healthcare-related conversations. |
| AlpaGasus: Training A Better Alpaca with Fewer Data | 07-2023 | Shows that filtering low-quality examples from the Alpaca dataset allows for training a better model with significantly less data. |
| BEAVERTAILS: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset | 07-2023 | Introduces a dataset that separately annotates helpfulness and harmlessness of model responses for better safety alignment. |
| WizardCoder: Empowering Code Large Language Models with Evol-Instruct | 06-2023 | Introduces Evol-Instruct, a method for generating complex instruction data to train more capable code-generating language models. |
| Textbooks Are All You Need | 06-2023 | Argues that training language models on high-quality, "textbook-style" data yields better performance than training on unfiltered web data. |
| Orca: Progressive Learning from Complex Explanation Traces of GPT-4 | 06-2023 | Introduces Orca, a model that learns to imitate the reasoning process of larger models like GPT-4 by training on their detailed explanation traces. |
| TinyStories: how Small Can Language Models Be and Still Speak Coherent English? | 05-2023 | Explores the minimum size for language models to generate coherent English by training them on synthetic children's stories. |
| WizardLM: Empowering Large Language Models to Follow Complex Instructions | 04-2023 | Creates large amounts of instruction-following data with varying complexity to improve how language models follow complex instructions. |
| Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data | 04-2023 | Creates a multi-turn chat dataset by having ChatGPT converse with itself, then uses this data to train an open-source chat model. |
| HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge | 04-2023 | Details fine-tuning the LLaMA model with a large Chinese medical dataset to create a specialized medical language model. |
| Instruction Tuning with GPT-4 | 04-2023 | Demonstrates using GPT-4 to generate a large dataset of instruction-following examples for fine-tuning other language models. |
| Alpaca: A Strong, Replicable Instruction-Following Model | 03-2023 | Presents a language model fine-tuned on 52,000 instruction-following demonstrations generated by a larger model. |
| CHATGPT OUTPERFORMS CROWD WORKERS FOR TEXT-ANNOTATION TASKS | 03-2023 | Shows that for several text annotation tasks, ChatGPT performs better and is more cost-effective than human crowd workers. |
| AugGPT: Leveraging ChatGPT for Text Data Augmentation | 02-2023 | Proposes using ChatGPT to rephrase training sentences to create a more diverse and larger dataset for model training. |
| SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions | 12-2022 | Introduces a method for a language model to generate its own instruction-following data for self-improvement through fine-tuning. |
| DISCO: Distilling Counterfactuals with Large Language Models | 12-2022 | Proposes generating high-quality, large-scale counterfactual data by using a syntactic parser to guide a large language model. |
| CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation | 10-2022 | Presents a "retrieve-then-edit" framework where a model finds and modifies text to create counterfactual examples. |
| STaR: Self-Taught Reasoner: Bootstrapping Reasoning With Reasoning | 03-2022 | Introduces a method for a language model to learn reasoning by generating its own rationales for problems and then fine-tuning on the correct ones. |
🤖 Agentic Workflows
| Title | Publication Date | Overview |
|---|---|---|
| State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs | 06-2026 | Introduces StateGen, which orchestrates a persona-conditioned user simulator, an agent, a state-grounded tool simulator, and a multi-axis LLM judge to generate scored multi-turn tool-use training conversations. |
| ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories | 06-2026 | Proposes ISE, which builds ~44K structured user intents, simulates role-locked multi-turn interactions, and executes every tool call in isolated OS environments to capture authentic failure-recovery trajectories. |
| BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution | 06-2026 | Introduces BenchEvolver, an evolutionary framework that transforms existing coding problems into harder verifiable variants by evolving reference solutions, yielding LiveCodeBench-Plus and reinforcement-learning training gains. |
| Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains | 05-2026 | Introduces K2V, which extends RLVR to knowledge-intensive domains via automated verifiable data synthesis from knowledge graphs and reasoning-process verification that rewards valid reasoning, not only correct final answers. |
| CausalSynth: Generating Structurally Sound Synthetic Data | 05-2026 | Proposes CausalSynth, which generates causal skeletons from a structural causal model, uses an LLM as a constrained realizer, and applies iterative consistency verification with closed-loop corrections to produce causally valid synthetic data. |
| Concordia: Self-Improving Synthetic Tables for Federated LLMs | 05-2026 | Concordia adapts federated LLMs on tabular tasks via a tri-level self-improving loop that trains LoRA on synthetic tables, reweights samples with utility scorers, and refines generators using group-relative policy optimization. |
| Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning | 04-2026 | Introduces COVERT, a two-stage pipeline producing verified tool-use trajectories via self-evolving synthesis with multi-level validation, then oracle-preserving augmentations that build RL-ready synthetic tool-use environments. |
| From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents | 01-2026 | Proposes EigenData, a self-evolving hierarchical multi-agent engine that synthesizes tool-grounded dialogues with executable per-instance checkers, feeding verifier-based reinforcement post-training of tool-using agents. |
| LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls | 11-2025 | LoopTool closes the data-training loop with three modules that probe model capability, verify labels, and expand error-driven samples, letting an 8B model trained on its synthetic tool-call data surpass its 32B generator. |
| Fara-7B: An Efficient Agentic Model for Computer Use | 11-2025 | Fara-7B is a screenshot-based on-device computer-use agent trained on trajectories produced by FaraGen, a synthetic data generation system for web tasks, performing competitively against larger systems on web benchmarks. |
| Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework | 11-2025 | Presents Matrix, a decentralized peer-to-peer multi-agent framework that scales tens of thousands of concurrent agentic workflows to generate dialogue, web-reasoning, and tool-use trajectory data. |
| Benchmarking Agentic Workflow Generation | 10-2024 | Introduces WORFBENCH, a comprehensive benchmark designed to evaluate how well large language model (LLM) agents can generate workflows—that is, structured plans of subtasks needed to solve complex problems. |
| BenchAgents: Automated Benchmark Creation With Agent Interaction | 10-2024 | Introduces a framework where AI agents interact with each other to automatically create new and challenging benchmarks. |
| ToolACE: Winning the Points of LLM Function Calling | 09-2024 | Introduces an automatic agentic pipeline that uses a self-evolution synthesis process and multi-agent interaction to generate accurate, complex, and diverse function-calling (tool-use) training data. |
| The Fellowship of the LLMs: Multi-Agent Workflows for Synthetic Preference Optimization Dataset Generation | 08-2024 | Uses a team of AI agents, each with a specific role, to generate high-quality datasets for training and aligning language models. |
| AgentInstruct: Toward Generative Teaching with Agentic Flows | 07-2024 | Introduces a framework to generate synthetic data for instruction-tuning. Instead of relying on human-curated prompts or limited seed sets, they use a network of AI agents, combined with tools and reflection loops—to turn raw content (e.g., documents, code) into 25 million prompt–response pairs. |
| Arena Learning: Build Data Flywheel for LLMs Post-Training via Simulated Chatbot Arena | 07-2024 | Proposes a simulated "arena" where chatbots compete, generating preference data that is used to continuously improve the models. |
| MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data | 06-2024 | Leverages a multi-agent system within a GAN framework to improve the generation of realistic synthetic tabular data. |
| Advancing LLM Reasoning Generalists With Preference Trees | 04-2024 | Improves a model's reasoning skills by having it explore and evaluate multiple reasoning paths, forming a "preference tree" of the best steps. |
| LAB: Large-Scale Alignment for Chatbots | 03-2024 | Introduces a large-scale dataset of human-AI conversations and a new method to more effectively align chatbots with human preferences. |
| Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation | 02-2024 | Proposes a framework where multiple AI agents work together to continuously update and evolve evaluation benchmarks for LLMs. |
| Synthetic data (almost) from scratch: Generalized instruction tuning for language models | 02-2024 | Uses a small set of human-written examples to bootstrap the generation of a large, diverse synthetic dataset for instruction tuning. |
| Orca-math: Unlocking the potential of SLMs in grade school math | 02-2024 | Creates a high-quality synthetic dataset of math problems to significantly improve the mathematical reasoning of smaller language models (SLMs). |
| Learning From Mistakes Makes LLM a Better Reasoner | 10-2023 | Improves a model's reasoning abilities by training it on data that includes common mistakes and provides corrective feedback. |
📖 Further Reading
| Title | Publication Date | Overview |
|---|---|---|
| The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data | 08-2026 | Shows that recursive training on self-generated data amplifies demographic bias—"fairness collapse"—with fairness degrading before standard language-modeling metrics reveal model collapse. |
| Fidelity-Diversity Metrics for Text | 07-2026 | Derives optimal-transport-based fidelity and diversity metrics for text that disentangle the two deficits, detecting diversity gaps in synthetic data that correlate with downstream fine-tuning degradation. |
| When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling | 07-2026 | Disentangles fixed-source synthesis from source expansion in synthetic-data scaling, showing per-question response budgets are a bounded axis while expanding seed questions wins at large budgets. |
| Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data | 06-2026 | Presents a model-agnostic statistical auditing framework distinguishing true from "phantom" disclosures in LLM-generated synthetic data, yielding tighter empirical privacy-leakage bounds without model access or shadow models. |
| An Information-Theoretic Criterion for Efficient Data Synthesis | 05-2026 | Provides an information-theoretic account showing synthetic data only improves models when the generation-training loop is information-open via external signals, otherwise the data processing inequality predicts collapse. |
| Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data | 05-2026 | Introduces a lightweight pre-pre-training stage on synthetic data with learnable temporal structure that improves LLM robustness to noisy pre-training data, cutting natural-text tokens by up to 49%. |
| Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning | 11-2025 | Shows fine-tuning on multi-source synthetic data mitigates distribution collapse and reduces self-preference bias, while synthetic data erodes safety safeguards despite higher output quality than human data. |
| Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls | 10-2025 | Conducts a large-scale empirical study (>1000 LLMs) deriving scaling laws for synthetic pre-training data, finding rephrased-text mixtures accelerate training while pure textbook-style data can trigger model collapse. |
| BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining | 08-2025 | Distills lessons from scaling synthetic data for LLM pretraining, showing that jointly optimizing many generation factors yields data that outperforms prior synthetic corpora like Cosmopedia and Nemotron-Synth. |
| Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning | 05-2025 | Introduces G-Vendi, a metric measuring training-data diversity via the entropy of model-induced gradients that strongly correlates with out-of-distribution reasoning performance, and uses it to guide synthetic data generation. |
| Scaling Laws of Synthetic Data for Language Models | 03-2025 | Presents SynthLLM, a framework revealing that synthetic pre-training data follows power-law scaling up to 300B tokens, and larger LLMs require fewer synthetic tokens to reach optimal performance |
| Evaluating Language Models as Synthetic Data Generators | 12-2024 | a benchmark that evaluates LLMs’ abilities to generate synthetic data by comparing outputs from multiple models and analyzing quality metrics (e.g., perplexity, difficulty), revealing that data-generation prowess doesn’t always match problem-solving strength |
| On the Diversity of Synthetic Data and its Impact on Training Large Language Models | 10-2024 | Introduces a diversity metric (“LLM cluster-agent”) to quantify synthetic data variety, demonstrating that data diversity boosts model performance—especially during fine-tuning—even for smaller-scale LLMs |
🔗 Related Repositories
| Awesome Synthetic Datasets | Practical resources for building synthetic text and vision datasets. |
|---|---|
| LLM Synthetic Data | Papers, tools, and blogs on LLM-generated data. |
| LLM-Datasets: | Curated datasets and tools for LLM post-training. |
| DataArc-SynData-Toolkit | Open-source, configuration-driven framework (visual UI + CLI) for multi-source, multimodal, and multilingual synthetic data generation via a modular, quality-controllable closed-loop pipeline. |
| GraphGen | Knowledge-graph-driven framework that builds fine-grained graphs from source text, locates an LLM's knowledge gaps via calibration error, and generates targeted QA, chain-of-thought, and multi-hop data for supervised fine-tuning. |
📝 Citation
If this work is helpful, please kindly cite as:
@inproceedings{alismail2025survey,
title={A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows},
author={Alismail, Ahmad and Lanquillon, Carsten},
booktitle={International Conference on Human-Computer Interaction},
pages={119--135},
year={2025},
organization={Springer}
}
agentic-workflow
llms
synthetic-data
synthetic-dataset-generation
Contributors
ahmad-alismail
28 commits
Similar repos
A curated and continuously updated collection of papers, tools, and datasets on synthetic data generation using LLMs and agentic workflows.
16
28 commits
updated Aug 12, 2026
README
A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows
📄Paper •
As AI systems become increasingly data-hungry, the need for high-quality datasets has never been greater. However, real-world data collection faces major challenges: scarcity, privacy constraints, and high acquisition costs. Synthetic Data Generation (SDG) offers a compelling alternative—creating artificial data that mimics real-world patterns without the associated drawbacks. Beyond training, synthetic data also plays a crucial role in evaluation and benchmark creation, enabling controlled, diverse, and scalable testing environments that are often impractical to build with real data.
📌 This repository complements the paper A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows. It serves as a continuously updated resource, curating references on LLM-based synthetic data generation to support ongoing learning, exploration, and collaboration within the research community.
💡 If you’d like to contribute or suggest additions, feel free to open a pull request or issue!
Table of Contents
📚 Surveys
| Title | Publication Date | Overview |
|---|---|---|
| Self-Improvement of Large Language Models: A Technical Overview and Future Outlook | 03-2026 | Reviews self-improving LLMs as a closed loop of data acquisition, data selection, model optimization, and inference refinement, where models autonomously generate, evaluate, and refine their own training data. |
| A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data | 01-2026 | Surveys evaluation of LLM-generated data across six modalities, proposing an LLM Data Auditor framework that categorizes intrinsic quality and trustworthiness metrics and exposes gaps in current evaluation practice. |
| A Survey on Efficient Large Language Model Training: From Data-centric Perspectives | 10-2025 | Surveys data-efficient LLM post-training, taxonomizing data selection, quality enhancement, synthetic data generation, distillation, and self-evolving data ecosystems, and outlining open research directions. |
| Synthetic Data Generation Using Large Language Models: Advances in Text and Code | 03-2025 | Reviews recent progress in using LLMs to generate synthetic text and code, covering methods, evaluation, and challenges. |
| Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation | 02-2025 | Surveys the evolution of LLM benchmarks from static datasets to dynamic systems to better handle data contamination. |
| A Survey on Data Synthesis and Augmentation for Large Language Models | 10-2024 | Provides a broad overview of techniques and strategies for creating and augmenting data specifically for training large language models. |
| On LLMs- Driven Synthetic Data Generation, Curation, and Evaluation: A Survey | 06-2024 | Offers a comprehensive survey of the entire pipeline for using LLMs to create synthetic data, from generation to evaluation. |
| Best Practices and Lessons Learned on Synthetic Data | 04-2024 | Summarizes key strategies, best practices, and lessons learned from enterprise applications of synthetic data generation. |
| Data Augmentation Using LLMs: Data Perspectives, Learning Paradigms, and Challenges | 03-2024 | Explores how LLMs can be used to augment data, discussing various data types, learning methods, and associated difficulties. |
| Large Language Models for Data Annotation and Synthesis: A Survey | 02-2024 | Surveys the use of LLMs for data annotation and synthesis, organizing methods by annotation generation, assessment, and utilization, and analyzing how models learn from LLM-produced labels. |
| A Survey on Data Augmentation in the Large Model Era | 01-2024 | Reviews the landscape of data augmentation techniques, focusing on methods that are particularly effective for large-scale models. |
| Comprehensive Exploration of Synthetic Data Generation: A Survey | 01-2024 | Provides a thorough survey of synthetic data generation, covering its history, methods, applications, and future challenges. |
🧪 SDG Methods
🧠 Traditional Architectures: Single LLM without External Tools
| Title | Publication Date | Overview |
|---|---|---|
| Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics | 08-2026 | Introduces Question-begets-Question, a self-evolving curriculum where a teacher LLM generates variants of problems the current checkpoint can mostly solve, breaking apparent performance ceilings on competition math. |
| Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning | 07-2026 | Shows collapse in iterative instruction tuning appears as polarization of competence, then proposes KITE, combining failure-guided synthetic data generation with knowledge-boundary-aware uncertainty curation. |
| CodeAlchemy: Synthetic Code Rewriting at Scale | 06-2026 | Presents CodeAlchemy, which rewrites publicly sourced code into 500B+ synthetic tokens via five strategies spanning quality rewriting, QA, developer tasks, dialogues, and execution traces. |
| Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation | 06-2026 | Shows that activation steering of open LLMs with language-identity and quality directions improves diversity and downstream utility of synthetic classification data for 11 low-resource languages over few-shot prompting. |
| Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning | 05-2026 | Introduces DOMINO, which synthesizes domain-specific data from reference examples by learning minimal domain representations via prompt tuning with a contrastive disentanglement objective separating domain patterns from sample noise. |
| Make LLM Learn to Synthesize from Streaming Experiences through Feedback | 05-2026 | Introduces StreamSynth, a setting where synthesis tasks arrive sequentially, and SynLearner, a framework letting a model accumulate reusable experience and feedback across tasks to improve later synthetic data generation. |
| Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection | 05-2026 | Proposes MSIFR, a training-free framework that terminates low-quality generation trajectories at intermediate checkpoints using fast rule-based validators, cutting token consumption while preserving accuracy in LLM synthetic data generation. |
| WRAP++: Web discoveRy Amplified Pretraining | 04-2026 | Proposes WRAP++, which discovers cross-document relationships from web hyperlinks and synthesizes joint QA over document pairs, amplifying ~8.4B Wikipedia tokens into 80B relational pretraining tokens. |
| How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data | 04-2026 | Systematically studies prompt design, generator model, and source data across ~90 experiments, deriving a recipe for generating trillions of synthetic pretraining tokens by rewriting web documents with a single small LLM (companion FinePhrase Space). |
| Data-efficient pre-training by scaling synthetic megadocs | 03-2026 | Shows that stitching or stretching multiple synthetic rephrases of a web document into long "megadocs" improves loss scaling and data efficiency by up to 1.8x in data-constrained pretraining. |
| Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts | 02-2026 | Introduces Persona Generators, which evolve persona-generating code with LLM mutation operators to produce diverse synthetic populations maximizing coverage of opinions and rare trait combinations. |
| FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale | 01-2026 | Proposes FineInstructions, which transforms internet-scale pretraining documents into billions of synthetic instruction–answer pairs via ~18M instruction templates, enabling pretraining from scratch with an instruction-tuning objective. |
| MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy | 08-2025 | Synthesizes hard mathematical problems from scratch using PlanetMath concept-explanation pairs and a reinforcement-learning policy that optimizes problem validity, complexity, and reasoning consistency. |
| CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks | 07-2025 | Proposes CoT-Self-Instruct, which prompts an LLM to reason via Chain-of-Thought over seed tasks before generating new synthetic examples, then filters them with automatic quality metrics for training. |
| Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models | 06-2025 | Proposes REWIRE, a guided-rewriting method that transforms low-quality discarded web documents into useful synthetic pre-training data, improving downstream performance over training on filtered web data alone. |
| NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities | 05-2025 | Proposes a methodology to generate synthetic and retrieval-based pre-training data tailored to a community's language, cultural heritage, and values, demonstrated by building NileChat for Egyptian and Moroccan Arabic. |
| A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment | 05-2025 | Introduces a modular framework adapting small language models into clinical experts via pre-instruction tuning, model merging, and alignment, producing the MediFlow synthetic dataset of 2.5 million medical instructions. |
| O1 Replication Journey – Part 2: Surpassing O1-preview through Simple Distillation Big Progress or Bitter Lesson? | 11-2024 | A base model can outperform O1-preview on mathematical reasoning through simple knowledge distillation from O1's API. |
| Self-Judge: Selective Instruction Following with Alignment Self-Evaluation | 09-2024 | Introduces a framework for a large language model to self-evaluate and decide whether to respond to a given prompt. |
| Automated test generation to evaluate tool-augmented LLMs as conversational AI agents | 09-2024 | Presents a method for automatically generating tests to evaluate how well tool-augmented LLMs perform as conversational agents. |
| Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources | 09-2024 | Introduces a scalable pipeline that generates synthetic examples with intermediate reasoning grounded in real data sources, then curates them by discarding low-quality generations to raise dataset quality. |
| Is Child-Directed Speech Effective Training Data for Language Models? | 08-2024 | Investigates the effectiveness of training language models on child-directed speech for better language acquisition. |
| Case2Code: Learning Inductive Reasoning with Synthetic Data | 07-2024 | Teaches large language models inductive reasoning by training them on synthetic input-output examples and their corresponding code. |
| Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability | 07-2024 | Improves the cross-lingual abilities of large language models by using their own translation capabilities to generate training data. |
| Scaling Synthetic Data Creation with 1,000,000,000 Personas | 06-2024 | Presents a method for generating vast amounts of diverse synthetic data by prompting a large language model with a billion different personas. |
| Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing | 06-2024 | Proposes generating large-scale alignment data by feeding simple pre-query templates to an already aligned model. |
| ToolCoder: Teach Code Generation Models to use API search tools | 05-2024 | Teaches code-generating models to use API search tools for finding relevant APIs for specific tasks. |
| DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data | 05-2024 | Proposes a framework and a large synthetic dataset to improve theorem-proving abilities of large language models in natural language. |
| Phi-3 Technical Report | 04-2024 | Details the architecture, training, and capabilities of the Phi-3 family of small, powerful, open-source multimodal models. |
| Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks | 04-2024 | Investigates how well ChatGPT can replicate human-generated data labels for various social computing tasks. |
| OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset | 02-2024 | Introduces a dataset of 1.8 million math problems and solutions to improve the mathematical reasoning of open-source language models. |
| Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling | 01-2024 | Presents a method for more efficient language model training by rephrasing web documents into different styles like Wikipedia articles. |
| Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models | 01-2024 | Shows that a language model can improve its reasoning and problem-solving skills by "playing" against itself to solve problems. |
| Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models | 12-2023 | Explores a self-training method where a model generates and filters its own training data to improve problem-solving skills. |
| Orca 2: Teaching Small Language Models How to Reason | 11-2023 | Presents a method to teach smaller language models different reasoning strategies for various tasks, enhancing their overall capabilities. |
| Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models | 11-2023 | Examines using large language models and techniques like retrieval-augmented generation to create structured medical text data. |
| HELPSTEER: Multi-attribute Helpfulness Dataset for STEERLM | 11-2023 | Introduces a dataset annotating model response helpfulness across multiple attributes (e.g., correctness, coherence) to improve training. |
| CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation | 10-2023 | Uses a large language model's uncertainty to decide whether a human or the model should handle data annotation for greater efficiency. |
| ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback | 10-2023 | Explores using large-scale AI-generated feedback to align language models with human preferences and presents a corresponding dataset. |
| Textbooks Are All You Need II: phi-1.5 technical report | 09-2023 | Describes phi-1.5, a 1.3B parameter model trained on high-quality "textbook" data for common sense reasoning. |
| MAMMOTH: BUILDING MATH GENERALIST MODELS THROUGH HYBRID INSTRUCTION TUNING | 09-2023 | Introduces open-source language models trained on a diverse set of math problems to improve general math problem-solving skills. |
| METAMATH: BOOTSTRAP YOUR OWN MATHEMATICAL QUESTIONS FOR LARGE LANGUAGE | 09-2023 | Improves mathematical reasoning in language models by rewriting existing math questions from various perspectives for a more diverse training set. |
| DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation | 08-2023 | Presents a language model trained on high-quality medical data for more accurate and reliable healthcare-related conversations. |
| AlpaGasus: Training A Better Alpaca with Fewer Data | 07-2023 | Shows that filtering low-quality examples from the Alpaca dataset allows for training a better model with significantly less data. |
| BEAVERTAILS: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset | 07-2023 | Introduces a dataset that separately annotates helpfulness and harmlessness of model responses for better safety alignment. |
| WizardCoder: Empowering Code Large Language Models with Evol-Instruct | 06-2023 | Introduces Evol-Instruct, a method for generating complex instruction data to train more capable code-generating language models. |
| Textbooks Are All You Need | 06-2023 | Argues that training language models on high-quality, "textbook-style" data yields better performance than training on unfiltered web data. |
| Orca: Progressive Learning from Complex Explanation Traces of GPT-4 | 06-2023 | Introduces Orca, a model that learns to imitate the reasoning process of larger models like GPT-4 by training on their detailed explanation traces. |
| TinyStories: how Small Can Language Models Be and Still Speak Coherent English? | 05-2023 | Explores the minimum size for language models to generate coherent English by training them on synthetic children's stories. |
| WizardLM: Empowering Large Language Models to Follow Complex Instructions | 04-2023 | Creates large amounts of instruction-following data with varying complexity to improve how language models follow complex instructions. |
| Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data | 04-2023 | Creates a multi-turn chat dataset by having ChatGPT converse with itself, then uses this data to train an open-source chat model. |
| HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge | 04-2023 | Details fine-tuning the LLaMA model with a large Chinese medical dataset to create a specialized medical language model. |
| Instruction Tuning with GPT-4 | 04-2023 | Demonstrates using GPT-4 to generate a large dataset of instruction-following examples for fine-tuning other language models. |
| Alpaca: A Strong, Replicable Instruction-Following Model | 03-2023 | Presents a language model fine-tuned on 52,000 instruction-following demonstrations generated by a larger model. |
| CHATGPT OUTPERFORMS CROWD WORKERS FOR TEXT-ANNOTATION TASKS | 03-2023 | Shows that for several text annotation tasks, ChatGPT performs better and is more cost-effective than human crowd workers. |
| AugGPT: Leveraging ChatGPT for Text Data Augmentation | 02-2023 | Proposes using ChatGPT to rephrase training sentences to create a more diverse and larger dataset for model training. |
| SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions | 12-2022 | Introduces a method for a language model to generate its own instruction-following data for self-improvement through fine-tuning. |
| DISCO: Distilling Counterfactuals with Large Language Models | 12-2022 | Proposes generating high-quality, large-scale counterfactual data by using a syntactic parser to guide a large language model. |
| CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation | 10-2022 | Presents a "retrieve-then-edit" framework where a model finds and modifies text to create counterfactual examples. |
| STaR: Self-Taught Reasoner: Bootstrapping Reasoning With Reasoning | 03-2022 | Introduces a method for a language model to learn reasoning by generating its own rationales for problems and then fine-tuning on the correct ones. |
🤖 Agentic Workflows
| Title | Publication Date | Overview |
|---|---|---|
| State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs | 06-2026 | Introduces StateGen, which orchestrates a persona-conditioned user simulator, an agent, a state-grounded tool simulator, and a multi-axis LLM judge to generate scored multi-turn tool-use training conversations. |
| ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories | 06-2026 | Proposes ISE, which builds ~44K structured user intents, simulates role-locked multi-turn interactions, and executes every tool call in isolated OS environments to capture authentic failure-recovery trajectories. |
| BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution | 06-2026 | Introduces BenchEvolver, an evolutionary framework that transforms existing coding problems into harder verifiable variants by evolving reference solutions, yielding LiveCodeBench-Plus and reinforcement-learning training gains. |
| Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains | 05-2026 | Introduces K2V, which extends RLVR to knowledge-intensive domains via automated verifiable data synthesis from knowledge graphs and reasoning-process verification that rewards valid reasoning, not only correct final answers. |
| CausalSynth: Generating Structurally Sound Synthetic Data | 05-2026 | Proposes CausalSynth, which generates causal skeletons from a structural causal model, uses an LLM as a constrained realizer, and applies iterative consistency verification with closed-loop corrections to produce causally valid synthetic data. |
| Concordia: Self-Improving Synthetic Tables for Federated LLMs | 05-2026 | Concordia adapts federated LLMs on tabular tasks via a tri-level self-improving loop that trains LoRA on synthetic tables, reweights samples with utility scorers, and refines generators using group-relative policy optimization. |
| Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning | 04-2026 | Introduces COVERT, a two-stage pipeline producing verified tool-use trajectories via self-evolving synthesis with multi-level validation, then oracle-preserving augmentations that build RL-ready synthetic tool-use environments. |
| From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents | 01-2026 | Proposes EigenData, a self-evolving hierarchical multi-agent engine that synthesizes tool-grounded dialogues with executable per-instance checkers, feeding verifier-based reinforcement post-training of tool-using agents. |
| LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls | 11-2025 | LoopTool closes the data-training loop with three modules that probe model capability, verify labels, and expand error-driven samples, letting an 8B model trained on its synthetic tool-call data surpass its 32B generator. |
| Fara-7B: An Efficient Agentic Model for Computer Use | 11-2025 | Fara-7B is a screenshot-based on-device computer-use agent trained on trajectories produced by FaraGen, a synthetic data generation system for web tasks, performing competitively against larger systems on web benchmarks. |
| Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework | 11-2025 | Presents Matrix, a decentralized peer-to-peer multi-agent framework that scales tens of thousands of concurrent agentic workflows to generate dialogue, web-reasoning, and tool-use trajectory data. |
| Benchmarking Agentic Workflow Generation | 10-2024 | Introduces WORFBENCH, a comprehensive benchmark designed to evaluate how well large language model (LLM) agents can generate workflows—that is, structured plans of subtasks needed to solve complex problems. |
| BenchAgents: Automated Benchmark Creation With Agent Interaction | 10-2024 | Introduces a framework where AI agents interact with each other to automatically create new and challenging benchmarks. |
| ToolACE: Winning the Points of LLM Function Calling | 09-2024 | Introduces an automatic agentic pipeline that uses a self-evolution synthesis process and multi-agent interaction to generate accurate, complex, and diverse function-calling (tool-use) training data. |
| The Fellowship of the LLMs: Multi-Agent Workflows for Synthetic Preference Optimization Dataset Generation | 08-2024 | Uses a team of AI agents, each with a specific role, to generate high-quality datasets for training and aligning language models. |
| AgentInstruct: Toward Generative Teaching with Agentic Flows | 07-2024 | Introduces a framework to generate synthetic data for instruction-tuning. Instead of relying on human-curated prompts or limited seed sets, they use a network of AI agents, combined with tools and reflection loops—to turn raw content (e.g., documents, code) into 25 million prompt–response pairs. |
| Arena Learning: Build Data Flywheel for LLMs Post-Training via Simulated Chatbot Arena | 07-2024 | Proposes a simulated "arena" where chatbots compete, generating preference data that is used to continuously improve the models. |
| MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data | 06-2024 | Leverages a multi-agent system within a GAN framework to improve the generation of realistic synthetic tabular data. |
| Advancing LLM Reasoning Generalists With Preference Trees | 04-2024 | Improves a model's reasoning skills by having it explore and evaluate multiple reasoning paths, forming a "preference tree" of the best steps. |
| LAB: Large-Scale Alignment for Chatbots | 03-2024 | Introduces a large-scale dataset of human-AI conversations and a new method to more effectively align chatbots with human preferences. |
| Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation | 02-2024 | Proposes a framework where multiple AI agents work together to continuously update and evolve evaluation benchmarks for LLMs. |
| Synthetic data (almost) from scratch: Generalized instruction tuning for language models | 02-2024 | Uses a small set of human-written examples to bootstrap the generation of a large, diverse synthetic dataset for instruction tuning. |
| Orca-math: Unlocking the potential of SLMs in grade school math | 02-2024 | Creates a high-quality synthetic dataset of math problems to significantly improve the mathematical reasoning of smaller language models (SLMs). |
| Learning From Mistakes Makes LLM a Better Reasoner | 10-2023 | Improves a model's reasoning abilities by training it on data that includes common mistakes and provides corrective feedback. |
📖 Further Reading
| Title | Publication Date | Overview |
|---|---|---|
| The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data | 08-2026 | Shows that recursive training on self-generated data amplifies demographic bias—"fairness collapse"—with fairness degrading before standard language-modeling metrics reveal model collapse. |
| Fidelity-Diversity Metrics for Text | 07-2026 | Derives optimal-transport-based fidelity and diversity metrics for text that disentangle the two deficits, detecting diversity gaps in synthetic data that correlate with downstream fine-tuning degradation. |
| When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling | 07-2026 | Disentangles fixed-source synthesis from source expansion in synthetic-data scaling, showing per-question response budgets are a bounded axis while expanding seed questions wins at large budgets. |
| Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data | 06-2026 | Presents a model-agnostic statistical auditing framework distinguishing true from "phantom" disclosures in LLM-generated synthetic data, yielding tighter empirical privacy-leakage bounds without model access or shadow models. |
| An Information-Theoretic Criterion for Efficient Data Synthesis | 05-2026 | Provides an information-theoretic account showing synthetic data only improves models when the generation-training loop is information-open via external signals, otherwise the data processing inequality predicts collapse. |
| Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data | 05-2026 | Introduces a lightweight pre-pre-training stage on synthetic data with learnable temporal structure that improves LLM robustness to noisy pre-training data, cutting natural-text tokens by up to 49%. |
| Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning | 11-2025 | Shows fine-tuning on multi-source synthetic data mitigates distribution collapse and reduces self-preference bias, while synthetic data erodes safety safeguards despite higher output quality than human data. |
| Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls | 10-2025 | Conducts a large-scale empirical study (>1000 LLMs) deriving scaling laws for synthetic pre-training data, finding rephrased-text mixtures accelerate training while pure textbook-style data can trigger model collapse. |
| BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining | 08-2025 | Distills lessons from scaling synthetic data for LLM pretraining, showing that jointly optimizing many generation factors yields data that outperforms prior synthetic corpora like Cosmopedia and Nemotron-Synth. |
| Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning | 05-2025 | Introduces G-Vendi, a metric measuring training-data diversity via the entropy of model-induced gradients that strongly correlates with out-of-distribution reasoning performance, and uses it to guide synthetic data generation. |
| Scaling Laws of Synthetic Data for Language Models | 03-2025 | Presents SynthLLM, a framework revealing that synthetic pre-training data follows power-law scaling up to 300B tokens, and larger LLMs require fewer synthetic tokens to reach optimal performance |
| Evaluating Language Models as Synthetic Data Generators | 12-2024 | a benchmark that evaluates LLMs’ abilities to generate synthetic data by comparing outputs from multiple models and analyzing quality metrics (e.g., perplexity, difficulty), revealing that data-generation prowess doesn’t always match problem-solving strength |
| On the Diversity of Synthetic Data and its Impact on Training Large Language Models | 10-2024 | Introduces a diversity metric (“LLM cluster-agent”) to quantify synthetic data variety, demonstrating that data diversity boosts model performance—especially during fine-tuning—even for smaller-scale LLMs |
🔗 Related Repositories
| Awesome Synthetic Datasets | Practical resources for building synthetic text and vision datasets. |
|---|---|
| LLM Synthetic Data | Papers, tools, and blogs on LLM-generated data. |
| LLM-Datasets: | Curated datasets and tools for LLM post-training. |
| DataArc-SynData-Toolkit | Open-source, configuration-driven framework (visual UI + CLI) for multi-source, multimodal, and multilingual synthetic data generation via a modular, quality-controllable closed-loop pipeline. |
| GraphGen | Knowledge-graph-driven framework that builds fine-grained graphs from source text, locates an LLM's knowledge gaps via calibration error, and generates targeted QA, chain-of-thought, and multi-hop data for supervised fine-tuning. |
📝 Citation
If this work is helpful, please kindly cite as:
@inproceedings{alismail2025survey,
title={A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows},
author={Alismail, Ahmad and Lanquillon, Carsten},
booktitle={International Conference on Human-Computer Interaction},
pages={119--135},
year={2025},
organization={Springer}
}
agentic-workflow
llms
synthetic-data
synthetic-dataset-generation
Similar repos
Contributors
ahmad-alismail
28 commits