NVIDIA NeMo Curator

GPU-accelerated data curation for training better AI models, faster. Scale from laptop to multi-node clusters with modular pipelines for text, images, video, and audio.

Part of the NVIDIA NeMo software suite for managing the AI agent lifecycle.

self.curator — the dataset / corpus curation component of the self.ai stack (feeds both fine-tuning data for self.llamolotl and eval task data for self.language-eval / self.code-eval). It is a fully vendored, GPLv3-relicensed fork of NVIDIA NeMo Curator (Apache-2.0), previously carried under the Python package nemo_curator; the package is now curator. Not affiliated with or endorsed by NVIDIA. See NOTICE for provenance and LICENSE (GPLv3); the original Apache-2.0 text is retained in LICENSE.nvidia-apache-2.0.

What You Can Do

Quick Start

# Install for your modality
uv pip install "nemo-curator[text_cuda12]"

# Run the quickstart example
python tutorials/quickstart.py

Full setup: Installation Guide • Docker • Tutorials

---

Features by Modality

Text Curation

Process and curate high-quality text datasets for large language model (LLM) training with multilingual support.

---

Image Curation

Curate large-scale image datasets for vision language models (VLMs) and generative AI training.

---

Video Curation

Process large-scale video corpora with distributed, GPU-accelerated pipelines for world foundation models (WFMs).

---

Audio Curation

Prepare high-quality speech datasets for automatic speech recognition (ASR) and multimodal AI training.

---

Why NeMo Curator?

Performance at Scale

NeMo Curator leverages NVIDIA RAPIDS™ libraries such as cuDF, cuML, and cuGraph along with Ray to scale workloads across multi-node, multi-GPU environments.

Proven Results:

• 16× faster fuzzy deduplication on 8 TB RedPajama v2 (1.78 trillion tokens)
• 40% lower total cost of ownership (TCO) compared to CPU-based alternatives
• Near-linear scaling from one to four H100 80 GB nodes (2.05 hrs → 0.50 hrs)

Quality Improvements

Data curation modules measurably improve model performance. In ablation studies using a 357M-parameter GPT model trained on curated Common Crawl data:

Results: Progressive improvements in zero-shot downstream task performance through text cleaning, deduplication, and quality filtering stages.

---

Learn More

---

Contribute

We welcome community contributions! For this fork, please refer to this repo's own CONTRIBUTING.md for guidelines, and follow self.ai's contribution model (DCO + Assisted-by: trailer).