Dapper

Dataset Absurdly Powerful Parser Engineered Recklessly

A dataset exploration and comparison tool with an interactive TUI. Currently optimized for AI conversation datasets, with a vision to become a general-purpose dataset comparer.

Features

• Interactive TUI - Browse and compare datasets in a terminal interface
• Side-by-Side Comparison - Compare two datasets or original vs. processed records
• Multi-Format Support - Load JSONL, JSON, Parquet, and CSV with automatic detection
• Dynamic Schema Detection - Automatically detects message, ID, and tool fields
• Dataset Mixer - Opinionated pipeline that combines specific HuggingFace datasets into a single unified Parquet training file (see Dataset Mixer below)
• CLI Tools - List, search, and analyze records from the command line
• Data Splitter - Split large datasets into N parts for parallel processing

Requirements

• Python 3.12+
• uv (recommended) or pip

Installation

Install Globally With uv

From a local checkout:

git clone <repository-url>
cd dataset-parser
uv tool install .

From a published package:

uv tool install dapper-datasets

From a Git repository:

uv tool install "dapper-datasets @ git+ssh://git@github.com/ai-blaise/dataset-parser.git"

For development, install the checkout as an editable uv tool so the dapper on your PATH follows local source changes:

uv tool install --force --reinstall --editable --python 3.12 .

The --editable . flag is the important part. Without it, the installed tool can keep running a previously built snapshot instead of the working tree. Check which copy you are actually running with readlink -f "$(which dapper)". A pipeline run started against a stale install silently uses the old code.

dapper-datasets is the package name. dapper is the terminal command. After installation, run Dapper from any directory:

dapper --help
dapper view /path/to/dataset.parquet
dapper list /path/to/dataset.jsonl -n 20
dapper mix /path/to/datasets -o /path/to/mixed.parquet

Local Development

uv sync
dapper --help

Using pip

git clone <repository-url>
cd dataset-parser
pip install -e .

Quick Start

Browse a dataset interactively

# Open a single file (read-only view)
dapper view dataset/conversations.jsonl

# Open with export mode (original vs. parsed side-by-side)
dapper view dataset/conversations.jsonl -x

# Open a directory (shows file picker)
dapper view dataset/

# Open data directly from GCS
dapper view gs://my-bucket/path/to/conversations.jsonl
dapper view gs://my-bucket/path/to/parquet-prefix/

# Open the configured GCS output prefix from dapper.yaml
dapper view --gcs

# Compare two GCS prefixes
dapper view gs://my-bucket/dataset-a/ --compare gs://my-bucket/dataset-b/

When a path or configured GCS prefix contains subdirectories, the TUI opens a browser first. Select a child prefix to descend or a supported file to render. JSONL, JSON, Parquet, CSV, and text files are supported; text files render one line per record.

Extract prompts (remove assistant responses)

# Output to stdout
dapper parse dataset/conversations.jsonl

# Output to a specific file
dapper parse dataset/conversations.jsonl -o prompts.json

# Output to a directory (creates train_parsed.json)
dapper parse dataset/train.jsonl -O parsed_datasets/

Mix datasets into unified training data

The dataset mixer combines multiple HuggingFace datasets into a single Parquet file with a unified schema.

Basic Usage

# Mix ALL datasets in datasets/ into a single Parquet file
dapper mix datasets/ -o output-datasets/full_mix_all_sources.parquet

# Dry-run: show record counts without writing output
dapper mix datasets/ --dry-run

Source Filtering

Filter which datasets to include or exclude using --include and --exclude. These flags support prefix matching (e.g., --include Nemotron matches both Nemotron-Terminal-Corpus and Nemotron-SFT-Agentic-v2-*):

# Only Nemotron family (Terminal Corpus + Agentic v2)
dapper mix datasets/ -o output-datasets/nemotron_only.parquet \
  --include Nemotron

# Only Nemotron Terminal Corpus (adapters + synthetic tasks)
dapper mix datasets/ -o output-datasets/nemotron_terminal_corpus_only.parquet \
  --include Nemotron-Terminal-Corpus

# Only Nemotron-SFT-Agentic-v2 (search + tool_calling combined)
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_combined.parquet \
  --include Nemotron-SFT-Agentic-v2

Random Sampling

Apply random sampling to Nemotron-SFT-Agentic-v2 records only (does NOT affect other sources):

# Full Agentic v2 (no sampling)
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_full.parquet \
  --include Nemotron-SFT-Agentic-v2

# 50% sample of Agentic v2 tool_calling (search stays 100%)
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_sample_50.parquet \
  --include Nemotron-SFT-Agentic-v2 \
  --tooling-sample-rate 0.5

# 40% sample of Agentic v2 tool_calling with seed for reproducibility
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_sample_40.parquet \
  --include Nemotron-SFT-Agentic-v2 \
  --tooling-sample-rate 0.40 \
  --sample-seed 42

# 20% sample of Agentic v2 tool_calling with seed
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_sample_20.parquet \
  --include Nemotron-SFT-Agentic-v2 \
  --tooling-sample-rate 0.2 \
  --sample-seed 42

Full Nemotron Family Mix Examples

# FULL Nemotron family (Terminal Corpus = 100% + Agentic v2 = 100%)
dapper mix datasets/ -o output-datasets/nemotron_full_family.parquet \
  --include Nemotron

# FULL Nemotron family + 40% sample of Agentic v2 tool_calling only
# (Terminal Corpus = 100%, search = 100%, tool_calling = 40%)
dapper mix datasets/ -o output-datasets/nemotron_mixed_sample.parquet \
  --include Nemotron \
  --tooling-sample-rate 0.40 \
  --sample-seed 42

Distillation Mix Datasets

The test-datasets/ directory contains real distillation mix datasets from HuggingFace. These are used directly as input to the mixer.

# Preview what would be mixed
dapper mix test-datasets/ --dry-run

Available datasets:

Adapter requirements:

Mix commands:

# Mix all distillation datasets into a single Parquet file
dapper mix test-datasets/ \
  -o output/distillation_mix.parquet

# Mix + shuffle + split into 4 chunks
dapper mix test-datasets/ \
  -o output/distillation_mix \
  --shuffle --shuffle-seed 42 \
  --num-chunks 4
# Creates: output/distillation_mix_part_1_of_4.parquet, ..., output/distillation_mix_part_4_of_4.parquet

# Only High-Coder datasets
dapper mix test-datasets/ \
  -o output/high_coder_only.parquet \
  --include "High-Coder"

# Only Hunter-Alpha datasets
dapper mix test-datasets/ \
  -o output/hunter_alpha_only.parquet \
  --include "Hunter-Alpha"

# Exclude large datasets for a quick mix
dapper mix test-datasets/ \
  -o output/quick_mix.parquet \
  --exclude "Hunter-Alpha-Programming-160000x" \
  --exclude "High-Coder-Reasoning-Multi-Turn"

Advanced Options

# Custom batch size for memory control (default: 2000)
dapper mix datasets/ -o output-datasets/custom.parquet \
  --batch-size 500

# Preview what will be included before running
dapper mix datasets/ --dry-run --include Nemotron

Pretraining corpus pipeline (GCS)

Four independent stages, each reading and writing a GCS prefix declared once in dapper.yaml. Every stage is separately runnable and re-runnable — none is a mode of another.

archive    →  staged-input/     text, JSONL, one dir per source
dedup      :  staged-input/     →  dedup-output/   text, Parquet, MinHash-deduplicated
tokenize   :  <any text corpus> →  tokens/         text + input_ids, Parquet

Prefixes are global config — commands take no path flags:

storage:
  provider: gcs
  bucket: pretraining-corpus
  dataset_prefix: dapper/pretraining/staged-input     # archive writes
  work_prefix:    dapper/pretraining/datatrove-work   # MinHash scratch (deletable)
  output_prefix:  dapper/pretraining/dedup-output     # dedup writes
  tokens_prefix:  dapper/pretraining/tokens           # tokenize writes

Auth is Application Default Credentials — run gcloud auth application-default login first. Dapper never handles a credential itself.

# 0. Inspect what is configured before moving any bytes
dapper catalog list
dapper catalog show fineweb

# 1. Archive HuggingFace sources into GCS. Streams straight to gs:// --
#    nothing is written to local disk and nothing is tokenized.
dapper archive --dry-run              # resolve catalog + bucket, write nothing
dapper archive check                  # count _SUCCESS markers by source
dapper archive --limit 1000           # small slice to prove the path works
dapper archive --sources fineweb --ray # full FineWeb across the Ray cluster

# 2. Deduplicate every currently complete archive with native DataTrove Ray.
#    This connects to the existing cluster; it does not start another service.
dapper dedup --gcs --ray
# Or freeze an explicit completed subset:
dapper dedup --gcs --ray --sources c4,cosmopedia

# 3. Tokenize into bin-partitioned WebDataset shards, in one pass.
dapper tokenize fineweb --dry-run     # resolve corpus + tokenizer, write nothing
dapper tokenize fineweb               # -> tokens/<bin>/shard-fineweb-*.tar
dapper tokenize --deduped             # one completed dedup run
dapper tokenize --deduped --dedup-run-id <id>  # required when several exist

# 4. Check a target mixture against what the corpus actually holds.
dapper mixture check                  # exits 3 if any cell is unsatisfiable

Tokenize output is the token artifact -- there is no Parquet intermediate:

tokens/8192/    shard-fineweb-00000-0000.tar    documents up to 8,192 tokens
     65536/     …                               8,193 - 65,536
     262144/    …                               overflow, unbounded
     _manifest/manifest.json                    capacities per bin/domain/subdomain
     _runs/<source>/                            markers, counts, logs

Each sample is <key>.npy (int32 token ids) plus <key>.json (id, url, domain, subdomain, token_count, …). Bins come from dedup.len_bins; the directory is the bin's inclusive upper edge, and the last bin absorbs everything above it. Shards are tag-pure, so a trainer picks whole shards by domain from the manifest rather than reading and discarding samples.

dapper tokenize takes a source name or --deduped, never both: the deduplicated corpus is partitioned by domain rather than by source, so there is no per-source prefix inside it to address.

For FineWeb, the default dapper tokenize fineweb workflow clusters staged raw text before tokenization and packing. It hashes word 1–2 grams and char_wb 3–5 grams into sparse features, L2-normalizes their weighted combination, fits 128 broad lexical/topic groups with scikit-learn MiniBatchKMeans on a deterministic 1,000,000-document sample, then assigns the full corpus in distributed Ray tasks. Document length is not a clustering feature. This is topic-local packing, not duplicate detection; corpus dedup uses DataTrove MinHash instead.

Run the whole pipeline in one sweep:

dapper run --limit 1000               # archive -> dedup -> tokenize
dapper run --yes                      # full corpus; --yes is required

--yes is mandatory for an unlimited dapper run because the full catalog commits to days of transfer and billable GCS egress.

Long runs

These are multi-hour jobs. Run them detached so a closed terminal does not SIGHUP the process group:

nohup dapper tokenize fineweb > tokenize.log 2>&1 &

Interrupted runs resume only when the immutable run identity is unchanged. Dedup strictly validates each exhaustive _SUCCESS marker against its JSONL inventory and object generations, freezes that source set, then checks every DataTrove rank marker before advancing. An archive that completes after dedup starts belongs to the next run.

Full FineWeb uses its commit-pinned native Parquet files as Ray work units:

dapper ray init
dapper archive --sources fineweb --ray

Ray workers are existing GCE VMs declared only in the untracked .env. Add one numbered DAPPER_RAY_WORKER_<NN>_INSTANCE / _ZONE pair per VM; Dapper discovers all pairs and derives the required cluster size. The instance value is the GCE VM name, while worker-01, worker-02, and so on are display aliases.

To run an archive on only the workers in one zone, keep all workers in the same .env and select the zone dynamically. The .env.example includes two fake workers in each example zone:

dapper ray init --zone us-central1-a
dapper archive --sources zyda-2 --ray

The zone filter selects only workers whose configured ZONE exactly matches the requested zone. Stop the selected group with dapper ray stop --zone us-central1-a.

The .env.example models four workers split across two zones: workers 01/02 in us-central1-a and workers 03/04 in us-east4-a. Select either pair with its zone.

The Ray head is the machine where dapper ray init runs; it is not worker 01 or the first worker in a zone. The worker INSTANCE and ZONE values are the only required per-worker settings.

For a persistent zonal mode shared by all Ray-backed commands, put DAPPER_RAY_ZONE=us-central1-a in the same .env. Then initialize the cluster with dapper ray init; archive, dedup, clustering, tokenization, and packing all use the connected zonal Ray cluster. Remove the variable to return to the full configured worker set.

The head resolves the Hugging Face manifest once. Each Ray task downloads one native file through Xet into a bounded RAM-disk spool, reads the Parquet file once in Arrow batches, and streams one deterministic JSONL object to GCS. The temporary file is released immediately. This avoids reopening roughly 1,000 row groups per 2 GiB FineWeb shard. Exact Parquet row counts and completion markers make every file independently resumable without accepting partial canary output. Only nodes registered in the private Ray cluster receive work. --force clears only the isolated configured archive_name before starting a new distributed archive.

The archive dashboard distinguishes active tasks from durable completions. A native shard contributes documents and bytes only after its GCS object closes, so the first scheduling wave is labeled warming up instead of presenting an unstable completion-based ETA. Its node table also reports live network RX/TX and /dev/shm spool consumption. Ray archive workers use bounded adaptive Xet clients; per-process high-performance mode is disabled because Dapper already fans out dozens of independent file transfers per node.

Tuning throughput

Hugging Face archive streams rows with datasets.load_dataset(..., streaming=True). With hf_xet installed, huggingface.xet_high_performance defaults to true, which sets HF_XET_HIGH_PERFORMANCE=1. hf_transfer is the older LFS accelerator and is no longer the recommended knob for current Hub-backed downloads.

Concurrency is config, not flags — both dedup and tokenize read it:

huggingface:
  download_mode: streaming
  xet_high_performance: true
  # xet_num_concurrent_range_gets: 16  # optional advanced override

dedup:
  datatrove:
    executor: local   # bare local command; --ray selects RayPipelineExecutor
    tasks: 1
    workers: 8
    ray:
      task_oversubscription: 4
      workers_per_bucket: 32
      signatures: {workers: auto, cpus_per_task: 1, memory_gb_per_task: 2}
      buckets: {workers: auto, cpus_per_task: 1, memory_gb_per_task: 2}
      clusters: {workers: 1, cpus_per_task: 8, memory_gb_per_task: 48}
      filter: {workers: auto, cpus_per_task: 1, memory_gb_per_task: 4}

On two 224-vCPU nodes, the document stages resolve to as many as 448 workers and 1,792 tasks; buckets resolve to 14 × 32 = 448 tasks. The global MinHash union/find stage remains one high-memory owner because stock DataTrove requires it. Ray queues dedup safely when an archive already reserves the cluster, and uses capacity from newly registered nodes without opening a second Ray port.

Before adding workers, find out what you are actually waiting on. These jobs are often network-bound rather than CPU-bound — tokenizing ~10B tokens is roughly 20 minutes of CPU, while uploading the ~50 GB of resulting Parquet over a 20 Mbps uplink is over five hours. When that is the case, more workers buy nothing; running the job in the bucket's own region is the fix.

Split a dataset into parts

# Split into 4 parts
dapper split dataset/conversations.jsonl -n 4

# Preview split without creating files
dapper split dataset/conversations.jsonl -n 10 --dry-run

Usage

CLI Commands

Pretraining corpus pipeline (GCS-backed, driven by dapper.yaml):

Command Coverage Status

The public dapper CLI exposes the core dataset workflows (exploration, TUI viewing, parsing, mixing, splitting) and the pretraining corpus pipeline (archive, dedup, tokenize). Some scripts in scripts/ are still internal or legacy and do not yet have public dapper wrappers, including rerollout variants, upload helpers, filtering helpers, and demo scripts.

TUI Keybindings

Dapper Parser Formats

Dataset Mixer

The Dataset Mixer is an opinionated pipeline built specifically to combine Nemotron family HuggingFace datasets into a single unified Parquet training file:

Each dataset has a dedicated adapter that handles its specific schema and normalizes records into a unified conversations-based output format with metadata columns. The source_dataset column tracks which HuggingFace dataset each record originated from.

Place datasets in datasets/ using their HuggingFace repository name as the directory:

datasets/
├── Nemotron-Terminal-Corpus/
└── Nemotron-SFT-Agentic-v2/

Source Filtering

Use --include and --exclude to produce filtered mix outputs from a single datasets/ directory. Filter values support prefix matching:

# Full Nemotron family (~380K records)
# Combines Terminal Corpus (100%) + Agentic v2 (100%)
dapper mix datasets/ -o output-datasets/nemotron_full_family.parquet \
  --include Nemotron

# Nemotron Terminal Corpus only (~366K records)
dapper mix datasets/ -o output-datasets/nemotron_terminal_corpus_only.parquet \
  --include Nemotron-Terminal-Corpus

# Nemotron-SFT-Agentic-v2 only (~14K records)
dapper mix datasets/ -o output-datasets/nemotron_agentic_v2_combined.parquet \
  --include Nemotron-SFT-Agentic-v2

# Full family with 40% sampling on tool_calling only (search stays 100%)
dapper mix datasets/ -o output-datasets/nemotron_mixed_40.parquet \
  --include Nemotron \
  --tooling-sample-rate 0.40 \
  --sample-seed 42

Filtering operates on the file list before any data is read. Both flags accept prefix matching (e.g., --include Nemotron matches both Nemotron-Terminal-Corpus and Nemotron-SFT-Agentic-v2-*).

Future Plans

The tool is currently optimized for AI conversation datasets but is designed to become a general-purpose dataset comparer:

• Configurable schema detection - Support any JSON structure, not just conversations
• ID-based record matching - Match records by key field instead of index
• Pluggable transformations - Optional processing instead of hardcoded Dapper Parser behavior
• Additional formats - Excel/XLSX support

Corpus sources (TODO)

The pretraining corpus is currently scoped to FineWeb only (corpus.sources.huggingface in dapper.yaml). The sources below were vetted but deferred — each is a plain HuggingFace dataset path and can be added as a one-line config entry when wanted.

Domains listed here are keyword-inferred and were never reviewed. Declare the real domain when promoting an entry.

Web / general

Code / math / specialist

PDF / long-form — HuggingFaceFW/finepdfs, finepdfs-edu

Dolma family — allenai/dolma, plus ten allenai/dolma3_* variants (_pool, mix-10B/50B/100B/150B/6T, date stamps 1025/1125). These are the same corpus at different sizes and blends; archiving all of them downloads the same documents repeatedly. Pick one.

Verify before adding

• nvidia/Nemotron-Pretraining-SFT-v1 — SFT data, not pretraining text
• HuggingFaceFW/ocr-annotations, finepdfs_lang_classification, finepdfs_fw_edu_labeled, finepdfs_eng_Latn_labeled — appear to be annotation/label sets rather than document corpora
• BLIP3o/BLIP3o-Pretrain — vision-language; may have no usable text column
• nvidia/Nemotron-Pretraining-Dataset-sample — a sample of a set already listed
• nvidia/Nemotron-Pretraining-Specialized-v1 … -v1.2 — successive releases; probably only the latest is wanted

dapper dedup --dry-run samples each configured source and reports whether a text field resolves, which settles the questionable entries empirically.

No loader yet — not HuggingFace datasets, so they need new handlers: togethercomputer/RedPajama-Data, EleutherAI/openwebtext2, facebookresearch/PhysicsLM4 (GitHub); common-pile (space); contrib/Nemotron/Nemotron-CC (archive); mlfoundations/datasets?search=dclm (search page).

Documentation

For detailed documentation, see the docs directory:

• Architecture Overview - System design and components
• Record Structure - JSONL data format reference
• CLI Reference - Complete CLI command documentation
• TUI Guide - Interactive terminal UI guide
• Dapper Parser - Transformation tool documentation
• Data Splitter - Dataset splitting utility
• Data Formats - Multi-format loading and schema normalization
• Verify Datasets - How to verify mixed training outputs against source datasets

Development

Running Tests

uv run pytest tests/

Project Structure

dapper/
├── dapper/               # Packaged Dapper commands and shared project logic
│   ├── cli.py            # Public dapper command dispatcher
│   ├── schema.py         # Universal --schema handling
│   ├── explore/          # dapper list/show/search/stats
│   ├── parser/           # dapper parse
│   ├── mix/              # dapper mix
│   ├── dedup/            # dapper dedup
│   ├── split/            # dapper split
│   └── tui/              # dapper view
├── utils/                # Core utilities (functional, memory-efficient)
│   ├── loader.py         # Multi-format data loading (load_records, etc.)
│   ├── detect.py         # Format detection (detect_format, etc.)
│   ├── normalize.py      # Schema normalization (normalize_record, etc.)
│   ├── sampling.py       # Reservoir sampling, shuffle, chunk
│   ├── streaming.py      # PyArrow RecordBatch transformation
│   ├── config.py         # Theme configuration
│   └── data.py           # Data transformation utilities
├── scripts/              # Standalone maintenance/rerollout utilities
│   ├── rerollout*.py     # Rerollout helpers
│   ├── filter_evals.py   # Evaluation filtering helper
│   └── upload_to_hf.py   # Hugging Face upload helper
├── tests/                # Test suite
├── datasets/             # HuggingFace datasets (gitignored)
├── docs/                 # Documentation
└── plans/                # Design plans

uv tool install "dapper-datasets @ git+ssh://git@github.com/ai-blaise/dataset-parser.git"

License

MIT License - see LICENSE for details.