🧬 Carbon Pretraining Corpus

Description

173M DNA & RNA sequences · 1.1 trillion nucleotides — the DNA pretraining mixture used to train Carbon, a genomic foundation model.

This dataset is a collection of data sources intended for training genomic foundation models, such as Carbon. It contains DNA and RNA sequences spanning eukaryote and prokaryote species. Across the four main configs it totals 1.1 T DNA base pairs (180B tokens with Carbon's 6-mer tokenizer). A pre-sampled 10B-token eukaryote subset (eukaryote_generator_10B_subset) is also provided for smaller / faster runs.

Nucleotides are counted in base pairs (Gbp = billion nucleotides).

Quick biology primer

DNA is the molecule that stores genetic information in all living things. It is a sequence of four letters — A, T, G, C — and a genome can be anywhere from thousands to billions of these letters long. Training a language model on DNA means treating those letters like tokens and learning the statistical patterns of life.

This corpus covers three major layers of biological complexity:

• Eukaryotes — organisms with a nucleus (animals, plants, fungi, protists). Complex, large genomes with introns, regulatory regions, and lots of repetition.
• Prokaryotes — bacteria and archaea. Simpler, denser genomes; genes are packed tightly with less non-coding "junk."
• mRNA — the processed, spliced transcript of a gene. This is what actually gets translated into protein. Much shorter than full genomic sequences. Including mRNA teaches the model about gene structure without the noise of intergenic regions.

DNA sequences in this corpus look like this:

ATGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGC

A 6-mer tokenizer, such as Carbon's, splits this into non-overlapping windows of 6 nucleotides — one token per 6 bases:

ATGCTA | GCTAGC | TAGCTA | GCTAGC | TAGCTA | GCTAGC | ...

Loading the data

from datasets import load_dataset

ds = load_dataset(
    "hf-carbon/carbon-pretraining-corpus",
    "mrna_evo2",          # or: eukaryote_generator | mrna_splice_evo2 | prokaryote_evo2 | eukaryote_generator_10B_subset
    split="train",
    streaming=True,
)

for example in ds.take(3):
    print(example)

Dataset composition

1. eukaryote_generator — Eukaryote Genomes

Source: GenerTeam/pretrain_data_eukaryote

Genomic sequences from eukaryotic organisms (fungi, plants, protozoa, invertebrate and vertebrate), packaged by the GenerTeam as part of their GENERator genomic foundation model. This is the main part of Carbon's training data, which is focused on eukaryote species. Each row carries full taxonomic metadata, gene type, strand orientation, and chromosome coordinates alongside the raw sequence.

Following GENERator, we filter out sequences longer than 100 kbp, as excluding them improves training performance on DNA downstream benchmarks. For long-context training, you can concatenate genes from the same contig to build longer samples.

Example row:

{
  "species_type": "fungi",
  "gene_type": "protein_coding",
  "strand": "+",
  "sequence": "ATGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGC...",
  "taxonomy": "Eukaryota; Fungi; Ascomycota; ...",
  "record_id": "NC_001224.1",
  "start": 1872,
  "end": 3503
}

2. mrna_evo2 — Messenger RNA Sequences

Source: arcinstitute/opengenome2 (mrna split)

Processed mRNA sequences — the "final edited" version of a gene after the cell has removed introns (non-coding interruptions) and kept only the exons. This is the sequence that gets read to produce protein. These sequences are shorter and more information-dense than raw genomic DNA.

3. mrna_splice_evo2 — Augmented mRNA Transcripts

Source: arcinstitute/opengenome2 (mrna_splice_promoter)

The same mRNA transcripts as mrna_evo2, with augmentations applied by the Evo2 team: each transcript gets an extra 1,024 bp of upstream promoter sequence prepended, and an extra 32 bp of flanking sequence around each exon boundary to expose splice sites. The resulting exon chunks are concatenated with a special @ separator into a single stitched sequence per transcript.

Sequences here are systematically longer than mrna_evo2 — the extra promoter and splice flanks are the only difference, not a different set of genes.

4. prokaryote_evo2 — Prokaryote Genomes

Upstream: arcinstitute/opengenome2 (gtdb_v220_imgpr split)

Long chromosomal chunks from bacteria and archaea, sourced from GTDB v220 (a curated taxonomy of 85 K prokaryote genomes) and IMG/PR (a DOE database of environmental prokaryote sequences). Prokaryote genomes are compact — genes sit back-to-back with minimal intergenic space — so these sequences are biologically rich per nucleotide.

5. eukaryote_generator_10B_subset — 10B-token eukaryote subsample

Source: derived from eukaryote_generator (this dataset).

A pre-sampled subset of eukaryote_generator totalling 60 Gbp (10 B tokens at 6-mer tokenization) for smaller / faster model trainings that don't need the full 423 Gbp eukaryote slice. Rows are uniformly sampled at the row level from the filtered eukaryote data, which keeps the species distribution proportional to the natural per-species base-pair count:

Same schema as eukaryote_generator (full structured metadata columns).

⚗️ Carbon Training Mixture

Data weights

The proportions below reflect the target mixture for Carbon's pure-DNA pretraining runs (1 T token target):

Metadata conditioning in Carbon training (eukaryote_generator only)

In Carbon we added optional metadata tags to a fraction of eukaryote sequences, so the model learns to use biological context when available but doesn't depend on it. Tags are prepended before the sequence with random dropout at tokenization time:

# 50% of sequences — no tags
<dna>ATGCTAGCTA...</dna>

# 16.7% — species + gene type
<species>fungi<gene_type>protein_coding<dna>ATGCTAGCTA...</dna>

# 16.7% — species only
<species>fungi<dna>ATGCTAGCTA...</dna>

# 16.7% — gene type only
<gene_type>protein_coding<dna>ATGCTAGCTA...</dna>

At inference time you can prompt with any combination of tags or none at all. The three OpenGenome2 subsets are tokenized as plain <dna>SEQUENCE</dna> without metadata conditioning.

License

This dataset is a mirror of two upstream sources with different permissive licenses:

Citation

@article{allal2026carbon,
  title={Carbon: Decoding the Language of Life},
  author={Allal, Loubna Ben and Li, Qiuyi and Fiusco, Maurizio and Tunstall, Lewis and Rasul, Kashif and Beeching, Ed and Aubakirova, Dana and Pati{\~n}o, Carlos and Frere, Thibaud and Lozhkov, Anton and others},
  journal={bioRxiv},
  pages={2026--05},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}