Ettin Mid-training Data

Phase 2 of 3: Higher-quality filtered data with context extension (250B tokens) used for mid-training of Ettin models.

This dataset contains the mid-training phase data used to train all Ettin encoder and decoder models. This phase focuses on higher-quality filtered data and context length extension to 8K tokens. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.

📊 Data Composition

🔧 Key Changes from Pre-training

Data Quality Improvements

• Filtered DCLM: Using Dolmino-filtered version instead of raw DCLM
• Enhanced Math: Dolmino-filtered mathematical content
• Curated StackExchange: Higher-quality Q&A content
• Removed Noisy Sources: Dropped CC Head, CC News, and general StackExchange

Technical Improvements

• Context Extension: Increased from 1K to 8K token sequences
• RoPE Updates: Modified positional encoding for longer context
• Learning Schedule: Inverse square root decay from peak LR

🚀 Usage

For pre-training see the ModernBERT repo: https://github.com/AnswerDotAI/ModernBERT

Direct Access

from streaming import StreamingDataset

# Load the streaming dataset
dataset = StreamingDataset(
    remote='https://huggingface.co/datasets/jhu-clsp/ettin-extension-data',
    local='/tmp/ettin-extension-data',
    shuffle=True
)

# Access samples (note: these will be longer sequences)
for sample in dataset:
    text = sample['text']  # Up to 8K tokens
    # Process your data...

📁 Structure

Each folder contains filtered, higher-quality data sources in MDS format:

• arxiv/ - Academic papers from ArXiv
• books/ - Literature and reference books
• dclm_dolmino/ - Dolmino-filtered web crawl data (primary source)
• math_dolmino/ - Filtered mathematical content
• pes2o/ - Scientific papers
• reddit/ - Reddit discussion threads
• stackexchange_dolmino/ - Filtered StackExchange Q&A
• starcoder/ - Code from GitHub repositories
• tulu_flan/ - Instruction-following examples
• wikipedia/ - Wikipedia articles

🔗 Related Resources

• Models: Ettin Model Suite (17M-1B parameters)
• Phase 1: Pre-training Data (1.7T tokens)
• Phase 3: Decay Phase Data (50B tokens)
• Training Order: Batch-level Data Order
• Paper: Arxiv link
• Code: GitHub Repository

Citation

@misc{weller2025seqvsseqopen,
      title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders}, 
      author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2507.11412},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11412}, 
}