Cosmos Corpus Encoded for Embedding Distillation

This dataset is a pre-tokenized version of alibayram/cosmos-corpus-0-05-with-embeddings, designed for efficient embedding distillation training of MFT and TabiBERT models.

Dataset Description

• Source: alibayram/cosmos-corpus-0-05-with-embeddings
• Language: Turkish
• Task: Embedding Distillation (Teacher-Student Training)
• Total Examples: 224,807 (Filtered from 300,000)
• Max Sequence Length: 2048 tokens

Pre-processing & Filtering

The dataset was processed using two different tokenizers to support multiple student architectures:

1. MFT Tokenizer: A custom morphologically informed tokenizer.
2. TabiBERT Tokenizer: A BERT-based tokenizer with 32k vocabulary.

Filtering:

• Original size: 300,000 examples.
• Filtered size: 224,807 examples (~75%).
• Criterion: Both mft_input_ids and tabi_input_ids must be <= 2048 tokens.
• Sequences longer than 2048 tokens were dropped to ensure efficient training within context limits.

Dataset Structure

The dataset contains the following columns:

Data Instances

{
  'text': 'Makine öğrenmesi, verilerden öğrenen algoritmaların çalışılmasıdır.',
  'mft_input_ids': [124, 5921, ...],
  'tabi_input_ids': [101, 2341, ...],
  'teacher_embedding_final': [0.021, -0.054, ...]  # 3584-dimensional vectors
}

Usage

This dataset is optimized for the EmbeddingDistillationTrainer. You can load it directly without needing to re-tokenize during training.

from datasets import load_dataset

dataset = load_dataset("alibayram/cosmos-corpus-encoded")

Training Example

To train a model using the mft_input_ids column:

from embedding_trainer import EmbeddingDistillationTrainer, EmbeddingTrainerConfig

config = EmbeddingTrainerConfig(
    student_model="alibayram/mft-downstream-task-embeddinggemma",
    input_ids_column="mft_input_ids",  # or "tabi_input_ids"
    embedding_column="teacher_embedding_final",
    loss_type="cosine",
    batch_size=256
)

trainer = EmbeddingDistillationTrainer(config)
trainer.train("alibayram/cosmos-corpus-encoded")

Creation Details

• Created by: Ali Bayram
• Date: 2026-01-25
• Teacher Model: google/gemma-2-9b-it (Embeddings extracted via sartify-llm/Gemma-2-9b-it-v2-embedding)
• Processing Script: prepare_dataset.py

License

MIT