Sanskrit-English Parallel Translation Dataset (Saamayik)

Table of Contents

• Dataset Description
• Dataset Structure
• Dataset Creation
• Usage
• Limitations
• Citation
• License

Summary

The Saamayik Sanskrit-English Parallel Corpus is a contemporary prose-focused translation dataset containing around 53,000 parallel sentences in Sanskrit and English (48,326 in this main dataset, with an additional 4,047 Mann Ki Baat sentences). Sāmayik, meaning "sayings of the contemporary world" in Sanskrit, specifically addresses the gap in existing Sanskrit corpora which predominantly feature classical poetry. This dataset emphasizes contemporary Sanskrit usage in prose form, compiled from diverse modern sources including educational materials, technical tutorials, spiritual texts, and religious content.

Paper: ACL Anthology | arXiv

Languages

• Sanskrit (sa): Contemporary Sanskrit in Devanagari script (with some classical content)
• English (en): Modern English

Supported Tasks

This dataset is designed for:

• Machine Translation: Training and evaluating Sanskrit↔English translation models
• Cross-lingual Understanding: Studying linguistic patterns between Sanskrit and English
• Sanskrit NLP Research: Supporting various Sanskrit language processing tasks
• Educational Applications: Assisting in Sanskrit language learning and teaching

Dataset Structure

Data Instances

Each instance contains a parallel translation pair:

{
  "translation": {
    "en": "Save it with Ctrl, S.",
    "sa": "Ctrl, S नुत्वा रक्षन्तु।"
  }
}

Data Fields

• translation: A dictionary containing:
  • en (string): The English text
  • sa (string): The Sanskrit text in Devanagari script

Data Splits

The dataset is divided into three splits:

Note: The complete dataset on Hugging Face includes an additional test_ood split containing 4,047 sentences from Mann Ki Baat (MKB) for out-of-domain evaluation.

Source Distribution

The dataset combines content from multiple sources:

1. Bible Translations: Parallel verses from Sanskrit and English Bible translations (7,838 pairs)
2. Gitasopanam: Spiritual and philosophical texts with translations (5,885 pairs)
3. NIOS (National Institute of Open Schooling): Educational content in both languages (11,356 pairs)
4. Spoken Tutorials: Technical and instructional content translations (23,835 pairs)

Note: The Mann Ki Baat (MKB) dataset (4,047 pairs) mentioned in the Sāmayik paper is provided separately as an out-of-domain evaluation set and is not included in this main dataset.

Dataset Creation

Curation Rationale

This dataset was created to address the scarcity of high-quality Sanskrit-English parallel corpora. Sanskrit, despite being a classical language with rich literary heritage, lacks sufficient digital resources for modern NLP applications. This dataset aims to:

• Provide contemporary Sanskrit prose for modern NLP applications
• Address the underrepresentation of prose in Sanskrit corpora
• Enable better machine translation for contemporary Sanskrit content
• Support the contemporary usage of Sanskrit beyond classical texts

Source Data

Data Collection

The data was collected from publicly available sources that provide Sanskrit-English parallel texts:

1. Religious Texts: Carefully aligned translations of religious scriptures
2. Educational Materials: Curriculum content from educational institutions
3. Technical Content: Modern technical tutorials translated into Sanskrit
4. Literary Works: Classical texts with established English translations

Data Processing

The preprocessing pipeline included:

• Text extraction from various formats (CSV, Excel, TXT)
• Sentence alignment verification
• Removal of empty or misaligned pairs
• Character normalization for Sanskrit text
• Quality filtering based on length ratios

The dataset underwent standard cleaning including removal of exact duplicate translation pairs, though individual sentences may appear multiple times with different translations.

Annotations

The dataset uses naturally occurring parallel translations rather than synthetic annotations. The translations were created by:

• Sanskrit scholars and linguists
• Educational institutions
• Religious organizations
• Professional translators

Personal and Sensitive Information

The dataset has been filtered to remove any personal information. The content is primarily educational, religious, and technical in nature.

Usage

Loading the Dataset

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("acomquest/Saamayik")

# Access specific splits
train_data = dataset['train']
val_data = dataset['validation']
test_data = dataset['test']
# Note: The complete dataset includes test_ood (Mann Ki Baat)

# Example usage
for i in range(5):
    example = train_data[i]
    print(f"English: {example['translation']['en']}")
    print(f"Sanskrit: {example['translation']['sa']}")
    print("---")

Considerations for Using the Data

Social Impact

Positive impacts:

• Cultural Preservation: Helps preserve Sanskrit language and literature
• Educational Access: Enables Sanskrit learning through parallel texts
• Research Advancement: Supports computational linguistics research for low-resource languages
• Knowledge Dissemination: Makes ancient Sanskrit texts accessible to English speakers

Potential concerns:

• Translation Quality: Some translations may not capture cultural nuances
• Domain Bias: Heavy representation of religious and philosophical content
• Script Limitations: Focuses on Devanagari script, not other Sanskrit writing systems

Discussion of Biases

The dataset may exhibit several biases:

1. Domain Bias: Overrepresentation of religious and educational content
2. Style Bias: Formal and literary language style predominates
3. Temporal Bias: Mix of ancient and modern content may affect consistency
4. Cultural Bias: Translations may reflect specific cultural interpretations

Other Known Limitations

• Limited coverage of conversational Sanskrit
• May not include recent Sanskrit neologisms
• Some technical translations might be non-standard
• Sentence-level alignment may miss discourse-level features

Additional Information

Dataset Curators

This dataset was curated as part of the Saamayik project for Sanskrit language preservation and computational linguistics research.

Licensing Information

This dataset is released under the MIT License.

Citation Information

If you use this dataset in your research, please cite the original Sāmayik paper:

@inproceedings{maheshwari-etal-2024-samayik,
    title = "Sāmayik: A Benchmark and Dataset for {E}nglish-{S}anskrit Translation",
    author = "Maheshwari, Ayush and 
              Gupta, Ashim and 
              Krishna, Amrith and 
              Singh, Atul Kumar and 
              Ramakrishnan, Ganesh and 
              Kumar, G. Anil and 
              Singla, Jitin",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1245",
    pages = "14244--14252",
}

Contributions

We welcome contributions to improve and expand this dataset. Please consider:

• Reporting alignment errors or quality issues
• Contributing additional parallel texts
• Improving preprocessing scripts
• Adding more metadata or annotations

Contact: https://ayushbits.github.io