Modotte

---

Note: This dataset is part of the lineup CodeX by Modotte. You can get lots of datasets in this same lineup, with the main focus on providing very high-quality datasets for model training and fine-tuning.

This dataset is curated from high-quality public sources and enhanced with synthetic data from both closed and open-source models. It serves as a strong foundation for instruction-based model tuning and fine-tuning, offering one of the most refined and extensive corpora available for coding tasks.

Key Features

• Scale: 7 million examples of highly curated coding data
• Diversity: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
• Quality: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
• Non-Thinking Focus: Direct code solutions without step-by-step reasoning chains, optimized for efficient instruction training
• Accuracy: Verified code executions and correctness validation using automated testing frameworks

Dataset Overview

CodeX-7M-Non-Thinking is a meticulously curated coding dataset designed specifically for instruction-based model tuning and fine-tuning of existing models with enhanced code generation capabilities. This represents one of the largest and most comprehensively filtered corpora of publicly available coding data on the Hugging Face platform, with a non-thinking approach that emphasizes direct, concise code outputs for rapid model training.

How to Use?

pip install -U datasets fsspec

from datasets import load_dataset
dataset = load_dataset("Modotte/CodeX-7M-Non-Thinking")

Key Features

• Scale: 7 million examples of highly curated coding data
• Diversity: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
• Quality: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
• Non-Thinking Focus: Direct code solutions without step-by-step reasoning chains, optimized for efficient instruction training
• Accuracy: Verified code executions and correctness validation using automated testing frameworks

Data Curation Process

This dataset has been carefully constructed through a multi-source approach, selectively collecting and merging examples from premium sources, along with customly generated examples to enrich the overall dataset for generation models.

Data Sources

• High-Quality Existing Datasets: Curated from multiple premium coding datasets available online (e.g., from NVIDIA, OpenAI-inspired repositories, and Modotte's internal collections)
• Synthetic Generation: Generated using both closed-source and open-source language models (Modotte)
• Expert Validation: Human-verified code solutions and implementations (Modotte)

Filtering Pipeline

Our rigorous filtering process includes open and closed-source filtering techniques, ensuring only the highest-quality examples are retained:

1. Deduplication: Removal of duplicate problems and code solutions
2. Normalization: Code formatting standardization and syntax cleanup
3. Stopword Processing: Intelligent removal of non-essential comments or boilerplate
4. Quality Scoring: Multi-dimensional quality assessment using metrics like code complexity, readability, and efficiency
5. Ranking-Based Filtering: Advanced ranking algorithms to prioritize top-tier examples based on relevance, novelty, and utility
6. Expert Selections: Manual curation by coding experts to select exemplary samples
7. Answer Verification: Automated testing and execution validation using frameworks like pytest or unit tests
8. Content Filtering: Removal of inappropriate, outdated, or incorrect code
9. Diversity Balancing: Ensuring balanced representation across languages and domains through algorithmic sampling

Problem Complexity Distribution

• Basic Level (30%): Fundamental programming concepts, simple syntax, and basic operations
• Intermediate Level (30%): Multi-function problems requiring modular code and basic algorithms
• Advanced Level (40%): Complex challenges involving data structures, optimization, and system design

Programming Domains Covered

• Algorithms and Data Structures
• Web Development and Frameworks
• Machine Learning and AI Implementations
• System Programming and Operating Systems
• Database Management and SQL/NoSQL
• Software Engineering Best Practices
• Competitive Programming Problems

Note: Domains are for reference only. The actual data is very diverse and covers more domains than stated. The actual data includes more complex and high-level questions than stated, spanning multiple programming languages such as Python, Java, C++, JavaScript, and others.

Use Cases

• Fine-tuning code generation capabilities in language models
• Training instruction-following models with a coding focus
• Benchmarking model performance on coding tasks and problem-solving
• Research in AI-assisted programming and automated code completion
• Educational applications requiring direct code examples and solutions

Dataset Format

Each example contains:

• Problem Statement: Clear coding challenge or task description
• Code Solution: Direct, response without intermediate reasoning

Quality Assurance

• Automated Verification: All code solutions verified using execution environments and testing suites
• Correctness Guarantee: Only problems with verified correct and functional code are included
• Human Review: Sample validation by coding experts
• Automated Checks: Static analysis, linting, and runtime verification where applicable
• Open and Closed-Source Filtering: Integration of proprietary and community-driven tools for enhanced quality control

Performance Metrics

Models trained on this dataset show significant improvements in:

• Code generation accuracy
• Efficiency in producing concise solutions
• Problem-solving speed
• Cross-language and cross-domain code transfer
• Reduction in hallucinated or erroneous code outputs

Acknowledgments

Special thanks to our partners and contributors:

• NVIDIA, Magpie-Align, Magpie-Align - Reference datasets; CodeX contains many examples taken from their existing datasets
• Microsoft - Inspirational datasets and methodologies; CodeX includes adapted examples from Microsft-related repositories
• Modotte Team - Dataset curation, quality assurance, along with customly generated examples

Citation

Anyone can freely use and modify this dataset.

License

This dataset is released under [apache-2.0].

@dataset{codex2024,
  title={CodeX-7M-Non-Thinking: Large-Scale Coding Dataset},
  author={Parvesh Rawal at Modotte},
  year={2024},
  publisher={Modotte},
  url={https://huggingface.co/datasets/Modotte/CodeX-7M-Non-Thinking}
}

Contact

For questions, suggestions, or collaboration opportunities:

• Email: Modotte
• Twitter: [@Modotte]
• GitHub: [Modotte]

---

Built with ❤️ by Modotte - Advancing AI through high-quality data