Modotte

---

Note: This dataset is part of the lineup CodeX by Modotte. You can get lots of datasets in this same lineup, with the main focus on providing very high-quality datasets for model training and fine-tuning.

This dataset is fully synthetic, curated from high-quality public sources and enhanced with synthetic data generated using both closed and open-source models. It serves as a strong foundation for instruction-based model tuning and fine-tuning, offering one of the most refined and extensive corpora available for coding tasks with reasoning.

Key Features

• Scale: 2 million examples of highly curated coding data
• Diversity: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
• Quality: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
• Thinking Focus: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
• Accuracy: Verified code executions and correctness validation using automated testing frameworks

Dataset Overview

CodeX-2M-Thinking is a meticulously curated coding dataset designed specifically for instruction-based model tuning and fine-tuning of existing models with enhanced code generation and reasoning capabilities. This fully synthetic dataset represents a large and comprehensively filtered corpus of coding data on the Hugging Face platform, emphasizing a thinking approach with step-by-step reasoning for deeper model training.

How to Use?

pip install -U datasets fsspec

from datasets import load_dataset
dataset = load_dataset("Modotte/CodeX-2M-Thinking")

Key Features

• Scale: 2 million examples of highly curated coding data
• Diversity: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
• Quality: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
• Thinking Focus: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
• Accuracy: Verified code executions and correctness validation using automated testing frameworks

Data Curation Process

This dataset has been carefully constructed through a fully synthetic approach, selectively generating and merging examples to enrich the overall dataset for generation models.

Data Sources

• High-Quality Existing Datasets: Curated from multiple premium coding datasets available online (e.g., from NVIDIA and Modotte's internal collections)
• Synthetic Generation: Fully generated using both closed-source and open-source language models (Modotte)
• Expert Validation: Human-verified code solutions, reasoning, and implementations (Modotte)

Filtering Pipeline

Our rigorous filtering process includes open and closed-source filtering techniques, ensuring only the highest-quality examples are retained:

1. Deduplication: Removal of duplicate problems and code solutions
2. Normalization: Code formatting standardization and syntax cleanup
3. Stopword Processing: Intelligent removal of non-essential comments or boilerplate
4. Quality Scoring: Multi-dimensional quality assessment using metrics like code complexity, readability, and efficiency
5. Ranking-Based Filtering: Advanced ranking algorithms to prioritize top-tier examples based on relevance, novelty, and utility
6. Expert Selections: Manual curation by coding experts to select exemplary samples
7. Answer Verification: Automated testing and execution validation using frameworks like pytest or unit tests
8. Content Filtering: Removal of inappropriate, outdated, or incorrect code
9. Diversity Balancing: Ensuring balanced representation across languages and domains through algorithmic sampling

Problem Complexity Distribution

• Basic Level (30%): Fundamental programming concepts, simple syntax, and basic operations
• Intermediate Level (30%): Multi-function problems requiring modular code and basic algorithms
• Advanced Level (40%): Complex challenges involving data structures, optimization, and system design

Programming Domains Covered

• Algorithms and Data Structures
• Web Development and Frameworks
• Machine Learning and AI Implementations
• System Programming and Operating Systems
• Database Management and SQL/NoSQL
• Software Engineering Best Practices
• Competitive Programming Problems

Note: Domains are for reference only. The actual data is very diverse and covers more domains than stated. The actual data includes more complex and high-level questions than stated, spanning multiple programming languages such as Python, Java, C++, JavaScript, and others.

Use Cases

• Fine-tuning code generation and reasoning capabilities in language models
• Training instruction-following models with a coding and reasoning focus
• Benchmarking model performance on coding tasks, problem-solving, and logical reasoning
• Research in AI-assisted programming, automated code completion, and explainable AI
• Educational applications requiring step-by-step code explanations and reasoning

Dataset Format

Each example contains:

• Problem Statement: Clear coding challenge or task description
• Step-by-Step Solution: Detailed reasoning process
• Code Solution: Final executable code with integrated reasoning

Quality Assurance

• Automated Verification: All code solutions verified using execution environments and testing suites
• Correctness Guarantee: Only problems with verified correct and functional code are included
• Human Review: Sample validation by coding experts
• Automated Checks: Static analysis, linting, and runtime verification where applicable
• Open and Closed-Source Filtering: Integration of proprietary and community-driven tools for enhanced quality control

Performance Metrics

Models trained on this dataset show significant improvements in:

• Code generation accuracy with reasoning
• Efficiency in producing detailed, step-by-step solutions
• Problem-solving speed and logical coherence
• Cross-language and cross-domain code transfer
• Reduction in hallucinated or erroneous code outputs through better reasoning

Acknowledgments

Special thanks to our partners and contributors:

• NVIDIA - Reference datasets; CodeX contains many examples taken from NVIDIA's existing datasets
• Modotte Team - Dataset curation, quality assurance, along with customly generated examples

Citation

Anyone can freely use and modify this dataset.

License

This dataset is released under [apache-2.0].

@dataset{codex2024,
  title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},
  author={Parvesh Rawal at Modotte},
  year={2024},
  publisher={Modotte},
  url={https://huggingface.co/datasets/Modotte/CodeX-2M-Thinking}
}

Contact

For questions, suggestions, or collaboration opportunities:

• Email: Modotte
• Twitter: [@Modotte]
• GitHub: [Modotte]

---

Built with ❤️ by Modotte - Advancing AI through high-quality data