NotSoTiny: A Large, Living Benchmark for RTL Code Generation

---

Table of Contents

• Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Data Processing
• Considerations for Using the Data
• Additional Information
  • License
  • Citation Information
  • Acknowledgements

Summary

NotSoTiny is a large, structurally rich, and "living" benchmark designed to assess Large Language Models (LLMs) on the generation of context-aware RTL (Register-Transfer Level) code. Built from hundreds of real hardware designs produced by the Tiny Tapeout community, this benchmark overcomes the limitations of prior static datasets by periodically incorporating new designs, making it resilient to data contamination.

Unlike previous benchmarks which rely on standalone modules or explicit specifications, NotSoTiny focuses on contextual module completion. In this setup, models are presented with a full design context, where one module is masked. The LLM must infer the missing module's functionality and interface solely from the surrounding implementation, mirroring real-world development scenarios where new components must integrate into existing systems.

This dataset includes the 25-12 release, with 1,114 deduplicated and curated tasks derived from real, taped-out hardware designs, making it significantly larger and more complex than existing RTL benchmarks.

Supported Tasks and Leaderboards

• Task: module-completion (Hardware (Verilog) Code Completion).
• Leaderboard: Solutions are evaluated using syntex checks and functional correctness through Formal Equivalence Checking (via Yosys). A leaderboard tracking model performance is available in the TuRTLe Leaderboard.

Languages

The dataset contains:

• Verilog (for hardware designs, prompts, and golden modules).
• English (for system messages and metadata).

Dataset Structure

Data Instances

Each row in the dataset represents a single module completion task. The model is given a system_message and a prompt (containing the context file task.v), and is expected to generate the missing Verilog code that matches the functionality of the golden_module.

Data Fields

• task_id (string): A unique identifier for the task, formatted as <shuttle_name>_<incremental_number>.
• shuttle_name (string): The identifier of the Tiny Tapeout shuttle from which the design originated (e.g., TT06, TT08).
• project_name (string): The name of the original Tiny Tapeout project containing the task.
• task_name (string): The name of the task directory, corresponding to the specific design module being tested.
• system_message (string): A unified system prompt (same for all tasks) used to instruct the LLM.
• prompt (string): The content of task.v. This contains the surrounding design context with the target module missing, serving as the input for the LLM.
• golden_module (string): The content of original_module.v. This is the ground truth implementation of the maked module, used for formal verification.

Data Splits

The 25-12 release consists of a single split containing 1,114 validated tasks. These tasks are derived from multiple Tiny Tapeout shuttles (TT06, TT07, TT08, TT09, TT10 IHP-02, TT10 IHP-25a, and TTsky25a).

Dataset Creation

Curation Rationale

Current RTL benchmarks suffer from insufficient scale, shallow verification protocols, and a high risk of training data contamination. NotSoTiny was created to address these issues by:

1. Scale & Complexity: Providing tasks with deeper hierarchies and complex control/datapath interactions typical of real hardware.
2. Living Nature: Utilizing the continuous release schedule of Tiny Tapeout shuttles to constantly refresh the benchmark, keeping it ahead of LLM training data.
3. Rigorous Verification: Using formal equivalence checking to ensure functional correctness, as simulation testbenches were found to have low coverage.

Source Data

The designs are sourced from the Tiny Tapeout project repositories on GitHub. These are open-source digital, mixed-signal, and analog circuits submitted by researchers and engineers for fabrication.

Data Processing

The dataset construction pipeline follows these steps:

1. Filtering: Projects are filtered to ensure they contain valid src/ and test/ directories, a Makefile, and a valid info.yaml.
2. Module Aggregation: The vppreproc tool is used to merge project files into a single self-contained Verilog file, preserving internal hierarchies.
3. Task Building: Each aggregated design is decomposed into multiple tasks. For each task, the body of one module is removed (to be generated), while the remaining modules serve as context.
4. Temporal Deduplication: To prevent data leakage and redundancy, designs are deduplicated across shuttles using MinHash and Locality-Sensitive Hashing (LSH). If duplicates exist, only the version from the oldest shuttle is retained.
5. Self-Verification: A final validation step ensures compatibility by verifying the golden solution against itself using the general Yosys formal verification script. Only tasks that pass this check are included.

Considerations for Using the Data

To use this dataset for benchmarking, it is highly recommended to integrate with the TuRTLe framework. TuRTLe is a unified evaluation framework designed to automate the entire benchmarking pipeline for RTL generation.

Specifically, the framework handles:

• Model Serving: It manages model interactions and generation requests.
• Processing & Verification: It executes all necessary processing steps, from initial syntax validation to rigorous Formal Equivalence Checking (using Yosys) against the golden_module.
• Final Reporting: It calculates and outputs the final performance results hsing standardized metrics.

Additional Information

License

The dataset is released under the Apache License 2.0.

Citation Information

@misc{ghorab2025notsotinylargelivingbenchmark,
      title={NotSoTiny: A Large, Living Benchmark for RTL Code Generation}, 
      author={Razine Moundir Ghorab and Emanuele Parisi and Cristian Gutierrez-Gomez and Miquel Albert\'i-Binimelis and Miquel Moreto and Dario Garcia-Gasulla and Gokcen Kestor},
      year={2025},
      eprint={2512.20823},
      archivePrefix={arXiv},
      primaryClass={cs.AR},
      url={[https://arxiv.org/abs/2512.20823](https://arxiv.org/abs/2512.20823)}, 
}

Acknowledgements

The HPAI team behind NotSoTiny would like to thank the Tiny Tapeout community for the open source efforts, which made possible this contribution. Special thanks to Matt Venn for his support.