tpcgen-rs: Quickly and Easily Create TPC-* Benchmark Data

[!NOTE] Made with ❤️ by @clflushopt, @alamb and @kevinjqliu

Originally clflushopt/tpchgen-rs, the project is now maintained at datafusion-contrib/tpcgen-rs

Modern, blazing fast and easy to use TPC-H and TPC-DS benchmark data generators.

Goal

Democratize the comparison of analytical systems by making it easy to efficiently generate benchmark data in modern formats such as Apache Parquet, Apache Arrow, and CSV.

Features

1. Blazing Speed 🚀
2. Easy to Use: CLI or embeddable libraries
3. Obsessively Tested: byte-for-byte identical with the reference generators 📋
4. Resource Efficient: multi-core and constant memory use 🧠

Quick Start

tpcgen-cli is distributed as a Python package. Run it with uvx:

uvx tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpch
uvx tpcgen-cli tpcds parquet -s 1 --output-dir /tmp/tpcds

Or install it with pip:

python -m pip install tpcgen-cli
tpcgen-cli tpch parquet -s 1 --output-dir /tmp/tpch

See the tpcgen-cli README for more examples.

Representative Performance (10x Faster)

Time to generate TPC-H Scale Factor 100 data in Parquet format. See the performance spreadsheet for details and more comparisons.

Sub projects

Testing

We go to great lengths to ensure our data generators produce the same exact bytes as the reference implementations, both for single and multi-part output. Our multi-level strategy for ensuring correctness is detailed in TESTING.md and includes comparing the output of our generators to the original reference implementations as part of CI.

Contributing

Pull requests are welcome. For major changes, please open an issue first for discussion. See our contributors guide for more details.

Architecture

Please see architecture guide for details on how the code is structured.

License

The project is licensed under the Apache License 2.0.