LLVM Opt Benchmark

IMPORTANT: This repository has moved to https://github.com/dtcxzyw/llvm-opt-benchmark-nightly.

---

LLVM Opt Benchmark is an LLVM IR dataset for data-driven compiler optimization research. This repository is also used by LLVM developers to evaluate the impact of their patches on real-world applications.

Don't submit PR to add new benchmarks. You can request new open-source C/C++/Rust repos here.

Please cite this work with the following BibTex entry:

@misc{opt-benchmark,
  title = {LLVM Opt Benchmark},
  url = {https://github.com/dtcxzyw/llvm-opt-benchmark},
  author = {Yingwei Zheng},
  year = {2023},
}

FAQs

LLVM developers use this corpus to assess the impact of your patches on real-world applications. If you see a link to this repository in your PR, it means that the target PR demonstrates some performance regressions or improvements caused by your changes. Here are some common questions you may have:

How can I reproduce the regression locally?

You should be able to reproduce the regression locally in the following steps:

# Apply your patch and rebuild opt.
...
# Download the source IR. Note that you should replace `optimized` with `original`.
wget https://raw.githubusercontent.com/dtcxzyw/llvm-opt-benchmark/refs/heads/main/bench/<program_name>/original/<file_name>.ll
# Run opt to generate the optimized IR.
bin/opt -O3 -disable-loop-unrolling -vectorize-loops=false -vectorize-slp=false -S <file_name>.ll -o opt.ll

Note that you don't need to clone the whole repository.

How can I evaluate my patch on this benchmark locally?

It is not recommended, as you can use the online service to evaluate your patch on GitHub if you have commit access to the LLVM repository.

You can use python3 ./scripts/gen_optimized.py bench <path-to-opt>. It will update the optimized IR files. Then you can review the diff with git.

The compile-time evaluation shows a huge impact on some files. What should I do?

Don't worry about it. If it doesn't affect the compile-time of the parent projects, it is generally acceptable. Otherwise, you may need to adjust the threshold or just handle simple cases.

What should I do when I see a regression?

Don't panic. Perfect is the enemy of good. We never ask the contributors to fix all the regressions before landing their patches.

Please follow the InstCombineContributorGuide to generalize your patch to cover the regression. If it doesn't work, try to find the pattern and file a separate issue. If it is hard to be caught by a separate transformation, try to bail out on the regression case. If we cannot make it better, the patch can still be accepted if the net effect is positive. Ask your reviewer to help you with the decision.

My method is expensive in compile time. But it shows some optimization opportunities. Should I abandon it?

Though we cannot accept the patch, we still encourage you to explore alternative approaches to handle the exposed optimization opportunities. As the distribution of the real-world code is not uniform, in general, a simple heuristic is good enough to cover most of the cases.

The evaluation result shows my patch has no effect on the benchmark. What does it mean?

We ask the issue reporter and the contributor to provide a motivating example from real-world scenarios. This benchmark only provides additional evidences to support the claim. It is highly recommended to run this benchmark if the real-world use case is missing, or it is found by fuzzers and super-optimizers. See also InstCombineContributorGuide.

The following patches may not be suitable for this benchmark:

• SLPVectorizer/LoopVectorize/LoopUnroll patches. Vectorization and loop unrolling are disabled since the diff is huge and hard to review. The performance is highly dependent on the target machine so the running time may be more representative.
• Sanitizer/Instrumentation/GPU patches. The related patterns are not included in this corpus.
• Patches which handle scalable vectors. This corpus only contains fixed-width vectors (generated from X86 intrinsics).

Do the regressions in IR diff imply the run-time performance regressions?

Not necessarily. The IR diff is only a proxy for the run-time performance. Generally fewer instructions at IR level implies better analysis result and less instructions at run-time. However, it depends on the target micro-architecture and the LLVM CodeGen components. For example, a canonicalization in InstCombine may cause the SelectionDAG to not recognize certain patterns, leading to bad codegen. Please refer to llvm-codegen-benchmark for frequent isel patterns. Anyway, the run-time performance should be the golden metric. The IR diff only helps us to find the root cause of regressions.

In addition, most of IR snippets are not the hot paths in the real-world applications. I choose to keep all the source IR files instead of only keeping the hot spots, as it is useful for monitoring the code size changes, which is also critical for the frontend performance on modern devices. Another reason is that we cannot find the hot paths in large applications like LLVM and verilator-generated simulators. BTW the training data for PGO in some programs is unavailable or highly biased, you know :).

The IR diff looks weird. It contains some invalid instructions. Is it a bug?

Many IR diffs only change the name of instructions and basic blocks. Previously, I used llvm-diff to reduce meaningless changes. However, it is slow and ineffective. Now I use a heuristic name-remapping algorithm to reduce the noise. The algorithm can reduce up to 70% of line changes. However, as it works on textual diff and does not understand the semantics of LLVM IR, it may produce some invalid instructions. Please check the raw diff in the previous commit pre-commit: Update.

The IR diff contains hundreds of file changes. How can I review it efficiently?

To fit the GitHub's limit of diff rendering, only part of the files are picked to be committed. It is chosen by a heuristic algorithm to improve the diversity of the dataset.

In the diff mode, a summary of the diff is also provided. It contains some key information to allow you to quickly review the changes:

• The number of files changed, lines added and removed (provided by git diff --shortstat). It is different from the numbers on the GitHub page, as it counts the statistics before diff reduction.
• A summary of the top-10 LLVM statistics changes.
• The number of line changes in each file (provided by git show <base>..HEAD --numstat --oneline). You can use this to quickly find the file with the most line additions or deletions (e.g., cat log | awk '{print $1 - $2, $3}' | sort -n).
• A summary from LLM (powered by Qwen). It provides a high-level overview of the changes. However, it always gives a positive response, so it may not be very useful. You can use it to find the files that are worth reviewing in detail.

From my own experience, the patterns are likely to be similar in the same project. So you can skip the whole project after you review the first few files in the same project. If your patch optimizes the C++/Rust standard library and other widely-used libraries, you can also skip the files with similar bb names in the hunk header (e.g., _ZNSt6vector...).

The IR diff is totally unrelated to my patch. Why?

Your changes may break existing optimizations. Please reproduce it locally and try to provide a minimal phase-ordering regression test. Then follow the instructions for dealing with regressions above.

Online services (previously hosted by PLCT Lab, ISCAS/currently hosted by SUSTech ARiSE Lab)

Special Acknowledgement: Thank @goldsteinn for providing additional computational resources to meet the growing demand for testing!

• Fuzzy DAG matching

  Please file an issue to provide LLVM IR with a single function. I will add the grep label to trigger CI.

  Example: https://github.com/dtcxzyw/llvm-opt-benchmark/issues/1072

• Middle-end optimization pre-commit testing

  Ping me if you want to see what is affected by your PR. It is useful for reviewers to find potential performance regressions and new optimization opportunities.

  For convenience, all llvm members are authorized to request pre-commit tests in https://github.com/dtcxzyw/llvm-opt-benchmark/issues/1312. Some basic PR editing commands are also supported by leaving a comment starts with /:

  • /close : Close the PR
  • /reopen: Reopen the PR
  • /add-label labels: Add labels (separated by comma). Available labels: reviewed, regression, crash, hang and miscompilation.
  • /remove-label labels: Remove labels.

• Codegen pre-commit testing

  See also llvm-codegen-benchmark.

• Weekly coverage report:

  https://dtcxzyw.github.io/llvm-opt-benchmark/

Benchmark List

Currently, this repository contains the following libraries/applications: