chipStar

chipStar enables compiling and running HIP and CUDA applications on platforms which support SPIR-V as the device intermediate representation. It supports OpenCL and Level Zero as the low-level runtime alternatives.

• User documentation
• Developer documentation
• A list of (un)supported features

The chipStar article provides an overview of chipStar. It is the preferred citation for academic publications referring to chipStar somehow.

The name chipStar comes from cuda and hip and the word Star which means asterisk, a typical shell wildcard, denoting the intention to make "CUDA and HIP applications run everywhere". The project was previously called CHIP-SPV.

Library Support

chipStar supports a growing set of HIP/ROCm libraries. Use install_chipstar.py to install chipStar along with any of these libraries:

# Interactive mode with TUI
./install_chipstar.py

# Install specific components
./install_chipstar.py --components chipstar,rocprim,hipcub

# Install everything
./install_chipstar.py --all --install-dir ~/install/HIP

# List available components
./install_chipstar.py --list

Note: When run from within the chipStar repository, the installer will use the current repository instead of cloning a fresh copy. This is useful when testing local changes or working with a specific branch.

Platform-Independent Libraries

These libraries have been ported to work with chipStar on any supported platform:

• rocPRIM - Parallel primitives library
• hipCUB - CUB-like primitives for HIP
• rocThrust - Thrust parallel algorithms
• rocRAND - Random number generation
• hipRAND - HIP random number interface
• rocSPARSE - Sparse matrix operations
• hipSPARSE - HIP sparse matrix interface
• hipMM - HIP memory manager (RMM port)

Intel MKL-Based Libraries

These libraries are ported for Intel GPUs using oneAPI MKL as backend:

• H4I-MKLShim - Intel MKL shim layer (required by libraries below)
• H4I-HipBLAS - hipBLAS via MKL
• H4I-HipFFT - hipFFT via MKL
• H4I-HipSOLVER - hipSOLVER via MKL

If there is a library that you need that is not yet supported, please open an issue stating which libraries you require and what application you are trying to build.

Applications

chipStar has so far been tested using the following applications:

• libCEED Our fork includes some workarounds.
• GAMESS Source code is not public.
• HeCBench CUDA Benchmarks.

Getting Started

Quickest way to get started is by using a prebuilt Docker container. Please refer to Docker README If you want to build everything yourself, you can follow a detailed Getting Started

Development Status and Maturity

While chipStar 1.3 can already be used to run various large HPC applications successfully, it is still heavily in development mode with plenty of known issues and unimplemented features. There are also known low-performance optimizations that are still to be done. However, we consider chipStar ready for wider-range testing and welcome community contributions in form of reproducible bug reports and good quality pull requests.

Release notes for 1.3, 1.2, 1.1, 1.0 and 0.9.

Prerequisites

• Cmake >= 3.20.0
• Clang and LLVM 21, 22, 23 (plus an experimental latest option tracking the maintained chipStar-llvm-23 branch)
  • Can be installed, for example, by adding the LLVM's Debian/Ubuntu repository and installing packages 'clang-21 llvm-21 clang-tools-21'.
  • For the best results, build Clang/LLVM with the chipStar patches applied (from llvm-patches/llvm-<version>/). See below for a scripted way to build and install the patched versions.
• SPIRV-LLVM-Translator from a branch matching the LLVM major version: (e.g. llvm_release_210 for LLVM 21, llvm_release_220 for LLVM 22) , llvm-spirv.
  • Make sure the built llvm-spirv binary is installed into the same path as clang binary, otherwise clang might find and use a different llvm-spirv, leading to errors.
• SPIRV-Tools and SPIRV-Headers:
  • If available on the system, the system installation will be used
  • If not found, the submodules will be automatically initialized and built when needed

Compiling Clang, LLVM and SPIRV-LLVM-Translator

It's recommended to build LLVM with the chipStar patches applied (a few fixes not yet in the upstream release branches). For this you can use a script included in the chipStar repository:

./scripts/configure_llvm.sh
Usage: ./scripts/configure_llvm.sh --version <version> --install-dir <dir> --link-type static/dynamic(default) [--variant translator|native] [--with-binutils [path]] [--configure-only] [-N]
--version: LLVM version 21, 22, 23, or latest (experimental, tracks the maintained chipStar-llvm-23 branch, no patches)
--install-dir: installation directory
--link-type: static or dynamic (default: dynamic)

./scripts/configure_llvm.sh --version 22 --install-dir /opt/install/llvm/22.0
cd llvm-project/llvm/build_22
make -j 16
<sudo> make install

Or you can do the steps manually (clone the release branches, apply the patches from llvm-patches/llvm-<version>/, then build):

git clone --depth 1 https://github.com/llvm/llvm-project.git -b release/22.x
cd llvm-project
for p in /path/to/chipStar/llvm-patches/llvm-22/llvm/*.patch; do git apply "$p"; done
cd llvm/projects
git clone --depth 1 https://github.com/KhronosGroup/SPIRV-LLVM-Translator.git -b llvm_release_220
cd SPIRV-LLVM-Translator
for p in /path/to/chipStar/llvm-patches/llvm-22/spirv-translator/*.patch; do git apply "$p"; done
cd ../../..

# DLLVM_ENABLE_PROJECTS="clang;openmp" OpenMP is optional but many apps use it
# DLLVM_TARGETS_TO_BUILD Speed up compilation by building only the necessary CPU host target
# CMAKE_INSTALL_PREFIX Where to install LLVM

cmake -S llvm -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_PROJECTS="clang;openmp" \
  -DLLVM_TARGETS_TO_BUILD=X86 \
  -DCMAKE_INSTALL_PREFIX=$HOME/local/llvm-22
make -C build -j8 all install

OpenCL Backend

• An OpenCL 2.0 or 3.0 driver with at least the following features supported:
  • Coarse-grained buffer shared virtual memory (SVM)
  • SPIR-V input
  • Generic address space
  • Program scope variables
• Further OpenCL extensions or features might be needed depending on the compiled CUDA/HIP application. For example, to support warp-primitives, the OpenCL driver should support also additional subgroup features such as shuffles, ballots and cl_intel_required_subgroup_size.

Level Zero Backend

• Intel Compute Runtime or oneAPI
• oneAPI Level Zero Loader
• For HIP-SYCL and HIP-MKL Interoperability: oneAPI

Downloading Sources

You can download and unpack the latest released source package or clone the development branch via git. We aim to keep the main development branch stable, but it might have stability issues during the development cycle.

To clone the sources from Github:

git clone https://github.com/CHIP-SPV/chipStar.git
cd chipStar
git submodule update --init --recursive

Building and Installing

mkdir build && cd build

# LLVM_CONFIG_BIN is optional if LLVM can be found in PATH or if not using a version-sufficed
# binary (for example, llvm-config-19)

cmake .. \
    -DLLVM_CONFIG_BIN=/path/to/llvm-config
    -DCMAKE_INSTALL_PREFIX=/path/to/install
make all build_tests install -j8

NOTE: If you don't have libOpenCL.so (for example from the ocl-icd-opencl-dev package), but only libOpenCL.so.1 installed, CMake fails to find it and disables the OpenCL backend. This issue describes a workaround.

Build Options

CHIP_ENABLE_DEVICE_PROGRAM_SCOPE_GLOBALS (default: ON)

Controls whether chipStar emits program-scope (SPIR-V CrossWorkgroup) global variables for device-side features that require them:

• device-side dynamic memory allocation (malloc/free, backed by the __chipspv_device_heap global), and
• clock() / clock64() / wall_clock() / wall_clock64() (backed by the __chip_clk_counter global).

Program-scope globals are initialized at module load through shadow kernels, and some OpenCL drivers (for example rusticl/radeonsi) cannot consume them at all (#1279). Disabling this option omits those globals, which avoids their per-module initialization overhead (#582) and restores compatibility with such drivers, at the following cost:

• device-side malloc/free become unavailable, and
• clock* / wall_clock* remain callable but return 0.

To build with these features disabled:

cmake .. -DCHIP_ENABLE_DEVICE_PROGRAM_SCOPE_GLOBALS=OFF ...

Building on ARM + Mali

To build chipStar for use with an ARM Mali G52 GPU, use these steps:

1. build LLVM and SPIRV-LLVM-Translator as described above

2. build chipStar with -DCHIP_MALI_GPU_WORKAROUNDS=ON cmake option

There are some limitations - kernels using double type will not work, and kernels using subgroups may not work.

Note that chipStar relies on the proprietary OpenCL implementation provided by ARM. We have successfully managed to compile and run chipStar with an Odroid N2 device, using Ubuntu 22.04.2 LTS, with driver version OpenCL 3.0 v1.r40p0-01eac0.

Building on RISC-V + PowerVR

To build chipStar for use with a PowerVR GPU, the default steps can be followed. There is an automatic workaround applied for an issue in PowerVR's OpenCL implementation.

There are some limitations: kernels using double type will not work, kernels using subgroups may not work, you may also run into unexpected OpenCL errors like CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST and other issues.

Note that chipStar relies on the proprietary OpenCL implementation provided by Imagination Technologies. We have successfully managed to compile and run chipStar with a VisionFive2 device, using VisionFive2's pre-built Debian image 202403, driver version 1.19. Other SBCs may require additional workarounds.

Running Unit Tests

There's a script check.py which can be used to run unit tests and which filters out known failing tests for different platforms. Its usage is as follows.

BUILD_DIR={path to build directory. Make sure that build_tests target has been built}

BACKEND={opencl/level0}
^ Which backend/driver/platform you wish to test:
"opencl" = Intel OpenCL runtime, "level0" = Intel LevelZero runtime 

DEVICE={cpu,igpu,dgpu,pocl}         # What kind of device to test.
^ This selects the expected test pass lists.
  'igpu' is a Intel Iris Xe iGPU, 'dgpu' a typical recent Intel dGPU such as Data Center GPU Max series or an Arc.

export CHIP_PLATFORM=N         # If there are multiple OpenCL platforms present on the system, selects which one to use.

You can always verify which device is being used by chipStar by:
CHIP_LOGLEVEL=info ./build/hipInfo

python3 $SOURCE_DIR/scripts/check.py $BUILD_DIR $DEVICE $BACKEND

Please refer to the user documentation for instructions on how to use the installed chipStar to build CUDA/HIP programs.

Tools

chipStar includes several utility tools for working with SPIR-V binaries and OpenCL compilation. These tools are built as part of the chipStar build process and installed in the bin directory.

SPIR-V Extractor Tool

The spirv-extractor tool extracts and validates SPIR-V binaries from HIP fatbinaries.

Usage

spirv-extractor [--check-for-doubles] [--validate] [-o <output_filename>] [-h] <fatbinary_path> [<additional_args>...]

Options

• --check-for-doubles: Check if SPIR-V uses double precision and skip test if so
• --validate: Perform comprehensive SPIR-V verification (syntax) and validation (spec compliance)
• -o <filename>: Output SPIR-V to specified file (both binary .spv and text .txt formats)
• -h: Show help message

Examples

Extract SPIR-V from a fatbinary and display as text:

./build/bin/spirv-extractor my_kernel.fatbin

Extract and save SPIR-V to files:

./build/bin/spirv-extractor -o kernel.spv my_kernel.fatbin

Validate SPIR-V compliance:

./build/bin/spirv-extractor --validate my_kernel.fatbin

Check for double precision usage (used internally by test framework):

./build/bin/spirv-extractor --check-for-doubles my_kernel.fatbin

The validation feature performs both SPIR-V specification validation and HIP-specific constraint checking, including verification of memory models, execution models, capabilities, and other requirements for HIP kernel compatibility.

OpenCL SPIR-V Compiler Tool

The opencl-spirv-compiler tool compiles SPIR-V binaries or assembly files using the OpenCL runtime to generate device-specific binaries.

Usage

opencl-spirv-compiler <file(s) or directory>

Features

• Accepts both SPIR-V binary files (.spv) and SPIR-V assembly files (.spvasm)
• Automatically detects file type using the file command
• Can process individual files or entire directories recursively
• Generates device-specific binary files with _device.bin suffix
• Provides detailed error messages including OpenCL build logs

Examples

Compile a single SPIR-V binary:

./build/bin/opencl-spirv-compiler kernel.spv

Compile a SPIR-V assembly file:

./build/bin/opencl-spirv-compiler kernel.spvasm

Process all SPIR-V files in a directory:

./build/bin/opencl-spirv-compiler /path/to/spirv/files/

The tool will create output files with the naming pattern <input_filename>_device.bin containing the compiled device-specific binary code.

Environment Variables

CHIP_BE=<opencl/level0>                         # Selects the backend to use. If both Level Zero and OpenCL are available, Level Zero is used by default
CHIP_PLATFORM=<N>                               # If there are multiple platforms present on the system, selects which one to use. Defaults to 0
CHIP_DEVICE=<N>                                 # If there are multiple devices present on the system, selects which one to use. Defaults to 0
CHIP_DEVICE_TYPE=<gpu/cpu/accel/fpga/pocl> or empty  # Selects which type of device to use. Cannot be used with CHIP_PLATFORM/CHIP_DEVICE. Defaults to empty.
CHIP_LOGLEVEL=<trace/debug/info/warn/err/crit>  # Sets the log level. If compiled in RELEASE, only err/crit are available
CHIP_DUMP_SPIRV=<ON/OFF(default)>               # Dumps the generated SPIR-V code to a file
CHIP_DUMP_PROCESSED_SPIRV=/path/to/dir          # Dumps processed SPIR-V binaries to the given directory (for debugging)
CHIP_JIT_FLAGS=<flags>                          # Additional JIT flags
CHIP_L0_COLLECT_EVENTS_TIMEOUT=<N(30s default)> # Timeout in seconds for collecting Level Zero events
CHIP_L0_EVENT_TIMEOUT=<N(0 default)             # Timeout in seconds for how long Level Zero should wait on an event before timing out
CHIP_SKIP_UNINIT=<ON/OFF(default)>              # If enabled, skips the uninitialization of chipStar's backend objects at program termination
CHIP_MODULE_CACHE_DIR=/path/to/desired/dir      # Module/Program cache dir. Defaults to $HOME/.cache/chipStar, if caching is undesired, set to empty string i.e. export CHIP_MODULE_CACHE_DIR=
CHIP_VERIFY_MODE=<off/failures/all>             # Controls LLVM IR and SPIR-V verification output during compilation. 'off' disables verification, 'failures' (default) shows table only when SPIR-V validation fails, 'all' always shows the verification table

Example:

╭─pvelesko@cupcake ~
╰─$ clinfo -l
Platform #0: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) Arc(TM) A380 Graphics
Platform #1: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) UHD Graphics 770

Based on these values, if we want to run on OpenCL iGPU:

export CHIP_BE=opencl
export CHIP_PLATFORM=1
export CHIP_DEVICE=0

NOTE: Level Zero doesn't have a clinfo equivalent. Normally if you have more than one Level Zero device, there will only be a single platform so set CHIP_PLATFORM=0 and then CHIP_DEVICE to the device you want to use. *You can check the name of the device by running a sample which prints the name such as build/samples/0_MatrixMultiply/MatrixMultiply

Module Cache

Compiled device binaries are cached under CHIP_MODULE_CACHE_DIR (default $HOME/.cache/chipStar). The cache key covers the SPIR-V, the option strings the driver receives, the linked device-library modules, device identity, the set of libraries the runtime loaded during initialization (so upgrading or LD_LIBRARY_PATH-swapping the driver or the Intel Graphics Compiler invalidates entries, even when the driver version string does not change), and compiler-relevant environment variables. Known limitations:

• Intel Compute Runtime settings supplied through an igdrcl.config / neo.config file in the working directory are not part of the key. Note NEO reads that file instead of the environment when it exists, so the same variable set via the file and via the environment produce different driver behavior with the file route invisible to the cache. If you use these files, disable the cache or clear it when changing them.
• Library observation uses /proc/self/maps (Linux only), and only sees libraries loaded during chipStar's backend initialization. If the application initializes OpenCL or Level Zero itself before the first HIP call, the compiler identity degrades to a fixed token and a compiler-only upgrade will not invalidate the cache (a warning is logged).

Troubleshooting

Clang++ Cannot Find libstdc++ When Building chipStar

This occurs often when the latest installed GCC version doesn't include libstdc++, and Clang++ by default chooses the latest found one regardless, and ends up failing to link C++ programs. The problem is discussed here.

The issue can be resolved by defining a Clang++ configuration file which forces the GCC to what we want. Example:

echo --gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/11 > ~/local/llvm-19/bin/x86_64-unknown-linux-gnu-clang++.cfg

Missing Double Precision Support

When running the tests on OpenCL devices which do not support double precision floats, there will be multiple tests that will error out.

It might be possible to enable software emulation of double precision floats for Intel iGPUs by setting two environment variables to make kernels using doubles work but with the major overhead of software emulation:

export IGC_EnableDPEmulation=1
export OverrideDefaultFP64Settings=1

If you device does not support emulation, you can skip these tests providing -DSKIP_TESTS_WITH_DOUBLES=ON option at cmake configure time.