cudametal

Compiles a useful subset of CUDA C++ (.cu) into a real Metal binary and runs it on macOS/Apple Silicon GPUs -- no NVIDIA GPU needed.

For simple example look into examples/rescale.

How it works

bin/cudametalcc takes a .cu file and runs it through a real compiler pipeline, entirely built from public, documented tools (no attempt to reverse-engineer NVIDIA's or Apple's own internals):

foo.cu (real CUDA syntax: __global__, threadIdx, blockIdx, <<<>>>, ...)
  -> clang -cc1 -x cuda            (unpatched upstream Clang, device IR,
                                     spirv64 target)
  -> llvm-link, opt                (always-inline + globaldce)
  -> llc -mtriple=spirv64-unknown-unknown -> spirv-dis
  -> spirv-to-msl/kernel_spirv_to_msl.py   (this project's own from-
                                             scratch SPIR-V -> MSL
                                             translator)
  -> xcrun metal / metallib        (a real .metallib)
  -> clang -x cuda --cuda-host-only -fcuda-include-gpubinary
                                    (embeds the .metallib; Clang's own
                                     CUDA host-side codegen auto-generates
                                     the __cudaRegisterFatBinary/
                                     __cudaRegisterFunction calls)
  -> link against runtime/libcudametalrt (a libcudart-compatible host
                                           shim: cudaMalloc/cudaMemcpy/
                                           <<<>>> launch, backed by real
                                           Metal buffers and command
                                           queues)
  -> a native, runnable macOS executable

kernel_spirv_to_msl.py is a purpose-built translator for exactly the Kernel/OpenCL-flavor SPIR-V this pipeline produces (SPIRV-Cross only accepts Shader/Vulkan-flavor SPIR-V, which Clang's CUDA path never emits). See DESIGN.md for the full design and supported-feature list, and spirv-to-msl/README.md for the translator's own running bug/fix log.

Translating a real third-party CUDA project

repos_build_artifacts/ holds build configs for fetching and building real, independently-written CUDA projects through cudametal -- a much better way to find translator gaps than writing kernels to order (see spirv-to-msl/README.md's "Status update" log for what this has already found and fixed).

Example: Karpathy's llm.c

llm.c is a real, independently written GPT-2 training implementation in raw CUDA. Its fp32-only training program (train_gpt2_fp32.cu) builds and trains a real GPT-2-124M model end to end through cudametal, completely unmodified.

Fetch, drop in cudametal's own Makefile.cudametal, and build:

$ bin/fetch_build_llm_c
+ updating existing checkout at repos_build_artifacts/llm.c/checkout
HEAD is now at f1e2ace Merge pull request #801 from ngc92/ngc92/fix-test
+ copying repos_build_artifacts/llm.c/Makefile.cudametal -> .../checkout/
+ building (CUDAMETAL_DIR=/Users/mkunc/src/cudametal)
rm -f train_gpt2 train_gpt2cu_metal train_gpt2fp32cu_metal
cc -Ofast ... train_gpt2.c -lm -o train_gpt2
bin/cudametalcc train_gpt2_fp32.cu -o train_gpt2fp32cu_metal
...
cudametalcc: built train_gpt2fp32cu_metal (kernels: encoder_forward_kernel3,
  encoder_backward_kernel, layernorm_forward_kernel3, permute_kernel,
  permute_kernel_backward, unpermute_kernel, unpermute_kernel_backward,
  softmax_forward_kernel5, residual_forward_kernel, gelu_forward_kernel,
  gelu_backward_kernel, matmul_backward_bias_kernel4,
  layernorm_backward_kernel2, softmax_autoregressive_backward_kernel,
  adamw_kernel2, fused_classifier_kernel3, matmul_forward_kernel4)
--- train_gpt2cu (bf16, cublasLt fused epilogue) is a known,
    documented cudametal gap -- run 'make -f Makefile.cudametal
    build-cuda-bf16' directly to see it fail and capture why.

built:
  repos_build_artifacts/llm.c/checkout/train_gpt2
  repos_build_artifacts/llm.c/checkout/train_gpt2fp32cu_metal

Then, with the starter-pack data files in place (see repos_build_artifacts/README.md for where to get them), run a real training job:

$ cd repos_build_artifacts/llm.c/checkout
$ ./train_gpt2fp32cu_metal
+-----------------------+----------------------------------------------------+
| Parameter             | Value                                              |
+-----------------------+----------------------------------------------------+
| train data pattern    | dev/data/tinyshakespeare/tiny_shakespeare_train.bin |
| val data pattern      | dev/data/tinyshakespeare/tiny_shakespeare_val.bin  |
| batch size B          | 4                                                  |
| sequence length T     | 1024                                               |
+-----------------------+----------------------------------------------------+
| device                | Apple M4 Max                                       |
+-----------------------+----------------------------------------------------+
| vocab_size V          | 50257                                              |
| num_layers L          | 12                                                 |
| num_parameters        | 124475904                                          |
+-----------------------+----------------------------------------------------+
| train_num_batches     | 74                                                 |
+-----------------------+----------------------------------------------------+
allocated 474 MiB for model parameters
allocated 5706 MiB for activations
val loss 4.513920
step    1/74: train loss 4.287299 (642.053000 ms, 6379 tok/s)
step    2/74: train loss 4.724538 (579.333000 ms, 7070 tok/s)
...
step   20/74: train loss 3.680017 (597.657000 ms, 6853 tok/s)
val loss 3.728942
generating:
---
O, but laugh it; that it's too late.

<|endoftext|>I will I shall I cause you, God being
to leave him.
---
...
step   74/74: train loss 3.379404 (661.166000 ms, 6195 tok/s)
val loss 3.491368
generating:
---
BUCKINGHAM:
But of my fair cousin Grey the head-horse,
Is wind-tout'd with sweet-tongued beards;
To wound her in some bitter advocate
Is free smile on the face plaiden of her curst bark:
Though I might allay her
---
total average iteration time: 629.044446 ms

A real, complete, 74-step GPT-2 training run: finite, decreasing validation loss throughout, coherent generated Shakespeare-style text -- zero changes to llm.c's own source. See repos_build_artifacts/README.md for the full details on this and how to add another project.

Example: tugot17's PMPP worked solutions

tugot17/pmpp is a real, independently-written set of worked solutions to Kirk & Hwu's Programming Massively Parallel Processors. Unlike llm.c, its own original files aren't directly buildable through cudametalcc as-is (PyTorch host code, benchmark harnesses); cudametal's own examples/pmpp/ is the adapted port -- 19 files across 15 chapters, each already part of the e2e suite.

$ bin/fetch_build_pmpp
+ cloning https://github.com/tugot17/pmpp.git -> repos_build_artifacts/pmpp/checkout
+ copying repos_build_artifacts/pmpp/Makefile.cudametal -> .../checkout/
+ building (CUDAMETAL_DIR=/Users/mkunc/src/cudametal)
bin/cudametalcc examples/pmpp/ch02_vector_multiplication/vecmul.cu -o out/vecmul
cudametalcc: built out/vecmul (kernels: vecMulKernel)
...
bin/cudametalcc examples/pmpp/ch18_electrostatic_gather/electrostatic_gather.cu -o out/electrostatic_gather
cudametalcc: built out/electrostatic_gather (kernels: cenergyGatherKernel)

built (in repos_build_artifacts/pmpp/checkout/out/): 19 binaries

Every binary is a self-contained PASS/FAIL check (real Metal execution verified against a CPU reference) -- run one directly:

$ ./repos_build_artifacts/pmpp/checkout/out/rgb_to_grayscale
PASS: 128x128 image verified correct (PMPP ch3 rgb_to_grayscale)

or run all 19 at once with make -C repos_build_artifacts/pmpp/checkout -f Makefile.cudametal CUDAMETAL_DIR="$(pwd)" run -- see repos_build_artifacts/README.md for the full details.