cugpt — C Translation of Karpathy's microgpt

Two C implementations of Andrej Karpathy's microgpt.py: a minimal GPT that trains a character-level language model on a names dataset and generates hallucinated names.

Quick Start

make
./microgpt    # 452× faster than Python
./cugpt       # 2476× faster than Python

Both download input.txt automatically on first run. The original Python reference is at the gist (not included here).

Model

1-layer GPT-2 variant · vocab=27 · n_embd=16 · n_head=4 · block_size=16 · 4,192 params RMSNorm · ReLU · no biases · Adam (lr=0.01, β₁=0.85, β₂=0.99, linear decay) · 1,000 steps

Benchmark

Machine: Ubuntu 22.04 / WSL2, gcc 14.2.0, Python 3.13.3 — 1,000 training steps on names dataset.

Python baseline

microgpt.c — compiler flag sweep

cugpt.c — compiler flag sweep

Summary

Key observations

• -ffast-math is the biggest single-flag win (~2×) — it lets the compiler reassociate FP ops and use faster math approximations. Safe here since we only need gradient descent, not bit-exact reproducibility.
• -funroll-loops helps microgpt.c (tight loops over tape entries) but hurts cugpt.c (BLAS already handles inner loops).
• -flto wins for cugpt.c by inlining the thin BLAS wrapper functions across translation units.
• Even unoptimized cugpt.c -O0 (0.189 s) beats the best microgpt.c build (0.323 s) — eliminating the autograd tape is an algorithmic win, not a compiler win.

Implementation Notes

microgpt.c translates the Python scalar-Value autograd graph to a flat Wengert tape (a pre-allocated 4M-entry arena, reset each step). Backward is a single reverse scan — no pointer chasing, no heap allocation, no recursive topo-sort.

cugpt.c discards the tape entirely. Forward activations are saved in a struct array (Activations act[BLOCK_SIZE]). The backward pass is written analytically. Every matrix-vector product is dispatched through cblas_dgemv; weight gradient accumulation uses cblas_dger (rank-1 outer-product update).

Build

./configure && make        # detects compiler, OpenBLAS, cblas.h
make check                 # smoke test (10 steps each)
make bench                 # full flag sweep + Python baseline
make install               # to /usr/local (or ./configure --prefix=~/.local)
make dist                  # creates cugpt-1.0.tar.gz

Dependencies:

• microgpt.c — C compiler + libm (always present)
• cugpt.c — C compiler + libopenblas-dev (apt install libopenblas-dev)

See INSTALL for manual build instructions.

License

MIT — see COPYING. Original microgpt.py by Andrej Karpathy, shared under the MIT convention used across his educational ML projects (micrograd, makemore).