Benchmarking a single‑prompt Rust OS kernel implementation, comparing Claude Code Fable, Opus, and Sonnet.
82
51 commits
updated Sep 10, 2026
README
JiegeOSBench
A benchmark evaluating how well LLM coding agents can autonomously implement a RISC‑V OS kernel from scratch — running an unmodified Linux nginx binary on QEMU, serving HTTP from the host.
Prompt: You are the AI-Jiege. Your task is to write a RISC-V OS kernel in Rust from scratch, with the goal of running a Linux nginx server in QEMU, accessible from outside. You must run the official nginx binary — modifying the target is not allowed. Design and implement it yourself; do not ask me any questions, I will not answer or provide help. You have all permissions, including searching the web, but must work in the current directory. Keep working until the goal is achieved.
| # | Model | Effort | Harness | First HTTP 200 | Total | Context | Cost | QEMU runs | Test date | Tier |
|---|---|---|---|---|---|---|---|---|---|---|
| 🏅 | GPT-6 Astra | High | Codex | 6min | 9min | 83K | $4 | 9 | 2026-09-05 | 👑 Jiege |
| 🥈 | DeepSeek V4.1 Flash | High | DSH | 30min | 47min | 350K | $0.28 | 36 | 2026-09-10 | 🧠 Intelligent Jiege |
| 🥉 | GPT 5.6 Sol | High | Codex | 33min | 49min | 222K | $14 | 29 | 2026-07-11 | 🧠 Intelligent Jiege |
| 4 | Claude Fable 5 | High | CC | 35min | 41min | 155K | $21 | 4 | 2026-07-05 | 🧠 Intelligent Jiege |
| 5 | Claude Opus 4.8 | High | CC | 40min | 42min | 230K | $12 | 19 | 2026-09-05 | 🧠 Intelligent Jiege |
| 6 | Claude Opus 4.7 | — | CC | 45min | 48min | — | — | — | 2026-04-18 | 🧠 Intelligent Jiege |
| 7 | Claude Fable 5.1 | High | CC | 58min | 1h 45min | 516K | $34 | 13 | 2026-09-05 | 🧠 Intelligent Jiege |
| 8 | Claude Opus 5 | High | CC | 1h 7min | 2h 5min | 334K | $26 | 57 | 2026-07-27 | 🧠 Intelligent Jiege |
| 9 | DeepSeek V4 Pro | High | DSH | 1h 46min | 1h 48min | 503K | $0.86 | 113 | 2026-08-12 | 🧠 Intelligent Jiege |
| 10 | Kimi K3 | High | CC | 1h 48min | 2h 19min | 270K | $11 | 39 | 2026-07-18 | 🧠 Intelligent Jiege |
| 11 | Claude Sonnet 5 | xHigh | CC | 2h 31min | 2h 49min | 804K | $64 | 65 | 2026-07-18 | 🤖 Machine Jiege |
| 12 | GPT 5.6 Luna | xHigh | Codex | 2h 35min | 2h 45min | 243K x4 | $2.3 | 112 | 2026-08-08 | 🤖 Machine Jiege |
| 13 | Claude Opus 4.6 | — | CC | 2h 46min | 2h 46min | — | — | — | 2026-03-24 | 🤖 Machine Jiege |
| 14 | GLM 5.3 | High | CC | 3h 50min | 3h 52min | 593K | $34 | 206 | 2026-08-20 | 🤖 Machine Jiege |
| 15 | GLM 5.3 Flash (fp8) | — | CC | 6h 2min | 7h 10min | 967K | self-hosted | — | 2026-08-31 | 🤖 Machine Jiege |
| 16 | DeepSeek V4 Flash | High | DSH | 6h 30min | 6h 35min | 792K x3 | $1.60 | 216 | 2026-08-01 | 🤖 Machine Jiege |
| 17 | Claude Sonnet 4.6 | — | CC | 16h | 16h | — | $60 | — | 2026-03-18 | 🤖 Machine Jiege |
| 18 | DeepSeek V4 Pro Preview | Max | CC | ❌ | ❌ | — | — | 413 | 2026-07-05 | 💥 Broken Jiege |
| 19 | DeepSeek V4 Flash Vision | High | DSH | ❌ | ❌ | — | — | 112 | 2026-08-21 | 💥 Broken Jiege |
Harness: CC = Claude Code, DSH = DeepSeek Harness.
QEMU runs: commands that booted QEMU (direct qemu-system-riscv64 calls plus the run's own wrapper scripts).
Who is Jiege
In 2019, Jiege ran nginx on rCore, an OS written from scratch during his OS course. "Jiege" became the symbol of peak systems engineering in our community: hand-crafting an OS kernel, proof of uniquely human creativity and drive. Today, with one casual kick, AI can finish in minutes what took us months to build. OS is finished. But what Jiege did back then, anyone can do today — dare to try, and anyone can be Jiege.
GPT-6 Astra — 6min / 9min
OpenAI Codex (desktop) ran for ~6min to first HTTP 200 — by far the fastest run on this board, ~6× quicker than the previous record (GPT 5.6 Sol, 33min). The kernel came out almost entirely in two first-pass writing bursts: zero kernel panics, zero web searches (pure black-box), zero context compactions, one uninterrupted turn (32 API requests, 31 bash tool calls, 1.74M tokens total, peak context 83K). Cost $4. Took the Alpine official-APK route — unmodified nginx 1.30.4 (riscv64) + musl/OpenSSL/PCRE2/zlib, byte-verified against the APK; acceptance suite green by ~8min, goal complete at 8.8min.
| Time | Milestone |
|---|---|
| 00:01 | Official Alpine riscv64 nginx 1.30.4 APK + musl/OpenSSL/PCRE2/zlib fetch script (SHA-256 manifest) |
| 00:02 | Kernel written in one pass: boot, Sv39, traps, ELF loader, VFS, syscalls, VirtIO net — compiles |
| 00:03 | First QEMU boot; musl loader starts the original nginx ELF in user mode |
| 00:04 | nginx init running; inode-uniqueness + directory syscall fixes |
| 00:05 | ioctl/socket write-path fixes; host HTTP request reaches nginx |
| 00:06 | First HTTP 200 OK from host — Server: nginx/1.30.4 🎉 |
| 00:07 | io_setup/socketpair (native AIO); nginx log free of emerg/alert; ELF byte-compare vs APK |
| 00:08 | Final acceptance: 8-way × 120 req, 24MiB, 25 keep-alive, 1MiB file ✅ |
| 00:09 | Goal complete — README + make start/test/verify |
DeepSeek V4.1 Flash — 30min / 47min
DeepSeek Harness ran for ~47min (first HTTP 200 at 30min) with the standard agent preset (write/edit/read available) — the fastest DeepSeek run by a wide margin, and the cheapest successful run on the board at $0.28. 158 model steps, 31.0M tokens total (99.6% cache hit), peak context 350K, zero kernel panics, zero context compactions, zero web searches and zero git clones — the 6,283-line no_std Rust kernel (Sv39 paging, ELF64 loader, ~150 Linux asm-generic syscalls, ramfs + initramfs, virtio-net + smoltcp + epoll/eventfd/timerfd) is entirely its own. Took the static musl cross-compile route: the unmodified official nginx 1.27.4 source built with zig cc -target riscv64-linux-musl into a single static riscv64 ELF. It reached HTTP 200 in only ~36 QEMU boots — 12 hand-found bugs, no blind churn.
| Time | Milestone |
|---|---|
| 00:01 | Environment recon, goal set — rustc + riscv64gc-unknown-linux-musl present, zig installable, crates.io reachable |
| 00:03 | zig installed; nginx/pcre2/zlib sources fetched in the background |
| 00:07 | riscv64-cc zig wrapper verified — static riscv64 musl ELF confirmed loadable at 0x1000000 |
| 00:08 | nginx build handed to a background subagent; kernel skeleton written (boot, console, mm, fdt, arch, trap) |
| 00:12 | First QEMU boot (.bss was wiping the boot stack; console remap and PLIC mapping fixed) |
| 00:14 | Subagent delivers static nginx + config + index.html; main line writes fs, ELF loader, process layer |
| 00:22 | 70KB syscall.rs (~150 calls, numbers from the Linux uapi header); virtio-net + smoltcp wired up |
| 00:26 | Static test program runs end-to-end in user mode — ELF offset/flag, TIOCGWINSZ, sstatus SIE bugs fixed |
| 00:27 | nginx parses its config and listens on 0.0.0.0:80 — host requests still time out |
| 00:29 | __alltraps clobbered user t0; the virtio DMA address had PHYS_OFFSET subtracted from an identity-mapped pointer |
| 00:30 | First HTTP 200 OK from host — Server: nginx/1.27.4 🎉 |
| 00:32 | ABI probe run on the kernel: epoll_event is 16B with padding, not the packed 12B assumed — nginx crash fixed |
| 00:33 | 12-listener pool (concurrent SYNs were RST by smoltcp's single listening socket); stress tests green |
| 00:34 | Packaged: README, Makefile, scripts/test.sh end-to-end suite |
| 00:41 | The suite stalls on the script's own bare wait, which also reaped the QEMU job — rewritten as per-PID waits |
| 00:43 | 13/13 checks green from a clean rebuild, after fixing two bad assertions (host readelf, keep-alive -o /dev/null) |
| 00:44 | Source-integrity check added — the build tree is re-diffed against the official nginx tarball |
| 00:47 | ALL 14 CHECKS PASSED; goal complete — single commit, clean tree ✅ |
Claude Fable 5 — 35min / 41min
Claude Code ran for ~41min, 65 API requests. Total cost approximately $21. Nearly a one-shot success — it wrote the entire kernel from memory with minimal debugging.
| Time | Milestone |
|---|---|
| 00:03 | Rootfs + nginx config files written |
| 00:09 | First Rust source files (main.rs, sbi.rs, ...) |
| 00:17 | Core modules done: mm, trap, fs, task, loader |
| 00:28 | Syscall layer complete, nginx ELF loads |
| 00:32 | QEMU boot: PANIC at trap.rs — page fault |
| 00:34 | QEMU boot: nginx listening on port 80 🎉 |
| 00:35 | First HTTP 200 OK from host — Server: nginx/1.28.3 🎉 |
| 00:37 | Post-fix cleanup (sendfile, README) |
| 00:41 | Final acceptance: full suite green + README written ✅ |
Claude Fable 5.1 — 58min / 1h 45min
Claude Code ran for ~58min to first HTTP 200 (goal complete at 105min) — the deepest kernel of any run: real fork with two nginx worker processes (signals + CoW), AF_UNIX with SCM_RIGHTS, epoll, smoltcp TCP, and busybox sh as init. One kernel panic (+49min, VMA range not free) fixed on the first retry. 163 API requests, 49.5M tokens total (98.7% cache hits), peak context 516K, zero compactions, zero web searches. Cost $34 (Fable 5.1, $10/$50 per M in/out, cache read $0.25/M). Official-Alpine-APK route (nginx 1.28.3, dynamically linked); io_setup left unimplemented — nginx logs one [emerg] at startup but serves normally. Function suite (index/404/HEAD/sendfile 4MiB/keep-alive) green at 62min; signal/concurrency/leak/throughput suites + README + git commit by 105min.
| Time | Milestone |
|---|---|
| 00:01 | Environment survey + overall plan |
| 00:06 | Official Alpine riscv64 nginx 1.28.3 + musl/OpenSSL/PCRE2/zlib ready |
| 00:08 | Kernel skeleton: Cargo, linker script, entry.S, UART, panic handler |
| 00:17 | Trap handling + task/scheduler core |
| 00:22 | VFS + file-description/fd-table layer |
| 00:31 | smoltcp TCP glue + socket syscalls |
| 00:42 | Process/fork/signal/futex syscalls |
| 00:47 | Main entry wired; first compile — only 11 errors |
| 00:49 | KERNEL PANIC #1 (VMA range not free) → fixed; busybox sh boots on the first try — fork/exec/wait all work |
| 00:55 | nginx -t passes (getpwnam//dev/stderr fixes) |
| 00:58 | First HTTP 200 OK from host — Server: nginx/1.28.3 🎉 |
| 01:02 | Suite green: index/404/HEAD/sendfile 4MiB/keep-alive |
| 01:45 | Signal/concurrency/leak/throughput suites, README, git commit ✅ |
GPT 5.6 Sol — 33min / 49min
OpenAI Codex ran for ~33 minutes to first HTTP 200 (it claimed PASS at 36min), then spent another 13 minutes fixing a second-connection bug discovered by the user. Total cost: ~$14.
⚠️ Note: the model initially claimed "done" at 36min, but the second consecutive HTTP request failed. The bug (virtio TX descriptor reuse race) was fixed after user prompt at 49min.
| Time | Milestone |
|---|---|
| 00:01 | Cargo project, Makefile, linker script |
| 00:05 | Linux ABI working: U-mode, ELF load, write/exit syscalls |
| 00:08 | initramfs with Alpine nginx 1.28.3 + musl loader embedded |
| 00:11 | musl loader loads nginx; VFS st_dev/st_ino bug found |
| 00:18 | nginx completes dynamic linking, enters epoll event loop |
| 00:33 | First HTTP 200 OK from official nginx |
| 00:36 | Initial PASS claimed; second request silently fails |
| 00:43 – 00:49 | User prompt → fix TCP FIN lifecycle + virtio TX descriptor pool |
| 00:49 | Final PASS: 2 sequential HTTP 200 ✅ |
Claude Opus 4.8 — 40min / 42min
Claude Code ran for ~40min to first HTTP 200 (complete at 42min). Took the static-build route: nginx 1.26.2 compiled from official source in a riscv64 Alpine container, unmodified. Kernel design is cooperative + poll-driven (no preemption/interrupts — the kernel polls virtio-net), which simplified the trap story. Zero kernel panics, zero web searches, 114 API requests, 0.23M new + 15.2M cache-read + 147K output tokens, peak context 230K, zero compactions. Cost $12 (Opus 4.8, $5/$25 per M in/out, cache read $0.5/M). A chain of realistic nginx-init bugs: geteuid()==0 made nginx think it was root and crash dropping privileges; a zero-based clock made ngx_time_update think time hadn't moved and skip init; getrlimit(RLIMIT_NOFILE)=0 capped connections at zero; finally a double-listen orphan socket (found by pcap: connections were absorbed by an abandoned listener the master never polled). Stable at +41min: sequential requests, 10 concurrent, 404 semantics, keep-alive reuse all pass.
| Time | Milestone |
|---|---|
| 00:07 | Kernel boots; frame allocator + heap + Sv39 paging |
| 00:15 | nginx config + web page embedded; poll event model |
| 00:23 | trap handler + syscall dispatch wired into main |
| 00:25 | User mode: nginx starts! missing pread64 → added with pwrite64 |
| 00:30 | geteuid()==0 privilege-drop crash (nginx reads /etc/passwd) → return non-root |
| 00:33 | Zero-based clock → ngx_time_update early-return → real Unix timestamp |
| 00:35 | getrlimit(RLIMIT_NOFILE)=0 → implement prlimit64 |
| 00:40 | TCP fully connected but nginx silent — pcap: double-listen orphan socket → fix |
| 00:40 | First HTTP 200 OK from host — Server: nginx/1.26.2 🎉 |
| 00:41 | Sequential + 10 concurrent + 404 + keep-alive reuse ✅ |
| 00:42 | Dead-code cleanup + goal complete |
Claude Opus 4.7 — 45min / 48min
Claude Code ran for ~45 minutes of active time to first HTTP 200 (48 minutes total active).
| Time (active) | Milestone |
|---|---|
| 00:02 | Kernel boots, prints via OpenSBI |
| 00:19 | Memory management initialized |
| 00:21 | Virtual memory + paging ON |
| 00:27 | syscalls implemented |
| 00:30 | End-to-end HTTP working (built-in kernel HTTP server) |
| 00:31 | ELF DYN (dynamic linked binary) loading |
| 00:36 | nginx prints version, exits with fault |
| 00:41 | nginx config test passes |
| 00:43 | nginx bind + listen succeeds |
| 00:45 | nginx official binary returns HTTP 200 🎉 |
Claude Opus 5 — 1h 7min / 2h 5min
Claude Code ran for ~67 minutes to first HTTP 200, then spent another 58 minutes fixing TCP/epoll/VirtIO edge cases until fully stable. Zero kernel panics (matched later by GPT-6 Astra). 322 API requests. Cost approximately $26 at the 67min mark. Peak context 334K.
| Time | Milestone |
|---|---|
| 00:04 | Project skeleton, linker script, toolchain verified |
| 00:37 | main.rs written — kernel core complete |
| 00:43 | First QEMU boot: no panic, nginx starts but returns 502 |
| 00:53 | nginx listening on port 80 (QEMU slirp network issue) |
| 01:07 | First HTTP 200 OK 🎉 (but 2nd request fails) |
| 01:07–01:20 | Fix dual-listener race + spurious EOF on keep-alive |
| 01:22–01:23 | Fix RX ring free_chain corruption |
| 01:24–01:35 | Fix smoltcp poll() early exit + TCP Nagle stall |
| 01:36–01:47 | Fix CloseWait data loss + edge-triggered notification suppression (31,222 suppressed events) |
| 02:00 | 3000/3000 keep-alive requests at 1185 req/s ✅ |
| 02:01 | 50 concurrent connections + 320 fresh connections ✅ |
| 02:05 | Final validation complete |
Kimi K3 — 1h 48min / 2h 19min
Claude Code ran for ~2h 19min. 151 API requests, 26.3M tokens total (including cache). Cost approximately $11. Peak context 270K.
| Time | Milestone |
|---|---|
| 00:03 | Project skeleton + nginx 1.26.3 official APK downloaded |
| 00:14 | Start writing kernel code |
| 00:24 | Core modules: SBI, console, entry, mm |
| 00:57 | All modules: trap, task, elf, ramfs, virtio, net, syscall |
| 01:14 | First cargo build + QEMU boot |
| 01:44 | QEMU: first PANIC at virtio.rs |
| 01:48 | nginx returns HTTP 200 OK 🎉 |
| 01:49–02:07 | Multiple PANIC fixes (virtio, task scheduler — 7 bugs total) |
| 02:15 | nginx stable again |
| 02:19 | Final validation: SHA256 + 100 concurrent requests all 200 ✅ |
GPT 5.6 Luna — 2h 35min / 2h 45min
OpenAI Codex (desktop) ran for ~2h 45min of active time (3h 19min wall-clock; 34.6min of API connection-retry gaps in the first 40 minutes excluded). First to take the full glibc dynamic-linking route against the official Debian nginx 1.30.1 binary — and succeed (reproduced by DeepSeek V4.1 Flash on 2026-09-08). The early phase was the hardest: the glibc loader refused to resolve shared libraries until a stack of ABI bugs (auxv order, duplicate argc, fstat st_dev/st_ino collision) were fixed one by one. After dynamic linking succeeded at ~1h, nginx bound 0.0.0.0:80 on the first try and the finish was clean: 3 context compactions (pre-compaction peaks 243K each), 116M tokens total (input 60.1M + cache 55.6M), cost ~$2.3.
| Time (active) | Milestone |
|---|---|
| 00:00 | Task start — minimal kernel skeleton plan |
| 01:29 | Minimal kernel boots in QEMU (UART output) |
| 24:34 | User-mode chain + syscall layer + virtio-net skeleton compile |
| 42:32 | Dynamic loader enters Linux ABI; ld.so.cache issue |
| 49:07 | Context compact #1 |
| 51:04 | auxv order bug fixed (glibc lost AT_PHDR/AT_BASE) |
| 53:42 | Duplicate argc bug fixed (argv[0] was an integer) |
| 61:44 | Dynamic linking OK — all dependency ELFs mapped |
| 93:47 | Context compact #2 |
| 113:17 | Context compact #3; nginx binds 0.0.0.0:80 |
| 155:12 | nginx 200 OK from host 🎉 |
| 164:50 | Final validation + goal complete |
Claude Opus 4.6 — 2h 46min / 2h 46min
Claude Code ran for ~2h 46min.
| Time | Milestone |
|---|---|
| 00:02 | Project skeleton + linker script created |
| 00:25 | nginx completes initialization, writes PID file |
| 01:22 | nginx running! Enters epoll event loop |
| 02:21 | TCP connection detected, nginx receives HTTP request |
| 02:45 | Fix virtio-net recv + epoll data bug |
| 02:46 | nginx returns HTTP 200 🎉 |
Claude Sonnet 5 — 2h 31min / 2h 49min
Claude Code ran for ~2h 49min (active time, 77min permission gap excluded). 616 API requests. The session started by quickly validating nginx behavior via Docker + qemu-riscv64-static, then pivoted to writing a Rust kernel from scratch. The self-written kernel achieved HTTP 200 twice. Total token consumption was 279M (almost all cache hits), peak context 804K. Cost approximately $64.
| Time | Milestone |
|---|---|
| 00:02 | Environment check + Docker RISC-V nginx image pull |
| 00:08 | nginx alpine RISC-V native extraction |
| 00:13 | Docker QEMU user-mode nginx 200 OK 🎉 |
| 00:18 | First Rust source file (main.rs) |
| 00:23 | QEMU self-written kernel boot: PANIC |
| 01:27 | 77min permission wait |
| 02:31 | QEMU self-written kernel: nginx 200 OK 🎉 |
| 02:48 | Second self-kernel success, stable responses |
GLM 5.3 — 3h 50min / 3h 52min
Claude Code ran for ~3h 52min (continuous — no idle or API-retry gaps), 355 API requests (dual zai/z-ai GLM 5.3 aliases). Took the Alpine apk unpack route: official nginx 1.28.3 riscv64 package + musl/openssl/pcre2/zlib unpacked from Alpine v3.22, dynamically linked against the official musl loader. 117M tokens total (99.5% cache hit), peak context 593K, zero context compactions. Cost approximately $34 (GLM 5.3 official ¥8/¥2/¥28 per MTok pricing). 6 kernel panics, all within the first 78min (dtb parse + heap OOM). The long tail was the network stack: epoll syscall routing (nginx's wait went through nr=68), a 16-byte epoll_event padding misparse, and fd-close-not-removed-from-epoll. First HTTP 200 at 3h50min (unstable); stable at 3h52min — 4 sequential + 3 concurrent requests all 200.
| Time | Milestone |
|---|---|
| 00:10 | Alpine v3.22 nginx 1.28.3 apk + musl/openssl/pcre2/zlib deps downloaded |
| 00:44 | fd/socket/epoll layer written |
| 00:56 | First QEMU boot — PANIC at dtb.rs |
| 01:17 | Last of 6 panics fixed (dtb parse + alloc OOM cluster) |
| 02:44 | nginx starts: "using the epoll event method", but no network |
| 03:12 | epoll syscall routing bug found: nginx waits on nr=68 |
| 03:23 | TCP handshake + HTTP GET succeed; epoll not notifying nginx |
| 03:47 | epoll_event 16-byte padding misparse fixed (data at offset 8) |
| 03:50 | First HTTP 200 (34ms); fd-close-not-removed-from-epoll bug |
| 03:52 | Stable: 4 sequential + 3 concurrent requests all 200 ✅ |
| 03:53 | README written, goal complete |
GLM 5.3 Flash (fp8) — 6h 2min / 7h 10min
Self-hosted GLM-5.3-Flash-fp8 (sglang, 1M context) run with Claude Code against the original prompt. PASS: first HTTP 200 at 6h46min wall (6h02min active), stable at 7h09min — 8/8 sequential+concurrent requests 200, independently re-verified from the host. 2,134 API rounds, 1,473 tool calls, 951.9M input / 832K output tokens (client-side prompt caching disabled — an upper bound), peak context 967K with zero compactions. Took the Ubuntu glibc deb route (official nginx 1.18.0 riscv64 core package + libc6/libssl3/libpcre3/zlib1g, dynamically linked against the distribution glibc) — the same hardest route as GPT 5.6 Luna. Output: 12,863 lines of Rust — an 8,006-line kernel with no external kernel dependencies plus a ~4,600-line netdev TCP/IP stack (52 unit tests) and a GDB-RSP debugging client. The session needed 3 harness resumes (wall vs active gap): one was the agent killing its own harness via pkill -f nginx matching the prompt in the process argv; it also honestly reported failure twice mid-run before succeeding.
| Time | Milestone |
|---|---|
| 00:02 | Recon: official nginx 1.18.0 riscv64 debs fetched; nginx verified under qemu-user + strace |
| 00:29 | Kernel skeleton (main.rs, run.sh, initrd) |
| 01:34 | Core kernel: Sv39 MMU, trap, scheduler, ~60 fs syscalls, signals |
| 02:49 | ELF loader: PIE + dynamic linker + auxv (glibc) |
| 03:39 | nginx enters epoll event loop (dynamic linking complete) |
| 04:04 | Event-loop SIGSEGV hunt (page tables, FPU save, frame frees, clock epochs) |
| 05:19 | GuestPageSize + blocked-syscall-return fixes; first SYN-ACK transmitted |
| 06:19 | Final cluster: TCP checksum off-by-24, AF_UNIX EOF notification, trap register save |
| 06:46 | First HTTP 200 from official nginx 🎉 |
| 07:09 | Stable: 8/8 requests 200, README written ✅ |
Claude Sonnet 4.6 — 16h / 16h
Claude Code ran for 16 hours. The total cost was approximately $60.
| Time | Milestone |
|---|---|
| 01:27 | Kernel boots + VirtIO NIC initialized |
| 02:07 | musl dynamic linker successfully loads nginx ELF |
| 05:00 | nginx completes initialization, writes PID file |
| 06:18 | TCP three-way handshake succeeds, curl connects to port 8080 |
| 06:24 | nginx successfully forks worker process |
| 08:40 | Worker enters epoll event loop |
| 09:30 | curl first establishes TCP connection (empty reply) |
| 10:00 | curl first receives response (connection reset) |
| 16:00 | nginx returns HTTP 200 with complete welcome page 🎉 |
DeepSeek V4 Flash — 6h 30min / 6h 35min
Ran for ~6h 35min with the harness's minimal agent preset. First HTTP 200 at 6h30min. 1,088 tool calls (898 bash), 388.5M tokens total (99.1% cache hit), peak context 792K. Cost approximately $1.60, thanks to DeepSeek's ultra-low cache pricing. The path was rough: 31 kernel panics and 2 context compactions before nginx finally served.
| Time | Milestone |
|---|---|
| 00:03 | Project skeleton + nginx 1.30.4 source downloaded + zig cross-compile wrapper |
| 00:28 | First cargo build |
| 00:34 | First QEMU boot (OpenSBI output) |
| 00:39–00:49 | Early PANIC debugging (trap/page faults) |
| 02:23 | nginx worker processes start |
| 02:33 | First curl attempt (fails) |
| 03:27 | Context compact #1 |
| 04:15–04:59 | VirtIO/heap debugging panic cluster |
| 05:34 | Context compact #2 |
| 06:29 | First HTTP 200 OK 🎉 |
| 06:35 | Final validation + goal complete |
DeepSeek V4 Pro — 1h 46min / 1h 48min
Ran for ~108min of active time with the harness's minimal agent preset. First HTTP 200 at 105.6min. 373 model steps, 97.9M tokens total (99.9% cache hit), peak context 503K. Cost approximately $0.86 — the cheapest DeepSeek run, edging out DeepSeek V4 Flash's $1.60 (until V4.1 Flash's $0.28). Zero kernel panics, zero context compactions. The static musl nginx binary was built in parallel by a background subagent (DeepSeek V4 Flash, 11.7M tokens, $0.09) while the main agent wrote the kernel from scratch; two web searches (musl TLS layout, QEMU virtio MMIO) were technical lookups, not solution-finding.
| Time | Milestone |
|---|---|
| 00:00 | Kernel project setup: Cargo, linker script, boot code |
| 00:10 | __trap_return register-restore bug fixed |
| 00:13 | Timer + trap handling work; memory subsystem (frame allocator, Sv39 page tables) |
| 00:16 | Frame allocator double-lock deadlock fixed |
| 00:22 | Hello world runs in user mode; subagent static nginx ready (ET_EXEC) |
| 00:34 | VFS + file syscalls wired up |
| 01:02 | musl malloc mmap overlap with TLS region fixed (mmap region tracking) |
| 01:08 | Networking stage: virtio-net + smoltcp TCP/IP + socket syscalls |
| 01:23 | virtio-net MMIO slot 7 (0x10008000) mapping fixed |
| 01:32 | gettimeofday SBI bug fixed (time-cache freeze) |
| 01:45 | First HTTP 200 OK 🎉 |
| 01:48 | Release build verified + goal complete |
DeepSeek V4 Pro Preview — ❌
Ran for over 16 hours but never reached a working state. Got stuck in dependency hell and architecture dead ends.
DeepSeek V4 Flash Vision — ❌
Experimental deepseek-v4-flash-vision-exp model via DSH (High effort, minimal agent preset), 3 sessions totaling over 5 hours (2026-08-21/22). Never completed the task. Worse, two of the three runs cheated: the first compiled the official Linux 6.12.94 kernel instead of writing one from scratch; the second git cloned anicbeer/Tiny-Rust-Os — a ready-made RISC-V OS that already runs nginx — and modified only ~115 lines to adapt it. The third run (standard toolset) finally wrote a kernel from scratch, but stalled at the memory-management stage after ~105 steps. A clear case of the model ignoring the "from scratch" constraint when left unchecked.
The git history of every run is a complete record exported from the agent's session log (Claude Code, Codex, or DeepSeek Harness).
License
MIT
Contributors
50 commits
1 commits
Benchmarking a single‑prompt Rust OS kernel implementation, comparing Claude Code Fable, Opus, and Sonnet.
82
51 commits
updated Sep 10, 2026
README
JiegeOSBench
A benchmark evaluating how well LLM coding agents can autonomously implement a RISC‑V OS kernel from scratch — running an unmodified Linux nginx binary on QEMU, serving HTTP from the host.
Prompt: You are the AI-Jiege. Your task is to write a RISC-V OS kernel in Rust from scratch, with the goal of running a Linux nginx server in QEMU, accessible from outside. You must run the official nginx binary — modifying the target is not allowed. Design and implement it yourself; do not ask me any questions, I will not answer or provide help. You have all permissions, including searching the web, but must work in the current directory. Keep working until the goal is achieved.
| # | Model | Effort | Harness | First HTTP 200 | Total | Context | Cost | QEMU runs | Test date | Tier |
|---|---|---|---|---|---|---|---|---|---|---|
| 🏅 | GPT-6 Astra | High | Codex | 6min | 9min | 83K | $4 | 9 | 2026-09-05 | 👑 Jiege |
| 🥈 | DeepSeek V4.1 Flash | High | DSH | 30min | 47min | 350K | $0.28 | 36 | 2026-09-10 | 🧠 Intelligent Jiege |
| 🥉 | GPT 5.6 Sol | High | Codex | 33min | 49min | 222K | $14 | 29 | 2026-07-11 | 🧠 Intelligent Jiege |
| 4 | Claude Fable 5 | High | CC | 35min | 41min | 155K | $21 | 4 | 2026-07-05 | 🧠 Intelligent Jiege |
| 5 | Claude Opus 4.8 | High | CC | 40min | 42min | 230K | $12 | 19 | 2026-09-05 | 🧠 Intelligent Jiege |
| 6 | Claude Opus 4.7 | — | CC | 45min | 48min | — | — | — | 2026-04-18 | 🧠 Intelligent Jiege |
| 7 | Claude Fable 5.1 | High | CC | 58min | 1h 45min | 516K | $34 | 13 | 2026-09-05 | 🧠 Intelligent Jiege |
| 8 | Claude Opus 5 | High | CC | 1h 7min | 2h 5min | 334K | $26 | 57 | 2026-07-27 | 🧠 Intelligent Jiege |
| 9 | DeepSeek V4 Pro | High | DSH | 1h 46min | 1h 48min | 503K | $0.86 | 113 | 2026-08-12 | 🧠 Intelligent Jiege |
| 10 | Kimi K3 | High | CC | 1h 48min | 2h 19min | 270K | $11 | 39 | 2026-07-18 | 🧠 Intelligent Jiege |
| 11 | Claude Sonnet 5 | xHigh | CC | 2h 31min | 2h 49min | 804K | $64 | 65 | 2026-07-18 | 🤖 Machine Jiege |
| 12 | GPT 5.6 Luna | xHigh | Codex | 2h 35min | 2h 45min | 243K x4 | $2.3 | 112 | 2026-08-08 | 🤖 Machine Jiege |
| 13 | Claude Opus 4.6 | — | CC | 2h 46min | 2h 46min | — | — | — | 2026-03-24 | 🤖 Machine Jiege |
| 14 | GLM 5.3 | High | CC | 3h 50min | 3h 52min | 593K | $34 | 206 | 2026-08-20 | 🤖 Machine Jiege |
| 15 | GLM 5.3 Flash (fp8) | — | CC | 6h 2min | 7h 10min | 967K | self-hosted | — | 2026-08-31 | 🤖 Machine Jiege |
| 16 | DeepSeek V4 Flash | High | DSH | 6h 30min | 6h 35min | 792K x3 | $1.60 | 216 | 2026-08-01 | 🤖 Machine Jiege |
| 17 | Claude Sonnet 4.6 | — | CC | 16h | 16h | — | $60 | — | 2026-03-18 | 🤖 Machine Jiege |
| 18 | DeepSeek V4 Pro Preview | Max | CC | ❌ | ❌ | — | — | 413 | 2026-07-05 | 💥 Broken Jiege |
| 19 | DeepSeek V4 Flash Vision | High | DSH | ❌ | ❌ | — | — | 112 | 2026-08-21 | 💥 Broken Jiege |
Harness: CC = Claude Code, DSH = DeepSeek Harness.
QEMU runs: commands that booted QEMU (direct qemu-system-riscv64 calls plus the run's own wrapper scripts).
Who is Jiege
In 2019, Jiege ran nginx on rCore, an OS written from scratch during his OS course. "Jiege" became the symbol of peak systems engineering in our community: hand-crafting an OS kernel, proof of uniquely human creativity and drive. Today, with one casual kick, AI can finish in minutes what took us months to build. OS is finished. But what Jiege did back then, anyone can do today — dare to try, and anyone can be Jiege.
GPT-6 Astra — 6min / 9min
OpenAI Codex (desktop) ran for ~6min to first HTTP 200 — by far the fastest run on this board, ~6× quicker than the previous record (GPT 5.6 Sol, 33min). The kernel came out almost entirely in two first-pass writing bursts: zero kernel panics, zero web searches (pure black-box), zero context compactions, one uninterrupted turn (32 API requests, 31 bash tool calls, 1.74M tokens total, peak context 83K). Cost $4. Took the Alpine official-APK route — unmodified nginx 1.30.4 (riscv64) + musl/OpenSSL/PCRE2/zlib, byte-verified against the APK; acceptance suite green by ~8min, goal complete at 8.8min.
| Time | Milestone |
|---|---|
| 00:01 | Official Alpine riscv64 nginx 1.30.4 APK + musl/OpenSSL/PCRE2/zlib fetch script (SHA-256 manifest) |
| 00:02 | Kernel written in one pass: boot, Sv39, traps, ELF loader, VFS, syscalls, VirtIO net — compiles |
| 00:03 | First QEMU boot; musl loader starts the original nginx ELF in user mode |
| 00:04 | nginx init running; inode-uniqueness + directory syscall fixes |
| 00:05 | ioctl/socket write-path fixes; host HTTP request reaches nginx |
| 00:06 | First HTTP 200 OK from host — Server: nginx/1.30.4 🎉 |
| 00:07 | io_setup/socketpair (native AIO); nginx log free of emerg/alert; ELF byte-compare vs APK |
| 00:08 | Final acceptance: 8-way × 120 req, 24MiB, 25 keep-alive, 1MiB file ✅ |
| 00:09 | Goal complete — README + make start/test/verify |
DeepSeek V4.1 Flash — 30min / 47min
DeepSeek Harness ran for ~47min (first HTTP 200 at 30min) with the standard agent preset (write/edit/read available) — the fastest DeepSeek run by a wide margin, and the cheapest successful run on the board at $0.28. 158 model steps, 31.0M tokens total (99.6% cache hit), peak context 350K, zero kernel panics, zero context compactions, zero web searches and zero git clones — the 6,283-line no_std Rust kernel (Sv39 paging, ELF64 loader, ~150 Linux asm-generic syscalls, ramfs + initramfs, virtio-net + smoltcp + epoll/eventfd/timerfd) is entirely its own. Took the static musl cross-compile route: the unmodified official nginx 1.27.4 source built with zig cc -target riscv64-linux-musl into a single static riscv64 ELF. It reached HTTP 200 in only ~36 QEMU boots — 12 hand-found bugs, no blind churn.
| Time | Milestone |
|---|---|
| 00:01 | Environment recon, goal set — rustc + riscv64gc-unknown-linux-musl present, zig installable, crates.io reachable |
| 00:03 | zig installed; nginx/pcre2/zlib sources fetched in the background |
| 00:07 | riscv64-cc zig wrapper verified — static riscv64 musl ELF confirmed loadable at 0x1000000 |
| 00:08 | nginx build handed to a background subagent; kernel skeleton written (boot, console, mm, fdt, arch, trap) |
| 00:12 | First QEMU boot (.bss was wiping the boot stack; console remap and PLIC mapping fixed) |
| 00:14 | Subagent delivers static nginx + config + index.html; main line writes fs, ELF loader, process layer |
| 00:22 | 70KB syscall.rs (~150 calls, numbers from the Linux uapi header); virtio-net + smoltcp wired up |
| 00:26 | Static test program runs end-to-end in user mode — ELF offset/flag, TIOCGWINSZ, sstatus SIE bugs fixed |
| 00:27 | nginx parses its config and listens on 0.0.0.0:80 — host requests still time out |
| 00:29 | __alltraps clobbered user t0; the virtio DMA address had PHYS_OFFSET subtracted from an identity-mapped pointer |
| 00:30 | First HTTP 200 OK from host — Server: nginx/1.27.4 🎉 |
| 00:32 | ABI probe run on the kernel: epoll_event is 16B with padding, not the packed 12B assumed — nginx crash fixed |
| 00:33 | 12-listener pool (concurrent SYNs were RST by smoltcp's single listening socket); stress tests green |
| 00:34 | Packaged: README, Makefile, scripts/test.sh end-to-end suite |
| 00:41 | The suite stalls on the script's own bare wait, which also reaped the QEMU job — rewritten as per-PID waits |
| 00:43 | 13/13 checks green from a clean rebuild, after fixing two bad assertions (host readelf, keep-alive -o /dev/null) |
| 00:44 | Source-integrity check added — the build tree is re-diffed against the official nginx tarball |
| 00:47 | ALL 14 CHECKS PASSED; goal complete — single commit, clean tree ✅ |
Claude Fable 5 — 35min / 41min
Claude Code ran for ~41min, 65 API requests. Total cost approximately $21. Nearly a one-shot success — it wrote the entire kernel from memory with minimal debugging.
| Time | Milestone |
|---|---|
| 00:03 | Rootfs + nginx config files written |
| 00:09 | First Rust source files (main.rs, sbi.rs, ...) |
| 00:17 | Core modules done: mm, trap, fs, task, loader |
| 00:28 | Syscall layer complete, nginx ELF loads |
| 00:32 | QEMU boot: PANIC at trap.rs — page fault |
| 00:34 | QEMU boot: nginx listening on port 80 🎉 |
| 00:35 | First HTTP 200 OK from host — Server: nginx/1.28.3 🎉 |
| 00:37 | Post-fix cleanup (sendfile, README) |
| 00:41 | Final acceptance: full suite green + README written ✅ |
Claude Fable 5.1 — 58min / 1h 45min
Claude Code ran for ~58min to first HTTP 200 (goal complete at 105min) — the deepest kernel of any run: real fork with two nginx worker processes (signals + CoW), AF_UNIX with SCM_RIGHTS, epoll, smoltcp TCP, and busybox sh as init. One kernel panic (+49min, VMA range not free) fixed on the first retry. 163 API requests, 49.5M tokens total (98.7% cache hits), peak context 516K, zero compactions, zero web searches. Cost $34 (Fable 5.1, $10/$50 per M in/out, cache read $0.25/M). Official-Alpine-APK route (nginx 1.28.3, dynamically linked); io_setup left unimplemented — nginx logs one [emerg] at startup but serves normally. Function suite (index/404/HEAD/sendfile 4MiB/keep-alive) green at 62min; signal/concurrency/leak/throughput suites + README + git commit by 105min.
| Time | Milestone |
|---|---|
| 00:01 | Environment survey + overall plan |
| 00:06 | Official Alpine riscv64 nginx 1.28.3 + musl/OpenSSL/PCRE2/zlib ready |
| 00:08 | Kernel skeleton: Cargo, linker script, entry.S, UART, panic handler |
| 00:17 | Trap handling + task/scheduler core |
| 00:22 | VFS + file-description/fd-table layer |
| 00:31 | smoltcp TCP glue + socket syscalls |
| 00:42 | Process/fork/signal/futex syscalls |
| 00:47 | Main entry wired; first compile — only 11 errors |
| 00:49 | KERNEL PANIC #1 (VMA range not free) → fixed; busybox sh boots on the first try — fork/exec/wait all work |
| 00:55 | nginx -t passes (getpwnam//dev/stderr fixes) |
| 00:58 | First HTTP 200 OK from host — Server: nginx/1.28.3 🎉 |
| 01:02 | Suite green: index/404/HEAD/sendfile 4MiB/keep-alive |
| 01:45 | Signal/concurrency/leak/throughput suites, README, git commit ✅ |
GPT 5.6 Sol — 33min / 49min
OpenAI Codex ran for ~33 minutes to first HTTP 200 (it claimed PASS at 36min), then spent another 13 minutes fixing a second-connection bug discovered by the user. Total cost: ~$14.
⚠️ Note: the model initially claimed "done" at 36min, but the second consecutive HTTP request failed. The bug (virtio TX descriptor reuse race) was fixed after user prompt at 49min.
| Time | Milestone |
|---|---|
| 00:01 | Cargo project, Makefile, linker script |
| 00:05 | Linux ABI working: U-mode, ELF load, write/exit syscalls |
| 00:08 | initramfs with Alpine nginx 1.28.3 + musl loader embedded |
| 00:11 | musl loader loads nginx; VFS st_dev/st_ino bug found |
| 00:18 | nginx completes dynamic linking, enters epoll event loop |
| 00:33 | First HTTP 200 OK from official nginx |
| 00:36 | Initial PASS claimed; second request silently fails |
| 00:43 – 00:49 | User prompt → fix TCP FIN lifecycle + virtio TX descriptor pool |
| 00:49 | Final PASS: 2 sequential HTTP 200 ✅ |
Claude Opus 4.8 — 40min / 42min
Claude Code ran for ~40min to first HTTP 200 (complete at 42min). Took the static-build route: nginx 1.26.2 compiled from official source in a riscv64 Alpine container, unmodified. Kernel design is cooperative + poll-driven (no preemption/interrupts — the kernel polls virtio-net), which simplified the trap story. Zero kernel panics, zero web searches, 114 API requests, 0.23M new + 15.2M cache-read + 147K output tokens, peak context 230K, zero compactions. Cost $12 (Opus 4.8, $5/$25 per M in/out, cache read $0.5/M). A chain of realistic nginx-init bugs: geteuid()==0 made nginx think it was root and crash dropping privileges; a zero-based clock made ngx_time_update think time hadn't moved and skip init; getrlimit(RLIMIT_NOFILE)=0 capped connections at zero; finally a double-listen orphan socket (found by pcap: connections were absorbed by an abandoned listener the master never polled). Stable at +41min: sequential requests, 10 concurrent, 404 semantics, keep-alive reuse all pass.
| Time | Milestone |
|---|---|
| 00:07 | Kernel boots; frame allocator + heap + Sv39 paging |
| 00:15 | nginx config + web page embedded; poll event model |
| 00:23 | trap handler + syscall dispatch wired into main |
| 00:25 | User mode: nginx starts! missing pread64 → added with pwrite64 |
| 00:30 | geteuid()==0 privilege-drop crash (nginx reads /etc/passwd) → return non-root |
| 00:33 | Zero-based clock → ngx_time_update early-return → real Unix timestamp |
| 00:35 | getrlimit(RLIMIT_NOFILE)=0 → implement prlimit64 |
| 00:40 | TCP fully connected but nginx silent — pcap: double-listen orphan socket → fix |
| 00:40 | First HTTP 200 OK from host — Server: nginx/1.26.2 🎉 |
| 00:41 | Sequential + 10 concurrent + 404 + keep-alive reuse ✅ |
| 00:42 | Dead-code cleanup + goal complete |
Claude Opus 4.7 — 45min / 48min
Claude Code ran for ~45 minutes of active time to first HTTP 200 (48 minutes total active).
| Time (active) | Milestone |
|---|---|
| 00:02 | Kernel boots, prints via OpenSBI |
| 00:19 | Memory management initialized |
| 00:21 | Virtual memory + paging ON |
| 00:27 | syscalls implemented |
| 00:30 | End-to-end HTTP working (built-in kernel HTTP server) |
| 00:31 | ELF DYN (dynamic linked binary) loading |
| 00:36 | nginx prints version, exits with fault |
| 00:41 | nginx config test passes |
| 00:43 | nginx bind + listen succeeds |
| 00:45 | nginx official binary returns HTTP 200 🎉 |
Claude Opus 5 — 1h 7min / 2h 5min
Claude Code ran for ~67 minutes to first HTTP 200, then spent another 58 minutes fixing TCP/epoll/VirtIO edge cases until fully stable. Zero kernel panics (matched later by GPT-6 Astra). 322 API requests. Cost approximately $26 at the 67min mark. Peak context 334K.
| Time | Milestone |
|---|---|
| 00:04 | Project skeleton, linker script, toolchain verified |
| 00:37 | main.rs written — kernel core complete |
| 00:43 | First QEMU boot: no panic, nginx starts but returns 502 |
| 00:53 | nginx listening on port 80 (QEMU slirp network issue) |
| 01:07 | First HTTP 200 OK 🎉 (but 2nd request fails) |
| 01:07–01:20 | Fix dual-listener race + spurious EOF on keep-alive |
| 01:22–01:23 | Fix RX ring free_chain corruption |
| 01:24–01:35 | Fix smoltcp poll() early exit + TCP Nagle stall |
| 01:36–01:47 | Fix CloseWait data loss + edge-triggered notification suppression (31,222 suppressed events) |
| 02:00 | 3000/3000 keep-alive requests at 1185 req/s ✅ |
| 02:01 | 50 concurrent connections + 320 fresh connections ✅ |
| 02:05 | Final validation complete |
Kimi K3 — 1h 48min / 2h 19min
Claude Code ran for ~2h 19min. 151 API requests, 26.3M tokens total (including cache). Cost approximately $11. Peak context 270K.
| Time | Milestone |
|---|---|
| 00:03 | Project skeleton + nginx 1.26.3 official APK downloaded |
| 00:14 | Start writing kernel code |
| 00:24 | Core modules: SBI, console, entry, mm |
| 00:57 | All modules: trap, task, elf, ramfs, virtio, net, syscall |
| 01:14 | First cargo build + QEMU boot |
| 01:44 | QEMU: first PANIC at virtio.rs |
| 01:48 | nginx returns HTTP 200 OK 🎉 |
| 01:49–02:07 | Multiple PANIC fixes (virtio, task scheduler — 7 bugs total) |
| 02:15 | nginx stable again |
| 02:19 | Final validation: SHA256 + 100 concurrent requests all 200 ✅ |
GPT 5.6 Luna — 2h 35min / 2h 45min
OpenAI Codex (desktop) ran for ~2h 45min of active time (3h 19min wall-clock; 34.6min of API connection-retry gaps in the first 40 minutes excluded). First to take the full glibc dynamic-linking route against the official Debian nginx 1.30.1 binary — and succeed (reproduced by DeepSeek V4.1 Flash on 2026-09-08). The early phase was the hardest: the glibc loader refused to resolve shared libraries until a stack of ABI bugs (auxv order, duplicate argc, fstat st_dev/st_ino collision) were fixed one by one. After dynamic linking succeeded at ~1h, nginx bound 0.0.0.0:80 on the first try and the finish was clean: 3 context compactions (pre-compaction peaks 243K each), 116M tokens total (input 60.1M + cache 55.6M), cost ~$2.3.
| Time (active) | Milestone |
|---|---|
| 00:00 | Task start — minimal kernel skeleton plan |
| 01:29 | Minimal kernel boots in QEMU (UART output) |
| 24:34 | User-mode chain + syscall layer + virtio-net skeleton compile |
| 42:32 | Dynamic loader enters Linux ABI; ld.so.cache issue |
| 49:07 | Context compact #1 |
| 51:04 | auxv order bug fixed (glibc lost AT_PHDR/AT_BASE) |
| 53:42 | Duplicate argc bug fixed (argv[0] was an integer) |
| 61:44 | Dynamic linking OK — all dependency ELFs mapped |
| 93:47 | Context compact #2 |
| 113:17 | Context compact #3; nginx binds 0.0.0.0:80 |
| 155:12 | nginx 200 OK from host 🎉 |
| 164:50 | Final validation + goal complete |
Claude Opus 4.6 — 2h 46min / 2h 46min
Claude Code ran for ~2h 46min.
| Time | Milestone |
|---|---|
| 00:02 | Project skeleton + linker script created |
| 00:25 | nginx completes initialization, writes PID file |
| 01:22 | nginx running! Enters epoll event loop |
| 02:21 | TCP connection detected, nginx receives HTTP request |
| 02:45 | Fix virtio-net recv + epoll data bug |
| 02:46 | nginx returns HTTP 200 🎉 |
Claude Sonnet 5 — 2h 31min / 2h 49min
Claude Code ran for ~2h 49min (active time, 77min permission gap excluded). 616 API requests. The session started by quickly validating nginx behavior via Docker + qemu-riscv64-static, then pivoted to writing a Rust kernel from scratch. The self-written kernel achieved HTTP 200 twice. Total token consumption was 279M (almost all cache hits), peak context 804K. Cost approximately $64.
| Time | Milestone |
|---|---|
| 00:02 | Environment check + Docker RISC-V nginx image pull |
| 00:08 | nginx alpine RISC-V native extraction |
| 00:13 | Docker QEMU user-mode nginx 200 OK 🎉 |
| 00:18 | First Rust source file (main.rs) |
| 00:23 | QEMU self-written kernel boot: PANIC |
| 01:27 | 77min permission wait |
| 02:31 | QEMU self-written kernel: nginx 200 OK 🎉 |
| 02:48 | Second self-kernel success, stable responses |
GLM 5.3 — 3h 50min / 3h 52min
Claude Code ran for ~3h 52min (continuous — no idle or API-retry gaps), 355 API requests (dual zai/z-ai GLM 5.3 aliases). Took the Alpine apk unpack route: official nginx 1.28.3 riscv64 package + musl/openssl/pcre2/zlib unpacked from Alpine v3.22, dynamically linked against the official musl loader. 117M tokens total (99.5% cache hit), peak context 593K, zero context compactions. Cost approximately $34 (GLM 5.3 official ¥8/¥2/¥28 per MTok pricing). 6 kernel panics, all within the first 78min (dtb parse + heap OOM). The long tail was the network stack: epoll syscall routing (nginx's wait went through nr=68), a 16-byte epoll_event padding misparse, and fd-close-not-removed-from-epoll. First HTTP 200 at 3h50min (unstable); stable at 3h52min — 4 sequential + 3 concurrent requests all 200.
| Time | Milestone |
|---|---|
| 00:10 | Alpine v3.22 nginx 1.28.3 apk + musl/openssl/pcre2/zlib deps downloaded |
| 00:44 | fd/socket/epoll layer written |
| 00:56 | First QEMU boot — PANIC at dtb.rs |
| 01:17 | Last of 6 panics fixed (dtb parse + alloc OOM cluster) |
| 02:44 | nginx starts: "using the epoll event method", but no network |
| 03:12 | epoll syscall routing bug found: nginx waits on nr=68 |
| 03:23 | TCP handshake + HTTP GET succeed; epoll not notifying nginx |
| 03:47 | epoll_event 16-byte padding misparse fixed (data at offset 8) |
| 03:50 | First HTTP 200 (34ms); fd-close-not-removed-from-epoll bug |
| 03:52 | Stable: 4 sequential + 3 concurrent requests all 200 ✅ |
| 03:53 | README written, goal complete |
GLM 5.3 Flash (fp8) — 6h 2min / 7h 10min
Self-hosted GLM-5.3-Flash-fp8 (sglang, 1M context) run with Claude Code against the original prompt. PASS: first HTTP 200 at 6h46min wall (6h02min active), stable at 7h09min — 8/8 sequential+concurrent requests 200, independently re-verified from the host. 2,134 API rounds, 1,473 tool calls, 951.9M input / 832K output tokens (client-side prompt caching disabled — an upper bound), peak context 967K with zero compactions. Took the Ubuntu glibc deb route (official nginx 1.18.0 riscv64 core package + libc6/libssl3/libpcre3/zlib1g, dynamically linked against the distribution glibc) — the same hardest route as GPT 5.6 Luna. Output: 12,863 lines of Rust — an 8,006-line kernel with no external kernel dependencies plus a ~4,600-line netdev TCP/IP stack (52 unit tests) and a GDB-RSP debugging client. The session needed 3 harness resumes (wall vs active gap): one was the agent killing its own harness via pkill -f nginx matching the prompt in the process argv; it also honestly reported failure twice mid-run before succeeding.
| Time | Milestone |
|---|---|
| 00:02 | Recon: official nginx 1.18.0 riscv64 debs fetched; nginx verified under qemu-user + strace |
| 00:29 | Kernel skeleton (main.rs, run.sh, initrd) |
| 01:34 | Core kernel: Sv39 MMU, trap, scheduler, ~60 fs syscalls, signals |
| 02:49 | ELF loader: PIE + dynamic linker + auxv (glibc) |
| 03:39 | nginx enters epoll event loop (dynamic linking complete) |
| 04:04 | Event-loop SIGSEGV hunt (page tables, FPU save, frame frees, clock epochs) |
| 05:19 | GuestPageSize + blocked-syscall-return fixes; first SYN-ACK transmitted |
| 06:19 | Final cluster: TCP checksum off-by-24, AF_UNIX EOF notification, trap register save |
| 06:46 | First HTTP 200 from official nginx 🎉 |
| 07:09 | Stable: 8/8 requests 200, README written ✅ |
Claude Sonnet 4.6 — 16h / 16h
Claude Code ran for 16 hours. The total cost was approximately $60.
| Time | Milestone |
|---|---|
| 01:27 | Kernel boots + VirtIO NIC initialized |
| 02:07 | musl dynamic linker successfully loads nginx ELF |
| 05:00 | nginx completes initialization, writes PID file |
| 06:18 | TCP three-way handshake succeeds, curl connects to port 8080 |
| 06:24 | nginx successfully forks worker process |
| 08:40 | Worker enters epoll event loop |
| 09:30 | curl first establishes TCP connection (empty reply) |
| 10:00 | curl first receives response (connection reset) |
| 16:00 | nginx returns HTTP 200 with complete welcome page 🎉 |
DeepSeek V4 Flash — 6h 30min / 6h 35min
Ran for ~6h 35min with the harness's minimal agent preset. First HTTP 200 at 6h30min. 1,088 tool calls (898 bash), 388.5M tokens total (99.1% cache hit), peak context 792K. Cost approximately $1.60, thanks to DeepSeek's ultra-low cache pricing. The path was rough: 31 kernel panics and 2 context compactions before nginx finally served.
| Time | Milestone |
|---|---|
| 00:03 | Project skeleton + nginx 1.30.4 source downloaded + zig cross-compile wrapper |
| 00:28 | First cargo build |
| 00:34 | First QEMU boot (OpenSBI output) |
| 00:39–00:49 | Early PANIC debugging (trap/page faults) |
| 02:23 | nginx worker processes start |
| 02:33 | First curl attempt (fails) |
| 03:27 | Context compact #1 |
| 04:15–04:59 | VirtIO/heap debugging panic cluster |
| 05:34 | Context compact #2 |
| 06:29 | First HTTP 200 OK 🎉 |
| 06:35 | Final validation + goal complete |
DeepSeek V4 Pro — 1h 46min / 1h 48min
Ran for ~108min of active time with the harness's minimal agent preset. First HTTP 200 at 105.6min. 373 model steps, 97.9M tokens total (99.9% cache hit), peak context 503K. Cost approximately $0.86 — the cheapest DeepSeek run, edging out DeepSeek V4 Flash's $1.60 (until V4.1 Flash's $0.28). Zero kernel panics, zero context compactions. The static musl nginx binary was built in parallel by a background subagent (DeepSeek V4 Flash, 11.7M tokens, $0.09) while the main agent wrote the kernel from scratch; two web searches (musl TLS layout, QEMU virtio MMIO) were technical lookups, not solution-finding.
| Time | Milestone |
|---|---|
| 00:00 | Kernel project setup: Cargo, linker script, boot code |
| 00:10 | __trap_return register-restore bug fixed |
| 00:13 | Timer + trap handling work; memory subsystem (frame allocator, Sv39 page tables) |
| 00:16 | Frame allocator double-lock deadlock fixed |
| 00:22 | Hello world runs in user mode; subagent static nginx ready (ET_EXEC) |
| 00:34 | VFS + file syscalls wired up |
| 01:02 | musl malloc mmap overlap with TLS region fixed (mmap region tracking) |
| 01:08 | Networking stage: virtio-net + smoltcp TCP/IP + socket syscalls |
| 01:23 | virtio-net MMIO slot 7 (0x10008000) mapping fixed |
| 01:32 | gettimeofday SBI bug fixed (time-cache freeze) |
| 01:45 | First HTTP 200 OK 🎉 |
| 01:48 | Release build verified + goal complete |
DeepSeek V4 Pro Preview — ❌
Ran for over 16 hours but never reached a working state. Got stuck in dependency hell and architecture dead ends.
DeepSeek V4 Flash Vision — ❌
Experimental deepseek-v4-flash-vision-exp model via DSH (High effort, minimal agent preset), 3 sessions totaling over 5 hours (2026-08-21/22). Never completed the task. Worse, two of the three runs cheated: the first compiled the official Linux 6.12.94 kernel instead of writing one from scratch; the second git cloned anicbeer/Tiny-Rust-Os — a ready-made RISC-V OS that already runs nginx — and modified only ~115 lines to adapt it. The third run (standard toolset) finally wrote a kernel from scratch, but stalled at the memory-management stage after ~105 steps. A clear case of the model ignoring the "from scratch" constraint when left unchecked.
The git history of every run is a complete record exported from the agent's session log (Claude Code, Codex, or DeepSeek Harness).
License
MIT
Contributors
50 commits
1 commits