benchtest results — Apple Silicon framework × harness benchmark

Raw benchmark data from benchtest: one Mac (Apple M5 Max, 128 GB), local LLM serving frameworks (OMLX / MTPLX / MLX-VLM / MLX-Serve) crossed with six agent harnesses (raw API, raw+, pi, opencode, aider, hart) on three build tasks (tetris, code review, agent console).

Datasets

Each folder holds one JSON per task run (per-cell records: framework, harness, model actually served, status, latency, tokens, TPS, server-measured PP/TGS, iterations, QA reliability score) plus per-cell JSONL crash-safety logs and a manifest README.

Headline finding

Flash-Next (MTP) is ~1.7× faster on MTPLX (84.9 tok/s) and OMLX (87.6) but a wash on MLX-VLM (39.0) — the framework decides whether the model architecture pays off. MLX-Serve cannot load the flash-next build at all on 128 GB (memory preflight needs ~112.5 GB).

Metric dictionary

• tps — overall tokens/wall-second (client); for agent harnesses this includes tool-execution time
• server_tgs / server_pp — server-measured decode / prefill tok/s (the honest cross-harness speed metric)
• iterations — model calls per cell
• qa_score — deterministic artifact reliability checks (0–100); quality judgment stays human
• tokens_source: server — token count from the framework's own counters

Generated by benchtest v1.0 · 2026-10.