Quick Rundown of MLPerf v6.0 Training

— 2026/06/18

MLPerf Training v6.0 was released on June 16, 2026. MLCommons and participating companies have already published their own highlights, results, and platform narratives. This rundown is not meant to re-summarize those announcements. Instead, it adds analysis on the parts I care about as an observer:

• The new MoE pretraining workloads, DeepSeek-v3 (671B) and GPT-OSS (20B), including notable optimization choices and configs.
• Scaling efficiency across GB200 and GB300 platforms. MI Instinct does not yet have enough data points for scaling analysis, but kudos for its first multi-node training submissions across MI300X, MI325X, and MI350X.
• Debut of AMD MXFP4 recipe submissions, enabling MXFP4/NVFP4 recipe comparisons on 8-GPU Llama 3.1 8B across B200, B300, GB200, GB300, MI350X, and MI355X.

Links:

• v6.0 Release by MLCommons, Supplemental Discussion pdf
• Result Dashboard, Submitters' code
• Nvidia: blog, tech dive
• AMD: blog, tech dive, tutorials
• Coreweave, Nebius, Lambda, Azure
• My rundown on previous v5.1 (Nov'25)

---

DeepSeek-v3 (671B)

• DeepSeek-v3 (DSv3) is a new pretraining workload in v6.0, bringing Mixture-of-Expert (MoE) model family into the benchmark. It is also the largest workload by parameter count 671B.
• Briefly, DSv3 is the base model behind DeepSeek-R1, the DeepSeek's flagship reasoning model that triggered a major market reaction in early 2025. While Google pioneered large-scale MoE earlier, DSv3 arguably brought MoE into the mainstream for open-weight frontier models.
• More on how MLCommons integrates DSv3 for benchmarking here, especially around how a 50-step trained checkpoint circumvents the high load imbalance and variance during early training.
• Given its sheer size, DeepSeek-v3 is a cluster-scale benchmark. NVIDIA and CoreWeave submitted results across a wide range of GPU counts, allowing us to estimate scaling efficiency for GB200 NVL72 and GB300 NVL72.
• Scaling Efficiency: 85.5% on GB200, 88.2% on GB300, solid strong-scaling overall. See how we fit log-linear and arrive at estimated efficiency.
• Fastest result: 8,192 GB300 gpus took only 2.021 mins to train DSv3 for 3B+ tokens. Key techniques: MXFP8 recipe including attention on top typical linear, full-iteration cuda-graph, EP communication using HybridEP and overlapping it via 1F1B PP schedule. See official technical highlights and discussion at depth in this technical report.
• Worth noting: Nvidia tech blog previews another ~1.3× training throughput uplift in NeMo 26.06 via full stack codesign on GB300. P/S on 26/07/23, a recent Nvidia blog discuss this uplift and techniques used are: A2A overlap, Full-iteration Cuda Graph, Advanced CuteDSL fusion.

---

GPT-OSS (20B)

• GPT-OSS 20B is another MoE workload introduced in v6.0, with accessibility as the main goal, it is benchmarkable with as little as a single 8-GPU node. Unlike DSv3, GPT-OSS does not start from a pretrained checkpoint. Instead, the benchmark recipe tunes hyperparameters of Adam and weight initialization to bound routing variance and improve benchmarking fairness.
• How to arrive at the plot above? 41 available entries, so pruning and deduplication are needed for readability. Entries are first grouped by GPU model, then deduplicated by GPU count. The legend shows the submitter combination behind each series.
• Although GPT-OSS 20B was benchmarked on GB300 across a wide range of GPU counts, we do not report scaling efficiency here. Based on our analysis, the runs are not a clean strong- or weak-scaling study, because each scale point uses different global batch size and parallelism mapping, taking uneven number of gradient updates, as well as number of tokens to converge. The relative ordering between gpu type still make sense.
• Just a thought: the GB300 curve shows visible curvature at higher scale. For example, the 32-GPU and 512-GPU runs use the same GBS and converge in the same number of steps, but the time-to-train speedup is only 3.9×, far below the ideal 16× from scaling 32 → 512 GPUs. My suspicion (1) EP comm overhead of EP=8 in 512x case while DP-only in 32x case; (2) gradient-reduce collectives of the 512-GPU case can become increasingly exposed, i.e. DP=64 and thin MBS=1.
• It is interesting to observe EP is only turned on at >72xGB200/300, 64xB200/300. TODO: deeper analysis

---

MXFP4, NVFP4, FP8 (Llama3.1 8B on 8 x GPUs)

Datasheet: MI355X, MI350X, B200, GB200, B300, GB300

source: 5.1 rundown

• Contributed by AMD, training in native MXFP4 has made its debut into benchmark this time round. This completes current-generation FP4 coverage in the benchmark across the two main FP4 formats: MXFP4 and NVFP4.
• Quick recap: MXFP4 quantizes every 32 elements using a power-of-two FP8 (e8m0) scale, while NVFP4 uses 16-element groups with FP8 (e4e3) scales. Both use FP4 (e2m1) as the quantized datatype. More discussion and references here.
• MXFP4 on MI355X time-to-train is slight longer than B200 NVFP4. MXFP8 vs MXFP4, 99.7/86.8=1.148x on MI355X, 122.9/109.8=1.119x on MI350X. Per AMD's blog, its MXFP4 recipe resembles NVFP4: applying a Hadamard rotation before quantization and fusing the full quantization pipeline into a single kernel, with model-specific, pre-calibrated tile sizes. For additional context, this paper may help explain the recipe.
• NVFP4: *B300 has noticeable but insignificant improvement between 5.1 and 6.0 while *B200 fares about the same.