JiwenJ/Awesome-Initialization

TeX

30 commits

updated Sep 23, 2026

README

Awesome μP

Curated, evidence-backed resources on maximal-update parametrization (μP), μTransfer, Hyperball optimization, and scale-aware hyperparameter transfer.

This repository tracks papers, implementations, engineering reports, and teaching material that directly derive, test, extend, criticize, or materially apply μP / muP, μTransfer, and maximal-update scaling. Dedicated Hyperball and scale-aware hyperparameter-transfer sections cover complementary optimizer geometry and proxy-to-target scaling methods under the same evidence standards.

μP / HPT snapshot: 2026-09-20. The μP collection contains 161 directly relevant papers, plus 54 learning resources and 96 implementation / artifact links. The complementary HPT collection adds 17 direct papers outside the μP and Hyperball bibliographies.

Start here
Why μP matters
Recent μP / μTransfer directions
Hyperball
Scale-aware hyperparameter transfer
Full paper list
Learning resources and blogs
Implementations and artifacts
Repository layout
Contributing

Start Here

Order	Theme	Resource
1	Theory foundation	Feature Learning in Infinite-Width Neural Networks
2	Core μTransfer recipe	Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
3	Adaptive optimizers	Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
4	Depth scaling	Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
5	Empirical LR transfer	An Empirical Study of μP Learning Rate Transfer
6	Practical caveat	Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

For the detailed guide and practical checklist, see μTransfer / μP Hyperparameter Transfer.

Why μP Matters

μP is not just an initialization trick. It is a scaling-aware parameterization recipe covering initialization scales, learning-rate multipliers, readout and embedding treatment, and optimizer parameter groups. The practical goal of μTransfer is:

Convert the target architecture to μP.
Tune key hyperparameters on a small proxy model.
Transfer those hyperparameters to a much larger model without re-running the expensive sweep.

The μP collection includes the direct Tensor Programs lineage, coordinate checks, architecture-specific extensions, critical evaluations, and application reports with substantive μP evidence. Generic initialization, Muon, scaling-law, batch/token, scheduler, or ordinary HPT papers are intentionally excluded when μP is only an analogy, background citation, or unvalidated recipe mention.

Recent μP / μTransfer Directions

Width transfer for LLMs — original μTransfer framing and practical Transformer results.
Depth transfer — Depth-μP, effective-depth laws, and limitations in modern multi-layer residual blocks.
Embedding and readout scaling — recent evidence that embedding-layer learning rate can dominate apparent μP gains in AdamW language-model training.
Architecture-specific μP — GNNs, GQA-μP, diffusion Transformers, probabilistic Transformers, Fourier neural operators, MoE, LoRA, and sparse models.
Training-system variants — u-μP for unit-scaled / low-precision training, SμPar for sparse training, and maximal-update rules for matrix-preconditioned optimizers.
Boundaries and negative results — experiments that directly test when μP does or does not transfer across optimizer, architecture, batch, duration, or parameter groups.

Topic	What to look for
Width transfer	Original μTransfer framing and practical Transformer results.
Depth transfer	Depth-μP, effective-depth laws, and limits in modern residual blocks.
Embedding / readout scaling	Cases where embedding-layer learning rate or output scaling controls transfer quality.
Architecture-specific μP	GNNs, GQA, diffusion Transformers, probabilistic Transformers, Fourier neural operators, sparse models, and LoRA.
Optimizer-specific transfer	Adaptive optimizers, Muon / hypersphere optimization, and optimizer-dependent scaling rules.
Batch and duration	Work that tests these axes while μP remains a substantive part of the method or experimental design.

Hyperball

A companion collection on Hyperball, AdamH, MuonH, their analyses, implementations, and applications. Snapshot: 2026-09-23.

Hyperball controls selected weight matrices and optimizer updates through Frobenius normalization. This collection covers the original method, substantive comparisons and criticism, applications, and explicitly identified extensions of its fixed-sphere idea. It follows the repository's primary-source, deduplication, and factual-description rules; an independent μP result is not required here. The μP collection retains its original scope, while the scale-aware HPT guide compares Hyperball and HyperP with other transfer routes.

The collection contains 13 papers, 22 learning resources / reports, and 24 implementation / artifact entries. Two papers (HyperP and MACRO) are also in the μP collection; collection totals overlap. MD Decoupling is explicitly labeled a related extension, and contextual readings below are outside the paper count. Model/data releases and mirrors are grouped with their corresponding code entry.

The complete collection, reading guide, and latest source audit are included below. BibTeX: hyperball.bib.

Reading path
Mechanism and interpretation
Papers
Learning resources and reports
Implementations, models, and data
Related foundations and distinctions
Evidence and limitations
Search coverage
Full source audit and verification details
Excluded and unresolved candidates

Hyperball Reading Path

Read the original paper and its author note for the wrapper and motivation.
Read HyperP for scale-transfer rules and limits, then Hyperball May Not Be a Free Lunch for scheduling evidence.
Compare Effective Learning Rate Governs Loss Dynamics with HyperTransfer: approximate loss alignment and conditional optimizer-trajectory equivalence are different claims.
Inspect a versioned implementation below before adapting the recipe to new tensor groups or distributed layouts.

Hyperball Mechanism and Interpretation

For a constrained matrix, let $R>0$ be its fixed radius, $u_t$ the base optimizer's proposed update, and $N(X)=X/\lVert X\rVert_F$. The original wrapper is

$$ W_{t+1}=R,N!\left(W_t-\eta_t R,N(u_t)\right). $$

The paper sets $R=\lVert W_0\rVert_F$ and applies the wrapper to attention/MLP matrices. Adam and Muon supply different directions, giving AdamH and MuonH. The proposed step has norm $\eta_tR$; the final displacement after projection need not. Implementations need defined behavior for zero updates and cannot infer a positive radius from a zero-initialized matrix. See Algorithm 1 and §2.

Fixed matrix norms do not imply identical feature dynamics across arbitrary architectures. Actual angular motion also depends on the parameter–update angle; parameter routing, normalization gains, radius conventions, and schedules remain part of the recipe. HyperP and Free Lunch investigate these boundaries.

Hyperball Papers

Dates are first public manuscript dates, in reverse order. Later versions are identified where relevant. The original 2025 author note and its living versions are resources in the same lineage as the 2026 formal paper, not additional papers. BibTeX: hyperball.bib.

Date	Paper	Main contribution	Tags
2026-09-07	HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball	Maps base and Hyperball optimizers through an online proxy norm, gradient/state rescaling, and induced LR schedules; proves conditional scale-invariant trajectory equivalence and studies a non-invariant extension.	theory, optimizer equivalence, effective LR, state mapping
2026-08-28	Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining	Proposes MuonM, adding curvature-conditioned fast/slow momentum to a learnable-radius sphere with parallel transport; Section 6.2 directly compares tuned MuonH, SSO, MuonS, and Muon on a 0.12B dense model, while the proposed method is evaluated on 0.12B–2.3B dense/MoE models.	direct comparison, related extension, MuonM, MuonH, learnable radius, momentum
2026-08-27	Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090	Uses MuonH for attention/MLP matrices, studies effective-LR matching and schedules, and retains MuonH during supervised fine-tuning; reviewed v2, September 3.	application, MuonH, pretraining, SFT, scheduling
2026-08-25	Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining	Tests MuonH/MuonW loss-trajectory alignment through effective-LR interventions and predicts held-out Hyperball runs without refitting its scaling law; accuracy depends on normalization and slowly varying dynamics.	empirical analysis, effective LR, loss dynamics, scaling laws
2026-07-24	Hyperball May Not Be a Free Lunch	Analyzes angular effective LR and radial/tangential updates; controlled MuonWD/MuonH schedule matching suggests effective-step evolution explains much of the difference, while faster early convergence can impair later performance.	criticism, scheduling, angular dynamics, MuonH
2026-07-22	Muon Reduces the Training Cost of Regulatory DNA Transformers	Compares AdamW, AdamH, MuonW, and MuonH on 26M–420M regulatory-DNA Transformers; independent weight decay works better with Muon in this setting, with relative-step and spectral diagnostics.	application, DNA, optimizer comparison, negative boundary evidence
2026-06-28	On the Nonlinearity of Learning Rate Scaling for LLM Training	Uses AdamH to test a weight-norm explanation of nonlinear LR scaling; 64M-model AdamH experiments make data-horizon extrapolation more nearly log-linear and reduce extrapolation cost relative to AdamW.	AdamH, effective LR, token horizon, scaling analysis
2026-06-24	Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors	Extends the fixed-sphere idea with learnable row/column magnitude gains and ablates sphere axes and gains; uses a different update-scaling convention from exact AdamH/MuonH. Reviewed v2, July 17.	related extension, MD Decoupling, magnitude gains, width transfer
2026-06-15	Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization	Introduces the AdamH/MuonH wrapper and its weight-decay motivation; reports 20–30% token-equivalent gains against an AdamW scaling-law baseline on Qwen3-style models up to 1.2B and reduced optimal-LR drift in width/depth sweeps.	original method, Hyperball, AdamH, MuonH, transfer
2026-06-10	Redesign Mixture-of-Experts Routers with Manifold Power Iteration	Evaluates MPI routers with AdamH and MuonH, adopts MuonH for 3B/11B MoE pretraining, and transfers a router-scale constant from smaller sweeps using Hyperball norm control.	application, MoE, MPI routers, MuonH
2026-05-06	Demystifying Manifold Constraints in LLM Pre-training	Compares MACRO with Frobenius and spectral MuonH variants on 120M–1B Qwen3-like models, isolates tangent-projection and weight-decay effects, and tests μP-compatible width transfer.	comparison, MACRO, MuonH, manifold constraints, μP
2026-03-30	Rethinking Language Model Scaling under Transferable Hypersphere Optimization	Builds HyperP around MuonH/AdamH, derives width/depth rules, fits token-horizon scaling, and adds SqrtGate for MoE granularity; finds Hyperball alone insufficient for depth transfer.	extension, HyperP, μP, depth, token horizon, MoE
2026-01-29	Manifold constrained steepest descent for smooth and closed-set optimization	The August 13 v2 explicitly formulates Hyperball and explains why projecting an ambient steepest direction need not guarantee constrained descent; its counterexample is geometric, not an LLM benchmark.	theoretical boundary, MCSD, projection, stationarity, v2 evidence

Hyperball Learning Resources and Reports

Author resources, experimental reports, talks, and tutorials are labeled separately from papers. Live reports are snapshots, not promises of completed runs. Translations and redirect aliases are grouped with their originals.

Resource	Type	Why it matters
Demystifying Manifold Constraints — JHU seminar	Shiqian Ma companion author seminar, 2026-09-14	Institutional abstract for the indexed MACRO comparison paper, discussing constraints, RMS normalization, and rotational equilibrium; no public recording or slides verified.
Agnes 2B pretraining protocol	Proposed application / technical protocol, 2026-09-03	Specifies selected-matrix MuonH with AdamW fallbacks, FP8, curriculum, and preregistered checks; training time and capability targets are projections or release gates, not completed Agnes results.
Marin 535B-A23B launch note	Ongoing training report, 2026-09-03	The public run specification uses MuonH matrices and AdamH readout updates; 18T tokens is the planned budget, not a completed result.
Blog: Survey of Optimizers	Ruoran Xu optimizer survey, 2026-08-28	Section 5.2 explains Hyperball through weight norms, angular learning rates, and matrix-optimizer shape scaling; synthesis of prior work, not an independent Hyperball benchmark.
Hyperball, effective lr, and the shape of peak-then-decay	Jiaxuan Zou technical essay, 2026-08-25	Synthesizes effective-LR replay, Free Lunch, and scheduling interpretations; explanatory analysis, not an independent benchmark. Chinese version.
Marin MoE Hyperball norm-grouping investigation	Primary implementation discussion and ablation, 2026-08-24	Documents cross-expert Frobenius normalization in the 535B stack; a d768 comparison found 3.015 versus 3.018 Paloma loss for grouped versus per-expert norms, described as within noise. Maintainers retained the grouping and monitored expert norms.
MarinDNA: A 1B standard Transformer rivals Evo 2 40B on variant effect prediction	Primary application report, 2026-08-03	Applies a Complete(d)-inspired AdamH recipe to genomic model scaling, with proxy sweeps and target checks across size, batch, and token horizon. Also indexed in the μP collection.
Fantastic Pretraining Optimizers II — FAI-Seminar	Kaiyue Wen author seminar, Chinese, 2026-07-24	Official schedule links the Hyperball paper, announcement, and Bilibili recording; recording content was not independently reviewed.
Magnitude–Direction Decoupling	Companion author post, 2026-06-15	Explains the fixed-norm direction and learned-magnitude extension, with ablations; its update scaling differs from the original Hyperball wrapper.
Improving our LLM Pretraining Efficiency	Larry Dial / Open Athena report, 2026-06-03	Reports AdamH MoE scaling and MuonH ablations across four compute scales; distinguishes theoretical compute gains from kernel/runtime effects and combined-recipe improvements.
Scaling Laws That Extrapolate 300× Past the Fit	Will Held / Delphi report, 2026-05-11	Combines AdamH with calibrated token-horizon scaling and width-sweep checks; the empirically chosen horizon exponent is not a universal Hyperball law. Also indexed in the μP collection.
A Genealogy of Optimizers	Siddharth Choudhary, with Claude; tutorial, 2026-05	Section 13 introduces MuonH through weight-norm control; interactive toy examples are illustrative rather than an optimizer benchmark.
Fantastic Pretraining Optimizers — IOS slides	Kaiyue Wen author talk, 2026-03-21	The author-linked deck introduces AdamH/MuonH and transfer experiments in slides 16–19.
On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism	Jiaxuan Zou technical essay, 2026-03-07	Derives SGDH, AdamH, and MuonH feature-space scaling under stated statistical/geometric assumptions; no independent transfer benchmark. Also indexed in the μP collection.
Does Muon improve regulatory DNA learning? Part 1.	Viraj Doshi author experiments, 2026-03-05	Explains Hyperball geometry with AdamH/MuonH LR sweeps and cases favoring weight decay; companion to the later regulatory-DNA preprint.
Hyperball Optimizer — Princeton PLI	Xingyu Dang author talk, 2026-02-05	Institutional talk page and recording, covering norm control, rotational equilibrium, and transfer; recording content was not independently reviewed.
Nanochat: Hyperball/MuonH Experiments (Negative Result)	Upstream development log, 2026-01-29	Records unsuccessful d12 MuonH/AdamH integration attempts, including LR sweeps, zero-initialized projection and readout-scale issues; a setting-specific result.
Optimization 1 — Norm reparametrization	Ziming Liu conceptual tutorial, 2026-01-23	Starts from Hyperball's fixed-radius motivation and explores learned magnitude and norm reparameterization in a two-dimensional toy problem; related extension, not an LLM Hyperball benchmark.
Yao Class Seminar 86: Fantastic Pretraining Optimizers I & II	Kaiyue Wen author seminar, 2025-12-13	Official early Hyperball announcement linking the original note; no separate public recording was verified.
Fantastic Pretraining Optimizers 2.1: Hyperball Optimization	Living author note / original research lineage	Redirects to the paper-linked Notion note. The legacy combined note has an author-supplied 2025-11-30 citation; both belong to the formal paper's lineage.
Fantastic Pretraining Optimizers 2.2: The Hitchhiker's Guide to the Weight Norm Theory	Living author theory tutorial	Develops noise-model explanations of weight norms and angular step sizes, with interactive simulations; exact publication date was not established.
Marin Agent MoE Experiment Digest	Project experiment digest; data snapshot 2026-08-20	Summarizes 80 experiments, including AdamH/MuonH, gradient-aware Hyperball, and optimizer ablations; maintainer-hosted, agent-assisted records rather than peer-reviewed evidence.

Hyperball Implementations and Artifacts

Links below were inspected for method support and provenance, not executed as reproductions. Pinned files identify the inspected implementation; upstream APIs may differ. Community ports and configurable variants are labeled separately from author code and framework integrations; inclusion does not establish benchmark gains or production readiness.

Artifact	Framework / method	What it provides
Marin / Levanter Hyperball optimizers	JAX / Optax / Haliax; author-associated stack	Versioned AdamH and MuonH, with matrix normalization, reprojection, and explicit parameter routing.
Marin Grug MoE optimizers	JAX / Optax; project implementation	The 535B optimizer snapshot uses recipe-specific grouping; issue #8621 documents per-expert Newton–Schulz but cross-expert Frobenius normalization for stacked experts, rather than fixed individual expert norms.
NVIDIA NeMo Emerging-Optimizers	PyTorch; framework integration	MuonHyperball and HyperballHook; this snapshot requires an explicit nonzero radius and validates initial parameter norms.
modded-nanogpt Track 3	PyTorch; public optimization benchmark	Source/log submissions for AdamH, MuonH, NorMuonH, KL-SOAP-H, SOAP-H, PSGD-H, and MuonH with retuned auxiliary Adam; bundled initialization/schedule changes make these recipe comparisons.
Hyperball May Not Be a Free Lunch — experiments	PyTorch; official paper artifact	Training programs, released scalar CSVs, and plotting code for effective-LR decomposition, alignment, and schedules; figure recreation can use CSVs without GPUs.
Puro-Megatron and Puro-2B releases	PyTorch / Megatron; official code, models, and data	Versioned documentation covers MuonHyperball, logical QKV/SwiGLU groups, tensor-parallel radii, and effective-LR diagnostics; model collection, base checkpoint, and training data accompany the paper.
microsoft/ArchScale	PyTorch / LitGPT; official HyperP code	HyperP, MuonH, and SqrtGate, with sharded global Frobenius-norm handling and hybrid parameter groups. Also indexed in the μP collection.
Router with Manifold Power Iteration	PyTorch / TorchTitan; official MPI-router code	Router implementation and advanced optimizer support for the paper's AdamH/MuonH comparisons; repository documents FSDP/expert-layout limits.
MD Decoupling — dense models	PyTorch / Megatron; official related-extension code	Fixed-norm direction and learned-magnitude optimizer, called `master` in research code; MoE branch accompanies the same paper. This is not the exact original Hyperball wrapper.
Open-Athena/marin-dna	JAX / Marin; primary application hub	Genomic models, experiment pointers, and AdamH-based training context. Also indexed in the μP collection.
Marin Complete(d)-inspired AdamH recipe	JAX / Levanter; application recipe, models, and data	Combines AdamH with batch/token-dependent settings and an empirically selected token exponent; shared by Delphi and MarinDNA. Includes a Delphi 447M / 122B-token checkpoint and Delphi plotting data with six configurations and per-row W&B links. Recipe also indexed in the μP collection.
Author HyperballAdam toy	PyTorch; historical author example	Normalized MLP example with fixed-radius reprojection; its step omits the paper's explicit radius multiplier, so its LR convention is radius-dependent.
Dragon / Segurant AdamH and AdEMAMixH	PyTorch; community implementation	The Segurant training code includes AdamH and AdEMAMixH, normalizing 2D matrices or individual 3D slices; grouped with the original HF release, without a verified transfer benchmark.
TitanPrecond	PyTorch / TorchTitan; community implementation	Experimental manifold optimizer with a `muonh` option and Frobenius/spectral constraints; update-alignment conventions change the LR scale.
CMU 18660 Hyperball Project	PyTorch; coursework implementation	MLP/NanoGPT hMuon comparisons; a community learning artifact, not an official paper reproduction or a separate formal paper.
PaddlePaddle Muon / Adam Hyperball	Paddle; upstream framework integration, merged 2026-09-21	MuonH/AdamH routing and update with FP32 full-matrix or per-expert Frobenius projection and whole-tensor sharding; radius is recomputed from current weights, subject to rounding and epsilon.
HeavyBall HyperBallAdamW	PyTorch; optimizer-library integration	Compiled update uses Adam direction and fixed-initial-norm Hyperball for rank ≥ 2, ordinary AdamW for vectors; higher-rank tensors are whole norm groups, with optional cautious/decay variants.
rollfast Hyperball transforms	JAX / Optax; community library	Fixed-initial-norm terminal transform and Adam/Muon/PRISM/RMNP/Kron/Aurora compositions, matrix masks, Adam fallback routing, and named-axis reductions; optional nonzero decay or caution changes the proposed direction.
MarinSkyRL MuonH / AdamH port	PyTorch / DTensor; Marin RL integration, merged 2026-08-03	Implementation routes hidden/expert matrices to MuonH, readout to AdamH, and auxiliary groups to Adam; zero decay, separate Adam LR, expert parallelism limited to one. Marin fork, not upstream SkyRL support or a verified RL-quality gain.
Author nanochat MuonH submission	PyTorch; author implementation, unmerged PR	Optimizer source and public run; combines NorMuon-based Hyperball with normalization, initialization, and schedule changes. Distinct from the earlier d12 negative log; recipe-level results, not an isolated optimizer swap.
ANCORA / ancora-cutile	CUDA Tile; community systems prototype	Device-resident MuonH/AdamH, FP32 master/norm arithmetic, BF16 views, per-expert constraints, and a NumPy oracle. Limited hardware/runtime validation; no matched multi-seed model-quality evidence or stable packaged API verified.
Tiny Shakespeare Hyperball sandbox	PyTorch; community learning artifact	Configurable matrix directions with RMS-to-Frobenius radius conversion; default `retract` follows the normalized-step wrapper, while optional tangent-projected `slerp` uses different geometry. Parameterization notes.
Palingenesis Hyperball wrapper	PyTorch; community fine-tuning wrapper	Normalizes the base optimizer's displacement and retracts to initial radius with decay disabled; `angular_lr=0` calibrates a separate per-matrix rate from the first base update, unlike a shared tuned Hyperball LR. No independent fine-tuning gain verified.
Chess-engine-4 Hyperball experiments	PyTorch; community application and mixed-result report	AdamHyperball source, paired seeds, controls, and W&B links. Initial d128 gate failed; later d256/d512 comparisons favored AdamH with width-indexed LR, while larger MXFP8 runs had spikes. MoE validation remains outstanding.

These contextual links do not increase the Hyperball paper count and are not asserted to implement Hyperball.

Reading	Connection and distinction
Spherical Motion Dynamics	Earlier analysis of normalization, SGD, weight decay, and motion on a sphere.
Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes	Earlier fixed-sphere learning-dynamics analysis; useful context for effective-LR regimes.
Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks	Analyzes norm/angular-update equilibria and explicit rotation control before Hyperball.
nGPT: Normalized Transformer with Representation Learning on the Hypersphere	Architectural/vector normalization differs from a matrix-wise Frobenius optimizer wrapper. The official illustrative code is a small reference implementation, not the internal training stack.
Learning Rate Transfer in Normalized Transformers	Shows that original nGPT does not by itself give width/token-horizon LR transfer, then introduces νGPT's alignment-based width, depth, and duration rules. νGPT remains row/vector-normalized rather than whole-matrix Frobenius Hyperball and is indexed in the μP collection.
Training nGPT	Extends normalized nGPT to a hybrid Mamba-2–Transformer MoE ladder through 30B total parameters. Its v2 uses model-dependent coefficients in (\eta_{\text{peak}}=C/\sqrt{d_{\text{model}}}) and explicitly leaves complete hyperparameter scaling laws for future work, so it is a scaling recipe rather than demonstrated zero-shot transfer or Hyperball.
Controlled LLM Training on Spectral Sphere	Spectral-sphere constraints and SSO/Muon Sphere differ from Hyperball's Frobenius sphere.
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration	Transfer theory adopted by later AdamH recipes; the original paper does not introduce Hyperball.
Summer-22B	Uses row-wise tangent-projected Adam and unit-row retraction; retained in the μP index, not labeled a Hyperball application.
Mano and Spherical Cautious Optimizers	Related tangent-space/oblique or cautious-update methods; checked texts do not establish direct AdamH/MuonH evaluation.

Hyperball Evidence and Limitations

Transfer is conditional. The original paper's finite width/depth sweeps are not a theorem for every architecture. HyperP supplies additional depth and MoE rules; its token-horizon exponent is fitted. Original experiments, HyperP.
Scheduling remains consequential. Free Lunch reports heuristic bidirectional schedule matching; ELR reports conditional loss alignment, including an initial transient. HyperTransfer additionally maps optimizer states under explicit assumptions. Its non-scale-invariant extension compares a rescaled representative, not necessarily the raw fixed-radius network's loss. Free Lunch, ELR, HyperTransfer.
Separate optimizer evidence from full-recipe gains. Puro combines hardware, precision, curriculum, and optimizer changes. The DNA study's family-level efficiency gains are not evidence that MuonH beats MuonW. Puro, DNA companion experiments.
Inspect norm conventions. Whole-matrix, row-wise, expert-wise, and global sharded norms define different algorithms. NeMo's inspected source requires an explicit radius; older online API documentation differs. Marin's expert-group investigation shows why per-expert direction computation does not necessarily imply per-expert radius constraints. Parameter-group exceptions are part of each linked implementation.
Read source beyond method names. Some repositories named Hyperball only project an ordinary optimizer step or clip to a ball; those operations omit parts of the original normalized-step sphere wrapper. See the implementation exclusions. Community ports may additionally change decay, angular-LR calibration, or the norm axes.

Hyperball Search Coverage and Maintenance

The search covered public sources through 2026-09-23, including arXiv version histories and full text, bioRxiv metadata, OpenReview records, author and institutional pages, GitHub source/logs/PRs, and Hugging Face releases. It followed Hyperball, AdamH, MuonH, SGDH, NorMuonH, AdEMAMixH, SOAPH, sphere-constrained Muon variants, HyperP, HyperTransfer, and the Fantastic Pretraining Optimizers II / 2.1 / 2.2 aliases. The September 23 pass verified unchanged latest versions for all 12 indexed arXiv papers, recovered six learning resources and nine implementation entries, and expanded existing model/data links. The bioRxiv page could not be re-read in that pass; its metadata is retained from the earlier audit. HyperTransfer, first submitted September 7, remains the latest directly verified paper by first-public date. Indexed search cannot establish absolute completeness.

See the full search audit, September 19 incremental audit, September 20 correction audit, and September 23 source audit for inclusion decisions, version issues, and unresolved artifacts. Graph-centrality HyperBall, geometric ball packing, generic “hyperball” terminology, games, mirrors, and automatic paper summaries are outside this optimizer collection. Follow CONTRIBUTING.md when adding entries.

Hyperball Source Audit — 2026-09-23

This audit extends the September 20 inventory through 2026-09-23, combining paper, learning-resource, and source-code checks. The inventory contains 13 papers, 22 learning resources / reports, and 24 implementation / artifact families. No additional qualifying research paper was verified; six resources and nine implementation families were recovered, including an upstream framework integration merged on September 21.

Collection	Previous	Added	Current
Research papers, including the labeled MD Decoupling extension	13	0	13
Learning resources / reports	16	6	22
Implementation / artifact families	15	9	24

All nine new implementation families belong in the main artifact index; their inclusion does not imply equivalent maturity, identical norm conventions, or demonstrated training gains. Puro and Delphi releases/data, Segurant, and additional Track 3 submissions enrich existing families without increasing their count. The review inspected public records and source; it did not execute third-party tests or reproduce training. Project-reported measurements remain attributed results.

Paper Search and Version Check

No qualifying paper first public from September 8 through September 23 was verified in this pass. The twelve existing arXiv records were reopened and their official Submission history checked; all latest versions match the previous inventory. Mirror crawl dates and “LastUpdated” labels were not treated as manuscript revisions.

Existing paper	Latest version verified
HyperTransfer	v1, 2026-09-07
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining	v1, 2026-08-28
Puro-2B	v2, 2026-09-03
Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining	v1, 2026-08-25
Hyperball May Not Be a Free Lunch	v1, 2026-07-24
On the Nonlinearity of Learning Rate Scaling for LLM Training	v1, 2026-06-28
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors	v2, 2026-07-17
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization	v1, 2026-06-15
Redesign Mixture-of-Experts Routers with Manifold Power Iteration	v1, 2026-06-10
Demystifying Manifold Constraints in LLM Pre-training	v1, 2026-05-06
Rethinking Language Model Scaling under Transferable Hypersphere Optimization	v2, 2026-04-05
Manifold constrained steepest descent for smooth and closed-set optimization	v2, 2026-08-13

HyperTransfer remains September 7 v1 despite a mirror displaying September 9 as an update date. The bioRxiv DNA paper could not be freshly read, so its version status was not independently reverified. Its July 22 posting date is retained from the earlier audit; the July 17 date embedded in the DOI is not substituted for the posting date.

Six Added Learning Resources / Reports

FAI-Seminar author talk — 2026-07-24

The official FAI schedule lists Kaiyue Wen's Chinese-language Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization talk. The row links the paper, announcement, and official Bilibili recording. It is a separate event from the existing Yao Class, Princeton, and IOS materials. The recording was not watched; Bilibili/WeChat targets failed extraction, so the verified schedule is the stable entry point.

Optimization 1 — Norm reparametrization — 2026-01-23

Ziming Liu's tutorial explicitly starts from Hyperball's fixed-radius motivation and explores learned magnitude through a two-dimensional Adam/MSE example and Colab. It is a related norm-direction extension, not an AdamH/MuonH LLM benchmark or a general counterexample for scale-invariant architectures.

Marin expert-group norm investigation — 2026-08-24

Issue #8621 documents cross-expert Frobenius normalization for stacked [layers, experts, in, out] tensors while Newton–Schulz acts on individual expert matrices. An August 25 comparison reports d768 Paloma macro loss 3.015 with grouped experts versus 3.018 per expert, described by the maintainer as within noise. The implementation and interpretation correction is detailed below; the issue is primary engineering evidence rather than a formal paper.

Agnes 2B pretraining protocol — 2026-09-03

The Agnes Foundation Model Team's report specifies selected-matrix MuonH with AdamW fallbacks, FP8, and a two-stage curriculum for a proposed 2.032B dense model. The date follows the first publication commit; the document labels itself Technical Report V1.1, September 2026. It explicitly describes a falsifiable protocol: training time, capability targets, and release floors are projections, reference-system measurements, or acceptance criteria, not completed Agnes training results. The repository supplies PDF/Markdown; no runnable full training release was independently verified.

Blog: Survey of Optimizers — 2026-08-28

Ruoran Xu's arXiv survey remains v1. Section 5.2 explains Hyperball through weight norms and angular learning rates; Sections 4.3, 10.5, and 14.3 add geometry and evaluation context. This is substantive explanatory coverage, but its Hyperball results summarize prior work; count it as a resource rather than another independent experiment or research-paper entry.

MACRO companion seminar at Johns Hopkins — 2026-09-14

The institutional event page identifies Shiqian Ma's Demystifying Manifold Constraints in LLM Pre-Training seminar. Its connection is the already-indexed MACRO comparison paper. The abstract discusses constraints, RMS normalization, and rotational equilibrium but does not itself name Hyperball. No public recording or slides were verified; this entry is an author seminar record, not a claim about viewed video content.

Nine Added Implementation / Artifact Families

1. PaddlePaddle upstream Muon / Adam Hyperball

PR #79792 merged on 2026-09-21, merge commit 8a0db499edcfa4ce6a54bc1d5d94c32f67e4da3c. Inspected Muon source routes use_hyperball and use_muon to MuonH, AdamH, ordinary Muon, or AdamW. _hyperball_apply computes FP32 radius and normalized update, steps by lr * radius, then reprojects; Hyperball routes omit weight decay. Trailing-two-axis norms distinguish full 2D matrices and individual 3D expert matrices. The sharding implementation keeps Muon/Hyperball tensors whole on an owner rank. Radius is recomputed rather than saved as an initialization checkpoint; exact preservation remains subject to epsilon and rounding. A merged integration does not establish availability in every released Paddle version.

2. HeavyBall HyperBallAdamW

The public class cites the Hyperball note and routes rank-two-or-higher tensors to Hyperball, with ordinary AdamW for vectors. The actual update normalizes the direction, steps by learning rate times stored initial norm, and projects back, with precision promotion and stochastic copy-back. Default weight decay is zero; optional decay/cautious masking changes the direction before normalization. Higher-rank tensors use whole-tensor norms, not automatic per-expert constraints.

3. rollfast Hyperball transforms

Pinned source stores initial L2 norms in HyperballState, uses FP32/named-axis reductions, and applies normalized direction, radius-scaled step, and fixed-radius projection as a terminal Optax transform. The README and tests expose AdamW, Muon, PRISM, RMNP, Kron, Aurora, and Riemannian-Aurora compositions, masks, and separate fallback learning rates. PyPI supplies a distribution entry. Optional nonzero decay/caution changes the underlying direction; no LLM superiority benchmark was verified.

4. MarinSkyRL PyTorch port

PR #249 merged on 2026-08-03; source at the PR head implements MuonH hidden/expert matrices, AdamH output head, and ordinary Adam auxiliary groups. MuonH/AdamH share a learning-rate track; ordinary Adam has another. _hyperball_delta uses trailing matrix axes and DTensor-aware materialization, with FP32 state and BF16 compute. The port rejects expert parallelism above one, nonzero weight decay, and unsupported options. The PR reports JAX-oracle, FSDP2/checkpoint, and four-H100 lifecycle checks, not an RL-quality comparison. This is Marin's SkyRL fork; it is not evidence of upstream SkyRL support.

5. Author nanochat MuonH submission

dangxingyu's PR #498 is unmerged. Pinned optimizer code supplies cached initial norms and hyperball_step_fused around a NorMuon direction. The submission links FP8 and BF16 runs plus schedule discussion #499. It also changes parameterized RMSNorm, zero-initialized vector output multipliers, matrix learning-rate depth scaling, and separate cooldowns: this is a recipe comparison, not an isolated optimizer swap. Reported d24/8-H100 results include 167.91 minutes and CORE 0.2645; these were not reproduced. Keep it distinct from the already-indexed January 29 d12 negative experiment.

6. ANCORA / ancora-cutile

The Hyperball kernel and NumPy oracle, AdamH head, and MuonH wiring establish the implementation beyond a README claim. It uses CUDA Tile/device-resident updates, FP32 master/norm arithmetic, BF16 views, per-expert constraints, MuonH hidden/expert matrices, AdamH untied head, and ordinary Adam auxiliary groups. The README describes a systems candidate with single-GPU Windows 11/CUDA 13.3/sm_120a validation, no stable package/API, and limited performance measurements rather than matched multi-seed quality evidence.

7. Tiny Shakespeare Hyperball sandbox

Optimizer source stores RMS radii and converts them to Frobenius radii with sqrt(numel) for normalized stepping and retraction. Its configurable ULMO directions default to hidden Gram Newton–Schulz. Default retract follows the wrapper geometry for that direction; optional slerp tangent-projects and uses an exponential-map update, a different variant. The parameterization note supports its role as a community learning/experimentation artifact, not an official reproduction.

8. Palingenesis fine-tuning wrapper

Current source snapshots parameter buckets, disables decay, obtains a direction from the base optimizer displacement, normalizes it, and applies an angular step plus initial-radius projection. Tests accompany the implementation. Positive angular_lr selects a common rate; zero calibrates a separate rate for each matrix from its first base update. Some prose still describes projection alone, so cite source for semantics. Repeated 20–30% claims derive from the original pretraining paper, not a verified fine-tuning gain.

9. Chess-engine-4 application and mixed-result report

Pinned source implements AdamHyperball with stored FP32 initial radii, radius-scaled updates, retraction, and zero-update handling; other parameter families use Adam. The experiment report includes per-arm runs, controls, paired seeds, and width/token-budget studies. Its initial d128 noninferiority gate failed (+0.01383/+0.01136 loss versus the best light AdamW); later d256/d512 arms favored AdamH, but learning rates remained width-indexed. Larger-width MXFP8 runs had spikes and MoE validation remained outstanding. This is useful community application evidence, not unchanged-learning-rate transfer or a formal paper.

Existing Families Expanded Without Double-Counting

Dragon / Segurant: Segurant AdamH and AdEMAMixH implement normalized updates and retraction, with 2D or per-first-axis-slice norms; group these training sources with the existing Dragon lineage.
Track 3: the pinned ledger additionally links #277 NorMuonH outer updates, #293 KL-SOAP-H, #302 SOAP-H, #316 PSGD + Hyperball, and #324 MuonH auxiliary tuning.
The latter three report 3125 steps/n=6, 3375/n=5, and 3250/n=10 respectively. #324 preserves matrix LR but retunes auxiliary parameters; these remain recipe-level records within one collection. No later Hyperball row than #324 was found in the inspected ledger.
Delphi: public blog data exposes six configurations and 4,117 rows, including 102 delphi-ladder and 3,737 hparam-scaling rows, fits, held-out validation, and per-row W&B links. The 447M / 122B-token model card identifies AdamH and the Complete(d)-inspired recipe; both enrich the existing recipe entry.
Puro: the author collection supplies base/phase checkpoints and curriculum/averaging inputs, alongside materialized training data. A root-review follow-up read the base-model card using a query-parameter URL after the clean URL failed extraction; it identifies random initialization and MuonH. These releases enrich the Puro-Megatron entry without another artifact count.
Marin / Levanter: the standalone AdamH source is the same implementation lineage. Individual agent experiment issues are not separate papers or implementation families.

Cross-Expert Norm Correction and Live-Run Status

Marin's 535B stack must be described as using recipe-specific cross-expert norm groups, not as independently fixing every expert matrix to its own initial Frobenius norm. The maintainer's acknowledgment says this grouping was unintended but retained after earlier comparisons; the author response accepts retaining it while monitoring individual expert norms. The 3.015-versus-3.018 d768 result does not establish a robust advantage. An August 31 stability update is an intermediate observation. The hero tracker remained open, last updated 2026-09-22 22:33 UTC; nothing inspected establishes completed 18T-token training by the cutoff. The agent digest explicitly summarizes 80 experiments as of August 20, so it must not be presented as covering all September issues.

Hyperball Exclusions and Unresolved Candidates

Author repository name is insufficient: dangxingyu/Megatron-LM-Hyperball had only main, with a complete non-truncated tree at 622a06af26348c999848531c2fa231507245e809; code search found no Hyperball. Its Muon source is ordinary tensor-parallel Muon. The verified author addition is nanochat PR #498.
PyTorchOptimizer: scoped search found no Hyperball source; generic Muon support does not establish AdamH/MuonH support. Existing NeMo MuonHyperball remains valid but requires an explicit matching nonzero radius.
Different geometry: Odyssey uses one-sided ball clipping; koochak projects after an ordinary optimizer step without the normalized radius-scaled direction. Neither is the exact original wrapper.
Configurable toolkit: MODULUS offers sphere/ball/tangent/target-angle variants and scalar/row/column/leaf choices; its defaults do not automatically reproduce initialization-radius whole-matrix Hyperball.
Code-only candidate: optimstep matrix-step source normalizes a base optimizer displacement and reprojects; base decay/options and a possible subsequent anchor-decay branch matter. No published result was verified, so retain it as supplementary rather than a counted benchmark family.
AdamH name collision: FRAMES-VQA, also copied into vlm_robustness, uses Adam with conditional decay toward initialization, not fixed-radius projection.
Copied records: optimizer-fingerprints, nanogpt-optimizer-benchmark, and many nanochat/modded-nanogpt forks expose copied benchmark files; copies are not independent implementation evidence.
Adjacent theory: Weight-norm Criticality discusses norm shrinkage and loss spikes but has no Hyperball/Muon text match or established direct comparison. Existing nGPT, SSO, Mano, Nora, PC Layer, AngularMuown, and OmniOpt classifications were not upgraded without new direct evidence.
Unconfirmed paper: OpenReview 39sAm5aTYZ appears in search with a Hyperball bibliography entry, but browser challenges/API failures prevented title and substantive-body verification; a citation alone is insufficient.
Other collisions: graph-centrality HyperBall/HyperANF, hyperbolic packing, conceptual-space balls, Deep SVDD boundaries, cryptographic sampling, games/paintball, Mien muonh, and unrelated SGDH/ADAMH/AdamHD abbreviations are excluded. No distinct official SGDH release was established.
Automatic summaries, translations, social mirrors, and speculative “MuonH Stiefel ERM” descriptions without an identified primary paper do not supply independent evidence. Nearly constant RMS in released weights, including Muse Glimmer observations, cannot establish the optimizer used.

Access Limits and Uncounted Resource Leads

Tencent ELR: official page and its English variant failed extraction; direct access returned HTTP 567. A public bookmark, translations, and HyperTransfer's bibliography corroborate existence, not a fresh primary-body review. The reported August 5 date remains unconfirmed; no translation is counted separately.
Author material: the Hyperball 2.1 redirect reached a Notion URL that returned 404 here, and 2.2 yielded no extractable body. IOS Google Slides export was inaccessible; its prior content description is inherited, not freshly reviewed. These failures do not establish withdrawal.
Princeton recording: the official event linked Xingyu Dang — PLI Lunch Series 20260205; add it to the existing entry, but the recording was not watched.
Unreleased/future talks: Kaiyue Wen's homepage records an April 2 Kimi talk without verified public media. The September 30 Rochester MACRO event is future-dated at this cutoff. Neither increases completed accessible resources.
Snowball application: the August 16 SFT report does not identify its optimizer in the inspected body; related implementation routing does not isolate Hyperball's contribution to its reported gains.

Search Coverage and Interpretation

Paper discovery combined exact Hyperball/AdamH/MuonH/HyperP/HyperTransfer queries, September date restrictions, arXiv/OpenReview searches, and full-text follow-up on sphere constraints, effective learning rate, and weight norms. All twelve arXiv version histories were checked directly; new candidate inclusion required primary metadata and a substantive section, experiment, or implementation connection. Resource searches covered English/Chinese author names and combinations with seminar, lecture, slides, video, Bilibili, tutorial, Tencent ELR, Open Athena, Marin, and Snowball. Existing author pages, institutional schedules, blogs, issue discussions, and the Marin digest/hero tracker were revisited, including recent September issue searches. GitHub discovery used Hyperball, MuonH, AdamH, NorMuonH, AdEMAMixH, and SOAPH, followed by repository-scoped code, PR metadata, branches, commit histories, benchmark ledgers, and Hugging Face collections. Global AdamH/SOAPH queries were noisy, making source-level scoped follow-up essential. Parameter routing, norm axes, initialization radius, normalized direction, projection, and optional decay determined implementation classification. This is a reproducible account of inspected public evidence, not a guarantee of exhaustive coverage of private repositories, unindexed forks, inaccessible pages, or newly published material.

Scale-Aware Hyperparameter Transfer

Dedicated complementary collection on transferring optimization hyperparameters from affordable proxy runs to larger models or more expensive training configurations. Snapshot: 2026-09-19.

The collection contains 17 complementary direct papers, 10 learning resources, and 14 implementation / artifact entries. Direct papers already counted by the strict μP or Hyperball collections are cross-linked from the guide instead of duplicated; resources and artifacts may overlap. It covers width, depth, model/data scale, token horizon, batch size, schedules, sparsity, expert configuration, adaptation rank, and post-training scale. Cross-dataset AutoML transfer and performance-only scaling laws are outside its direct scope.

Read the scale-aware HPT guide for the taxonomy, cross-collection index, practical protocol, and distinctions between μP, Hyperball, HyperP, nGPT, and νGPT. See BibTeX and the search audit.

Complementary Direct Papers

Date	Paper	Main contribution	Transfer axes
2026-09-08	Hyperparameter Scaling Laws Across MoE Sparsity	Fits learning-rate and batch-size laws that explicitly include MoE activation ratio, then validates joint scale-and-sparsity extrapolation on a held-out 12B-total-parameter model with 1/64 activation.	compute, tokens, MoE sparsity, expert granularity
2026-09-01	Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search	Introduces PLES, a cost-aware multi-fidelity acquisition rule that selects proxy runs to reduce uncertainty in an entire power-law hyperparameter fit; reported experiments need less than one tenth of grid-search compute.	model scale, data scale, experiment budget
2026-08-28	Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss	Jointly models optimal learning rate and batch size over model/data scale and tests whether settings transfer between the stable and decay phases of WSD schedules; releases the underlying pretraining-run collection.	model size, data, batch, WSD phase
2026-07-08	Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks	Gives an exact failure case for data-agnostic depth rules and derives a data-dependent correction whose dynamics are nearly depth independent in the analyzed scalar networks.	depth, data distribution
2026-07-01	How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size	Splits data budget into batch size and training steps in a three-term loss law, recovering optimal and suboptimal batch-size scaling from runs that need not all use an optimal batch.	model size, steps, batch, token allocation
2026-06-04	Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training	Learns proxy laws from compute budget to optimal learning rate and batch size, estimates a checkpoint's equivalent pretraining compute, and reports up to 90% lower search overhead for continued pretraining.	continued-pretraining state, compute, batch
2026-06	Post-Training Science for Supervised Fine-Tuning	Measures whether learning-rate and batch-size choices transfer across Qwen3 and Llama, dense and MoE models, LoRA and full fine-tuning, datasets, and a model ladder reaching 235B parameters; recommendations include uncertainty estimates.	post-training scale, family, data, LoRA/full SFT
2026-03-22	On the Role of Batch Size in Stochastic Conditional Gradient Methods	Derives regime-dependent batch-size and step-size rules under fixed token budgets for momentum conditional-gradient methods, proposes an adaptive batch/sequence strategy, and checks the predicted regimes in NanoGPT.	batch, step size, token budget, sequence length
2026-03-16	Deriving Hyperparameter Scaling Laws via Modern Optimization Theory	Derives learning-rate, momentum, and batch-size power laws from optimization bounds for LMO-based methods including normalized SGD, signSGD, and Muon; the model size is held fixed.	iterations, tokens, batch, momentum
2026-02-06	Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate	Uses a weak-convexity-inspired loss bound to fit learning-rate laws and reports extrapolation up to 80× in training horizon and 70× in model size.	model size, training horizon, schedule
2026-02-04	Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model	Derives horizon-dependent optimal schedules, batch ramps, and momentum behavior in a solvable model, then shows that horizon transfer differs between easy and hard regimes in simple vision and language experiments.	horizon, schedule shape, batch, momentum
2025-03-06	Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining	Fits optimal learning rate as a function of model and data scale and optimal batch size primarily as a function of data, using 3,700 runs across dense/MoE shapes and data recipes; releases code, data, and checkpoints.	parameters, data, batch, model shape, dense/MoE
2025-02-24	Function-Space Learning Rates	Introduces FLeRM: record layerwise function-space update scales on a cheap model, then adjust target parameter-space learning rates to match them across width, depth, initialization scale, and LoRA rank.	width, depth, initialization, LoRA rank
2025-02-07	Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient	Alongside its loss/compute study, derives and tests an optimal-learning-rate law using active non-embedding parameters and expert count, including expert-count interpolation and extrapolation.	active parameters, expert count, MoE scale
2025-01-31	The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training	Uses a convex-optimization proxy to transfer an optimal learning rate across schedule extensions and continued-training horizons in 124M and 210M Llama-style models.	schedule length, continued training
2024-05-23	Scalable Optimization in the Modular Norm	Recursively defines an architecture-level modular norm and normalizes any base optimizer's updates so one learning rate transfers across width and block count/depth in Transformers, ResMLPs, and ResNets.	width, depth, architecture, base optimizer
2024-01-05	DeepSeek LLM: Scaling Open-Source Language Models with Longtermism	Section 3.1 fits optimal learning rate and batch size as power laws of training compute on proxy runs, validates at a larger held-out compute budget, and uses the laws in the 7B/67B scaling recipe.	compute, learning rate, batch size

Scale-Aware HPT Learning Resources

Resource	Type	Why it matters
μTransfer: A technique for hyperparameter tuning of enormous neural networks	Microsoft Research explainer	Practical introduction to base shapes, proxy sweeps, and zero-shot width transfer with μP.
Greg Yang's Tensor Programs reading guide	Author-maintained guide	Organizes the Tensor Programs lineage and links talks, papers, and code behind μP.
Quickstart Guide: Hyperparameter selection	Learning Mechanics tutorial	Connects width/depth parameterization choices to transfer experiments and concrete diagnostics.
Step Law project	Official project and calculator	Interactive entry point for the empirical model/data learning-rate and batch-size laws, with released data and checkpoints.
The Modula Docs	Official documentation	Explains modular norms, architecture composition, optimizer wrapping, and the implementation used for modular-norm transfer.
Fantastic Pretraining Optimizers 2.1: Hyperball Optimization	Living author note	Original Hyperball research lineage and geometric motivation for fixed-radius optimizer wrappers.
The Hitchhiker's Guide to the Weight Norm Theory	Living author tutorial	Develops weight-norm and angular-step interpretations needed to reason about Hyperball schedules.
On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism	Technical essay	Works through assumptions connecting SGDH, AdamH, MuonH, feature-space scaling, and μP.
Scaling Laws That Extrapolate 300× Past the Fit	Primary technical report	Documents a practical Complete(d)P/AdamH scaling workflow, failed initial assumptions, held-out checks, and a hyperparameter calculator.
Hyperparameter Optimization in Machine Learning	Survey	Broad HPO reference useful for separating model-scale hyperparameter transfer from cross-task AutoML transfer and ordinary search methods.

Scale-Aware HPT Implementations and Artifacts

Artifact	Framework / method	What it provides
microsoft/mup	PyTorch; μP / μTransfer	Reference base-shape tooling, μP layers, optimizer parameter groups, coordinate checks, and examples.
modula-systems/modula	JAX; modular norm	Official package for recursively composing modules and normalizing base-optimizer updates for width/depth learning-rate transfer.
function-space-learning-rates-paper	PyTorch; FLeRM	Official experiments and measurement code for matching layerwise function-space learning rates across scales.
step-law/steplaw	LLM pretraining; Step Law	Official training code, loss measurements, checkpoints, and optimal-hyperparameter estimator.
OpenEuroLLM dense English scaling laws	LLM pretraining; empirical laws	Official scripts and records for the OpenEuroLLM learning-rate, batch-size, loss, and WSD phase study.
OpenEuroLLM scaling-law releases	Data and models	Training measurements and model artifacts accompanying the OpenEuroLLM fits.
microsoft/ArchScale	PyTorch / LitGPT; HyperP	Official HyperP, MuonH, SqrtGate, and width/depth/MoE scaling experiments.
VITA-Group/principled_scaling_lr_init	Architecture-aware HPT	Official code for topology-aware initialization and maximal-learning-rate scaling across computation graphs.
S-Lab-System-Group/Hydro	Distributed HPO; μP proxies	Uses small μP surrogates to preserve multi-hyperparameter rankings and reduce target-scale HPO cost.
fabian-sp/lr-scheduling	PyTorch; schedule transfer	Official experiments for convex-proxy learning-rate scheduling and transfer across schedule extensions.
Joint MoE scaling-law releases	Models and inference	Author-released MoE checkpoints and inference code accompanying the expert-count and active-parameter scaling study.
deepseek-ai/DeepSeek-LLM	Models and training utilities	Official 7B/67B release accompanying the DeepSeek scaling recipe; it is not a standalone reproduction of the hyperparameter-law sweeps.
NVIDIA/ngpt	PyTorch; normalized Transformer	Illustrative code for nGPT's row/vector-normalized baseline; νGPT supplies the later transfer-specific scaling rules.
NVIDIA NeMo Emerging-Optimizers	PyTorch; Hyperball	Framework implementations of MuonHyperball and Hyperball hooks; radius and tensor-group conventions must match the intended recipe.

Full Paper List

The table is ordered by arXiv published date, OpenReview public date, official report date, or venue date in reverse chronological order. The current search window covers 2024-08-25 to 2026-09-19; earlier rows retain the direct theoretical lineage and historically important applications.

The 2026-08-13 audit re-read all 239 previously listed papers and removed 114 non-substantive, adjacent-only, or foundation-only entries. Inclusion now requires substantive derivation, implementation, experimental validation, direct criticism, or material application of μP / μTransfer. The audit also cross-checked arXiv, OpenReview, venue-only records, full-text application reports, official projects, and the active francesco-innocenti/mup-papers index; alternate records of the same work are collapsed into one entry.

The 2026-08-18 incremental audit found no new direct μP record first released after August 13, but recovered seven earlier false negatives through full-text and venue-level checks, removed one residual-scaling / Muon paper that only cited μP-related work, tightened fourteen ambiguous descriptions, and reclassified companion artifacts by what they actually implement.

The 2026-08-24 full-text audit added one newly released direct record and restored nine earlier false negatives, including Depth-μP / Muon routing analysis, μP-controlled scaling-law and warmup studies, progressive depth expansion, Hydro, Cerebras-GPT, MiniCPM, and the Feature Speed Formula. It also removed an abstract-only record whose public text did not expose any μP result, refreshed official code, checkpoints, and documentation, and continued to reject work where μP is only a citation, analogy, or unvalidated configuration.

The 2026-08-25 completion audit found no new direct record released on August 24–25, but restored eleven historical false negatives through primary-source full-text checks. It removed a dynamic-sparsity paper whose learning-rate rule was only motivated by μP, moved the TASI lectures to teaching resources, corrected descriptions that overclaimed transfer validation, and retained the Depth-μP / Muon MoE study with an explicit note that it does not test multi-depth optimal-LR transfer.

The 2026-09-05 incremental search cross-checked arXiv, OpenReview, venue records, author resources, and the community μP index. It added AK-Momentum (formerly DeltaMomentum; renamed on September 3) and the CCN 2025 SwiFT V2 extended abstract, plus experimental and teaching resources. New entries require primary-source μP derivations, coordinate checks, or explicit proxy-to-target transfer; related-work-only matches are excluded. This is a best-effort literature search through the snapshot date, not a claim that every unindexed or unpublished work has been found.

The 2026-09-14 audit recovered two application papers (ATLAS GN2 and token-level data filtering), fourteen teaching / technical resources, and eight implementation links. It checked primary full texts, author code, venue records, and community-index gaps while preserving the substantive μP scope. See the search audit for evidence, date conventions, excluded general-HPT candidates, and unresolved records.

The 2026-09-19 incremental audit found no new direct μP record after September 14. It promoted previously excluded non-μP scale-transfer work into the separate scale-aware HPT collection, preserving this table's stricter requirement that μP be a substantive method or object of analysis. See the combined audit.

Date	Paper	Main contribution	Tags
2026-08-20	Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts	Adapts μP to MLA-MoE models trained with Muon, validates one optimal learning rate while jointly widening the model and increasing total experts, then fits a proxy token-horizon law used for 155B-total / 17B-active pretraining over 10T tokens.	application report, μP, MoE, Muon, MLA, width-expert transfer, token horizon
2026-08-19	Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule	Derives a width-invariant delta-rule momentum coefficient under μP, checks activation and effective-update scales across widths 128–2048, and transfers a 67M proxy's hyperparameters to 370M / 1B language models; formerly titled DeltaMomentum.	μP extension, AK-Momentum, optimizer, width transfer, coordinate check
2026-07-30	Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers	Builds HeteroP from μP / CompleteP, scaling tensors by functional fan-in and block depth so one proxy-tuned base learning rate transfers across a heterogeneous diffusion Transformer / MoE family.	application report, μP extension, HeteroP, width-depth transfer, MoE
2026-07-06	Hyperparameter Transfer in Graph Neural Networks	Extends Tensor Programs IV/V maximal-update reasoning to GNNs under SGD, Adam, and AdamW, yielding stable feature updates and learning-rate transfer across width and depth.	μP extension, GNNs, width-depth transfer
2026-06-16	On the Residual Scaling of Looped Transformers: Stability and Transferability	Applies maximal-update conditions to weight-tied looped Transformers and derives residual scaling that preserves learning-rate transfer across loop counts.	maximal update, looped Transformers, residual scaling
2026-06-16	Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion	Extends CompleteP-style depth scaling to constrained spectral optimizers such as Scion, supporting width- and depth-wise learning-rate transfer.	Scion, depth transfer
2026-06-04	Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss	Shows activation preconditioning preserves magnitude-based μP rules, so AdamW, Muon, and Shampoo learning-rate / weight-decay width scalings port to DoPr, and validates DoPr-AdamW width transfer.	μP compatibility, DoPr, width transfer, LR-WD scaling
2026-06-02	Unlocking Feature Learning in Gated Delta Networks at Scale	Derives μP scaling rules for Gated Delta Networks and validates width learning-rate transfer under AdamW and SGD.	Gated Delta Networks, sequence models
2026-05-29	Fast Learning Rate Transfer for Gradient Descent in Sketched Linear Regression	Analyzes the finite-width convergence rate behind μTransfer in a solvable sketched-linear model, proving unconditional fast optimal-LR transfer at fixed horizons and conditional fast-transfer regimes when width and horizon grow jointly.	μTransfer, fast transfer, optimal LR, finite-width theory, sketched linear regression, HiLD 2026
2026-05-29	Why Routers Freeze: Infinite Width Learning Dynamics for Mixture of Experts	Uses Tensor Programs to show router saturation under standard parameterization and derives μP-MoE scaling for stable router dynamics.	MoE, routers, OpenReview
2026-05-29	Depth scaling and Muon enable balanced expert usage in MoE training	Extends 1/√L Depth-μP residual scaling to MoE routing, deriving depth-stable hidden-state concentration and router-balance guarantees and validating the mechanism at initialization and in toy pretraining with Muon; it does not test optimal-LR transfer across multiple depths.	Depth-μP extension, MoE, Muon, routing balance, HiLD 2026
2026-05-25	Scaling Limits of Deep Reinforcement Learning: A Stability Analysis with Maximal Update Parametrization	Compares width scaling of Deep Q-Networks under mean-field, NTK, and μP; μP prevents the tested large-width divergence and rank collapse, with more conservative returns.	μP, reinforcement learning, width scaling, stability, trade-off
2026-05-23	Feature Learning in Wide Neural Networks under μP: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field Limit	Studies identifiability, sparse-dictionary structure, and mean-field feature-learning limits for wide two-layer networks under μP.	mean field, identifiability
2026-05-22	Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models	Extends hyperparameter transfer across dense FFN and MoE settings via active-width μP and expert-capacity scaling.	MoE, Complete-muE
2026-05-20	Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate	Defines transfer-quality metrics and argues that embedding learning-rate scaling explains much of μP's practical AdamW benefit in LMs.	embedding LR, AdamW
2026-05-19	Toto 2.0: Time Series Forecasting Enters the Scaling Era	Tunes a 10M proxy once with u-μP and transfers the recipe to five 4M–2.5B time-series Transformers, while documenting distributed/FSDP implementation adaptations.	application report, u-μP, proxy-to-target transfer
2026-05-14	GQA-μP: The maximal parameterization update for grouped query attention	Derives μP scalings for grouped-query attention and studies transfer over GQA repetition and weight decay.	GQA, attention
2026-05-13	How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization	Analyzes MoE scaling regimes, shows where μP transfer can fail, and derives MSSP for robust learning-rate transfer.	MoE, MSSP
2026-05-13	When is Warmstarting Effective for Scaling Language Models?	Uses μP as the controlled width-transfer baseline for warmstarting, transferring the best width-48 configuration to larger MLPs and all language-model scales while testing when model growth remains compute-efficient.	warmstart, μP, width transfer, experimental control
2026-05-11	Hyperparameter Transfer for Dense Associative Memories	Extends μP-style maximal-update prescriptions to Dense Associative Memories with shared weights and sharp activations.	μP extension, DenseAM, HPT
2026-05-09	Sparse Layers are Critical to Scaling Looped Language Models	Extends μP to shared loop blocks, MoE experts, routers, and unembeddings, then validates learning-rate transfer from width 128 to 1024 with tightly matched loss.	application report, looped Transformer, MoE, μP, width LR transfer, weight sharing
2026-05-08	Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer	Tracks bulk and outlier spectral dynamics and shows μP yields width-consistent learning-rate transfer in deep linear settings.	spectral dynamics, theory
2026-05-06	Demystifying Manifold Constraints in LLM Pre-training	Tests MACRO with μP-compatible initialization and constraint radii, finding consistent optimal learning rates across widths 256–2048 for both Frobenius and spectral variants.	μP, MACRO, width transfer, manifold constraints
2026-05-05	Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer	Derives Nora's η∝n^-1/2 rule from μP feature-update conditions; its 60M and 135M experiments tune each scale separately and therefore do not independently validate zero-shot LR transfer.	μP, Nora, Muon, width LR scaling, transfer caveat
2026-04-29	Learning Rate Transfer in Normalized Transformers	Revisits μP for nGPT and proposes νGPT, enabling learning-rate transfer across width, depth, and token horizon.	nGPT, νGPT
2026-04-28	Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer	Derives a probabilistic-Transformer-specific μP parameterization and validates cross-scale hyperparameter transfer in masked-language-model experiments.	μP extension, probabilistic Transformer, cross-scale HPT
2026-03-30	Rethinking Language Model Scaling under Transferable Hypersphere Optimization	Derives HyperP from μP and Depth-μP under Frobenius-sphere constraints, then tests MuonH transfer across width, depth, architecture shape, and MoE granularity.	μP extension, Depth-μP, HyperP, MuonH, MoE
2026-03-10	On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer	Derives MOGA operator-geometry width scalings, exactly recovers μP for Adam/SignSGD, relates them to the spectral condition, and validates width-wise LR invariance.	μP, MOGA, operator norms, optimizers, width LR transfer
2026-02-28	Spectral Condition for μP under Width-Depth Scaling	Builds a unified spectral recipe for μP under joint width-depth scaling, including practical multi-transformation residual blocks.	width-depth, spectral conditions
2026-02-26	Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model	Transfers a hypersphere-constrained μP recipe from 30M to 1B parameters, while handling the separate 1K→100K training-horizon axis with empirically fitted √T and √B corrections.	application report, video, μP, width transfer, empirical horizon correction
2026-02-24	Extending μP: Spectral Conditions for Feature Learning Across Optimizers	Uses spectral conditions to derive μP-style transfer rules for AdamW, ADOPT, LAMB, Sophia, Shampoo, and Muon.	optimizers, spectral conditions
2026-02-11	μpscaling small models: Principled warm starts and hyperparameter transfer	Introduces μP-motivated upscaling and warm-start methods that preserve hyperparameter transfer when growing model width.	warmstart, upscaling
2026-02-07	Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks	Extends maximal-update / AM-μP reasoning to non-recurrent multi-path architectures and derives effective-depth laws for width-depth learning-rate transfer.	maximal update, AM-μP, depth, multi-path
2026-02-07	On the Infinite Width and Depth Limits of Predictive Coding Networks	Shows predictive-coding networks and backpropagation share the same μP width- and depth-stable feature-learning parameterizations.	μP, predictive coding, width-depth
2026-02-05	Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning	Introduces Maximal-Update Adaptation (µA), derived from μP feature-update conditions, and validates rank-wise LR scaling plus LoRA-to-full-finetuning transfer.	μA, LoRA, finetuning, LR transfer
2026-01-29	Shaping capabilities with token-level data filtering	Uses depth-matched width-512 μP proxies to select AdamW learning rate and weight decay for data-filtering experiments spanning a language-model family up to 1.816B parameters.	application report, μP, width transfer, depth-matched proxy, data filtering
2026-01-28	Hyperparameter Transfer with Mixture-of-Expert Layers	Extends μP-style Transformer parameterization to transfer hyperparameters across width, depth, expert count, and expert size.	μP extension, MoE, HPT
2026-01-19	Scaling laws for amplitude surrogates	Gives explicit μP initialization and learning-rate rules, validating width-wise LR invariance for MLP surrogates while reporting transfer instability for the LLoCa-Transformer family.	application report, scientific ML, μP, width LR transfer, negative result
2026-01-15	On the origin of neural scaling laws: from random graphs to natural language	Uses μP throughout synthetic-sequence scaling experiments and directly compares μP with standard parameterization on language, finding preliminary evidence for a larger model-size exponent and a different compute-optimal parameter/token allocation.	μP analysis, scaling laws, parameter efficiency, compute-optimal scaling
2026-01-13	Controlled LLM Training on Spectral Sphere	Matches a spectral-sphere optimizer to μP spectral conditions and transfers a single optimal learning rate across widths 256–2048 and 70M–1.8B language models while controlling weight and update scales.	spectral μP, optimal-LR transfer, width scaling, optimizer
2026-01-08	How to Set the Learning Rate for Large-Scale Pre-training?	Compares fitted LR scaling laws with transfer, extending μTransfer to MoE, depth, weight decay, and token horizons while documenting large-scale limits.	μTransfer, MoE, token horizon
2026-01-08	Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers	Introduces learnable row/column multipliers as a more expressive generalization of μP multipliers, compares against tuned μP baselines, and analyzes width scaling; a complete transfer parameterization remains open.	μP extension, multipliers, LLMs, transfer limitation
2026-01-06	From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence	Tunes learning rate on small models and applies μP plus CompleteP across width, depth, and aspect ratio when estimating learnable information in data.	application report, μP, CompleteP, width-depth-aspect transfer
2026-01-04	Towards a Principled Muon under μP: Ensuring Spectral Conditions throughout Training	Develops Muon++ to maintain μP spectral conditions throughout training without repeated weight normalization.	Muon, spectral conditions
2025-12-31	Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space	Uses decoupled μP parameterization for zero-shot hyperparameter transfer across widths and compression regimes.	application report, decoupled μP
2025-12-28	Understanding the Mechanisms of Fast Hyperparameter Transfer	Formalizes fast hyperparameter transfer and studies when μP transfer is compute-efficient versus when it fails.	mechanism, fast transfer
2025-12-26	Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration	Extends CompleteP-style transfer to modules, width, depth, batch size, training duration, and per-module hyperparameters; accepted as an ICLR 2026 Poster, after initially being submitted as “Transfer Paramatters.”	CompleteP, batch, duration
2025-12-24	Feature Learning Dynamics in Infinite-Depth Neural Networks	Gives a rigorous feature-learning limit for one-layer ResNets under depth-μP and quantifies how forward-backward weight-reuse effects vanish with depth.	depth-μP, infinite depth, theory
2025-12-22	DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations	Tunes a 50M proxy, applies μTransfer across the 13M–1.83B family, and reports a width-256–1024 coordinate check before its compute/data-scaling study.	application report, iEEG foundation model, μTransfer, coordinate check, model-data scaling
2025-12-20	Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale	Devotes a full chapter to maximal-update parameterization, μTransfer, optimizer dependence, activation/depth scaling, local learning, and u-μP implementation recipes.	survey, μP, μTransfer, optimization
2025-12-11	Scaling Behavior of Discrete Diffusion Language Models	Uses CompleteP to transfer a recipe from 25M / 50M proxies across width and depth in discrete diffusion LMs, while identifying batch size and token horizon as separate corrections.	application report, discrete diffusion LM, CompleteP, width-depth transfer, batch, token horizon
2025-12-09	Jais 2: A Family of Arabic-Centric Open Large Language Models	Tunes base learning rate, initialization, embedding / unembedding multipliers, and layerwise scales on a 100M, width-256, depth-68 μP proxy before transferring the recipe to the 8B and 70B models.	application report, Arabic LLM, μP, width transfer, depth-matched proxy
2025-12-05	Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales	Derives μP width/depth learning-rate and epsilon rules for Shampoo, SOAP, and Muon, then shows how blocking and spectral normalization reduce finite-width drift in transfer experiments from 190M to 1.4B parameters.	μP, matrix preconditioners, Muon, width-depth transfer, finite-width effects
2025-11-23	Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM	Uses μP to transfer hyperparameters from a 20M proxy to a 1.3B small language model.	application report, SLM
2025-11-14	Training Neural Networks at Any Scale	Reviews scale-agnostic optimization, μP, coordinate checks, and parameterization-aware rules for keeping training behavior stable across model scales.	survey, scale-agnostic optimization
2025-11-07	Scaling depth capacity via zero/one-layer model expansion	Uses μP spectral conditions to classify depth-expansion initializations and validates a Muon-NSGD μP-scaled recipe that reuses hyperparameters before and after zero/one-layer expansion; the focus is progressive depth growth rather than a general Depth-μP theorem.	μP analysis, depth expansion, hyperparameter transfer, Muon-NSGD, progressive training
2025-11-03	A Proof of Learning Rate Transfer under μP	Proves width learning-rate transfer for linear MLPs under μP and contrasts it with SP and NTP.	theory, LR transfer
2025-10-21	Weight Decay may matter more than muP for Learning Rate Transfer in Practice	Challenges the practical mechanism of μP transfer in LLM settings and argues weight decay often stabilizes representation updates after early training.	critique, weight decay
2025-10-17	Robust Layerwise Scaling Rules by Proper Weight Decay Tuning	Extends μP into the AdamW steady state: combining μP's matrix LR rule η∝d^-1 with λ∝√d preserves sublayer gains and restores zero-shot LR–WD transfer across width.	μP extension, weight decay, AdamW, width transfer
2025-10-05	Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets	Replaces per-layer maximal-update constraints with an average update criterion for CNNs and ResNets, yielding width-robust depth laws.	AM-μP, CNNs, ResNets
2025-09-29	Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs	Uses a 39M μP proxy sweep to align training hyperparameters from 111M to 3.9B parameters, then transfers a re-evaluation-timescale fit across sizes to design data curricula.	application report, μP, data curriculum, optimizer timescale, model scale
2025-09-29	Scaling with Collapse: Efficient and Predictable Training of LLM Families	Validates μP training-curve collapse at LLM scale when tokens per parameter, LR schedule, and AdamW timescale are scaled consistently.	μP application, TPP, AdamW
2025-09-18	CompleteP for RL: Maintaining Feature Learning When Scaling Deep Reinforcement Learning	Applies CompleteP to non-stationary reinforcement learning and demonstrates learning-rate transfer plus feature and policy consistency across width and depth; accepted at ICML 2026, with the earlier ICLR submission retained for version history.	CompleteP, reinforcement learning, ICML 2026
2025-09-18	Optimal Scaling Needs Optimal Norm	Implements the μP spectral condition with layerwise Scion operator norms, then finds an output-norm invariant across width, depth, batch, and token horizon while fitting the corresponding optimal LR–batch scaling law.	maximal update, spectral condition, Scion, multi-axis invariant, LR-batch scaling
2025-09-12	The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram	Extends CompleteP's proportional width–depth result to general non-proportional (L,M,D) limits and derives residual scaling necessary and sufficient for maximal local feature updates.	CompleteP extension, feature learning, width-depth limit, residual scaling, theory
2025-08-13	FM4NPP: A Scaling Foundation Model for Nuclear and Particle Physics	Derives μP-SSM rules for Mamba state-space matrices and validates them with layerwise activation and update coordinate checks, where standard or heuristic μP variants become unstable.	application report, scientific foundation model, μP-SSM, Mamba, coordinate check
2025-08-13	μ-Parametrization for Mixture of Experts	Derives μ-parameterization rules for MoE models and demonstrates learning-rate transfer across model sizes.	MoE, μTransfer
2025-08-12	SwiFT V2: Towards Large-scale Foundation Model for Functional MRI	Implements μP for 4D fMRI Transformers, verifies width-stable activations with coordinate checks, and transfers smaller-model hyperparameters to models up to 8.8B; the largest model was still training in this CCN extended abstract.	application report, μP, μTransfer, fMRI, coordinate check, CCN 2025
2025-07-30	Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance	Tunes architecture-specific μP multipliers on 300M–1.5B hybrid attention / SSM proxies and applies the transferred recipe across the 0.5B–34B Falcon-H1 family.	application report, hybrid LLM, μP, proxy-to-target transfer
2025-07-11	Pre-Training LLMs on a budget: A comparison of three optimizers	Tunes optimizer-specific μP recipes on a 50M proxy and transfers them through a 2.7B target; appendix width sweeps independently verify LR transfer for AdamW, Lion, and Sophia.	application report, μP, optimizer comparison, width LR transfer, LLM
2025-07-09	Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation	Introduces μP++ by combining μP, Depth-μP, and zero weight decay on vector/scalar-like parameters, then validates stability in scaling studies up to 3.4B parameters / 600B tokens; the final 3.8B production model used SP.	μP++, Depth-μP, width-depth scaling, weight decay, hybrid LLM, negative result
2025-07-06	Transfer Learning in Infinite Width Feature Learning Networks	Develops source-to-target transfer-learning dynamics in the μP / mean-field feature-learning limit and characterizes when limited source data or task misalignment causes negative transfer.	μP analysis, transfer learning, mean field, adaptive kernels, negative transfer
2025-07-02	Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks	Uses μP throughout a Transformer scaling ladder and directly ablates the parameterization while studying when suboptimal hyperparameter scaling breaks compute-optimal collapse.	μP application, scaling collapse, parameterization ablation
2025-06-27	HyperCLOVA X THINK Technical Report	Sweeps learning rate and regularization on sub-billion μP proxies, then zero-shot transfers the selected settings to multi-billion HyperCLOVA X THINK production models.	application report, μP, μTransfer, LLM, proxy-to-target
2025-06-24	Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators	Derives μTransfer-FNO for scaling Fourier Neural Operators by Fourier modes.	neural operators, PDE
2025-06-20	The Importance of Being Lazy: Scaling Limits of Continual Learning	Directly compares NTP and μP width scalings, finding a task-dependent optimal feature-learning level that transfers across widths in continual learning.	μP, NTP, width transfer, continual learning, DMFT
2025-06-17	Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size	Identifies the fixed-vocabulary assumption behind the usual μP embedding rule and derives a large-vocabulary regime in which the optimal hidden-to-embedding learning-rate ratio grows as the square root of width.	μP limitation, embedding LR, vocabulary, width scaling
2025-06-09	MiniCPM4: Ultra-Efficient LLMs on End Devices	Uses μP-backed small-model hyperparameter searches in the ModelTunnel / ScalingBench pipeline to validate transfer before training end-device LLMs.	application report, SLM, model wind tunnel
2025-05-28	On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling	Contrasts SP with NTP and μP, then proposes MUSOLI—a maximal-update rule under standard output-layer initialization—to explain large-learning-rate feature learning.	μP analysis, MUSOLI, SP, feature learning
2025-05-26	Variational Deep Learning via Implicit Regularization	Extends μP to variational mean and covariance parameters and validates proxy-selected learning-rate transfer from hidden width 128 to 2048.	variational inference, μP extension, width LR transfer
2025-05-21	Scaling Diffusion Transformers Efficiently via μP	Generalizes μP to diffusion Transformer families such as DiT, U-ViT, PixArt-α, and MMDiT.	diffusion, DiT
2025-05-20	The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models	Uses μP to scale its base encoder from width 512 to 1024 and 2048 while reusing the smaller model's hyperparameters, isolating how model size changes character-level capability emergence.	application report, μP, width transfer, language models
2025-05-19	Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training	Holds proxy-tuned μP learning rates fixed while deriving and validating weight-decay and batch-size laws across model and data scale.	μP application, weight decay, batch size
2025-05-19	μPC: Scaling Predictive Coding to 100+ Layer Networks	Derives μPC from Depth-μP and transfers separate weight and activity learning rates zero-shot across width and from 16 to 128 layers in predictive-coding networks.	predictive coding, Depth-μP, width-depth transfer, learning-rate transfer
2025-05-04	Practical Efficiency of Muon for Pretraining	Studies Muon plus maximal update parameterization and gives a telescoping algorithm for efficient μP hyperparameter transfer.	Muon, μP
2025-05-02	Don't be lazy: CompleteP enables compute-efficient deep transformers	Proposes CompleteP for depth-wise hyperparameter transfer while avoiding lazy learning in deep Transformers.	CompleteP, depth, Transformers
2025-05-01	Customizing the Inductive Biases of Softmax Attention using Structured Matrices	Derives μP initialization and AdamW learning-rate scalings for bilinear MLR/BTT structured-attention factors, then validates a shared optimal learning rate across OpenWebText model widths 512–1024.	μP extension, structured attention, MLR, BTT, width transfer
2025-05-01	On the Provable Separation of Scales in Maximal Update Parameterization	Provides theory for why μP can separate macro-variables from micro-variables, supporting small-scale hyperparameter tuning.	theory, scale separation
2025-04-10	On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognition	Tunes learning rate and initialization on a small fixed-depth proxy, then uses μP width transfer to study model, data, and compute scaling for gait recognition.	application report, computer vision, μP, width transfer, model-data scaling
2025-04-06	Prot42: a Novel Family of Protein Language Models for Target-aware Protein Binder Generation	Tunes an 81M μP proxy and transfers the selected training recipe to 500M and 1.1B protein language models.	application report, protein language model, μP, width transfer
2025-03-24	Scaling Laws for Emulation of Stellar Spectra	Tunes TransformerPayne μP hyperparameters on a small proxy, validates stable optimal learning rates across widths 64–512 and attention/token axes, and documents that transfer degrades for depth changes beyond roughly 0.5–2× the proxy depth.	application report, scientific ML, μP, width transfer, depth limitation
2025-03-20	Chem42: a Family of chemical Language Models for Target-aware Ligand Generation	Runs a 200-configuration sweep over initialization, embedding / output multipliers, and peak learning rate on a 38M μP proxy before scaling to 190M–1B chemical language models.	application report, chemical language model, μP, width transfer
2025-03-12	Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrization	Proves global convergence while preserving rich feature learning for L-layer infinite-width networks trained with SGD under μP.	theory, global convergence
2025-02-21	Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs	Uses μP proxy-to-target transfer to isolate how schedule shape, tokens per parameter, and weight decay affect LLM pretraining.	μP application, linear decay, LLM pretraining
2025-02-21	VaViM and VaVAM: Autonomous Driving through Video Generative Modeling	Sweeps 50 configurations on a width-256, 60M proxy and uses μP to scale the same recipe to 185M and billion-parameter video / action models.	application report, video generation, autonomous driving, μP, width transfer
2025-02-12	Distillation Scaling Laws	Validates a simplified μP parameterization across 198M–1.82B students and 546M–7.75B teachers, finding a shared near-optimal learning rate around 0.01 before fitting distillation scaling laws.	application report, distillation, μP, LR transfer
2025-02-11	Adaptive kernel predictors from feature-learning infinite limits of neural networks	Uses maximal-update parametrization as the rich infinite-width regime and derives deterministic data-adaptive Bayesian and tangent-kernel predictors for noisy gradient flow and weight-decayed gradient flow.	μP analysis, adaptive kernels, DMFT, feature learning, weight decay
2025-02-11	Training Deep Learning Models with Norm-Constrained LMOs	Derives Scion's layerwise norm-constrained updates from the μP spectral condition, proves a width-independent maximal-update learning rate, and validates zero-shot transfer from small proxies through nanoGPT models up to 3B parameters.	maximal update, spectral condition, Scion, Muon, width transfer, proxy-to-target
2025-02-09	μnit Scaling: Simple and Scalable FP8 LLM Training	Proposes μnit Scaling for FP8 LLM training with simple width-wise hyperparameter transfer and matched training/inference numerics.	FP8, u-μP
2025-02-04	Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer	Analyzes deep linear networks under μP, capturing maximal-update feature learning plus width/depth hyperparameter-transfer effects.	μP, deep linear networks, width-depth transfer
2025-01-23	A thorough reproduction and evaluation of μP	TMLR reproduction study evaluating claimed μP benefits, implementation sensitivity, and transfer behavior.	reproduction, TMLR
2024-12-27	Xmodel-2 Technical Report	Uses explicit μP search rules for width, depth, initialization, learning rate, and logits, transferring proxy-model choices to a 1.2B target.	application report, LLM, μP, cross-scale HPT
2024-12-23	YuLan-Mini: An Open Data-efficient Language Model	Tunes learning rate and batch size on a 0.05B proxy, applies explicit width- and depth-aware μP scaling to the 2.42B target, and documents a long-run large-LR stability limit addressed with WeSaR.	application report, μP, width-depth transfer, LLM, stability limitation, WeSaR
2024-11-11	Warmstarting for Scaling Language Models	Studies μTransfer-compatible warmstarting from smaller language models via shrink, zero-padding, and μP-scaled perturbations.	warmstart, LLMs
2024-11-07	Scaling Laws for Precision	Uses a 6e-4 peak LR on the smallest model and Depth-μP width/depth scaling for all larger models across 465 precision-scaling runs up to 1.7B parameters; it does not independently validate optimal-LR transfer.	application report, Depth-μP, width-depth scaling, low precision, transfer caveat
2024-11-06	Crystal: Illuminating LLM Abilities on Language and Code	Transfers a layer-wise μP recipe from smaller and shallower proxies to language-and-code pretraining targets; the earlier LLM360 report provides project history.	application report, code LLM, μTransfer
2024-11-04	Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation	Introduces maximal-update parameterizations for predictive coding and target propagation and validates μTransfer under local-loss training.	μP extension, local learning, predictive coding, target propagation
2024-10-31	μP²: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling	Extends maximal-update ideas to SAM by scaling layerwise perturbations so learning rate and perturbation radius transfer jointly.	SAM, perturbation scaling
2024-10-28	Modular Duality in Deep Learning	Unifies μP- and Shampoo-style operator-norm scaling and derives width-invariant update and learning-rate rules for scalable optimizers.	modular duality, operator norms, optimizer scaling
2024-10-08	Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit	Shows that optimal LR and critical batch size evolve with pretraining token budget even under μP, with the measured critical batch size growing proportionally to tokens.	μP, token horizon, batch size
2024-10-06	The Optimization Landscape of SGD Across the Feature Learning Strength	Parameterizes all studied networks with μP, verifies width-consistent dynamics, and maps how μP's free feature-learning-strength multiplier γ changes the optimal-learning-rate regimes across architectures and tasks.	μP, feature-learning strength, optimal LR, width consistency, ICLR 2025
2024-10-04	Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks	Uses a width-scaled μP recurrent-network parameterization, varies its feature-learning strength, and independently verifies that the resulting regime remains consistent across network widths on four tasks.	μP application, recurrent networks, width scaling, feature learning, coordinate validation
2024-10-04	Flavour Tagging with Graph Neural Network at ATLAS	Applies μP to GN2 flavour tagging, validating layer-scale stability and a shared optimal peak learning rate across embedding widths 64–256 to reduce hyperparameter-search cost.	application report, μP, GN2, particle physics, width LR transfer, coordinate checks
2024-09-30	Scaling Optimal LR Across Token Horizons	Directly tests μP at 25B–100B token horizons and shows that its optimal learning rate still decreases with training duration, so token-horizon transfer requires an additional fitted power-law correction.	μP critique, token horizon, learning-rate scaling, ICLR 2025
2024-09-25	On Feature Learning in Structured State Space Models	Shows that standard μP and spectral scaling conditions do not directly guarantee feature learning for structured state-space models such as Mamba.	SSM, Mamba, OpenReview
2024-08-29	Large-Scale Multi-omic Biosequence Transformers for Modeling Protein-Nucleic Acid Interactions	Tunes μP hyperparameters at roughly 10^6 parameters, then applies the fixed layerwise recipe across the 88M–2.3B OmniBioTE family trained on a 250B-token multi-omic corpus.	application report, biosequence, μP, proxy-to-target transfer
2024-08-23	Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler	Combines PowerLR with μP so one base recipe transfers jointly across token count, batch size, model size, and architecture in the reported experiments.	μP application, scheduler, tokens, batch, model scale
2024-07-29	Apple Intelligence Foundation Language Models	Uses simplified μParam throughout AFM pretraining, selecting a 0.01 production learning rate from a width-768 proxy sweep and reporting approximate optimal-LR stability across model sizes.	application report, simplified μP, proxy-to-target, LLM, transfer caveat
2024-07-24	u-μP: The Unit-Scaled Maximal Update Parametrization	Combines μP with Unit Scaling; aims for simpler defaults and low-precision / FP8-friendly training.	unit scaling, FP8
2024-07-08	Scaling Exponents Across Parameterizations and Optimizers	Large empirical/theoretical study of learning-rate scaling across optimizers and parameterizations; argues that transfer can occur beyond classical μP and highlights Adam epsilon scaling.	scaling exponents, optimizers
2024-06-13	Why Warmup the Learning Rate? Underlying Mechanisms and Improvements	Uses standard versus maximal-update parameterization as a central control in SGD and Adam warmup experiments, showing that μP's smaller, flatter initialization often reduces the benefit of warmup and documenting full versus simplified Adam μP recipes.	μP analysis, warmup, sharpness, SGD, Adam, NeurIPS 2024
2024-06-12	The Impact of Initialization on LoRA Finetuning Dynamics	Uses μP asymptotics to show how LoRA initialization direction changes the maximal stable learning-rate scaling with model width.	LoRA, initialization, LR scaling
2024-06-10	Compute Better Spent: Replacing Dense Layers with Structured Matrices	Derives and validates μP initialization and learning-rate rules for structured matrix layers so hyperparameters transfer across widths and structures.	μP extension, structured matrices, initialization, LR transfer
2024-05-31	μLO: Compute-Efficient Meta-Generalization of Learned Optimizers	Derives μP for learned optimizer architectures and improves generalization to wider, deeper, and longer-horizon tasks.	learned optimizers, μLO
2024-05-24	Infinite Limits of Multi-head Transformer Dynamics	Derives μP-compatible feature-learning limits jointly over Transformer width, depth, head count, and attention scaling.	maximal update, Transformers, width-depth-head limits, DMFT
2024-05-24	Sparse maximal update parameterization: A holistic approach to sparse training dynamics	SμPar extends maximal-update ideas to sparse neural networks and transfers hyperparameters across width and sparsity.	sparsity, SμPar
2024-05-22	How to set AdamW's weight decay as you scale model and dataset size	Shows that fixed weight decay breaks μP's AdamW learning-rate transfer later in training, and restores transfer by scaling weight decay with width to preserve the AdamW EMA timescale.	μP critique, AdamW, weight decay, width-data transfer
2024-05-16	LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting	Tunes optimizer and loss hyperparameters on a width-128 proxy with MUP-AdamW and zero-shot transfers them to the width-512 LaT-PFN forecasting model, reporting μP as essential for stability.	application report, μP, width transfer, MUP-AdamW, time-series foundation model
2024-04-30	The lazy (NTK) and rich (μP) regimes: a gentle tutorial	Tutorial explaining the richness scale between lazy NTK training and active μP feature learning.	tutorial, feature learning
2024-04-25	Tele-FLM Technical Report	Uses a 283M width-only μP proxy at the target depth of 64 layers to tune seven learning-rate, initialization, input, and output multipliers before training the 52.85B target.	application report, LLM, μP, width transfer, depth-matched proxy
2024-04-22	Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone	Reports transferring a μP recipe from a small proxy to the 7B phi-3-small model for improved training stability.	application report, LLM, μP, width transfer
2024-04-09	MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies	Applies Tensor Programs width and depth scaling, finds the optimal base learning rate remains near 0.01 across 0.04B–0.5B Model Wind Tunnel sweeps, and verifies the transferred setting on a 2.1B target.	application report, LLM, μP, Depth-μP, width-depth transfer
2024-04-08	An Empirical Study of μP Learning Rate Transfer	Empirically tests whether μTransfer gives near-optimal learning-rate transfer for Transformer architectures.	empirical, LR transfer
2024-02-27	Super Consistency of Neural Network Landscapes and Learning Rate Transfer	Shows that Hessian sharpness and loss landscapes remain consistent across width and depth under μP and its depth extension, explaining learning-rate transfer beyond the NTK regime.	μP, depth extension, sharpness, landscape, LR transfer
2024-02-27	Principled Architecture-aware Scaling of Hyperparameters	Generalizes the maximal-update condition to arbitrary computation graphs, derives topology-aware learning-rate scaling, and transfers a single hyperparameter search across architectures.	μP extension, architecture-aware HPT, maximal update
2024-02-19	LoRA+: Efficient Low Rank Adaptation of Large Models	Uses infinite-width / μP scaling to derive different learning rates for LoRA's two adapter matrices, preserving efficient feature learning as width grows.	LoRA, finetuning, LR scaling
2023-12-19	On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width	Derives optimizer-specific maximal-update parameterizations for K-FAC, Shampoo, and FOOF, with appendix width sweeps validating optimal-LR transfer.	maximal update, second-order, K-FAC, Shampoo, FOOF, width LR transfer
2023-12-10	Feature-Learning Networks Are Consistent Across Widths At Realistic Scales	Parameterizes networks in μP and verifies that their training dynamics, representations, and large-learning-rate effects converge consistently across realistic widths.	μP, finite width, width consistency, NeurIPS
2023-11-30	The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks	Derives a feature-speed formula containing the maximal-update μ-criterion, recovers Depth-μP for ResNets, and derives a new large-depth scaling for ReLU MLPs.	maximal update, Depth-μP, feature speed, ResNets, ReLU MLPs
2023-10-26	A Spectral Condition for Feature Learning	Shows how spectral-norm scaling yields feature learning and gives an elementary derivation of maximal update parametrization.	spectral condition, theory
2023-10-03	Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks	Studies depthwise parameterizations; proposes Depth-μP for single-layer residual blocks and discusses limitations for deeper blocks.	Depth-μP, ResNets, Transformers
2023-09-28	Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit	Combines μP with residual branch scaling to transfer hyperparameters across width and depth.	depth transfer, ResNets, ViTs
2023-09-25	Small-scale proxies for large-scale Transformer training instabilities	Tests μParam across Transformer scales, finding stable optimal learning rates but no loss or learning-rate-sensitivity gain and no replacement for QK normalization, while using activation, gradient, and logit scaling trends as small-scale instability diagnostics.	μP application, negative result, LR transfer, width-depth scaling, stability proxy, activation-gradient diagnostics
2023-09-20	BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model	Tunes learning rate, initialization, embedding, and output multipliers on a 40M μP proxy before transferring them to a 3B, 627B-token language model.	application report, LLM, μTransfer
2023-09-07	FLM-101B: An Open LLM and How to Train It with $100K Budget	Sweeps learning rate, initialization, and softmax temperature on a 40M proxy, transfers the μP recipe to 16B, then grows the model to 51B and 101B.	application report, LLM, μP, model growth
2023-08-30	Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models	Tunes batch size, learning rate, initialization, and embedding / logit multipliers on a 40M μTransfer proxy before scaling to 13B.	application report, Arabic LLM, μTransfer
2023-08-03	Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit	Derives maximal-update / μP infinite-width limits for adaptive entrywise optimizers, including Adam, and establishes their feature-learning dynamics.	Tensor Programs, μP, Adam, adaptive optimization
2023-07-10	Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters	Builds small μP/MU-parametrized surrogate models for HPO, preserves the ranking of joint learning-rate, batch-size, and momentum configurations across width scales, and validates proxy-to-target tuning on six workloads.	application system, μP, μTransfer, surrogate HPO, width transfer, OSDI 2023
2023-07-05	How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model	Uses maximal-update parameterization to preserve representation learning as width grows and contrasts the resulting compositional feature learning with the lazy regime.	μP application, compositional learning, feature learning, width scaling
2023-05-13	Depth Dependence of μP Learning Rates in ReLU MLPs	Studies how maximal-update learning rates depend on depth under mean-field initialization.	depth, learning rate
2023-04-14	nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales	Introduces μScaling, using μP to predict large-model pretraining loss from smaller counterparts.	μScaling, loss prediction
2023-04-06	Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster	Tunes μP hyperparameters on a 40M proxy, transfers the same learning rate through 2.7B parameters, and reports more stable norms and substantially lower cross-scale loss variance than standard parameterization.	application report, LLM, μP, μTransfer, width transfer
2023-04-06	Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks	Analyzes finite-width corrections around the mean-field / μP feature-learning limit, deriving O(width^-1/2) fluctuations of DMFT order parameters and width-dependent kernel / prediction variance beyond the lazy regime.	μP analysis, mean field, finite-width fluctuations, DMFT, feature learning
2022-11-29	Infinite-width limit of deep linear neural networks	Derives the infinite-width gradient-flow limit of deep linear networks under μP, including its initialization and layerwise learning-rate factors.	μP, deep linear networks, infinite width, gradient flow, feature learning
2022-10-31	Non-Gaussian Tensor Programs	Extends Tensor Programs universality—and the resulting NNGP, NTK, free-independence, dynamical-dichotomy, and μ-parameterization conclusions—from Gaussian to broad non-Gaussian initializations.	Tensor Programs, μP, non-Gaussian initialization, universality, NeurIPS 2022
2022-10-10	Meta-Principled Family of Hyperparameter Scaling Strategies	Derives a one-parameter family interpolating between NTK and mean-field / maximal-update hyperparameter scaling.	scaling strategies, theory
2022-05-19	Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks	Develops a self-consistent DMFT for a parametrization explicitly equivalent to μP, recovers the Tensor Programs μP feature process, and validates width-invariant loss and kernel dynamics in CNNs.	μP-equivalent, DMFT, feature learning, kernel dynamics, width invariance
2022-03-07	Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer	Introduces μTransfer; demonstrates transfer on Transformer and ResNet settings.	μP, μTransfer, LLMs
2022-01-28	Efficient Computation of Deep Nonlinear Infinite-Width Neural Networks That Learn Features	Identifies the computational barrier in the μP μ-limit, introduces an efficiently computable π-limit, and compares it with finite-width μ-parametrized networks.	μP, μ-limit, feature learning, infinite width, Tensor Programs
2021-10-29	Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limit	Studies mean-field integrable parameterizations and shows one training method is equivalent to a modification of μP.	mean field, integrable parameterizations
2020-11-30	Feature Learning in Infinite-Width Neural Networks	Introduces maximal-update parametrization and classifies the stable infinite-width regimes that retain nontrivial feature learning instead of collapsing to kernel dynamics.	Tensor Programs IV, μP foundation, feature learning

Learning Resources and Blogs

Resource	Type	Notes
Training and optimisation of large transformer models at CERN: an ATLAS case study on Kubeflow	ATLAS implementation talk + slides (2024)	Demonstrates GN2 μP conversion, layer-scale checks, and peak-LR transfer from width 64 to 256, with practical proxy-search cost measurements and a recorded talk.
Online KL Shampoo	Optimizer research report (2026)	Derives an optimizer-specific μP shape rule from spectral conditions, checks update spectra, and validates learning-rate transfer from 78.7M to 295.8M non-embedding parameters.
Theoretical Foundations of Deep Learning: Optimization, Generalization, and Scaling	Dissertation / μFT-Transfer (2024)	Chapter 6 derives μFT-Transfer via subsampling and rescaling pretrained μP networks, with CIFAR-10 experiments testing proxy-to-target finetuning learning-rate transfer.
Quickstart Guide: Hyperparameter selection (and why theorists should care)	Learning Mechanics tutorial (2025)	Explains μP width transfer, feature-learning strength, and depth-scaling prescriptions, connecting the included theory papers to concrete research questions.
Go small then go home - hyperparameter transfer for ML in HEP	FastML experiments + slides (2025)	Tests μP learning-rate transfer in particle-tracking MLPs, CICADA autoencoders, and particle Transformers, reporting small-proxy failures and imperfect batch-size transfer.
An Overview of Maximal Update Parametrization (μP)	Author tutorial slides (2026, Chinese)	Chenyu Zheng's April 9 Tsinghua tutorial explains spectral μP, Adam/Muon scaling, Diffusion Transformer transfer, and the transition from Depth-μP to CompleteP and joint width-depth scaling.
Spectral Condition for μP under Width-Depth Scaling — workshop slides	Author talk slides (2026)	Official ICLR Delta Workshop slides explain spectral width-depth conditions, multi-layer residual blocks, and their connection to Depth-μP, CompleteP, and Muon-Kimi transfer.
muP 漫游：What is μP and why do we need it?	Author tutorial (2026, Chinese)	Chenyu Zheng's introductory article derives feature-update instability under standard parameterization in a two-layer model and motivates layerwise μP scaling for proxy-to-target transfer.
MarinDNA: A 1B standard Transformer rivals Evo 2 40B on variant effect prediction	Author technical report (2026)	Validates an adapted Complete(d) / AdamH recipe from ~25M proxies to 255M–1B genomic Transformers across model size, batch, and token horizon, with target learning-rate sweeps and explicit departures from the original prescription.
Scaling Laws That Extrapolate 300× Past the Fit	Author technical report (2026)	Documents Delphi's failed initial μP-informed recipe and an empirically revalidated Complete(d)P / AdamH adaptation, including a hyperparameter calculator, 24 width–batch–duration checks, and scaling to 25B parameters.
Stanford CS336 Lecture 11: Scaling details	Course slides (2025)	Tatsunori Hashimoto's lecture connects Cerebras-GPT and MiniCPM proxy recipes to spectral μP derivations and empirical learning-rate-transfer limitations involving normalization, optimizers, and weight decay.
Dive into Deep Learning: Scaling Up	Author-hosted textbook chapter (accessed 2026)	Executable PyTorch and JAX examples compare standard and maximal-update parameterization through width learning-rate sweeps, layerwise Adam rules, coordinate checks, and spectral scaling.
On the Hypersphere: μP Scaling of Optimizers with the Hyperball Mechanism	Researcher technical essay (2026)	Jiaxuan Zou derives feature-space scaling rules for SGDH, AdamH, and MuonH under stated spherical-dynamics assumptions; an analytical explanation rather than an independent transfer benchmark.
μP for Mamba and Mamba-2: implementation and transfer experiments	Author implementation note (2024)	Documents empirical parameter-group choices, coordinate checks, and WikiText learning-rate sweeps across widths 64–2048, including short-run loss and stability caveats; these experiments do not establish a general SSM scaling theorem.
Hyperparameter Transfer: A Recipe for Efficient and Robust Scaling	Conference slides (2026)	Gage DeZoort's August 19 HAMLET-PHYSICS presentation on proxy-to-target learning-rate transfer for GNNs and Coupled Particle-Edge Networks under SGD, Adam, and AdamW, with jet-identification and stellar-stream applications.
Optimization (from SGD/Adam to Shampoo/Muon)	Lecture slides (2026)	UC Berkeley EE 290/194 optimizer lecture deriving maximal-update and spectral initialization / learning-rate rules and connecting them across SGD, Adam, Shampoo, and Muon.
Statistical Properties of Training & Generalization	Review / teaching material (2026)	Physics-oriented review whose §2.1 and Table 1 explain μP / CompleteP proxy-to-target initialization, learning-rate, weight-decay, readout, and depth scalings, with a discussion of transfer limitations; not a new μP method.
Training Diffusion Transformers with Muon	Experimental technical report (2026)	Sven Lüpke transfers a width-256 / 31M DiT proxy's learning rate to width 1024 / 465M, using μP for Adam and RMS-to-RMS update scaling for Muon; documents architecture, optimizer groups, and target-model results without independently sweeping the target LR optimum.
Revisiting Neural Network Parameterizations for Optimal Performance	Independent reproduction / technical report (2026)	Reproduces μP versus modified-SP width transfer and coordinate checks, then tests CompleteP-style depth scaling; reports the training and initialization setup and explicitly limits its own sweeps to global LR on dense Adam-trained models.
The Practitioner's Guide to the Maximal Update Parameterization	Guide / blog	Practical implementation guide linked by `EleutherAI/nanoGPT-mup`; useful for coordinate checks and small GPT experiments.
Train an LLM using Maximal Update Parameterization	Documentation	End-to-end Cerebras guide to proxy-to-target μTransfer, base dimensions, initialization scaling, layer-wise learning-rate groups, supported models, and validation practice.
Hydro documentation	Documentation	Official guide to Hydro's μP/MU-parametrized surrogate construction, model shrinking, tuning workflow, and supported workloads.
On infinitely wide neural networks that exhibit feature learning	Research blog	Microsoft Research introduction to feature learning at infinite width and the original maximal-update construction.
μTransfer: A technique for hyperparameter tuning of enormous neural networks	Blog	Microsoft Research explainer for μTransfer and the Tensor Programs V workflow.
Introducing AuraFlow v0.1	Project / technical blog	Official fal.ai report on zero-shot μP learning-rate transfer across MMDiT widths and a proxy search for a transferable aspect ratio.
VaViM and VaVAM project page	Project page	Official Valeo AI page for the μP-scaled video and action models, with qualitative results, driving demonstrations, failure cases, paper metadata, and code links.
Jais 2: A Blueprint for Sovereign AI	Technical blog	Official Cerebras account of transferring learning-rate and optimization settings from small μP proxies to the Jais 2 8B / 70B models.
Greg Yang's Tensor Programs and μP reading guide	Author guide	First-author map of Tensor Programs IV–VI, μP / μTransfer papers and code, and a recommended entry point into the theory.
Infinite Limits of Neural Networks	Institutional guide	Kempner Institute comparison of NTK and μP feature-learning limits, realistic-width convergence, and hyperparameter transfer across width and depth.
Infinite Widths (& Depths) Part III: The Maximal Update Parameterisation	Researcher guide	Compact guide connecting width μP, Depth-μP, feature learning, and the main extensions.
In awe at the scale of these tensors: a gentle introduction to unit-scaled maximal-update parametrization	Research blog	Official Aleph Alpha companion to u-μP, explaining unit-scaled tensor rules, width transfer, and the large-scale training motivation.
Scaling Predictive Coding to 100+ Layer Networks	Author guide	First-author explanation of μPC, Depth-μP, and zero-shot learning-rate transfer from 16- to 128-layer predictive-coding networks.
Can We Scale Predictive Coding?	Author guide	First-author explanation of why predictive coding and backpropagation share width- and depth-stable feature-learning parameterizations.
How To Scale	Independent technical guide	Broad practitioner synthesis of μP derivation, abc-parameterization, width/depth/data/horizon limits, learning-rate and batch scaling, and Muon's relation to μP; the author explicitly flags possible inaccuracies.
Apple Intelligence Foundation Language Models	Research page	Official Apple research page for the AFM report and its simplified μParam proxy-to-production training recipe.
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration	Research page	Apple research page for Complete(d)P and transfer across width, depth, batch size, and duration.
Rethinking Maximal Update Parametrization: Steepest Descent on the Spectral Ball	Technical essay	Geometric reinterpretation of maximal updates with spectral-ball constraints and learning-rate-transfer experiments.
Rethinking Maximal Update Parametrization: Steepest Descent on Finsler-Structured Geometries	Technical essay	Detailed derivation of maximal-update-aware steepest descent through Finsler geometry and dual ascent.
Lecture Notes on Infinite-Width Limits of Neural Networks	Lecture notes	Pedagogical derivation of infinite-width limits and width-only μP for MLPs.
TASI Lectures on Physics for Machine Learning	Lecture notes	Broad physics-for-ML notes whose §4.3 derives the abc constraints leading to maximal-update parametrization; retained as teaching material rather than an original μP paper.
How Width Scaling Affects Neural Networks: Generalization, Optimal Hyperparameters, Feature Learning and Beyond	Dissertation	Moritz Haas's systematic treatment of width scaling, hyperparameter transfer, feature learning, and μP / μP².
Scaling and Renormalization in Statistical Learning	Dissertation	Alexander Atanasov's treatment of scaling laws, infinite-width feature learning, μP, and the approach of realistic networks to their limiting dynamics.
Reduce Model Tuning Costs with MuP	Engineering blog (2023)	Speechmatics account of μP encoder-Transformer learning-rate-transfer experiments and a PyTorch implementation walkthrough; its depth observations are empirical rather than a general depth-transfer guarantee.
Deriving Muon	Author technical essay	Jeremy Bernstein derives Muon from RMS-to-RMS operator norms and explains why its normalized updates recover μP-style learning-rate transfer across width.
Complete-muE — Tune Dense Once, Transfer to All MoE Configurations	Author guide	First-author worked recipe for composing active-width μP with batch/duration scaling to transfer one dense-model sweep across MoE width, depth, capacity, granularity, and token workload.
francesco-innocenti/mup-papers	Curated list	Active community list of mean-field / maximal-update parameterisation papers, grouped by theory and extensions.
unit-scaling documentation	Documentation	Docs and examples for the PyTorch unit-scaling library used by u-μP.
Inferno maximal-update parametrization documentation	Documentation	Official API reference for variational μP's `MaximalUpdate` parametrization, including layerwise initialization and optimizer learning-rate scales.
Non-Gaussian Tensor Programs supplemental	Paper supplement	Official proof appendix and NNGP / NTK simulation notebooks supporting the paper's non-Gaussian Tensor Programs universality result; this is not a μP training implementation.
Non-Gaussian Tensor Programs talk	Talk / video	Official NeurIPS 2022 presentation of Tensor Programs universality beyond Gaussian initialization and its consequences for μ-parameterization.
Tensor Programs V / μTransfer talk	Talk / video	Author talk linked from the official `microsoft/mup` repository, covering the transfer recipe, large-scale results, and theoretical motivation.
Renormalizing the optimal hyperparameters of a neural network	Talk / slides	Greg Yang's concise μTransfer talk with a recording, 25-page slide deck, and effective-field-theory analogy.
Timothy Nguyen conversation on μP and Tensor Programs	Video	Long-form discussion touching Tensor Programs, μP, and scaling limits.
AutoML Seminar: scaling exponents across parameterisations	Talk	Seminar companion for scaling-exponent work across parameterizations and optimizers.

Implementations and Artifacts

Resource	Related method	Notes
tilde-research/online-kl-shampoo-release	Online KL Shampoo / μP	Official PyTorch optimizer with μP shape scaling, spectral initialization, and Scaled CANS inverse-root updates; the production distributed offload system is not included.
livaage/mup_transfer_gnn_mlp	FastML particle-tracking μP experiments	Talk-linked Python experiment configurations, launchers, and timing scripts for the tracking-MLP study; requires the separate GNN Tracking environment and model setup described in its README.
livaage/cicada-teacher-hyperparameter-transfer	FastML CICADA μP experiments	Talk-linked TensorFlow autoencoder implementation with μP model variants, learning-rate sweeps, and width/depth/timing comparisons; includes experimental research configurations.
neilrathi/token-filtering	Data-filtering μP application	Official PyTorch code for token-level filtering experiments, with depth-matched proxy configurations and μP-scaled AdamW parameter groups in a nanoGPT-derived training stack.
Open-Athena/marin-dna	Complete(d)-inspired genomic scaling	Official research hub with experiment records and models; the report links JAX / Levanter training configurations in Marin.
Marin Complete(d) / AdamH scaling recipe	Adapted Complete(d) / AdamH	Author-linked JAX / Levanter recipe used by Delphi and MarinDNA; implements projection and Adam learning rates plus batch/token scaling of epsilon and beta2, with a modified 0.3 token-horizon exponent.
alxndrTL/mamba.py	Community Mamba / Mamba-2 μP	PyTorch implementation with initialization, readout, and optimizer-group scaling, plus coordinate-check and learning-rate-sweep scripts supporting its empirical width-transfer report.
EPFL LLM optimizer benchmark: μP models	PyTorch GPT / Llama μP	Research training stack with explicit μP GPT/Llama modules, base-width and depth factors, and training-dynamics logging; μP support was added after the original optimizer-benchmark paper.
sven-luepke/pixel-dit-muon	DiT μP / Muon transfer	Author-linked PyTorch training code for the 31M-to-465M pixel-space DiT experiment, supporting Adam μP transfer and Muon's RMS-to-RMS width scaling with separate non-hidden parameter groups.
er537/MuP	Speechmatics μP implementation	Author-linked lightweight PyTorch implementation of initialization and optimizer parameter-group scaling, with DDP / FSDP support, accompanying the Speechmatics encoder-Transformer transfer report.
microsoft/mup	μP / μTransfer	Reference PyTorch package with base-shape tooling, model and optimizer scaling, examples, and coordinate-check utilities.
dkarkada/mupify	μP tutorial implementation	Companion implementation for The lazy (NTK) and rich (μP) regimes, exposing width-aware layers, optimizer groups, and coordinate checks.
S-Lab-System-Group/Hydro	Hydro / μP surrogate HPO	Official OSDI 2023 implementation of μP-parametrized small surrogates for multi-hyperparameter proxy-to-target tuning.
edwardjhu/TP4	Tensor Programs IV / feature learning	Paper-declared official code reproducing the infinite-width feature-learning experiments on Word2Vec and Omniglot via MAML.
gvlassis/ant	Independent μP reproduction	Official code for the TMLR reproduction study, covering ResNet and GPT-family μP experiments plus Muon and matrix-preconditioned optimizers.
google-deepmind/nanodo	Small-scale Transformer stability proxies	Official JAX infrastructure whose first published fork supported the included μParam transfer, learning-rate-sensitivity, and coordinate-diagnostic experiments.
Yuan-Jinghui/Nora	Nora	Official Nora implementation; JiaxuanZou0714/Lrp contains the accompanying reproduction experiments.
moritzhaas/large-lr-width-scaling	Large-LR width scaling	Official SP / μP width sweeps and refined coordinate checks that decompose effective and propagating updates layer by layer.
tml-tuebingen/torch-module-monitor	Refined coordinate checks	Official Tübingen package for activation, gradient, and parameter statistics and for decomposing activation changes into effective and propagating updates, used by the large-LR width-scaling and MSSP-MoE studies.
microsoft/mutransformers	μP Transformers	Historical Microsoft demonstration for Hugging Face Transformer models; archived in June 2026 and based on Transformers 4.16.2.
Cerebras/modelzoo	Cerebras μP / μTransfer	Active training stack with μP / µParam scaling and configurations for GPT-style and other supported model families.
cerebras/Cerebras-GPT-Intermediate	Cerebras-GPT μP application	Official intermediate-checkpoint archive containing the 111M–2.7B μP models and matched standard-parameterization baselines used in the transfer study.
NVIDIA Megatron Core μP configuration	Production LLM μP	Native `use_mup` support with base-width, embedding, output, and attention scaling in Megatron-LM.
zanussbaum/mup-tf	TensorFlow μP	Community TensorFlow implementation of maximal update parameterization.
EleutherAI/nanoGPT-mup	μP for GPTs	Compact GPT-style implementation with marked μP changes, examples, and coordinate-check scripts.
EleutherAI/nanoGPT-mup/tree/supar	SμPar	Minimal implementation for sparse maximal update parameterization.
EleutherAI/nanoGPT-mup/tree/completep	CompleteP	Minimal implementation for CompleteP depth-wise transfer experiments.
graphcore-research/unit-scaling	u-μP	PyTorch library for Unit-Scaled Maximal Update Parameterization.
Aleph-Alpha-Research/scaling	u-μP large-scale training	Official large-scale u-μP training code accompanying the paper, with released Aleph Alpha checkpoints.
inferno-ml/inferno	Variational / probabilistic μP	Paper-declared official PyTorch library for Variational Deep Learning via Implicit Regularization; implements `MaximalUpdate` for Bayesian neural-network modules, with documentation and the `inferno-torch` package.
lucaslingle/mu_transformer	Large-scale μTransfer	Official JAX / Flax Transformer implementation with TPU/FSDP training, coordinate checks, and proxy-to-target configurations.
NanChanNN/muA-lora	μA / LoRA transfer	Official code for learning-rate scaling across LoRA ranks and transfer to full finetuning over language, vision, VLM, RLVR, and diffusion tasks.
lauyikfung/gated_delta_net_mup	Gated Delta Network μP	Official implementation with AdamW / SGD width-transfer configurations and FineWeb-Edu pretraining scripts; an author-maintained AWS Trainium port is also available.
vankadara-lab/mssp-moe	MSSP / MoE	Official code with coordinate checks and sweeps across width, depth, expert count, expert width, and routing regimes.
ruihanxx/MOGA	Operator-norm transfer	Official PyTorch implementation with GPT-2 / OpenWebText and LLaMA / C4 width-scaling and learning-rate-transfer experiments.
charliezchen/scaling-matrix-preconditioning	Matrix preconditioners	Official JAX / Optax code for μP-aware Shampoo, SOAP, and Muon transfer across width and depth.
SDLAML/disco	Scion / maximal-update norm transfer	Official distributed Scion / Muon implementation for Optimal Scaling Needs Optimal Norm, with logs from more than 2,000 training runs.
LIONS-EPFL/scion	Scion / maximal-update transfer	Official code for Training Deep Learning Models with Norm-Constrained LMOs, including the paper's μP-derived scale-agnostic optimizer and proxy-to-target experiments.
alokshah04/double-preconditioning	DoPr / μP-compatible preconditioning	Paper-declared official code for Double Preconditioning, with MuJoCo, Robomimic, and SFT experiments covering AdamW, Muon, AdaMuon, and activation-preconditioned variants.
shikaiqiu/compute-better-spent	Structured matrices	Official code for μP-derived initialization and learning-rate scaling of structured matrix layers across MLP, ViT, and GPT experiments.
YilunKuang/structured-attention	Structured-attention μP	Official ICML 2025 implementation of bilinear MLR/BTT attention, including its μP parameterization and width-transfer experiments.
VITA-Group/principled_scaling_lr_init	Architecture-aware HPT	Official code for architecture-aware initialization and maximal-learning-rate scaling across MLPs, CNNs, and varied computation graphs.
shikaiqiu/supercollapse	Scaling collapse / μP	Official JAX experiments, logs, and notebooks for the included study's μP parameterization ablation and scaling-collapse analysis.
bentherien/mu_learned_optimization	μLO	Official research code for μP learned-optimizer meta-training and cross-width / depth evaluation on Transformer and MLP tasks.
tiiuae/Falcon-H1	Falcon-H1 application	Official project hub documenting the hybrid architecture, customized μP scaling recipe, evaluation, deployment, and finetuning resources.
tiiuae/Falcon-H1 checkpoints	Falcon-H1 application	Official 0.5B–34B base, instruction-tuned, and quantized checkpoint collection.
cerebras/btlm-3b-8k-base	BTLM μP application	Released μP-trained 3B checkpoint with a PyTorch `get_mup_param_groups` continuation example and explicit μP configuration fields.
naver-hyperclovax/HyperCLOVAX-SEED-Think-14B	HyperCLOVA X THINK μP application	Official 14B model card and weights; the card identifies maximal update parameterization as part of the dense production architecture.
RUC-GSAI/YuLan-Mini	YuLan-Mini μP application	Official pretraining code, checkpoints, optimizer states, and ablation logs, including the Cerebras μP and stability studies.
StijnVerdenius/Lat-PFN	LaT-PFN μP application	Official training and tuning code for the width-128-to-512 MUP-AdamW transfer application.
Pehlevan-Group/dmft_fluctuations	Finite-width μP / DMFT	Official notebooks reproducing finite-width kernel and prediction fluctuation experiments around the μP feature-learning limit.
InceptionAI Jais models	Jais μTransfer application	Official Jais 13B foundation weights and accompanying Jais-chat checkpoint.
Inception Jais 2 model family	Jais 2 μP application	Official 8B / 70B checkpoint collection accompanying the depth-matched proxy-transfer report.
XiaoduoAILab/Xmodel-2	Xmodel-2 μP application	Official training code with `mup_search`, scaling-law experiments, `train_mup.py`, and the released 1.2B checkpoint.
XiaoduoAILab/Xmodel-2.5	Xmodel-2.5 μP application	Official Megatron-LM training and evaluation code with a dedicated `mup` module, released final checkpoints, and training-history checkpoints.
CofeAI/Tele-FLM	Tele-FLM μP application	Official 52B weights and custom inference code accompanying the proxy-to-target μP report.
Microsoft Phi-3 collection	Phi-3 μP application	Official model collection containing phi-3-small 7B, the target explicitly described by the technical report's μP proxy-transfer experiment; the collection's other models are not thereby claimed as μP-trained.
fal/AuraFlow-v0.3	AuraFlow μP application	Official 7B checkpoint and Diffusers integration; a released model artifact rather than μP pretraining reproduction code.
dvruette/gidd-easydel	Discrete-diffusion LM μP scaling	Author-maintained reproduction code for masked, uniform, and hybrid-noise scaling sweeps, released 3B / 10B checkpoints, and the exact pretokenized training split.
FM4NPP/PP_collision	FM4NPP μP-SSM application	Official checkpoint hub for the proton–proton collision model scales; no paper-specific training code was released.
shikaiqiu/epiplexity	Epiplexity μP application	Official PyTorch and JAX / Flax code for the model-size sweeps, scaling-law analysis, and figure notebooks.
DIVER-Project/DIVER-1	DIVER-1 μP scaling application	Official deanonymized training code and checkpoints for the μTransfer-based iEEG model family.
valeoai/VideoActionModel	VaViM / VaVAM μP application	Official code and 185M–1.2B weights, including `mup_shapes`, scaling-law notebooks, pretraining, finetuning, and action-learning pipelines.
CofeAI/FLM-101B	FLM-101B μP / model-growth application	Official 101B checkpoint and custom inference code; the paper-specific pretraining code is not public.
LLM360/crystalcoder-train	CrystalCoder μP application	Official training code with released intermediate checkpoints and pretraining data.
nyuolab/OmniBioTE	Biosequence μP application	Official code for multi-omic μP scaling with released OmniBioTE weights.
OpenBMB/MiniCPM	MiniCPM / MiniCPM4 applications	Official project and checkpoints for the original Model Wind Tunnel μP/Depth-μP transfer study and MiniCPM4's μP-backed ModelTunnel pipeline.
nikhilgsh/loraplus	LoRA+ / LoRA initialization	Author implementation shared by the LoRA+ and LoRA-initialization scaling papers.
modula-systems/modula	Modular norm / Modular Duality / μP	Official JAX package accompanying Scalable Optimization in the Modular Norm and later modular-duality work; normalizes architecture-composed updates for width/depth LR transfer and relates that geometry to μP-style scaling.
lchizat/2025-hidden-width-deep-resnet	Maximal local updates across width and depth	Official code reproducing the included Neural Mean ODE and joint depth–hidden-width maximal-update phase-diagram experiments.
lchizat/2022-wide-linear-NN	Deep linear networks under μP	Author-linked code for the included infinite-width μP gradient-flow and finite-width convergence experiments.
karl-hajjar/wide-networks	Integrable parameterizations / μP	Paper-declared reproduction code for the included study of integrable infinite-width training and its modified-μP equivalence.
lchizat/2023-BAFU	Feature Speed Formula	Author-linked Julia code reproducing the feature-speed, backward-feature-angle, and depth-scaling experiments.
pcsl-epfl/hierarchy-learning	Random Hierarchy Model / μP	Author-linked code, models, and datasets for the included compositional-learning experiments under maximal-update parameterization.
cosmaadrian/strawberry-problem	Character-level emergence / μP	Author-released code for the EMNLP study, including the width-512/1024/2048 μP experiment that reuses the base model's hyperparameters.
Pehlevan-Group/Richness_Sweep	μP feature-learning strength	Official code reproducing the μP width-consistency checks and the optimal-learning-rate phase diagrams across the feature-learning multiplier γ.
dayal-kalra/why-warmup	μP warmup analysis	Official code and data for the standard- versus maximal-update-parameterization warmup, sharpness, and optimizer experiments.
DataDog/toto	u-μP application	Official code for Toto 2.0, a time-series foundation-model family trained with a u-μP transfer pipeline.
Datadog/toto-20 checkpoints	u-μP application	Released Toto 2.0 model checkpoints accompanying the u-μP scaling report.
LithiumDA/muTransfer-FNO	μTransfer-FNO	Official implementation for zero-shot hyperparameter transfer in Fourier Neural Operators.
muTransfer-FNO data	μTransfer-FNO	Dataset release used by the μTransfer-FNO experiments.
cofe-ai/Mu-scaling	μScaling / nanoLM	Early μScaling code release for accurate loss prediction across scales using maximal update parametrization.
cofe-ai/nanoLM	nanoLM / μScaling	Expanded official benchmark supporting GPT, LLaMA, BERT, and T5 μP grid search, cross-width transfer, and loss prediction.
CofeAI/NanoData	nanoLM data	Official 97.12B-token curated pretraining dataset for cross-architecture μScaling studies.
ML-GSAI/Scaling-Diffusion-Transformers-muP	Diffusion Transformer μP	Official NeurIPS 2025 implementation with DiT / PixArt coordinate checks, proxy-model sweeps, and transferred large-model learning rates.
GSAI-ML/DiT-muP checkpoints	Diffusion Transformer μP	Released DiT-μP and PixArt-μP pretrained checkpoints accompanying the scaling study.
ML-GSAI/Width-Depth-muP	Width-depth μP	Official implementation for spectral conditions under joint width-depth scaling.
microsoft/ArchScale	μP++ / HyperP / Depth-μP	Official code for Decoder-Hybrid-Decoder μP++ and later HyperP / MuonH / SqrtGate extensions, with depth-transfer setup and architecture-scaling experiments.
Unakar/Spectral-Sphere-Optimizer	Spectral Sphere Optimizer	Official implementation for spectral-sphere μP-aligned optimization.
Unakar/Megatron-LM/tree/SSO_main	Spectral Sphere Optimizer	Megatron-LM implementation branch for large-scale SSO experiments.
JesseFarebro/flax-mup	Flax / Optax μP	Community Flax and Optax implementation of maximal update parametrization.
Melina-Jingting/mup-equinox	JAX / Equinox μP	Lightweight community library bringing μP-style modules and base-shape utilities to Equinox.
yuxinma98/mupscaling	μP warm starts / upscaling	Official implementation for function-preserving μP model and optimizer-state upscaling, with MLP, ResNet, GPT-2, equivalence tests, and transfer experiments.
Neeratyoy/warmstarting_exps	Warmstarting / SZP / μP	Official pipelines for both warmstarting studies, including shrink-zero-perturb, controlled μP width-transfer baselines, growth-factor sweeps, and scaling-law analysis.
thebuckleylab/jpc limits-paper experiments	Predictive-coding width / depth limits	Official JAX reproduction code for stable feature-learning parameterizations and width / depth transfer in predictive-coding networks.
thebuckleylab/jpc μPC experiments	μPC / Depth-μP	Official full experiment suite for the μPC paper; the repository also retains the compact tutorial notebook.

Repository Layout

Path	Purpose
docs/mup-transfer.md	Main μP / μTransfer reading guide, paper timeline, resources, code links, and practical checklist.
docs/hyperball.md	Companion copy of the complete Hyperball collection shown in this README.
docs/hyperparameter-transfer.md	Complementary scale-aware HPT papers, taxonomy, cross-collection map, artifacts, and transfer protocol.
papers/hyperball.bib	BibTeX for Hyperball and its substantive related extensions.
papers/hyperparameter-transfer.bib	BibTeX for complementary direct HPT papers outside the strict μP/Hyperball counts.
docs/hyperball-search-audit-2026-09-14.md	Hyperball search scope, source evidence, exclusions, and version checks.
docs/hyperball-search-audit-2026-09-20.md	Correction audit adding the recovered MuonM/MuonH comparison and extending the incremental search.
docs/hyperball-search-audit-2026-09-23.md	Archive copy of the full Hyperball source audit included above.
papers/mup-transfer.bib	BibTeX references for the μP / μTransfer collection.
docs/search-audit-2026-09-14.md	μP search scope, primary-source evidence, exclusions, and unresolved candidates.
docs/hyperparameter-transfer-search-audit-2026-09-19.md	Combined scale-aware HPT audit, incremental μP/Hyperball check, exclusions, and artifact verification.
papers/README.md	Notes on maintaining reference files.
CONTRIBUTING.md	Contribution scope and entry template.

Contributing

Useful additions include direct μP papers, implementation notes, coordinate-check scripts, substantive applications, and well-evidenced failed-transfer studies. The dedicated Hyperball collection accepts direct Hyperball research, comparisons, applications, and documented extensions. The scale-aware HPT collection accepts direct proxy-to-target rules and clearly scoped theory or experiments over model/training scale. Please include source links, arXiv IDs when available, and a one-sentence reason why the resource matters.

See CONTRIBUTING.md for the suggested format.

Contributors

JiwenJ

30 commits

Similar repos

Hiroki11x/awesome-muon-optimizer

KellerJordan/Muon

Muon is an optimizer for hidden layers in neural networks

2,855

schrodingercatss/tuning_playbook_zh_cn

一本系统地教你将深度学习模型的性能最大化的战术手册。

3,222

AGI-Arena/MARS

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large…

722

MoonshotAI/Moonlight

Muon is Scalable for LLM Training

1,547

microsoft/dion

Dion optimizer algorithm

548

ishandutta2007/Awesome-Compute-Optimal-Training

Variants of Compute-Optimal-Training 🌟 Star if you like it! 🌟

thu-pacman/Puro-Megatron

Megatron-LM adapted for Puro-2B

Languages

TeX

100.0%

JiwenJ/Awesome-Initialization

TeX

30 commits

updated Sep 23, 2026

See the code

README

Awesome μP

Curated, evidence-backed resources on maximal-update parametrization (μP), μTransfer, Hyperball optimization, and scale-aware hyperparameter transfer.

μP / HPT snapshot: 2026-09-20. The μP collection contains 161 directly relevant papers, plus 54 learning resources and 96 implementation / artifact links. The complementary HPT collection adds 17 direct papers outside the μP and Hyperball bibliographies.

Start here
Why μP matters
Recent μP / μTransfer directions
Hyperball
Scale-aware hyperparameter transfer
Full paper list
Learning resources and blogs
Implementations and artifacts
Repository layout
Contributing

Start Here

Order	Theme	Resource
1	Theory foundation	Feature Learning in Infinite-Width Neural Networks
2	Core μTransfer recipe	Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
3	Adaptive optimizers	Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
4	Depth scaling	Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
5	Empirical LR transfer	An Empirical Study of μP Learning Rate Transfer
6	Practical caveat	Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

For the detailed guide and practical checklist, see μTransfer / μP Hyperparameter Transfer.

Why μP Matters

Convert the target architecture to μP.
Tune key hyperparameters on a small proxy model.
Transfer those hyperparameters to a much larger model without re-running the expensive sweep.

Recent μP / μTransfer Directions

Width transfer for LLMs — original μTransfer framing and practical Transformer results.
Depth transfer — Depth-μP, effective-depth laws, and limitations in modern multi-layer residual blocks.
Embedding and readout scaling — recent evidence that embedding-layer learning rate can dominate apparent μP gains in AdamW language-model training.
Architecture-specific μP — GNNs, GQA-μP, diffusion Transformers, probabilistic Transformers, Fourier neural operators, MoE, LoRA, and sparse models.
Training-system variants — u-μP for unit-scaled / low-precision training, SμPar for sparse training, and maximal-update rules for matrix-preconditioned optimizers.
Boundaries and negative results — experiments that directly test when μP does or does not transfer across optimizer, architecture, batch, duration, or parameter groups.

Topic	What to look for
Width transfer	Original μTransfer framing and practical Transformer results.
Depth transfer	Depth-μP, effective-depth laws, and limits in modern residual blocks.
Embedding / readout scaling	Cases where embedding-layer learning rate or output scaling controls transfer quality.
Architecture-specific μP	GNNs, GQA, diffusion Transformers, probabilistic Transformers, Fourier neural operators, sparse models, and LoRA.
Optimizer-specific transfer	Adaptive optimizers, Muon / hypersphere optimization, and optimizer-dependent scaling rules.
Batch and duration	Work that tests these axes while μP remains a substantive part of the method or experimental design.