A zij (Arabic: زِيج, pronounced "zeej") is an astronomical handbook from the Islamic golden age: a set of tables and computational methods that astronomers consulted instead of re-deriving the field from scratch. The best known is the Zīj al-Sindhind of Muḥammad ibn Mūsā al-Khwārizmī (محمد بن موسى الخوارزمي, c. 820 CE). His Latinized name, Algoritmi, became the word algorithm, and his book al-Jabr gave us algebra.

This project takes the name in that spirit. It gathers the equation, the paper, and runnable code for the optimization methods used in machine learning.

Contents

• Installation
• Quick start
• Library
• Canon
  • First-Order Optimizers (226)
  • Memory-Efficient Optimizers (90)
  • Fractional-Order Optimizers (93)
  • Distributed and Communication-Efficient Optimizers (75)
  • Second-Order and Orthogonalized Optimizers (67)
  • Zeroth-Order Optimizers (38)
  • Privacy-Preserving Optimizers (40)
  • Sharpness-Aware Optimizers (28)
  • Quantum and Quantum-Inspired Optimizers (31)
  • Learning-Rate-Free Optimizers (24)
  • Learning Rate Schedulers (28)
• How zij compares
• Engineering standards
• Acknowledgments
• Citation
• License

Installation

pip install zij

From source, with the pinned environment:

git clone https://github.com/junaidaliop/zij.git
cd zij
conda env create -f environment.yml
conda activate zij-optim

Quick start

import zij

# torch.optim, vendored at tag v2.12.0
opt = zij.AdamW(model.parameters(), lr=3e-4)

# research optimizers, same interface
opt = zij.Muon([p for p in model.parameters() if p.ndim == 2], lr=2e-2)
opt = zij.Prodigy(model.parameters())                       # no learning rate to set
opt = zij.SAM(model.parameters(), base_optimizer=zij.SGD, lr=0.1, rho=0.05)

# memory-efficient low-rank training (per-group rank)
opt = zij.GaLoreAdamW(
    [{"params": params, "rank": 128, "update_proj_gap": 200, "scale": 0.25, "proj_type": "std"}],
    lr=1e-2,
)

# look up by name
zij.list_optimizers("adam*")
opt_cls = zij.load_optimizer("soap")

zij.optim mirrors torch.optim, so zij.optim.AdamW is the same class as zij.AdamW, and zij.optim.lr_scheduler is available. Use whichever import reads better in your code.

[!NOTE] A few families use a documented non-standard call protocol. Schedule-Free needs opt.train() and opt.eval(); the SAM family takes a closure or an explicit first_step / second_step pair; Adam-mini and LOMO are built from a model rather than a parameter list. Each class docstring states which.

Library

The PyTorch package ships 106 ready-to-use optimizers. zij.core mirrors torch.optim at tag v2.12.0 (Adam, AdamW, SGD, Muon, LBFGS, Adafactor, and the rest, plus lr_scheduler and swa_utils). zij.contrib adds research methods grouped by family: first-order, second-order, memory-efficient, learning-rate-free, and sharpness-aware. In every Canon table below, the zij column names the class where an implementation exists; a dash (—) means the method is listed but not yet implemented (paper-only, or its source is under a license that cannot be vendored).

zij is a PyTorch library today. The Canon is framework-agnostic: it covers each method regardless of the framework of its original code. JAX and TensorFlow ports are planned and will follow the same standards.

Canon

The Canon below covers 740 methods across 11 categories. Each row records the canonical name, venue, paper, the best available implementation, and the zij class where one exists.

First-Order Optimizers

First-order optimizers update parameters using only gradients and accumulated gradient statistics such as momentum and second-moment estimates. This page covers the stochastic gradient descent lineage, the Adam family, and more recent sign-based and variance-reduced methods. The zij column gives the class name for optimizers already implemented in the package.

Memory-Efficient Optimizers

Memory-efficient optimizers reduce the optimizer-state memory that dominates large-model training budgets, where Adam-style methods store two extra full-precision values per parameter. The methods below cover factored second moments, 8-bit and 4-bit state quantization, low-rank gradient projection, block-coordinate updates, zeroth-order gradient estimates, and stateless update rules.

Trainer integrations

HuggingFace transformers exposes many of these methods through the optim argument of TrainingArguments. Each string value below maps to a memory-efficient optimizer; all backing libraries except the built-in Adafactor must be installed separately.

Fractional-Order Optimizers

Fractional-order optimizers generalize the integer-order gradient step with fractional-calculus operators, most commonly the Caputo, Riemann-Liouville, or Grünwald-Letnikov derivative, which weight past gradient information through power-law memory kernels. The field is young: the first neural-network training results date to 2015, convergence theory is still being settled, and most papers ship no code.

Foundations

Recent advances

Surveys

Note: FAdam (arXiv 2405.12807) is a Fisher-information variant of Adam and is unrelated to fractional calculus despite the name.

Distributed and Communication-Efficient Optimizers

Optimizers in this category target training across many devices or nodes, where memory and inter-worker communication are the main bottlenecks. They shard optimizer state, compress gradient exchange, or synchronize infrequently so that training scales without a proportional increase in bandwidth. Some entries are standalone update rules, while others wrap an inner optimizer with a communication-efficient outer loop.

Second-Order and Orthogonalized Optimizers

Second-order and orthogonalized optimizers exploit curvature information or the matrix structure of gradients rather than purely elementwise first-order statistics. This group spans quasi-Newton and Hessian-diagonal methods (L-BFGS, AdaHessian, Sophia), full-matrix and Kronecker-factored preconditioning (PSGD, Shampoo, SOAP), and orthogonalized-update methods in the Muon family. Venues reflect peer-reviewed acceptance where applicable; otherwise the arXiv year is listed.

Zeroth-Order Optimizers

Zeroth-order (gradient-free) methods train models using only function evaluations, estimating gradients from randomized perturbations of the parameters instead of backpropagation. Because they need no backward pass or activation storage, they run at roughly inference-level memory, which has made them a practical option for fine-tuning large language models on constrained hardware. The lineage runs from SPSA in classical stochastic approximation to recent variance-reduced and low-rank variants built on MeZO.

Privacy-Preserving Optimizers

Privacy-preserving optimizers train models under differential privacy, typically by clipping per-sample gradients and adding calibrated noise to updates. This page lists differentially private optimization methods and reference libraries, from the original DP-SGD to later variants that reduce clipping bias, correct moment estimates, or filter privacy noise.

Sharpness-Aware Optimizers

Sharpness-aware methods seek parameters that lie in neighborhoods with uniformly low loss rather than at isolated minima, which tends to improve generalization. Introduced by SAM (Foret et al., ICLR 2021), these methods wrap a base optimizer such as SGD or AdamW and add a gradient ascent perturbation step before the descent update. Later work makes the perturbation scale-invariant, closes the surrogate gap, reweights the sharpness term, amortizes the extra forward-backward cost, or extends the idea to second-order optimization.

Quantum and Quantum-Inspired Optimizers

This page collects optimizers from two adjacent settings. The first is the optimization of variational quantum circuits, where shot noise and the quantum geometry of the parameter space drive the design of measurement-frugal, gradient-free, and natural-gradient methods. The second is quantum-inspired and quantum-hardware optimization of classical neural networks, where quantum fluctuations, adiabatic evolution, or annealer sampling replace or augment the classical training loop.

Optimizers for variational quantum circuits

Quantum-inspired and quantum-hardware methods

Learning-Rate-Free Optimizers

Learning-rate-free (also called parameter-free or tuning-free) optimizers select their step size automatically during training instead of requiring a manually tuned learning rate. Most methods in this family estimate a quantity such as the distance from the initial point to the solution and set the effective step size from observed gradients, while others wrap an existing base optimizer and tune its global scale factor online. The goal is to match the performance of a well-tuned baseline without a learning-rate search.

Learning Rate Schedulers

zij.core.lr_scheduler vendors the PyTorch core learning rate schedulers under their original class names. The first table lists every vendored class, including the LRScheduler base class, with the published work it derives from where one exists. The second table covers notable schedules from the literature that zij does not yet implement.

In zij

Notable schedules elsewhere

Schedule-Free is not a schedule on top of an optimizer but a replacement for scheduling, achieved through online iterate averaging inside the optimizer; see the learning-rate-free optimizers.

Weight averaging is available separately in zij.core.swa_utils, which provides stochastic weight averaging and exponential moving average utilities (AveragedModel, SWALR, update_bn, and the SWA/EMA averaging functions), following Averaging Weights Leads to Wider Optima and Better Generalization (Izmailov et al., UAI 2018).

How zij compares

Two kinds of project cover this ground: curated awesome-lists, and installable optimizer collections. zij (زِيج) is both.

Engineering standards

• The Canon and the code are one project. Every Canon row links the paper and, where it exists, the implementation. Every implementation links back to its source and paper.
• Provenance is explicit. Vendored files record their upstream repository, pinned commit, and license; THIRD_PARTY_NOTICES.md aggregates the attributions. Sources under GPL, non-commercial, or no license are not vendored and remain listed only.
• Mathematics is explicit. Each update rule is written in standard notation. Where an official implementation diverges from its own paper, the docstring records what the code computes.
• Everything is tested. Every registered optimizer has convergence and state-dict round-trip tests.

Contributing

New implementations, Canon entries, and corrections are welcome. See CONTRIBUTING.md. A Canon correction counts as much as a code change.

Acknowledgments

zij (زِيج) builds on the projects it learns from:

• APRIL-AIGC/Awesome-Optimizer: an awesome-list whose breadth helped inform this project's scope.
• kozistr/pytorch_optimizer: a comprehensive, maintained PyTorch optimizer collection, and a reference for several vendored implementations.
• jettify/pytorch-optimizer: an early community optimizer collection and the source of several classic implementations.
• timm: tested optimizer implementations and packaging conventions.
• PyTorch: the torch.optim core that zij.core mirrors.
• The optimizer authors: each method is someone's research. The canonical paper is cited in every Canon row and class docstring, and the original repository is credited per file in THIRD_PARTY_NOTICES.md.

Citation

If you use an optimizer from this library, cite two works: the original paper of the algorithm (linked in its Canon row and docstring), and zij as the software you ran. The paper credits the method; the software credits the implementation.

@software{raja_zij,
  author = {Raja, Muhammad Junaid Ali Asif},
  title  = {zij: A Canon and Library of Deep Learning Optimizers},
  year   = {2026},
  url    = {https://github.com/junaidaliop/zij}
}

Machine-readable metadata is in CITATION.cff.

License

Apache-2.0. Vendored components retain their original licenses; see THIRD_PARTY_NOTICES.md.

Contact

Muhammad Junaid Ali Asif Raja — muhammadjunaidaliasifraja@gmail.com