NousResearch/DisTrO

Distributed Training Over-The-Internet

1,064

15 commits

updated Oct 14, 2025

README

DisTrO

This is the repository for DisTrO (Distributed Training Over-The-Internet), a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude.

Aug. 26th, 2024: DisTrO (Preliminary Report)
Dec. 2nd, 2024: DeMo Optimization (Paper) (Code), original seed research/idea for DisTrO
Dec. 2nd, 2024: Nous trains a 15b model using DisTrO
May 14th, 2025: Psyche Network
May 14th, 2025: Nous Consilience 40b LLM, Huggingface
Oct. 14th, 2025: DeMo Optimization (Paper v2), (Production Code)

Join us on Discord if you're interested in helping research and build the future of distributed training.

Significant stargazers

Pedro Cuenca

887 followers · starred Aug 2024

Justin Johnson

9,557 followers · starred Sep 2024

Nouamane Tazi

474 followers · starred Nov 2024

Peter Kieltyka

831 followers · starred Aug 2024

Similar repos

Shenggan/awesome-distributed-ml

A curated list of awesome projects and papers for distributed training or inference

281

schneiderkamplab/DeToNATION

Decoupled Torch Network-Aware Training on Interlinked Online Nodes

PrimeIntellect-ai/prime-diloco

prime is a framework for efficient, globally distributed training of AI models over the internet.

856

PrimeIntellect-ai/OpenDiloco

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training

598

microsoft/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference…

43,238

deepspeedai/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference…

43,238

ByteDance-Seed/Triton-distributed

Distributed Compiler and Optimized Parallel Kernels

1,569

inpluslab-wuhui/Systems-for-Foundation-Models