Any-Winter-4079/DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

Python

3 commits

updated Oct 1, 2026

See what people are saying

Newest first

Most points

Source	Message	Score	Date
	DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts? (r/LocalLLaMA) Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary of my results ( code here:…	12	Oct 1, 2026

README

DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

This repository compares training on DDR4/PCIe 4 and DDR5/PCIe 5 workstations using RTX PRO 6000 WS GPUs. train_DDP.py supports one or two GPUs; train_PP.py uses two pipeline stages. The plot reports cumulative training-step time to validation loss ≤ 3.28, excluding validation and warm-up.

Running the benchmarks

Start up a container with the given image (PyTorch 2.10.0 + CUDA 12.8):

https://cloud.vast.ai/?ref_id=195884&creator_id=195884&name=anywinter4079%2Fpytorch%3A2.10.0-cu128

Clone and cd into this repository:

git clone https://github.com/Any-Winter-4079/DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training.git && \
cd DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

Download the dataset:

python data/fineweb-npy.py

Run, e.g., for pipeline parallelism:

torchrun --standalone --nproc_per_node=2 train_PP.py

Similar repos

thad0ctor/unsloth-5090-multiple

unsloth-5090-multiple

LambdaLabsML/distributed-training-guide

Best practices & guides on how to write distributed pytorch training code

633

anhvth/opensloth

244

tdrussell/diffusion-pipe

A pipeline parallel training script for diffusion models.

2,028

universuen/ViT-Fine-Tuning-with-ColossalAI

AI-Hypercomputer/gpu-recipes

Recipes for reproducing training and serving benchmarks for large machine learning models using…

141

local-inference-lab/rtx6kpro

RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

1,118

ulmentflam/llm.mojo

GPT-2 training in pure Mojo with hand-written CUDA and Metal GPU kernels. llm.c parity in bf16 on…

Languages

Python

100.0%

Any-Winter-4079/DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

Python

3 commits

updated Oct 1, 2026

See the code

See what people are saying

Newest first

Most points

Source	Message	Score	Date
	DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts? (r/LocalLLaMA) Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary of my results ( code here:…	12	Oct 1, 2026

README

DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

Running the benchmarks

Start up a container with the given image (PyTorch 2.10.0 + CUDA 12.8):

https://cloud.vast.ai/?ref_id=195884&creator_id=195884&name=anywinter4079%2Fpytorch%3A2.10.0-cu128

Clone and cd into this repository:

git clone https://github.com/Any-Winter-4079/DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training.git && \
cd DDR4-PCIe-4-vs-DDR5-PCIe-5-for-CUDA-training

Download the dataset:

python data/fineweb-npy.py

Run, e.g., for pipeline parallelism:

torchrun --standalone --nproc_per_node=2 train_PP.py

Similar repos

thad0ctor/unsloth-5090-multiple

unsloth-5090-multiple

LambdaLabsML/distributed-training-guide

Best practices & guides on how to write distributed pytorch training code

633

anhvth/opensloth

244

tdrussell/diffusion-pipe

A pipeline parallel training script for diffusion models.

2,028

universuen/ViT-Fine-Tuning-with-ColossalAI

AI-Hypercomputer/gpu-recipes

Recipes for reproducing training and serving benchmarks for large machine learning models using…

141

local-inference-lab/rtx6kpro

RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

1,118

ulmentflam/llm.mojo

GPT-2 training in pure Mojo with hand-written CUDA and Metal GPU kernels. llm.c parity in bf16 on…

Languages

Python

100.0%