nanosarvam

A compact, from-scratch implementation of a Mixture-of-Experts (MoE) language model inspired by the Sarvam-30B and DeepSeek-V2 architectures.

Designed to be readable, hackable, and trainable on consumer GPUs.

---

Architecture

Full config (~30 B parameters)

Tiny config (~300 M parameters, fits on 8 GB VRAM)

Enabled with --tiny. Suitable for experimentation on a consumer GPU.

---

Installation

git clone https://github.com/cneuralnetwork/nanosarvam
cd nanosarvam
pip install -r requirements.txt

Requirements: Python 3.11+, PyTorch 2.2+, a CUDA-capable GPU.

---

Quickstart

Train on TinyStories (8 GB GPU, recommended defaults)

export WANDB_API_KEY=your_key_here        # or pass --wandb_api_key

python train.py \
  --tiny \
  --use_fp16 \
  --use_8bit_adam \
  --grad_checkpoint \
  --batch_size 1 \
  --accumulation_steps 16 \
  --max_seq_len 512

Expected VRAM usage: ~2.5 GB weights + optimizer, ~0.4 GB activations.

Train on a custom Hugging Face dataset

Any public or gated Hugging Face dataset works. Specify the repo ID with --dataset, the text column with --text_field, and the split names:

# OpenWebText
python train.py --tiny --use_fp16 --use_8bit_adam \
  --dataset "Skylion007/openwebtext" \
  --text_field text \
  --val_split none          # openwebtext has no validation split

# FineWeb (10 BT sample, gated — requires HF token)
python train.py --tiny --use_fp16 --use_8bit_adam \
  --dataset "HuggingFaceFW/fineweb" \
  --dataset_name "sample-10BT" \
  --text_field text \
  --val_split none \
  --hf_token YOUR_HF_TOKEN

# SlimPajama
python train.py --tiny --use_fp16 --use_8bit_adam \
  --dataset "cerebras/SlimPajama-627B" \
  --text_field text \
  --train_split train \
  --val_split validation

---

Full argument reference

Dataset

Tokenizer

Model

Training

Memory optimisation

Checkpointing

Weights & Biases

---

API keys and secrets

Never hardcode API keys in source files. nanosarvam reads credentials from environment variables and falls back to CLI args as a convenience for one-off runs.

# Recommended: set in your shell profile or a .env file (never commit .env)
export WANDB_API_KEY=...
export HF_TOKEN=...         # only needed for gated datasets

python train.py --tiny --use_fp16 --use_8bit_adam ...

If no W&B key is found, the run is logged offline to ./wandb/.

---

VRAM reference (tiny config, seq_len=512)

---

Resuming a run

python train.py --tiny --use_fp16 --use_8bit_adam \
  --resume_from ./checkpoints/checkpoint_step_1000.pt

---

Project structure

nanosarvam/
├── model.py          # Model definition (RMSNorm, RoPE, GQA, SwiGLU, MoE)
├── train.py          # Training loop, dataset loading, CLI
├── requirements.txt
└── README.md

---