HPCNC (Hippocampus & Neocortex)

RWKV7-1.5B Training Benchmarks

Tested on NVIDIA GeForce RTX 3060 (12 GB VRAM) with full parameter training (no LoRA).

Training Results (10.5 GB VRAM limit)

Note: Standard AdamW OOMs - optimizer states alone require ~11.6 GB for 1.5B model.

Memory-Efficient Training Techniques

1. Gradient Checkpointing (grad_cp=1): Recompute activations during backward

   • Saves ~7x activation memory
   • Faster at high memory utilization (less allocation overhead)

2. Per-Parameter Optimizer: Run optimizer step during backward via register_post_accumulate_grad_hook

   • Each parameter is updated immediately when its gradient is computed
   • Gradient is freed right after update (param.grad = None)
   • Avoids storing any gradients in VRAM - only one gradient exists at a time

3. Infinite Context Mode (train_type="infctx"): This project always trains with infinite context length

   • Model names like rwkv7-g1a-0.1b-20250728-ctx4096 indicate the original training context length
   • This does NOT limit inference or training context - we always use unbounded context
   • Both ctx_len=sys.maxsize and chunk_ctx=sys.maxsize are set to allow arbitrarily long sequences