nGPT

My best attempt at replicating/teaching Nvidia's Normalized-GPT. The vast majority of it (see first commit from Nov 18, 2024) was written only from reading the paper and without awareness of the fact that Nvidia had open-sourced their own implementation (see also lucidrains' implementation). Click below to watch the video:

getting started

setting up

1. clone the repo, create your venv, and install everything in requirements.txt

learning

1. read thru tutorial.ipynb, watch the video and read the paper

using

training

1. edit values in config.py
2. run all cells in train.ipynb

inference

1. open inference.ipynb
2. set the variable model_name equal to the name of a sub-folder within models/. defaults to the tiny model I trained "nGPT-2m"
3. run the rest of the cells. Notice that temperature needs to be very weirdly small in order to get reasonable output; this has something to do with the fact that we're using cosine similarity and the scaling factor. Not sure if that quirk would persist in a model of proper size