What is this

This repo contains the collection of my toy models. For fun.

It may contain bugs. It may not work outside of my specific setup.

MIT License

IoI Circuit Analyzer

Code: learning/src/learning/ioi_circuit

Load a Pretrained GPT2 model into a custom Transformer to analyze how it detects indirect objects.

Reproduce paper: Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small", arXiv:2211.00593 [cs.LG]

ShakeSpeare Generator

Code: learning/src/learning/shakespeare_generator

Multi-Head Attention Transformer model to learn how to generate "ShakeSpeare"-like text.

Names Generator

Code: learning/src/learning/names_generator

MLP model for generating names given a country of origin

Names Classifier

Code: learning/src/learning/names_classifier

RNN, LSTM, GRU models for classifying the country of origin of a given name.