sradc/pretraining-BERT

Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch

Jupyter Notebook

9 commits

updated May 20, 2025

See the code

README

pretraining-BERT

This accompanies the blog post: sidsite.com/posts/bert-from-scratch

See the pretraining code in pretraining_BERT.ipynb.

Fine-tuning

It's possible to fine tune a model after pretraining, using the run_finetuning.sh script. Note that the parameters here are roughly based on Cramming, but I used different training parameters for two of the tasks. (TODO: add these.))

Significant stargazers

SuperDJY

38 followers · starred Jun 2023

Similar repos

MuhammetCanGumussu/Bert-Implementation-From-Scratch

pranavrao-qure/MiniTasks

Repo for misc scripts

UBC-NLP/LMBERT

JonasGeiping/cramming

Cramming the training of a (BERT-type) language model into limited compute.

1,368

uclanlp/newsbert

Model

Paper: Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

google-research/bert

TensorFlow code and pre-trained models for BERT

40,038

google-bert/bert-large-uncased-whole-word-masking-finetuned-squad

Model

BERT large model (uncased) whole word masking finetuned on SQuAD

191

HoweChen0517/finetuning-bert-for-IR

sradc/pretraining-BERT

Pre-train BERT from scratch, with HuggingFace. Accompanies the blog post: sidsite.com/posts/bert-from-scratch

Jupyter Notebook

9 commits

updated May 20, 2025

See the code

README

pretraining-BERT

This accompanies the blog post: sidsite.com/posts/bert-from-scratch

See the pretraining code in pretraining_BERT.ipynb.

Fine-tuning

Significant stargazers

SuperDJY

38 followers · starred Jun 2023

Similar repos

MuhammetCanGumussu/Bert-Implementation-From-Scratch

pranavrao-qure/MiniTasks

Repo for misc scripts

UBC-NLP/LMBERT

JonasGeiping/cramming

Cramming the training of a (BERT-type) language model into limited compute.

1,368

uclanlp/newsbert

Model

Paper: Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

google-research/bert

TensorFlow code and pre-trained models for BERT

40,038

google-bert/bert-large-uncased-whole-word-masking-finetuned-squad

Model

BERT large model (uncased) whole word masking finetuned on SQuAD

191

HoweChen0517/finetuning-bert-for-IR