nomic-bert-2048: A 2048 Sequence Length Pretrained BERT

nomic-bert-2048 is a BERT model pretrained on wikipedia and bookcorpus with a max sequence length of 2048.

We make several modifications to our BERT training procedure similar to MosaicBERT. Namely, we add:

• Use Rotary Position Embeddings to allow for context length extrapolation.
• Use SwiGLU activations as it has been shown to improve model performance
• Set dropout to 0

We evaluate the quality of nomic-bert-2048 on the standard GLUE benchmark. We find it performs comparably to other BERT models but with the advantage of a significantly longer context length.

Pretraining Data

We use BookCorpus and a 2023 dump of wikipedia. We pack and tokenize the sequences to 2048 tokens. If a document is shorter than 2048 tokens, we append another document until it fits 2048 tokens. If a document is greater than 2048 tokens, we split it across multiple documents. We release the dataset here

Usage

from transformers import AutoModelForMaskedLM, AutoConfig, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased') # `nomic-bert-2048` uses the standard BERT tokenizer

config = AutoConfig.from_pretrained('nomic-ai/nomic-bert-2048', trust_remote_code=True) # the config needs to be passed in
model = AutoModelForMaskedLM.from_pretrained('nomic-ai/nomic-bert-2048',config=config, trust_remote_code=True)

# To use this model directly for masked language modeling
classifier = pipeline('fill-mask', model=model, tokenizer=tokenizer,device="cpu")

print(classifier("I [MASK] to the store yesterday."))

To finetune the model for a Sequence Classification task, you can use the following snippet

from transformers import AutoConfig, AutoModelForSequenceClassification
model_path = "nomic-ai/nomic-bert-2048"
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
# strict needs to be false here since we're initializing some new params
model = AutoModelForSequenceClassification.from_pretrained(model_path, config=config, trust_remote_code=True, strict=False)

Join the Nomic Community

• Nomic: https://nomic.ai
• Discord: https://discord.gg/myY5YDR8z8
• Twitter: https://twitter.com/nomic_ai