Env Preparing

Setup venv:

python3.11 -m venv .venv
source .venv/bin/activate

Install requirements:

python.exe -m pip install --upgrade pip

pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install -r requirements.txt --upgrade

Test Cuda (optional):

nvidia-smi

Nougat-LaTeX-OCR

Nougat-LaTeX-based is fine-tuned from facebook/nougat-base with im2latex-100k to boost its proficiency in generating LaTeX code from images. Since the initial encoder input image size of nougat was unsuitable for equation image segments, leading to potential rescaling artifacts that degrades the generation quality of LaTeX code. To address this, Nougat-LaTeX-based adjusts the input resolution and uses an adaptive padding approach to ensure that equation image segments in the wild are resized to closely match the resolution of the training data. Download the model here 👈🏻.

Uses

fine-tune on your customized dataset

1. Prepare your dataset in this format
2. Change config/base.yaml
3. Run the training script

python tools/train_experiment.py --config_file config/base.yaml --phase 'train'

use it directly

use a pipeline as a high-level helper

from transformers import pipeline
pipe = pipeline("image-to-text", model="Norm/nougat-latex-base")

load model directly

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Norm/nougat-latex-base")
model = AutoModel.from_pretrained("Norm/nougat-latex-base")

Check the model application information in https://huggingface.co/Norm/nougat-latex-base You can find an example in examples_for_latex folder

python examples/run_latex_ocr.py --img_path "examples/test_data/eq1.png"