siddharth0112358/simpleT5_TRL

stars

commits

Python

primary language

Jan 4, 2026

updated

README

simpleT5-TRL

Train encoder-decoder models (T5, BART, T5Gemma, etc.) with Full Finetuning, LoRA, QLoRA, DPO, SimPO & RFT in just a few lines of code.

Built on top of HuggingFace Transformers and TRL.

Supported models: T5, MT5, ByT5, CodeT5, BART, mBART, Pegasus, LED, T5Gemma, and more!

Installation

From PyPI

pip install simplet5-trl

From Source

git clone https://github.com/siddharth0112358/simpleT5_TRL.git
cd simpleT5_TRL
pip install -e .

Quick Start

from simplet5_trl import SimpleT5_TRL
import pandas as pd

# Prepare data
train_df = pd.DataFrame({
    "source_text": ["summarize: This is a long article..."],
    "target_text": ["Short summary"]
})

# Train
model = SimpleT5_TRL()
model.from_pretrained("t5-base")
model.train(train_df=train_df, eval_df=train_df, max_epochs=3)

# Predict
model.load_model("outputs/checkpoint-xxx", use_gpu=True)
print(model.predict("summarize: Your text here"))

Training Methods

Method	Function	Data Columns	Use When
SFT	`train()`	`source_text`, `target_text`	Standard supervised finetuning
DPO	`train_dpo()`	`prompt`, `chosen`, `rejected`	You have preference pairs
SimPO	`train_simpo()`	`prompt`, `chosen`, `rejected`	Limited GPU memory (no ref model)
RFT	`train_rft()`	`source_text`, `target_text`	Curated high-quality samples

Choosing the Right Method

Have preference pairs (chosen vs rejected)?
├── Yes → Have enough GPU memory?
│         ├── Yes → DPO
│         └── No  → SimPO
└── No  → RFT (or standard train)

Finetuning Modes

All training methods support three finetuning modes via finetuning= parameter:

Mode	Memory	Quality	Use Case
`"full"`	High	Best	Small models, enough VRAM
`"lora"`	Low	Good	Large models, limited VRAM
`"qlora"`	Very Low	Good	Very large models, consumer GPUs

Hyperparameters

Common Parameters (All Methods)

# Training
max_epochs=3                    # Number of epochs
max_steps=-1                    # Max steps (-1 = use epochs)
batch_size=8                    # Batch size
learning_rate=1e-4              # Learning rate
precision="32"                  # "32", "16", "bf16"
seed=42                         # Random seed

# Optimizer
optim="adamw_torch"             # "adamw_torch", "sgd", "adafactor"
weight_decay=0.0                # Weight decay
warmup_steps=0                  # Warmup steps
warmup_ratio=0.0                # Warmup ratio
lr_scheduler_type="linear"      # "linear", "cosine", "constant", "polynomial"

# Gradient
gradient_accumulation_steps=1   # Gradient accumulation
gradient_checkpointing=False    # Memory-saving checkpointing

# Saving
outputdir="outputs"             # Output directory
save_strategy="epoch"           # "epoch", "steps", "no"
save_steps=500                  # Save every N steps
save_total_limit=None           # Max checkpoints to keep

# Evaluation
eval_strategy="epoch"           # "epoch", "steps", "no"
eval_steps=500                  # Eval every N steps

# Logging
logging_steps=1                 # Log every N steps
report_to=["tensorboard"]       # "tensorboard", "wandb"

# Finetuning
finetuning="full"               # "full", "lora", "qlora"

LoRA Parameters

lora_r=16                       # LoRA rank
lora_alpha=32                   # LoRA alpha
lora_dropout=0.05               # LoRA dropout
lora_target_modules=None        # Auto-detected if None

QLoRA Parameters

quantization="4bit"             # "4bit" or "8bit"
bnb_4bit_compute_dtype="float16"
bnb_4bit_quant_type="nf4"       # "nf4" or "fp4"
bnb_4bit_use_double_quant=True

DPO Parameters

model.train_dpo(
    beta=0.1,                   # Deviation from reference (lower = more deviation)
    loss_type="sigmoid",        # "sigmoid", "hinge", "ipo", "robust"
    label_smoothing=0.0,        # Label smoothing
    max_length=512,             # Max sequence length
    max_prompt_length=256,      # Max prompt length
)

SimPO Parameters

model.train_simpo(
    beta=2.0,                   # SimPO beta (higher than DPO)
    simpo_gamma=0.5,            # Target reward margin
    label_smoothing=0.0,
    max_length=512,
    max_prompt_length=256,
)

RFT Parameters

model.train_rft(
    max_seq_length=512,         # Max sequence length
    packing=False,              # Pack multiple examples
    dataset_text_field="text",  # Column for text data
)

Prediction Parameters

model.predict(
    source_text="input",
    max_length=512,
    num_beams=2,
    do_sample=True,
    temperature=1.0,
    top_k=50,
    top_p=0.95,
    repetition_penalty=2.5,
)

Loading Models

# Full finetuned
model.load_model("outputs/checkpoint-xxx", use_gpu=True)

# LoRA
model.load_model("outputs/checkpoint-xxx", finetuning="lora", base_model_name="t5-base")

# QLoRA
model.load_model("outputs/checkpoint-xxx", finetuning="qlora", base_model_name="t5-large", quantization="4bit")

Supported Models

Family	Examples
T5	`t5-small`, `t5-base`, `t5-large`, `t5-3b`
MT5	`google/mt5-small`, `google/mt5-base`
BART	`facebook/bart-base`, `facebook/bart-large`
CodeT5	`Salesforce/codet5-base`
Pegasus	`google/pegasus-xsum`
LongT5	`google/long-t5-local-base`
T5Gemma	`google/t5gemma-2-270m-270m`, `google/t5gemma-2b-2b-ul2`

Troubleshooting

Missing embed_positions.weight Warning

When loading BART or Pegasus models, you may see:

model.decoder.embed_positions.weight | MISSING
model.encoder.embed_positions.weight | MISSING

This is expected and harmless. These models use sinusoidal positional embeddings computed at runtime, not learned weights.

LongT5-tglobal NaN Issues

The google/long-t5-tglobal-* models may produce NaN values during training. Use the local variant instead:

model.from_pretrained("google/long-t5-local-base")  # Stable
# instead of "google/long-t5-tglobal-base"          # May produce NaN

Acknowledgements

simpleT5 by Shivanandroy
Transformers by HuggingFace
TRL by HuggingFace
PEFT by HuggingFace
bitsandbytes

License

MIT License

Similar repos

huggingface/trl

Train transformer language models with reinforcement learning.

19,286

tsis-mobile-technology/UseTRLFinetune

pszemraj/long-t5-tglobal-xl-16384-book-summary

Model

tuggspeedman-ai/SmolLM3-3B-summarize-dpo-lora

Model

Blealtan/RWKV-LM-LoRA

RWKV is a RNN with transformer-level LLM performance. It can be directly trained like a GPT…

410

huggingface/peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

21,641

Red-Hat-AI-Innovation-Team/training_hub

An algorithm-focused interface for common llm training, continual learning, and reinforcement…

meta-pytorch/torchtune

PyTorch native post-training library

5,808

Languages

Python

100.0%

siddharth0112358/simpleT5_TRL

stars

commits

Python

primary language

Jan 4, 2026

updated

README

simpleT5-TRL

Train encoder-decoder models (T5, BART, T5Gemma, etc.) with Full Finetuning, LoRA, QLoRA, DPO, SimPO & RFT in just a few lines of code.

Built on top of HuggingFace Transformers and TRL.

Supported models: T5, MT5, ByT5, CodeT5, BART, mBART, Pegasus, LED, T5Gemma, and more!

Installation

From PyPI

pip install simplet5-trl

From Source

git clone https://github.com/siddharth0112358/simpleT5_TRL.git
cd simpleT5_TRL
pip install -e .

Quick Start

from simplet5_trl import SimpleT5_TRL
import pandas as pd

# Prepare data
train_df = pd.DataFrame({
    "source_text": ["summarize: This is a long article..."],
    "target_text": ["Short summary"]
})

# Train
model = SimpleT5_TRL()
model.from_pretrained("t5-base")
model.train(train_df=train_df, eval_df=train_df, max_epochs=3)

# Predict
model.load_model("outputs/checkpoint-xxx", use_gpu=True)
print(model.predict("summarize: Your text here"))

Training Methods

Method	Function	Data Columns	Use When
SFT	`train()`	`source_text`, `target_text`	Standard supervised finetuning
DPO	`train_dpo()`	`prompt`, `chosen`, `rejected`	You have preference pairs
SimPO	`train_simpo()`	`prompt`, `chosen`, `rejected`	Limited GPU memory (no ref model)
RFT	`train_rft()`	`source_text`, `target_text`	Curated high-quality samples

Choosing the Right Method

Have preference pairs (chosen vs rejected)?
├── Yes → Have enough GPU memory?
│         ├── Yes → DPO
│         └── No  → SimPO
└── No  → RFT (or standard train)

Finetuning Modes

All training methods support three finetuning modes via finetuning= parameter:

Mode	Memory	Quality	Use Case
`"full"`	High	Best	Small models, enough VRAM
`"lora"`	Low	Good	Large models, limited VRAM
`"qlora"`	Very Low	Good	Very large models, consumer GPUs

Hyperparameters

Common Parameters (All Methods)

# Training
max_epochs=3                    # Number of epochs
max_steps=-1                    # Max steps (-1 = use epochs)
batch_size=8                    # Batch size
learning_rate=1e-4              # Learning rate
precision="32"                  # "32", "16", "bf16"
seed=42                         # Random seed

# Optimizer
optim="adamw_torch"             # "adamw_torch", "sgd", "adafactor"
weight_decay=0.0                # Weight decay
warmup_steps=0                  # Warmup steps
warmup_ratio=0.0                # Warmup ratio
lr_scheduler_type="linear"      # "linear", "cosine", "constant", "polynomial"

# Gradient
gradient_accumulation_steps=1   # Gradient accumulation
gradient_checkpointing=False    # Memory-saving checkpointing

# Saving
outputdir="outputs"             # Output directory
save_strategy="epoch"           # "epoch", "steps", "no"
save_steps=500                  # Save every N steps
save_total_limit=None           # Max checkpoints to keep

# Evaluation
eval_strategy="epoch"           # "epoch", "steps", "no"
eval_steps=500                  # Eval every N steps

# Logging
logging_steps=1                 # Log every N steps
report_to=["tensorboard"]       # "tensorboard", "wandb"

# Finetuning
finetuning="full"               # "full", "lora", "qlora"

LoRA Parameters

lora_r=16                       # LoRA rank
lora_alpha=32                   # LoRA alpha
lora_dropout=0.05               # LoRA dropout
lora_target_modules=None        # Auto-detected if None

QLoRA Parameters

quantization="4bit"             # "4bit" or "8bit"
bnb_4bit_compute_dtype="float16"
bnb_4bit_quant_type="nf4"       # "nf4" or "fp4"
bnb_4bit_use_double_quant=True

DPO Parameters

model.train_dpo(
    beta=0.1,                   # Deviation from reference (lower = more deviation)
    loss_type="sigmoid",        # "sigmoid", "hinge", "ipo", "robust"
    label_smoothing=0.0,        # Label smoothing
    max_length=512,             # Max sequence length
    max_prompt_length=256,      # Max prompt length
)

SimPO Parameters

model.train_simpo(
    beta=2.0,                   # SimPO beta (higher than DPO)
    simpo_gamma=0.5,            # Target reward margin
    label_smoothing=0.0,
    max_length=512,
    max_prompt_length=256,
)

RFT Parameters

model.train_rft(
    max_seq_length=512,         # Max sequence length
    packing=False,              # Pack multiple examples
    dataset_text_field="text",  # Column for text data
)

Prediction Parameters

model.predict(
    source_text="input",
    max_length=512,
    num_beams=2,
    do_sample=True,
    temperature=1.0,
    top_k=50,
    top_p=0.95,
    repetition_penalty=2.5,
)

Loading Models

# Full finetuned
model.load_model("outputs/checkpoint-xxx", use_gpu=True)

# LoRA
model.load_model("outputs/checkpoint-xxx", finetuning="lora", base_model_name="t5-base")

# QLoRA
model.load_model("outputs/checkpoint-xxx", finetuning="qlora", base_model_name="t5-large", quantization="4bit")

Supported Models

Family	Examples
T5	`t5-small`, `t5-base`, `t5-large`, `t5-3b`
MT5	`google/mt5-small`, `google/mt5-base`
BART	`facebook/bart-base`, `facebook/bart-large`
CodeT5	`Salesforce/codet5-base`
Pegasus	`google/pegasus-xsum`
LongT5	`google/long-t5-local-base`
T5Gemma	`google/t5gemma-2-270m-270m`, `google/t5gemma-2b-2b-ul2`

Troubleshooting

Missing embed_positions.weight Warning

When loading BART or Pegasus models, you may see:

model.decoder.embed_positions.weight | MISSING
model.encoder.embed_positions.weight | MISSING

This is expected and harmless. These models use sinusoidal positional embeddings computed at runtime, not learned weights.

LongT5-tglobal NaN Issues

The google/long-t5-tglobal-* models may produce NaN values during training. Use the local variant instead:

model.from_pretrained("google/long-t5-local-base")  # Stable
# instead of "google/long-t5-tglobal-base"          # May produce NaN

Acknowledgements

simpleT5 by Shivanandroy
Transformers by HuggingFace
TRL by HuggingFace
PEFT by HuggingFace
bitsandbytes

License

MIT License

Similar repos

huggingface/trl

Train transformer language models with reinforcement learning.

19,286

tsis-mobile-technology/UseTRLFinetune

pszemraj/long-t5-tglobal-xl-16384-book-summary

Model

tuggspeedman-ai/SmolLM3-3B-summarize-dpo-lora

Model

Blealtan/RWKV-LM-LoRA

RWKV is a RNN with transformer-level LLM performance. It can be directly trained like a GPT…

410

huggingface/peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

21,641

Red-Hat-AI-Innovation-Team/training_hub

An algorithm-focused interface for common llm training, continual learning, and reinforcement…

meta-pytorch/torchtune

PyTorch native post-training library

5,808

Languages

Python

100.0%