GenRL Reinforcement Learning Framework for Visual Generation

GenRL is a scalable, modular reinforcement learning framework for optimizing visual generation models — from images to videos — with plug-and-play reward functions, multi-GPU distributed training, and first-class support for diffusion & flow-based generators.

🚀 Getting Started · 📖 Algorithms · 🏛️ Model Zoo · 📊 Performance · 🏗️ Architecture

✨ Highlights

• 🎯 Unified RL for Visual Generation — A single framework covering text-to-image (T2I), text-to-video (T2V), and image-to-video (I2V) generation
• 🏆 Production-Ready Models — High-performance RL-trained checkpoints available on HuggingFace, optimized for aesthetic quality, motion coherence, and text alignment
• 🔄 Multi-Paradigm Support — Native support for both Diffusion and Rectified Flow generation paradigms via unified SDE formulation
• 🧩 Modular Reward System — Plug-and-play reward functions: aesthetic scores, text-alignment, motion quality, OCR accuracy, and custom user-defined rewards
• 📝 High-Quality Training Data — Carefully curated ~200k filtered prompts for optimal RL training performance
• ⚡ Scalable & Efficient — Multi-node FSDP training with activation checkpointing, LoRA / full fine-tune, EMA, 8-bit Adam, and memory-efficient reward model offloading
• 🎛️ YAML-Driven Configuration — Everything from model choice, reward weights, training schedule to FSDP sharding strategy is controlled via a single YAML config
• 🔬 Reproducible by Design — Deterministic seeding across sampling, training, and logging for bit-exact experiment reproduction

📖 Supported Algorithms

💡 GenRL is designed to be algorithm-agnostic. Adding a new RL algorithm only requires implementing a new trainer — everything else (rewards, data, logging) is reusable. For GRPO-based algorithms, most implementations only need to modify a small amount of code in the trainer.

🤖 Supported Models

🎁 Supported Reward Functions

🔗 Multiple rewards can be composed with configurable weights — GenRL supports both reward-weighted and advantage-weighted composition modes.

📊 Performance

📝 Video OCR Training Comparison

FlowGRPO Original

GenRL FlowGRPO

Performance Comparison: Under the same settings (using config/default.yaml with 8 GPU training), the original FlowGRPO repository reaches an OCR score of approximately 0.28 at around 3k steps, while our GenRL implementation achieves an OCR score close to 0.3 at only 1.2k steps, demonstrating superior training efficiency and convergence.

📊 LongCat Multi-Reward Training

HPSv3 General

HPSv3 Percentile

VideoAlign-MQ

VideoAlign-TA

LongCat Reproduction: Our GenRL implementation successfully reproduces LongCat (not yet open-sourced) on the Wan2.1-T2V 1.3B model. Training with 64 H100 GPUs up to 1.5k steps, all four reward metrics continue to improve normally, demonstrating stable and effective multi-reward RLHF training. The training dataset consists of ~200k carefully filtered prompts (datasets/filtered_prompts/), ensuring high-quality training data for optimal RL performance.

📈 Visual Example

The following video demonstrates the quality improvement achieved by GenRL training:

Video Layout: The video consists of 8 sub-videos arranged in a 2×4 grid:

• Top row (4 videos): Baseline results from Wan-AI/Wan2.1-T2V-1.3B-Diffusers (pretrained model)
• Bottom row (4 videos): Results from our GenRL LongCat implementation after 1.5k steps of RL training on the same model

🚀 Getting Started

📋 Prerequisites

• Python 3.10+
• CUDA 12.x + PyTorch 2.6
• 8× A100/H100 GPUs (recommended for video training)
• Git LFS (required for downloading large dataset files)

0️⃣ Clone Repository

Important: This repository uses Git LFS for large dataset files. Make sure Git LFS is installed before cloning.

# Install Git LFS (if not already installed)
# Ubuntu/Debian
sudo apt-get install git-lfs

# macOS
brew install git-lfs

# Initialize Git LFS
git lfs install

# Clone the repository (Git LFS will automatically download large files)
git clone https://github.com/ModelTC/GenRL.git
cd GenRL

💡 Note: The datasets/filtered_prompts/ directory contains large JSON files (~300MB) stored with Git LFS. If you encounter download issues, you can manually pull LFS files with git lfs pull.

1️⃣ Install Dependencies

pip install -r requirements.txt

2️⃣ Initialize Submodules

git submodule update --init --recursive

3️⃣ Setup Reward Model Checkpoints

cd genrl/reward/VideoAlign/checkpoints
git lfs install
git clone https://huggingface.co/KwaiVGI/VideoReward
mv VideoReward/* .
mv VideoReward/.* . 2>/dev/null || true
rm -rf VideoReward
cd ../../../..

python -c "from paddleocr import PaddleOCR; ocr = PaddleOCR(use_angle_cls=False, lang='en', use_gpu=False, show_log=False)"

pip install flash-attn==2.7.4.post1 --no-build-isolation

4️⃣ Launch Training

# Single node, 8 GPUs (LoRA + FSDP)
accelerate launch train.py --config config/default.yaml

# Multi-node (8 nodes × 8 GPUs)
torchrun --nnodes=8 --nproc_per_node=8 \
  --rdzv_backend=c10d \
  --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \
  train.py --config config/longcat.yaml

5️⃣ Inference

Trained models (LoRA adapters or full-parameter checkpoints) can be directly used for inference (exemplified by Wan2.1-T2V-1.3B-Diffusers with LoRA training):

Using Diffusers Library (PEFT-compatible):

import torch
from diffusers import AutoencoderKLWan, WanPipeline
from diffusers.utils import export_to_video
from peft import PeftModel

# Available models: Wan-AI/Wan2.1-T2V-14B-Diffusers, Wan-AI/Wan2.1-T2V-1.3B-Diffusers
model_id = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
pipe = WanPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
pipe.to("cuda")

# Load LoRA adapter (if using LoRA training)
pipe.transformer = PeftModel.from_pretrained(
    pipe.transformer,
    "path/to/final_model/transformer"  # Path to LoRA checkpoint after training
)

prompt = "A cat walks on the grass, realistic"
negative_prompt = "Bright tones, overexposed, static, blurred details, worst quality, low quality"

output = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=480,
    width=832,
    num_frames=81,
    guidance_scale=5.0
).frames[0]
export_to_video(output, "output.mp4", fps=15)

Using LightX2V (for accelerated inference):

import torch
from diffusers import AutoencoderKLWan, WanPipeline
from peft import PeftModel

# Load base model
model_id = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
pipe = WanPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)

# Load and merge LoRA adapter
pipe.transformer = PeftModel.from_pretrained(
    pipe.transformer,
    "path/to/final_model/transformer"  # Path to LoRA checkpoint
)
pipe.transformer = pipe.transformer.merge_and_unload()  # Merge LoRA weights into base model

# Save merged model (optional)
merged_model_path = "path/to/merged_model"
pipe.save_pretrained(merged_model_path)

After merging, you can directly use LightX2V for accelerated inference. See the LightX2V inference script for example usage.

💡 Note: LoRA adapters saved by GenRL are fully compatible with the PEFT library, allowing seamless integration with standard diffusers workflows and third-party inference frameworks like LightX2V.

🏗️ Architecture

GenRL/
├── 🚀 train.py                        # Entry point
├── 📁 config/                          # YAML configs
│   ├── default.yaml                    #   Default (OCR, FlowGRPO)
│   └── longcat.yaml                    #   Multi-reward, LongCat
├── 📁 genrl/
│   ├── config.py                       # Config schema & loader
│   ├── constants.py                    # Global constants
│   ├── data.py                         # Dataset & dataloaders
│   ├── rewards.py                      # Multi-reward composition
│   ├── advantages.py                   # Advantage computation (GRPO)
│   ├── stat_tracking.py                # Per-prompt stat tracking
│   ├── ema.py                          # EMA wrapper
│   ├── 📁 trainer/
│   │   ├── base_trainer.py             #   Abstract base trainer
│   │   ├── wan_trainer.py              #   Wan model trainer
│   │   ├── sampling.py                 #   Sampling epoch logic
│   │   ├── evaluation.py               #   Eval & video logging
│   │   ├── diffusion.py                #   Log-prob computation
│   │   └── embeddings.py               #   Text embedding utils
│   ├── 📁 reward/
│   │   ├── ocr.py                      #   OCR reward
│   │   ├── hpsv3.py                    #   HPSv3 reward
│   │   ├── videoalign.py               #   VideoAlign rewards
│   │   ├── 📁 HPSv3/                   #   HPSv3 submodule
│   │   └── 📁 VideoAlign/              #   VideoAlign submodule
│   └── 📁 diffusers_patch/
│       └── wan_pipeline_with_logprob.py  # SDE step with log-prob
├── 📁 datasets/                        # Prompt datasets
└── 📁 scripts/
    └── launch.sh                       # Launch script

⚙️ Configuration

All training behavior is controlled by a single YAML file. Key sections:

run_name: my_experiment
seed: 42
num_epochs: 100000
height: 240
width: 416
frames: 33

reward_fn:
  video_ocr: 1.0

trainer: wan
use_lora: true

sample:
  batch_size: 8
  num_steps: 20
  guidance_scale: 4.5
  sde_type: flow_sde

train:
  learning_rate: 1.0e-4
  clip_range: 1.0e-3
  lora_r: 32
  ema: true

accelerate:
  distributed_type: FSDP
  mixed_precision: bf16
  num_processes: 8

📂 Output Structure

logs/
└── <experiment>/
    └── <run_name>_<timestamp>/
        ├── 📁 checkpoints/                    # Periodic checkpoints
        │   └── checkpoint-{step}/
        │       ├── ema/                        # EMA states (if enabled)
        │       ├── unwrapped_model/            # Model weights
        │       │   └── transformer/
        │       │       ├── adapter_config.json         # LoRA config (if LoRA)
        │       │       ├── adapter_model.safetensors   # LoRA weights (if LoRA)
        │       │       └── README.md                   # Model card
        │       ├── optimizer_0/                # Optimizer states (for resuming)
        │       ├── pytorch_model_fsdp_0/       # FSDP sharded model states
        │       ├── random_states_*.pkl         # Random states for each rank (for reproducibility)
        │       └── metadata.json               # Step & config metadata
        ├── 📁 final_model/                    # Final trained model
        │   └── transformer/
        │       ├── adapter_config.json         # LoRA config (if LoRA)
        │       └── adapter_model.safetensors   # LoRA weights (if LoRA)
        ├── 📁 eval_videos/                    # Evaluation videos
        └── 📁 sample_videos/                  # Training sample videos

🔑 Key Features at a Glance

🏛️ Model Zoo

We provide LoRA checkpoints fine-tuned with reinforcement learning (GRPO) on the Wan2.1-T2V-1.3B model. More checkpoints will be released soon.

📝 TODO

• Model support
  • Extend support for more text-to-image / image-to-image backbones beyond the current Wan / Hunyuan family
• Algorithmic extensions
  • Integrate more GRPO-family variants and related online RL algorithms
  • Add DPO / OnlineDPO, SFT / OnlineSFT style objectives alongside GRPO-style training
• Rollout & parallelism
  • Integrate LightX2V inference framework for accelerated rollout
  • Multi-level parallel rollout (e.g., SP, HSDP) for better hardware utilization
  • Asynchronous rollout workers with decoupled sampling/training pipelines
  • Improved multi-node orchestration utilities and monitoring for large-scale runs

🙏 Acknowledgements

GenRL is built upon the excellent work of the open-source community. We would like to thank:

• FlowGRPO — We reference their implementation for the GRPO-based algorithm and training framework.

📄 License

GenRL is licensed under the Apache License 2.0. See LICENSE.txt for the full license text.

📝 Citation

If you find GenRL useful in your research, please cite:

@misc{genrl,
  author = {GenRL Contributors},
  title = {GenRL: Reinforcement Learning Framework for Visual Generation},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ModelTC/GenRL}},
}