Official implementation of AV-GRPO: dataset, training, and inference.
Python
28
19 commits
updated Oct 5, 2026
AV-GRPO, to our knowledge, is the first GRPO framework designed for joint audio-video generation models. It enables full-parameter training or LoRA training of the 22B LTX-2.3 model on just 8 A800 GPUs.
cd /path/to/AV-GRPO/JavisDiT
pip install -r requirements/requirements-eval.txt
💡 All necessary pre-trained models will be automatically downloaded to
./checkpoints.
hf download google/gemma-3-12b-it-qat-q4_0-unquantized
hf download Lightricks/LTX-2.3
cd /path/to/AV-GRPO
pip install -r requirements.txt
Use a search command to find .../AV-GRPO/ in the file below and replace all occurrences with your actual file path. Make sure not to miss any.
| File | Description |
|---|---|
.../AV-GRPO/packages/ltx-trainer/configs/ltx2_av_lora.yaml | Training config |
Enter your WandB information at lines 367–381 of .../AV-GRPO/packages/ltx-trainer/src/ltx_trainer/trainer.py to log training status. Alternatively, you can comment out all WandB logging.
Our training dataset is already available at /AV-GRPO/Dataset/dataset.json. No separate download is required.
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python3 -m accelerate.commands.launch --config_file /path/to/AV-GRPO/packages/ltx-trainer/configs/accelerate/fsdp_compile.yaml train.py /path/to/AV-GRPO/packages/ltx-trainer/configs/ltx2_av_lora.yaml
⚠️ Note: All
/path/to/above should be replaced with your actual paths.
Full fine-tuning:
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python merge_full.py
LoRA:
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python merge_lora.py
huggingface-cli download --resume-download Dr-Loser/AV-GRPO AV-GRPO_FULL.safetensors
cd /path/to/LTX-2/packages/ltx-trainer/scripts
python3 inference.py \
--checkpoint CHECKPOINT_PATH \
--text-encoder-path TEXT_ENCODER_PATH \
--prompt "your prompt here" \
--output /path/to/output
| Prompt | LTX-2.3 | AV-GRPO FULL (ours) |
|---|---|---|
| There is a large building on fire with intense flames and a lot of smoke billowing out. The building is surrounded by water, and there are some boats visible. A loud explosion is heard, followed by the sound of fire crackling and burning. The sky is clear, and there are some buildings in the background. | ||
| A woman is playing the violin in an orchestra setting. She is wearing a black top and green pants. The background shows other musicians with various instruments, and there's a stained glass window behind them. The sound of the violin blends with the orchestra's accompaniment. | ||
| A small stream flows over rocks and grass, with the water clear and the rocks covered in moss. A cartoonish green character with a round body and a single eye jumps into the water, making a croaking sound. | ||
| A child pushes a toy truck, says \"Vroom,\" pulls it back, says \"Beep,\" then crashes it into a block and shouts \"Boom.\" | ||
| A colossal waterfall thunders down from mist-shrouded emerald cliffs into a sapphire abyss, its deafening crash echoing through the valley, sending up explosive plumes of spray that catch the morning light. | ||
| Two men face each other nose-to-nose in a tense confrontation within a dimly lit bank. The older man says, \"我们才不会害怕残忍的流氓.\" The clown replies, \"你知道吗？你让我想起了我的父亲，我恨我的父亲.\" The audio shows suffocating silence broken by tense dialogue. | ||
| In a medium close-up, a young woman with blonde shoulder-length hair stands in a lavender field under a twilight sky. She says, \"告诉我这条光滑的绿色带子见证了多少年的沉重。\" The audio features gentle, atmospheric singing establishing a calm and wistful mood. |
See individual submodule licenses (JavisDiT, LTX, etc.) for their terms.
107 followers · starred Oct 2026
Official implementation of AV-GRPO: dataset, training, and inference.
Python
28
19 commits
updated Oct 5, 2026
AV-GRPO, to our knowledge, is the first GRPO framework designed for joint audio-video generation models. It enables full-parameter training or LoRA training of the 22B LTX-2.3 model on just 8 A800 GPUs.
cd /path/to/AV-GRPO/JavisDiT
pip install -r requirements/requirements-eval.txt
💡 All necessary pre-trained models will be automatically downloaded to
./checkpoints.
hf download google/gemma-3-12b-it-qat-q4_0-unquantized
hf download Lightricks/LTX-2.3
cd /path/to/AV-GRPO
pip install -r requirements.txt
Use a search command to find .../AV-GRPO/ in the file below and replace all occurrences with your actual file path. Make sure not to miss any.
| File | Description |
|---|---|
.../AV-GRPO/packages/ltx-trainer/configs/ltx2_av_lora.yaml | Training config |
Enter your WandB information at lines 367–381 of .../AV-GRPO/packages/ltx-trainer/src/ltx_trainer/trainer.py to log training status. Alternatively, you can comment out all WandB logging.
Our training dataset is already available at /AV-GRPO/Dataset/dataset.json. No separate download is required.
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python3 -m accelerate.commands.launch --config_file /path/to/AV-GRPO/packages/ltx-trainer/configs/accelerate/fsdp_compile.yaml train.py /path/to/AV-GRPO/packages/ltx-trainer/configs/ltx2_av_lora.yaml
⚠️ Note: All
/path/to/above should be replaced with your actual paths.
Full fine-tuning:
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python merge_full.py
LoRA:
cd /path/to/AV-GRPO/packages/ltx-trainer/scripts
python merge_lora.py
huggingface-cli download --resume-download Dr-Loser/AV-GRPO AV-GRPO_FULL.safetensors
cd /path/to/LTX-2/packages/ltx-trainer/scripts
python3 inference.py \
--checkpoint CHECKPOINT_PATH \
--text-encoder-path TEXT_ENCODER_PATH \
--prompt "your prompt here" \
--output /path/to/output
| Prompt | LTX-2.3 | AV-GRPO FULL (ours) |
|---|---|---|
| There is a large building on fire with intense flames and a lot of smoke billowing out. The building is surrounded by water, and there are some boats visible. A loud explosion is heard, followed by the sound of fire crackling and burning. The sky is clear, and there are some buildings in the background. | ||
| A woman is playing the violin in an orchestra setting. She is wearing a black top and green pants. The background shows other musicians with various instruments, and there's a stained glass window behind them. The sound of the violin blends with the orchestra's accompaniment. | ||
| A small stream flows over rocks and grass, with the water clear and the rocks covered in moss. A cartoonish green character with a round body and a single eye jumps into the water, making a croaking sound. | ||
| A child pushes a toy truck, says \"Vroom,\" pulls it back, says \"Beep,\" then crashes it into a block and shouts \"Boom.\" | ||
| A colossal waterfall thunders down from mist-shrouded emerald cliffs into a sapphire abyss, its deafening crash echoing through the valley, sending up explosive plumes of spray that catch the morning light. | ||
| Two men face each other nose-to-nose in a tense confrontation within a dimly lit bank. The older man says, \"我们才不会害怕残忍的流氓.\" The clown replies, \"你知道吗？你让我想起了我的父亲，我恨我的父亲.\" The audio shows suffocating silence broken by tense dialogue. | ||
| In a medium close-up, a young woman with blonde shoulder-length hair stands in a lavender field under a twilight sky. She says, \"告诉我这条光滑的绿色带子见证了多少年的沉重。\" The audio features gentle, atmospheric singing establishing a calm and wistful mood. |
See individual submodule licenses (JavisDiT, LTX, etc.) for their terms.
107 followers · starred Oct 2026