MiniT2I: A Minimalist Baseline for Text-to-Image Generation

       

Official PyTorch/Diffusers re-implementation of MiniT2I.

MiniT2I is a simple direct-RGB text-to-image generator that trains a pixel-space MM-JiT denoiser with flow matching, conditioned on frozen FLAN-T5-Large text tokens. The recipe is intentionally plain: avoiding image tokenizers, cascaded generation, RL stages, and any auxiliary losses. Data used in training MiniT2I is fully public and easy to implement. For more details, please refer to our blog post.

This repository is a PyTorch reproduction codebase for MiniT2I, including:

• PyTorch/Diffusers inference for MiniT2I-B/16 and MiniT2I-L/16.
• LoRA adaptation code used for the downstream adaptation experiments.
• Full PyTorch training code for the ablation setting MiniT2I-B/32.
• GenEval, DPG-Bench, and FID evaluation entry points.

Table of Contents

• Model Zoo
• Repository Layout
• Installation
• Diffusers Inference & Colab Demo
  • Recommended Inference Settings
• LoRA Adaptation
• Evaluation
  • Evaluation Metrics
  • Evaluate Checkpoints
• Full Training
  • Dataset Preparation
  • Pretrain on CC12M
  • Fine-Tune on 120K mix
• Acknowledgments
• Citation

Model Zoo

The number in parentheses is the reproduction results by the JAX implementation.

For checkpoints, please refer to the Hugging Face repository -- it stores both MiniT2I-B/16 and MiniT2I-L/16 weights.

Repository Layout

.
├── main.py                     # unified train/eval entry point
├── configs/                    # pretraining, fine-tuning, and eval configs
├── mini_t2i/                    # PyTorch training and benchmark evaluation code
├── diffusers/                  # custom Diffusers pipeline source
├── lora/                       # LoRA adaptation entry points
└── tools/                      # dataset preparation helpers

Installation

Create a Python environment with a CUDA-enabled PyTorch build, then install the dependencies:

git clone <REPO_URL>
cd minit2i-torch

python -m pip install -r requirements.txt

For inference-only use, the core dependencies are torch, diffusers, transformers, safetensors, pillow, and huggingface_hub.

Edit the user settings at the top of mini_t2i/settings.py before training or evaluation:

DATA_ROOT = Path("/path/to/dataset/root")
OUTPUT_ROOT = Path("/path/to/output/root")
CHECKPOINT_ROOT = Path("/path/to/checkpoint/root")
EVAL_OUTPUT_ROOT = Path("/path/to/evaluation/output/root")
ASSET_ROOT = Path("/path/to/evaluation/assets")

HF_TOKEN_FILE = Path("/path/to/secrets/hf_token")
WANDB_API_KEY_FILE = Path("/path/to/secrets/wandb_api_key")

The rest of mini_t2i/settings.py derives benchmark, dataset, checkpoint, and output paths from these roots.

Diffusers Inference & Colab Demo

Try MiniT2I without setting up training data in the Colab demo!

import torch
from diffusers import DiffusionPipeline

HUB_MODEL_ID = "MiniT2I/MiniT2I"

pipe = DiffusionPipeline.from_pretrained(
    HUB_MODEL_ID,
    custom_pipeline=HUB_MODEL_ID,
    trust_remote_code=True,
)

image = pipe(
    "A lonely astronaut standing on a quiet beach under two moons.",
    model_type="b16",
    guidance_scale=2.5,
    num_inference_steps=100,
    torch_dtype=torch.bfloat16,
).images[0]
image.save("minit2i-b16.png")

image = pipe(
    "a transparent green backpack on a marble pedestal, with notebooks and a metal water bottle visible inside",
    model_type="l16",
    guidance_scale=6.0,
    num_inference_steps=100,
    torch_dtype=torch.bfloat16,
).images[0]
image.save("minit2i-l16.png")

Supported aliases include:

• b, b16, base, minit2i-b/16 (for using MiniT2I-B/16);
• l, l16, large, and minit2i-l/16 (for using MiniT2I-L/16).

Recommended Inference Settings

The following values are the defaults used by the Colab demo and the GenEval / DPG-Bench evaluation configs. They are reasonable starting points; CFG can be tuned per prompt.

LoRA Adaptation

The LoRA code attaches adapters to the attention projections, MLP projections, and text projection layers.

Here, we provide our default settings in the blog post: using rank-32 adapters on the Naruto BLIP-caption dataset as a domain-adaptation example.

python lora/train_lora.py \
  --pretrained_model_name_or_path MiniT2I/MiniT2I \
  --model_type b16 \
  --dataset_name lambdalabs/naruto-blip-captions \
  --output_dir lora_runs/naruto_lora \
  --train_batch_size 16 \
  --gradient_accumulation_steps 1 \
  --learning_rate 5e-4 \
  --rank 32 \
  --lora_alpha 32 \
  --mixed_precision bf16 \
  --loss_type velocity \
  --max_train_steps 400 \
  --validation_at_start \
  --validation_steps_interval 100 \
  --validation_prompt "anime portrait of a ninja with orange hair, high quality"

To sample a trained adapter:

python lora/sample_lora.py \
  --adapter_dir lora_runs/naruto_lora \
  --prompt "anime portrait of a ninja with orange hair, high quality" \
  --steps 100 \
  --cfg_scale 6.0 \
  --device cuda \
  --dtype bfloat16

For an end-to-end walkthrough of the same recipe, see the notebooks/minit2i_lora_naruto_finetune.ipynb notebook.

Evaluation

The evaluation pipeline supports GenEval and DPG-Bench prompt-following evaluation for the Hugging Face Diffusers checkpoints and local PyTorch checkpoints.

Evaluation Metrics

This repository does not vendor benchmark prompts, detector checkpoints, or VQA checkpoints. Download each benchmark asset separately and edit mini_t2i/settings.py.

GenEval. GenEval uses prompts and detector checkpoints from djghosh13/geneval. Follow that repository to download the prompt metadata and pretrained detector assets. With the default settings, place them as:

/path/to/evaluation/assets/geneval/evaluation_metadata.jsonl
/path/to/evaluation/assets/geneval/pretrained/

DPG-Bench. DPG-Bench uses the Jialuo21/DPG-Bench Hugging Face dataset and the mPLUG VQA checkpoint iic/mplug_visual-question-answering_coco_large_en. The Hugging Face dataset is stored as parquet with a test split, so the default config loads it directly with datasets instead of requiring a CSV download.

With the default settings, place the mPLUG checkpoint as:

/path/to/evaluation/assets/modelscope/iic/mplug_visual-question-answering_coco_large_en/

To use a local DPG-Bench file instead, set DPG_BENCH_DATA in mini_t2i/settings.py to a local CSV, parquet, JSON, or JSONL path.

Evaluate Checkpoints

Evaluate the Hugging Face Diffusers checkpoints directly from the Hub without downloading a local .pt file:

python main.py eval --config-file configs/eval_hf_b16.yml
python main.py eval --config-file configs/eval_hf_l16.yml

These configs use hf_model_id: MiniT2I/MiniT2I and model_type: b16 or model_type: l16. Edit configs/eval.yml only when evaluating a local PyTorch checkpoint:

checkpoint: /path/to/checkpoint.pt
checkpoint_target: ema
output_dir: /path/to/eval_output

Run all enabled benchmarks:

python main.py eval --config-file configs/eval.yml

Full Training

The PyTorch training interface is config-driven through main.py, configs/, and mini_t2i/.

We provide our default B/32 ablation settings in configs/pretrain.yml and configs/finetune.yml. For the training config of B/16 and L/16, please refer to our JAX code release.

Dataset Preparation

This repository does not vendor training datasets. Download the data separately, prepare it into the expected local layout, and edit mini_t2i/settings.py to point at your copies.

CC12M. Pretraining uses Conceptual Captions 12M data. Prepare local tensor chunks from CaptionEmporium/conceptual-captions-cc12m-llavanext:

python tools/prepare_cc12m_chunks.py \
  --dataset CaptionEmporium/conceptual-captions-cc12m-llavanext \
  --out /path/to/dataset/root/cc12m_tensor_chunks

With the default settings, the chunks should live at /path/to/dataset/root/cc12m_tensor_chunks.

For large runs, the public CC12M image URLs are often stale or access-denied. As an optional, more robust route, thanks to @kpetrovicc, you can let img2dataset handle parallel downloads and per-shard failure logging, then convert the downloaded WebDataset shards into the same chunk_*.pt tensor format:

DATA_ROOT=/path/to/dataset/root
WORK="$DATA_ROOT/cc12m_prep"

hf download CaptionEmporium/conceptual-captions-cc12m-llavanext \
  --repo-type dataset \
  --include "*.jsonl.gz" \
  --local-dir "$WORK/meta"

python tools/cc12m_jsonl_to_parquet.py \
  --jsonl "$WORK/meta/train.jsonl.gz" \
  --out "$WORK/parquet/metadata.parquet"

img2dataset \
  --url_list "$WORK/parquet" \
  --input_format parquet \
  --url_col url \
  --caption_col caption_llava \
  --output_folder "$WORK/wds" \
  --output_format webdataset \
  --image_size 512 \
  --resize_mode center_crop \
  --encode_format jpg \
  --processes_count 16 \
  --thread_count 64 \
  --number_sample_per_shard 10000 \
  --enable_wandb False \
  --retries 2

python tools/cc12m_wds_to_chunks.py \
  --wds "$WORK/wds" \
  --out "$DATA_ROOT/cc12m_tensor_chunks"

120K mix. Fine-tuning uses the 120K mix WebDataset layout assembled from three public sources:

• BLIP3o/BLIP3o-60k
• OpenDatasets/dalle-3-dataset
• FreedomIntelligence/ShareGPT-4o-Image

Stage the sources into the directory structure expected by configs/finetune.yml:

python tools/prepare_120k_mix_layout.py \
  --out /path/to/dataset/root/finetune_wds

• With the default settings, the WebDataset shards should live at /path/to/dataset/root/finetune_wds.
• The fine-tuning loader samples the three source groups with weights [0.06, 0.016, 0.04] for BLIP3o-60K, DALL-E3, and ShareGPT4o, roughly proportional to the number of samples.
• When local shards are absent and finetune_hf_streaming: true, the loader falls back to the public Hugging Face sources configured in TrainConfig; we recommend staging local WebDataset shards for speed.

Pretrain on CC12M

torchrun --standalone --nproc_per_node=8 \
  main.py train \
  --config-file configs/pretrain.yml \
  --output-dir "${OUT_DIR}" \
  --dataset-backend local_folder \
  --micro-batch-size 128 \
  --grad-accum-steps 1 \
  --attention-impl sdpa

MSCOCO-30K FID is the online monitoring metric used during pretraining. Download the caption file and reference statistics from MiniT2I/evaluation-assets:

huggingface-cli download MiniT2I/evaluation-assets \
  --repo-type dataset \
  --include "coco/*" \
  --local-dir /path/to/evaluation/assets

With the default settings, this command creates the expected coco/ asset directory. The repository FID implementation uses pytorch-fid's InceptionV3 feature extractor and compares generated-image features against mscoco_fid_stats_512.npz.

Pretraining enables online FID monitoring by default. configs/pretrain.yml sets:

fid_every: 20000
fid_num_samples: 30000
fid_batch_size: 64

The training loop evaluates both the raw model and the EMA model, then logs the FID metrics to W&B when enabled. Edit MSCOCO_CAPTION_FILE and MSCOCO_STATS_FILE in mini_t2i/settings.py before launching pretraining.

To run the same FID path manually for a locally trained pretrained checkpoint, enable the fid benchmark in configs/eval.yml:

checkpoint: /path/to/pretrained/checkpoint.pt
checkpoint_target: ema
output_dir: /path/to/eval_output

benchmarks:
  fid:
    enabled: true
    num_samples: 30000
    batch_size: 64

Fine-Tune on 120K mix

You can directly run the following script to reproduce our B/32 ablation-level results. Hyperparameters can be tuned at configs/finetune.yml.

torchrun --standalone --nproc_per_node=8 \
  main.py train \
  --config-file configs/finetune.yml \
  --resume-from "${PRETRAINED_CKPT}" \
  --no-auto-resume \
  --output-dir "${OUT_DIR}" \
  --micro-batch-size 64 \
  --grad-accum-steps 2 \
  --num-workers 8 \
  --attention-impl sdpa

Acknowledgments

This codebase builds on a number of open-source efforts:

• JAX MiniT2I -- the reference implementation that this PyTorch port mirrors.
• Hugging Face Diffusers and Transformers -- pipeline scaffolding and the FLAN-T5-Large text encoder.
• PEFT -- LoRA adapters used in lora/.
• djghosh13/geneval -- prompts and detector checkpoints for GenEval.
• Jialuo21/DPG-Bench and the mPLUG VQA model from ModelScope -- DPG-Bench evaluation.
• pytorch-fid -- InceptionV3 FID feature extraction.
• Public training data from CaptionEmporium/conceptual-captions-cc12m-llavanext, BLIP3o/BLIP3o-60k, OpenDatasets/dalle-3-dataset, FreedomIntelligence/ShareGPT-4o-Image, and lambdalabs/naruto-blip-captions.

Citation

@misc{minit2i2026,
  title  = {MiniT2I: A Minimalist Baseline for Text-to-Image Generation},
  author = {Wang, Xianbang and Zhao, Hanhong and Lu, Yiyang and Zhou, Kangyang and Ma, Linrui and He, Kaiming},
  year   = {2026},
  url    = {https://peppaking8.github.io/#/post/minit2i}
}