Transformers Continuous Batching Scripts

GPU inference scripts using transformers' native continuous batching (CB). No vLLM dependency required.

Why transformers CB?

• Instant new model support - works with any model supported by transformers, including newly released architectures. No waiting for vLLM to add support.
• No dependency headaches - no vLLM, flashinfer, or custom wheel indexes. Just transformers + accelerate.
• Simple HF Jobs setup - no Docker image needed. Just hf jobs uv run.
• ~95% of vLLM throughput - uses PagedAttention and continuous scheduling for near-vLLM performance.

Available Scripts

generate-responses.py

Generate responses for prompts in a dataset. Supports chat messages and plain text prompts.

Quick Start

# Local (requires GPU)
uv run generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --prompt-column question

# HF Jobs (single GPU)
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --prompt-column question \
    --max-tokens 1024

# HF Jobs (multi-GPU for larger models)
hf jobs uv run --flavor l4x4 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --model-id Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --messages-column messages \
    --max-batch-tokens 2048 \
    --max-tokens 4096

Example with SmolTalk2

# Generate responses for SmolTalk2 chat data
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    HuggingFaceTB/smoltalk2 username/smoltalk2-responses \
    --subset SFT \
    --split OpenHermes_2.5_no_think \
    --messages-column messages \
    --max-tokens 256

Parameters

Tuning --max-batch-tokens

This is the key performance parameter. It controls how many tokens the continuous batching scheduler processes per step:

• Too low (e.g., 128): GPU underutilized, slow throughput
• Too high (e.g., 8192): May cause out-of-memory errors
• Default 512: Conservative, works on most GPUs
• Recommended for A100/H100: 2048-4096
• Recommended for L4: 512-1024

If you hit OOM errors, reduce this value or switch to --dtype float16.

Current Limitations

• Single GPU only - device_map="auto" (pipeline parallelism) doesn't work with CB's PagedAttention cache. Transformers does have tensor parallelism (tp_plan="auto") for supported models, but it requires torchrun and is undocumented with CB. For now, use a model that fits on one GPU (e.g., 8B in bf16 on A10G/L4 with 24GB).
• Text-only - no vision-language model support yet.

When to use this vs vLLM

Rule of thumb: Use transformers CB when you want simplicity and broad model support. Use vLLM when you need maximum throughput with well-supported models.