ComfyUI-Lance

Custom ComfyUI nodes for ByteDance Lance — a unified 3B multimodal model supporting image and video understanding, generation, and editing in a single framework.

A video I posted about unified multimodal, and this model : https://youtu.be/fvEeEW6trXA

Official sources

• Model weights: bytedance-research/Lance on Hugging Face
• Inference pipeline: bytedance/Lance (inference_lance.py, inference_lance.sh)

This pack bundles the upstream modeling/ code in lance_source/ and calls the same validate_on_fixed_batch path as the official CLI.

Features

Installation

1. Clone into ComfyUI custom nodes

cd ComfyUI/custom_nodes/
git clone https://github.com/benjiyaya/ComfyUI-Lance.git

2. Install dependencies

pip install -r ComfyUI-Lance/requirements.txt

3. Download model weights

Download the full Hugging Face repo (or individual folders) into ComfyUI/models/lance/.

The HF layout uses Lance_3B / Lance_3B_Video; the loader also accepts lance_3b / lance_3b_video (same as upstream downloads/ in the GitHub repo).

ComfyUI/models/lance/
├── Lance_3B/               # Image tasks (HF name) — or lance_3b/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Lance_3B_Video/         # Video tasks (HF name) — or lance_3b_video/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Qwen2.5-VL-ViT/         # Vision encoder
│   ├── vit.safetensors
│   └── config.json
└── Wan2.2_VAE.pth          # VAE weights (repo root on HF)

Upstream places the same files under downloads/; ComfyUI uses models/lance/ via folder_paths.

The Lance model source code is bundled in lance_source/ — no separate clone needed.

Advanced: Set LANCE_REPO_PATH env var to use a custom Lance repo instead of the bundled source.

Hardware Requirements

• GPU: NVIDIA GPU with ≥40GB VRAM (e.g., A100 40GB, A6000)
• CUDA: 12.4+
• Python: 3.10+

Usage

Basic Text-to-Video

[Lance Model Loader] → [Lance Text-to-Video] → [ComfyUI Save Video]

1. Load the model with lance_3b_video variant
2. Enter your prompt in the Text-to-Video node
3. Set resolution, frames, CFG, seed
4. Connect to ComfyUI's native Save Video to export as MP4

Image Understanding

[Load Image] → [Lance Image Understanding]

1. Load the model with lance_3b or lance_3b_video
2. Connect an image and type your question
3. Output is a text string

Video Edit / Video Understanding

[Load Video] → [Lance Video Edit] / [Lance Video Understanding]

Connect frame batches from ComfyUI's native Load Video node or VideoHelperSuite's Load Video. The node converts frames to a temp video file for Lance processing.

Key Parameters

Aligned with official inference defaults:

Task ↔ model variant (official)

Model Caching

The model is loaded once and cached per GPU. Multiple nodes can share the same loaded model without re-loading.

License

This ComfyUI wrapper is provided as-is. The Lance model is licensed under Apache 2.0 by ByteDance. See https://github.com/bytedance/Lance for details.