Emu3.5 ComfyUI Nodes

ComfyUI integration for BAAI's Emu3.5 multimodal models

✅ STATUS: ALL NODES FULLY WORKING - Verified December 19, 2025

Overview

This repository provides ComfyUI custom nodes for running BAAI's Emu3.5 models for:

• Text-to-Image (T2I) - Generate images from text descriptions ✅
• Image Editing (X2I) - Transform/edit existing images ✅
• Interleaved Generation (Story Mode) - Create illustrated stories with text and images ✅
• Visual Q&A - Answer questions about images, OCR, image comparison ✅

Workflow Screenshots

Models Supported:

• Emu3.5-Image (34B params - T2I/X2I) - ✅ Working
• Emu3.5-Base (65B params - Story/Interleaved/VQA) - ✅ Working
• Vision Tokenizer (VQ-VAE for image encoding/decoding) - ✅ Working

What's New (December 2025)

V2 Nodes Released! 🎉

• Emu 3.5 Loader V2 - Improved model loading with memory management
• Emu 3.5 T2I Sampler V2 - Text-to-image with tiled decoding for large images
• Emu 3.5 X2I (Image Edit) - Transform/edit images with text prompts
• Emu 3.5 Interleaved - Generate stories/tutorials with multiple images
• Emu 3.5 VQA - Visual question answering
• Emu 3.5 Memory Manager - VRAM management utilities

Credits & Attribution

This project is built upon:

• BAAI Emu3.5 - Original model and codebase
  • Paper: Emu3.5: Native Multimodal Models are World Learners
  • Authors: Emu3.5 Team, Beijing Academy of Artificial Intelligence
  • License: Apache 2.0

Development Contributors:

• Eric Rollei - ComfyUI node development and integration
• Claude Opus 4.5 (Anthropic) - AI pair programming assistant for debugging, compatibility fixes, and feature implementation

Technical Contributions:

• Transformers 4.57+ compatibility patches (DynamicCache API changes)
• Blackwell GPU (sm_120) eager attention workaround
• VQA task presets and multi-image comparison support
• Memory management and VRAM optimization

All model weights and architecture remain property of BAAI under Apache 2.0 license.

Installation

Prerequisites

• ComfyUI installed
• Python 3.10+
• CUDA-capable GPU:
  • Full BF16: 48GB+ VRAM (RTX A6000, RTX 6000 Ada/Blackwell)
  • NF4 Quantized: 24GB+ VRAM (RTX 4090, RTX A5000)
• 100GB+ disk space for model weights

Method 1: Git Clone (Recommended)

cd ComfyUI/custom_nodes
git clone --recursive https://github.com/EricRollei/Emu35-Comfyui-Nodes.git emu35
cd emu35

# Install dependencies
pip install -r requirements.txt

Note: The --recursive flag automatically clones the patched Emu3.5 submodule with transformers 4.57+ compatibility fixes.

Method 2: Manual Download

1. Download this repository
2. Extract to ComfyUI/custom_nodes/emu35/
3. Download Emu3.5 repo and place in emu35/Emu3_5_repo/
4. Install requirements: pip install -r requirements.txt

Download Model Weights

Place models in ComfyUI/models/emu35/:

Option A: Full BF16 (48GB+ VRAM - Best Quality)

huggingface-cli download BAAI/Emu3.5-Image --local-dir models/emu35/Emu3.5-Image
huggingface-cli download BAAI/Emu3.5-VisionTokenizer --local-dir models/emu35/vision_tokenizer

Option B: NF4 Quantized (24GB+ VRAM)

huggingface-cli download wikeeyang/Emu35-Image-NF4 --local-dir models/emu35/Emu3.5-Image-NF4
huggingface-cli download BAAI/Emu3.5-VisionTokenizer --local-dir models/emu35/vision_tokenizer

Directory structure:

ComfyUI/
├── custom_nodes/
│   └── emu35/
│       ├── nodes.py
│       ├── nodes_v2.py        # New V2 nodes
│       ├── __init__.py
│       ├── patched_tokenization_emu3.py
│       └── Emu3_5_repo/       # Official repo
└── models/
    └── emu35/
        ├── Emu3.5-Image/      # (or Emu3.5-Image-NF4/)
        └── vision_tokenizer/

Nodes Reference

V2 Nodes (Recommended)

1. Emu 3.5 Loader V2

Loads the Emu3.5 model with improved memory management.

---

2. Emu 3.5 T2I Sampler V2

Text-to-image generation with improved quality and tiled decoding.

---

3. Emu 3.5 X2I (Image Edit) ⭐ NEW

Transform or edit existing images based on text instructions.

Example Prompts for X2I:

• "Transform this image into a realistic photo"
• "Change the background to a beach sunset"
• "Add sunglasses to the person"
• "Replace the dog with a cat"
• With 2+ images: "Replace the [object] in first image with [object] from second image"

---

4. Emu 3.5 Interleaved (Story/HowTo)

Generate text with multiple embedded images (stories, tutorials).

---

5. Emu 3.5 VQA (Visual Question Answering)

Analyze images, answer questions, describe content, and read text (OCR).

⚠️ Important: Use the emu35-base model for VQA tasks, not emu35-image!

Task Types:

Example Questions for VQA:

Basic Understanding:

• "Describe this image in detail."
• "What is the main subject of this image?"
• "What colors are dominant in this image?"

Object Identification:

• "What objects are visible in this image?"
• "Is there a person in this image? What are they doing?"
• "What type of animal is in the photo?"

Counting & Spatial:

• "How many people are in this image?"
• "What is to the left of the car?"
• "Where is the sun in this scene?"

OCR (Text Reading):

• "What text appears in this image?"
• "Read the sign in the background."
• "What does the label say?"

Analysis & Style:

• "What artistic style is this image?"
• "What time of day is shown?"
• "What emotion does this scene convey?"
• "Is this image indoor or outdoor?"

Comparison (with 2 images):

• "What's different between these two images?"
• "Which image shows more people?"
• "Compare the lighting in both photos."

Tips for Best Results:

1. Use emu35-base model - The Image model is optimized for generation, not understanding
2. Lower temperature (0.1-0.3) for factual answers like counting, OCR, or identification
3. Higher temperature (0.5-0.7) for creative descriptions or artistic analysis
4. Be specific - "What brand is the laptop?" works better than "What's in the image?"
5. For OCR, use task_type="ocr" which has an optimized prompt
6. For comparisons, connect both images (image + image2)

---

6. Emu 3.5 Memory Manager

Utilities for VRAM management.

---

Legacy Nodes (V1)

The original nodes are still available for compatibility:

• Emu35Loader - Original loader
• Emu35Sampler - Original T2I sampler
• Emu35VQA - Original VQA node
• Emu35ClearCache - Cache clearing

Performance

Tested Configurations

Generation Times

Technical Details

Architecture

• Model Size: 34B parameters
• Training: 10T+ multimodal tokens
• Image Tokenization: VQ-VAE (IBQ) with 262,144 codebook
• Visual Tokens: Token IDs 151855-413998
• Max Resolution: 2048x2048 (128x128 latents)

Special Token IDs

BOS = 151849  # <|extra_203|> Begin generation
EOS = 151850  # <|extra_204|> End generation  
IMG = 151851  # <|image token|>
BOI = 151852  # <|image start|>
EOI = 151853  # <|image end|>
EOL = 151846  # <|extra_200|> End of line
VISUAL_START = 151854  # First visual token

Task Templates

T2I (Text-to-Image):

<|extra_203|>You are a helpful assistant for t2i task. USER: {prompt} ASSISTANT: <|extra_100|>

X2I (Image Edit):

<|extra_203|>You are a helpful assistant for x2i task. USER: <|IMAGE|>{prompt} ASSISTANT: <|extra_100|>

Known Issues & Solutions

1. SDPA Attention on Blackwell GPUs

Issue: SDPA attention produces noise/garbage on Blackwell (sm_120) with CUDA 12.8. Solution: Use attention_implementation="eager" (default in V2 loader).

2. Tokenizer Crashes

Issue: Missing visual tokens in tokenizer. Solution: Patched tokenizer auto-synthesizes missing tokens.

3. Out of Memory

Solutions:

• Use NF4 quantization (24GB VRAM)
• Reduce image_area in X2I node
• Use smaller aspect ratios
• Enable tiled decoding

4. X2I Generates Only 1 Token

Issue: Stopping criteria triggered by reference image tokens. Solution: Fixed in V2 - stopping criteria now ignores input tokens.

GPU Compatibility

Example Workflows

Basic T2I

[Emu 3.5 Loader V2] → [Emu 3.5 T2I Sampler V2] → [Preview Image]
                              ↑
                    prompt: "a red apple on a wooden table, 
                             studio lighting, photorealistic"

Image Editing (X2I)

[Load Image] → [Emu 3.5 X2I] → [Preview Image]
                    ↑
[Emu 3.5 Loader V2] 
                    ↑
          prompt: "Transform into an oil painting"

Multi-Image Edit

[Load Image 1] → 
                  [Emu 3.5 X2I] → [Preview Image]
[Load Image 2] →        ↑
                        prompt: "Replace the background 
                                 from image 1 with the 
                                 scene from image 2"

Development

File Structure

emu35/
├── nodes.py           # V1 nodes (legacy)
├── nodes_v2.py        # V2 nodes (recommended)
├── __init__.py        # ComfyUI registration
├── patched_tokenization_emu3.py  # Fixed tokenizer
├── download_nf4.py    # NF4 download helper
├── Emu3_5_repo/       # Official Emu3.5 code
└── dev/               # Development/test scripts

Testing

cd ComfyUI/custom_nodes/emu35/dev

# Test tokenizer
python test_tokenizer.py

# Test minimal generation
python test_minimal.py

Contributing

Contributions welcome! Priority areas:

• vLLM integration for faster generation
• Additional sampling strategies
• Workflow examples
• Documentation improvements

License

• This integration code: MIT License
• Emu3.5 models and code: Apache 2.0 (BAAI)

Citation

@article{emu3.5,
  title={Emu3.5: Native Multimodal Models are World Learners},
  author={Emu3.5 Team},
  journal={arXiv preprint arXiv:2510.26583},
  year={2024}
}

Links

• Emu3.5 Project: https://emu.world/
• Paper: https://arxiv.org/pdf/2510.26583
• Official Code: https://github.com/baaivision/Emu3.5
• Model Weights: https://huggingface.co/BAAI/Emu3.5-Image
• NF4 Quantized: https://huggingface.co/wikeeyang/Emu35-Image-NF4
• ComfyUI: https://github.com/comfyanonymous/ComfyUI