ComfyUI-AITocha-Vision

GPU-native, NSFW-friendly image captioner for ComfyUI.

A drop-in replacement for the GGUF-based JoyCaption nodes that loads the JoyCaption model directly via HuggingFace transformers + bitsandbytes 4-bit quantization — no llama-cpp-python, no CUDA-compile fragility, no silent CPU fallback.

---

Why this exists

The popular ComfyUI JoyCaption GGUF nodes (JC_GGUF_adv etc.) depend on llama-cpp-python built with CUDA support. On many RunPod / Vast / Docker base images that build is CPU-only by default, and when JoyCaption falls back to CPU it silently returns None. Downstream CLIPTextEncode then crashes with:

AttributeError: 'NoneType' object has no attribute 'replace'

This node loads the same JoyCaption model the standard PyTorch way — via transformers — so it:

• ✅ Runs entirely on GPU (CUDA via standard PyTorch, no compilation step)
• ✅ Works on any ComfyUI install with a recent transformers + bitsandbytes
• ✅ Persists across pod restarts via the standard HF_HOME cache
• ✅ Returns a real STRING even on edge cases (never None)
• ✅ Loads ~2× faster and generates ~3× faster than partial-offload llama-cpp on the same GPU

Install

Via ComfyUI Manager (recommended)

Search "AITocha Vision" in ComfyUI Manager → install → restart ComfyUI.

Manually

cd ComfyUI/custom_nodes
git clone https://github.com/AITocha/ComfyUI-AITocha-Vision.git
cd ComfyUI-AITocha-Vision
pip install -r requirements.txt

Restart ComfyUI.

Nodes

All under the AITocha/Vision category.

Auto Caption (AITocha — one node)

All-in-one. Drop in, connect an IMAGE, get a STRING caption back. The model is cached at module level — reusing the same node across multiple queues doesn't reload weights.

Load Caption Model (AITocha) + Generate Caption (AITocha)

Split version for graphs that caption many images per run. Load once, caption N times.

Inputs

Why "Scene only (no identity)"?

Built specifically for img2img + LoRA workflows where you bring your own identity tokens (a character LoRA trigger, a named subject) and only want the captioner to fill in pose, clothing, setting, lighting, composition. The model is instructed not to describe facial features, hair color, eye color, age, or any identifying physical traits.

This avoids the typical img2img problem of the captioner over-specifying details that conflict with your LoRA, while still injecting accurate scene context.

Models

The first run downloads the JoyCaption model from HuggingFace (~14 GB) into your HF_HOME cache (default ~/.cache/huggingface). For RunPod / Docker setups, set HF_HOME=/workspace/huggingface so it persists across pod restarts.

Example

Connect any LoadImage → Auto Caption (AITocha — one node) → CLIPTextEncode.

For SDXL img2img with character LoRA + auto-prompting:

LoadImage ──► AutoCaption ──► [identity prompt] + caption ──► CLIPTextEncode ──► KSampler
                  └─► (Scene only preset, no facial features described)

Credits

• The JoyCaption model itself is by @fpgaminer. All credit for the actual VLM goes to them.
• This node is just a thin transformers wrapper for ComfyUI.

License

Apache 2.0 — see LICENSE.