RS Nodes for ComfyUI

A comprehensive custom node pack for ComfyUI focused on LTXV audio-video generation, LoRA training, image generation, prompt engineering, and post-processing. Built for real-world production workflows with emphasis on VRAM efficiency and quality control.

---

Table of Contents

• Installation
• Node Reference
  • Video Generation
    • RS LTXV Generate
    • RS LTXV Extend
    • RS LTXV Upscale
  • Image Generation
    • RS Flux2 Generate
    • RS Z-Image Generate
  • LoRA Training
    • RS LTXV Prepare Dataset
    • RS LTXV Train LoRA
  • Guidance & Control
    • RS IC-LoRA Guider
    • RS LTXV TTM Guider
    • RS Canny Preprocessor
  • Prompt Engineering
    • RS Prompt Parser
    • RS Prompt Formatter
    • RS Prompt Formatter Local
  • Audio
    • RS Audio Concat
    • RS Audio Save
    • RS MOSS TTS Loader (optional)
    • RS MOSS TTS Batch Save (optional)
  • Post-Processing & Utilities
    • RS Film Grain
    • RS Video Trim
    • RS Video Save (ProRes)
    • RS Frame Splitter
    • RS Frame Collector
    • RS Free VRAM
    • RS Counter
    • RS Sigma Scheduler
• Workflow Examples
• Tips & Troubleshooting

---

Installation

cd ComfyUI/custom_nodes
git clone https://github.com/richservo/rs-nodes.git
cd rs-nodes
install.bat

The install script handles everything: initializes the LTX-2 submodule (for LoRA training) and installs all Python dependencies.

Linux:

./install.sh

External Dependencies

• Ollama — Required for RS Prompt Formatter and dataset captioning (RS LTXV Prepare Dataset). Install and run locally. The nodes auto-pull models on first use.
• LTXV Models — The LTXV generation/extension nodes require LTXV model checkpoints and VAEs loaded through ComfyUI's standard model loading nodes.

---

Node Reference

All nodes appear under the RS Nodes category in ComfyUI's node menu.

---

Video Generation

RS LTXV Generate

All-in-one LTXV video generation with optional audio, keyframe injection, multimodal guidance, and integrated 2x latent upscaling.

Required Inputs:

Optional Inputs — Generation:

Optional Inputs — Frame Injection:

Optional Inputs — Audio (AV Dual-Tower):

Optional Inputs — Efficiency:

Optional Inputs — Upscale (2x Latent):

Outputs:

---

RS LTXV Extend

Extend an existing video with seamless temporal continuation using overlap blending.

Required Inputs:

Optional Inputs:

Also accepts the same Audio, Efficiency, Upscale, and Scheduler inputs as RS LTXV Generate.

Outputs:

---

RS LTXV Upscale

Standalone 2x video upscaler with optional temporal upscaling and first-frame I2V re-diffusion.

Required Inputs:

Optional Inputs:

Outputs:

Key Behaviors:

• Encodes input, applies spatial 2x via upscale model, re-diffuses with first-frame I2V guidance
• Optional temporal 2x increases frame count before spatial upscaling
• Tiled encoding/decoding for memory efficiency

---

Image Generation

RS Flux2 Generate

All-in-one Flux2.dev image generation with Kontext reference images, empirical mu scheduling, and optional two-pass latent upscaling.

Required Inputs:

Optional Inputs:

Outputs:

Key Behaviors:

• Kontext reference images auto-scaled to optimal aspect ratios and appended to conditioning
• Empirical mu scheduling based on image dimensions
• Optional two-pass: half-res first pass, then full-res re-diffusion

---

RS Z-Image Generate

All-in-one Z-Image Turbo generation with Qwen3-4B text encoder, RenormCFG, and system prompt support.

Required Inputs:

Optional Inputs:

Outputs:

---

LoRA Training

RS LTXV Prepare Dataset

Scans a folder of videos/images, optionally detects and crops faces, identifies speakers by voice, generates captions via Ollama with woven-in dialogue, and preprocesses latents for LoRA training.

Required Inputs:

Optional Inputs — Captioning:

Optional Inputs — Face Detection & Character ID:

Optional Inputs — Speech Transcription & Voice Attribution:

Optional Inputs — Other:

Outputs:

Key Behaviors:

Extraction & Selection:

• Two-phase processing: clip generation → latent preprocessing
• Face detection via InsightFace (SCRFD + ArcFace, antelopev2 model pack)
• Multi-character mode: drop face refs in character_refs_folder — auto-identifies cast per clip, balances quotas across characters
• Location mode: separate folder for set/location references — Gemma matches location per clip
• Pan & scan: face-aware cropping at any aspect ratio while keeping the face in frame
• Frame-count safety: clip extraction always overshoots target frame count and trims, so clips never come up short of the bucket minimum
• Incremental & resumable: per-clip atomic saves — interrupted runs continue from where they stopped; transcripts and conditions persist as they're generated
• Quarantine, never delete: clips that fail (hallucination, no usable speech, no matching characters, etc.) move to <output_dir>/rejected_clips/<reason>/ instead of being deleted, so false positives can be reviewed and restored

Speech Transcription:

• Demucs vocal isolation runs first to strip music/SFX before Whisper transcribes
• Default model is large-v3 for best accuracy on character voices, accents, and unusual vocabulary; configurable down to tiny for quick tests
• Silent clips get an empty transcript "" (recorded once, never re-attempted on subsequent runs)

Voice Attribution (when voice_refs_folder is set):

• Each character's voice reference is embedded with speechbrain ECAPA-TDNN (192-d L2-normalized vector, downloaded automatically — no HuggingFace token needed)
• For each Whisper segment in a clip, ECAPA produces a per-segment embedding that's matched against enrolled voice prints by cosine distance
• Face-detection hint: the on-screen character (from face detection) gets a small distance bonus, biasing ambiguous matches toward the visible speaker without overriding strong off-screen voice evidence
• Each segment is tagged with the matched character (or unknown for short utterances / unenrolled voices)
• Tagged segments are passed to the captioner as a DIALOGUE CONTEXT block, and Gemma weaves the lines naturally into the caption ("X stands by the window and says, '...'") instead of appending dialogue as a separate field

Caption Encoding (in-process when clip is connected):

• Captions stay clean in dataset.json (visual prose only — dialogue is woven in by the captioner)
• At text-encode time, captions are normalized: quote-wrapping single quotes are stripped (so 'pee-wee' becomes pee-wee without affecting Pee-wee's or it's); known character names are case-normalized using the canonical trigger (so a stray Cowboy curd is from a Whisper mistranscription still encodes as Cowboy Curtis)
• Original captions in dataset.json are never mutated — the normalization only affects the encoded .pt tensors

VRAM Management:

• All prepper-loaded models (Whisper, Demucs, ECAPA-TDNN, InsightFace) are unloaded between phases, with a final unload sweep before prepare() returns so unattended workflows that follow with training start with a clean GPU

Voice Reference Files:

• Format: any audio (.wav, .mp3, .flac, .m4a, .ogg, .opus) or video (.mp4, .mkv, .mov, .avi, .webm, .m4v) — audio is auto-extracted from video via ffmpeg
• Naming: filename stem must match the character_refs_folder entry (e.g. cowboy curtis.wav matches cowboy curtis.jpg)
• Length: 10-30 seconds of clean speech is the sweet spot; under 5 seconds risks unstable embeddings
• Variety helps: different sentences/intonations beat one repeated phrase. Concatenating short clips of the same character with ffmpeg works fine — each character can be built from 4-6 short single-speaker clips spliced together
• Demucs runs on enrollment too, so background music/effects in your reference clip get stripped automatically

---

RS LTXV Train LoRA

In-process LoRA training for LTX-2, reusing ComfyUI's already-loaded transformer. Includes a standalone training monitor that opens in a separate browser tab for full-size, resizable loss charts.

Required Inputs:

Optional Inputs:

Outputs:

Key Behaviors:

• In-process: reuses the loaded 22B transformer — no reload, no double memory
• Training monitor: click "Open Training Monitor" on the node to open a full-page loss chart in a new browser tab. Shows raw loss dots, EMA-smoothed line, color-coded trend line, step timing, and divergence status. Loads history from loss_history.json on refresh.
• ROSE optimizer: stateless optimizer — no momentum buffers, lower memory. ROSE-specific settings auto-hide when another optimizer is selected.
• Divergence detection: monitors EMA loss vs minimum. If loss rises above threshold% for too long, saves a checkpoint and attempts LR reset. If unrecoverable, stops training and rewinds to the pre-divergence checkpoint.
• LR cycle decay: progressive learning rate reduction across scheduler cycles (e.g., 0.9 = 10% reduction per cycle)
• Loss history persistence: saves loss_history.json every 50 steps for chart continuity across resume/restart
• Layer offloading: streams transformer blocks CPU↔GPU one at a time (~0.5 GB VRAM instead of ~11 GB)
• FP8 quantization: reduces model memory while preserving LoRA weights in float
• Resume support: restores optimizer state, RNG, loss history, and rebuilds LR schedule
• Presets: auto-configure module toggles and rank for common use cases (subject, style, motion)
• Quantization modifies the transformer in-place — reload checkpoint after training

---

Guidance & Control

RS IC-LoRA Guider

Structural control for LTXV using IC-LoRA (In-Context LoRA) with preprocessed control images.

Required Inputs:

Outputs:

---

RS LTXV TTM Guider

Time-to-Move (TTM) motion control guider for LTXV. Enforces reference motion via dual-clock denoising with mask-based control.

Required Inputs:

Optional Inputs:

Outputs:

---

RS Canny Preprocessor

Canny edge detection with automatic LTXV-safe resolution (128-aligned).

Inputs:

Outputs:

---

Prompt Engineering

RS Prompt Parser

Parse structured dialogue scripts with [s]tyle, [a]ction, and [d]ialogue tags into separate video and audio prompts.

Inputs:

Script Format:

[s] cinematic, natural lighting, handheld camera
[a] a man walks into a room and sits down
[d] Hello, how are you doing today?

Outputs:

---

RS Prompt Formatter

AI-powered prompt enhancement using a local Ollama model with reference image support and output caching.

Inputs:

Outputs:

Key Behaviors:

• Streams responses from Ollama with live token printing
• Auto-pulls missing models, strips <think> blocks
• Caches prompt + output as JSON — skips Ollama when input unchanged

Requires: Ollama running locally.

---

RS Prompt Formatter Local

Ollama-free prompt formatter reusing Gemma3 12B from ComfyUI's DualCLIPLoader. Supports optional reference images via vision embeddings.

Inputs:

Outputs:

Key Behaviors:

• No Ollama dependency — loads Gemma3 12B weights directly
• Vision support for up to 3 reference images
• JSON caching (text-only prompts)

---

Audio

RS Audio Concat

Concatenate up to 20 audio clips with per-clip trimming and configurable pauses.

Inputs: For each clip: audio file, start/end trim, pause after.

Outputs:

---

RS Audio Save

Export audio to disk with format selection (wav, flac, mp3, ogg).

---

RS MOSS TTS Loader

(Optional — requires transformers)

Load MOSS-TTS model variants for text-to-speech generation.

---

RS MOSS TTS Batch Save

(Optional — requires transformers)

Generate TTS audio from a dialogue list with automatic segmentation, per-clip trimming, and batch export. Supports one_shot, all, and single generation modes with Whisper-based word alignment.

---

Post-Processing & Utilities

RS Film Grain

Add realistic film grain with color variation and luminance-aware highlight protection.

Inputs:

Outputs:

---

RS Video Trim

Trim video frames and/or audio by time range.

Inputs:

Outputs:

---

RS Free VRAM

Passthrough utility that forces VRAM cleanup between pipeline stages.

Unloads all models, runs garbage collection, clears CUDA cache.

---

RS Counter

Persistent incrementing counter. State stored in counter_state.json across workflow executions.

Inputs:

Outputs:

---

Workflow Examples

Basic LTXV Video Generation

[Load LTXV Model] ──┐
[CLIP Text Encode] ──┼──→ [RS LTXV Generate] ──→ [Save Video]
[CLIP Text Encode] ──┤
[Load VAE] ──────────┘

LoRA Training Pipeline

[Load LTXV Model] ──┐
[Load VAE] ──────────┼──→ [RS LTXV Prepare Dataset] ──→ [RS LTXV Train LoRA]
[Load CLIP] ─────────┘                                         │
                                                    (click "Open Training Monitor")

Audio-Driven Video with TTS

[RS Prompt Parser] ──→ [RS Prompt Formatter] ──→ [CLIP Text Encode] ──┐
        │                                                              │
        └──→ [RS MOSS TTS Loader] ──→ [RS MOSS TTS Batch Save] ──┐    │
                                              │                    │    │
                                      [RS Audio Concat] ──────────┼────┼──→ [RS LTXV Generate]

IC-LoRA Structural Control

[Load Image] ──→ [RS Canny Preprocessor] ──→ [RS IC-LoRA Guider] ──→ [RS LTXV Generate]

Video Extension + Upscale

[RS LTXV Generate] ──→ [RS LTXV Extend] ──→ [RS LTXV Upscale] ──→ [Save Video]

---

Tips & Troubleshooting

VRAM Management

• Place RS Free VRAM between heavy inference nodes and post-processing to reclaim GPU memory.
• Use ffn_chunks (default: 4) and video_attn_scale (default: 1.03) to reduce VRAM during generation.
• Enable upscale_tiling for temporal tiling during upscale on long videos.
• LoRA training uses layer offloading to stream blocks one at a time (~0.5 GB instead of ~11 GB).

Generation Quality

• cfg: 2.5–4.0 works well for video; higher values can cause artifacts.
• audio_cfg: 5.0–9.0 is typical for audio.
• stg_scale: Start at 0. Small values (0.1–0.5) can improve temporal consistency.
• rescale: 0.7 is a good default.
• Upscaling: upscale_denoise=0.3–0.6 balances sharpness vs. faithfulness.

LoRA Training

• Use fp8-quanto quantization (no C++ build tools needed).
• Subject LoRAs: enable self-attention + cross-attention. Captions should describe everything except the subject.
• Style LoRAs: enable self-attention + feed-forward. Captions should describe the visual style in detail.
• ROSE optimizer: stateless, lower memory than AdamW. LR may need to be ~2x higher than AdamW (model-dependent — experiment).
• Divergence detection: uses EMA loss distance from minimum. Threshold of 15% is a good default. Auto-saves checkpoint at detection, attempts LR reset, stops and rewinds if unrecoverable.
• Resume: restores optimizer state, loss history, and LR schedule. The training monitor loads history from loss_history.json automatically.
• Training Monitor: open via the link on the training node. Full-page chart with raw loss dots, EMA-smoothed line, trend line (green=decreasing, red=increasing), step timing, and divergence status. Resizes with the browser window.

Prompt Workflow

• Write scripts with [s], [a], [d] tags and feed into RS Prompt Parser.
• RS Prompt Formatter caches to JSON — if the input prompt hasn't changed, Ollama is skipped.
• RS Prompt Formatter Local avoids Ollama entirely by loading Gemma3 12B directly.

---

License

See LICENSE for details.