anime-character-lcm-lora

高速なアニメキャラクター生成用の LCM-LoRA アダプター

This repository provides a LoRA adapter fine-tuned from Stable Diffusion v1.5 using LCM (Latent Consistency Models).

This repository contains LoRA adapter weights only. The base model must be loaded separately.

📊 Performance

🎨 Model Overview

This LCM-accelerated LoRA specializes in anime character generation:

• ✅ High-quality anime faces and character designs
• ✅ Fast inference (LCM 6-step generation)
• ✅ Compatible with Stable Diffusion v1.5
• ✅ Robust prompts (Gao et al. 2306.13103)

🎓 Research Foundation

This model is based on:

1. LCM-LoRA (Luo et al. 2311.05556)

   • Latent Consistency Models for acceleration
   • Skip-step training with Augmented PF-ODE

2. RobustPromptGenerator (Gao et al. 2306.13103)

   • Multi-redundant prompt generation
   • Typography robustness

3. ControlNet Integration (Zhang et al. 2302.05543)

   • Sketch-to-image pipeline support
   • Lineart mode for anime sketches

💻 Usage

Installation

pip install diffusers transformers torch peft

Basic Generation

from diffusers import StableDiffusionPipeline
from peft import PeftModel
import torch

# Load base model
base_model_id = "runwayml/stable-diffusion-v1-5"
adapter_id = "Shion1124/anime-character-lcm-lora"

pipe = StableDiffusionPipeline.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LCM-LoRA adapter
pipe.unet = PeftModel.from_pretrained(
    pipe.unet,
    adapter_id,
    adapter_name="lcm"
)

# Enable LCMScheduler for fast inference
from diffusers import LCMScheduler
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# Generate
prompt = "anime girl, beautiful face, masterpiece, best quality"
image = pipe(
    prompt=prompt,
    num_inference_steps=6,     # LCM: 4-8 steps recommended
    guidance_scale=1.5,        # LCM optimal value
    height=512,
    width=512
).images[0]

image.save("output.png")

With RobustPromptGenerator

# (See: https://github.com/Shion1124/anime-character-generator)
from prompt_optimizer_v2 import RobustPromptGenerator

optimizer = RobustPromptGenerator(use_google_api=True)
robust_prompt = optimizer.optimize_prompt(
    request="happy anime girl with long hair",
    mode="lcm_controlnet"
)

image = pipe(
    prompt=robust_prompt['prompt_variants'][0],
    num_inference_steps=6,
    guidance_scale=1.5
).images[0]

With ControlNet (Sketch-to-Image)

from diffusers import (
    StableDiffusionControlNetPipeline,
    ControlNetModel, LCMScheduler
)

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-lineart"
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LCM + anime LoRA
pipe.load_lora_weights(adapter_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# Generate from sketch
image = pipe(
    prompt="anime girl, masterpiece",
    image=sketch_image,
    num_inference_steps=6,
    guidance_scale=1.5,
    controlnet_conditioning_scale=0.8
).images[0]

📚 Training Details

Base Model

• Model: Stable Diffusion v1.5
• Framework: Diffusers library

LoRA Configuration

• Rank (r): 64
• Alpha: 32
• LCM Integration: LCM-LoRA (latent-consistency/lcm-lora-sdv1-5)

Acceleration

• Method: LCM (Latent Consistency Models)
• Skip-step Training: Multi-step consistency
• Augmented PF-ODE: Guidance encoding

Speed Optimization

• Inference Steps: 6 (vs 20 standard)
• Device: GPU (T4 or better)
• Data Type: float16 (for memory efficiency)

📖 Architecture

Input Prompt (Text)
    ↓
[Layer 1: CLIP Text Encoder]
    ↓
[Layer 2: SD v1.5 UNet + Anime LoRA]
    ↓
[Layer 3: LCM-LoRA Scheduler]
    ↓
Output Image (512×512)

Total Time: ~1.3 seconds per image

⚖️ License & Attribution

License

• LoRA Model: MIT License
• Base Model: Stable Diffusion v1.5 (OpenRAIL M License)

Citation

If you use this model in research, please cite:

@article{luo2023lcm,
  title={LCM-LoRA: A Universal Stable-Diffusion Acceleration Module},
  author={Luo, Simian and Sun, Yiqin and Kang, Longxiang and ...},
  journal={arXiv preprint arXiv:2311.05556},
  year={2023}
}

@article{gao2024robustness,
  title={Evaluating Robustness of Text-to-Image Models},
  author={Gao, Yiming and Chen, ...},
  journal={arXiv preprint arXiv:2306.13103},
  year={2024}
}

Acknowledgments

• Base Model: Hugging Face Stable Diffusion v1.5
• Acceleration: Latent Consistency Models (Luo et al.)
• Robustness: Text-to-Image Robustness (Gao et al.)
• ControlNet: Sketch Control (Zhang et al.)

📝 Notes

• This is a LoRA adapter and requires the base model
• Works with any anime-friendly prompt
• Can be combined with other LoRAs using set_adapters()
• GPU-based inference recommended for best performance

🐛 Issues & Feedback

If you encounter issues:

1. Check base model install: pip install diffusers
2. Verify GPU memory: 6GB+ recommended
3. Try adjusting num_inference_steps (4-8)

---

Created: 2026-03-05 13:44:11 Model Type: LoRA Adapter (Anime Character Generation) Framework: Diffusers + PEFT Base: Stable Diffusion v1.5