RaCig is designed to generate images based on textual prompts and reference images for characters (referred to as "Characters"). It leverages several models and techniques, including:
RaCigPipeline) to orchestrate the generation process.The pipeline can handle multiple characters ("Characters") in a single scene, defined by their names, gender, and reference images (face and clothes).
Clone the repository:
git clone https://github.com/ZulutionAI/RaCig.git
cd RaCig
Install dependencies:
pip install -r requirements.txt
Download necessary models and retrieval datasets:
Models: https://huggingface.co/ZuluVision/RaCig
Put the models under checkpoint as follow:
./models/
βββ action_direction_dino/
β βββ checkpoint_best_regular.pth
βββ controlnet/
β βββ model.safetensors
βββ image_encoder/
β βββ config.json
β βββ model.safetensors
β βββ pytorch_model.bin
βββ ipa_weights/
β βββ ip-adapter-plus-face_sdxl_vit-h.bin
β βββ ip-adapter-plus_sdxl_vit-h.bin
βββ sdxl/
βββ dreamshaper.safetensors
Retrieval datasets: https://huggingface.co/datasets/ZuluVision/RaCig-Data
./data
βββ MSDBv2_v7
βββ Reelshot_retrieval
βββ retrieve_info
python inference.py
output/ directory by default.
Β·python run_gradio.py
For more detailed instruction, see Gradio Interface Instructions (EN) or Gradio Interface Instructions (δΈζ)
We only train the controlnet, to make it recognize the feature map better. (The fused feature map after injecting IP information is quite hard for controlnet to constrain the pose, so we slightly finetune the controlnet)
We use the retrieval dataset to finetune it. The dataset structure is organized as above.
bash train.sh
This project is based on the work of the following open-source projects and contributors:
7 commits
Python
78.6%
Cuda
19.2%
C++
2.0%
RaCig is designed to generate images based on textual prompts and reference images for characters (referred to as "Characters"). It leverages several models and techniques, including:
RaCigPipeline) to orchestrate the generation process.The pipeline can handle multiple characters ("Characters") in a single scene, defined by their names, gender, and reference images (face and clothes).
Clone the repository:
git clone https://github.com/ZulutionAI/RaCig.git
cd RaCig
Install dependencies:
pip install -r requirements.txt
Download necessary models and retrieval datasets:
Models: https://huggingface.co/ZuluVision/RaCig
Put the models under checkpoint as follow:
./models/
βββ action_direction_dino/
β βββ checkpoint_best_regular.pth
βββ controlnet/
β βββ model.safetensors
βββ image_encoder/
β βββ config.json
β βββ model.safetensors
β βββ pytorch_model.bin
βββ ipa_weights/
β βββ ip-adapter-plus-face_sdxl_vit-h.bin
β βββ ip-adapter-plus_sdxl_vit-h.bin
βββ sdxl/
βββ dreamshaper.safetensors
Retrieval datasets: https://huggingface.co/datasets/ZuluVision/RaCig-Data
./data
βββ MSDBv2_v7
βββ Reelshot_retrieval
βββ retrieve_info
python inference.py
output/ directory by default.
Β·python run_gradio.py
For more detailed instruction, see Gradio Interface Instructions (EN) or Gradio Interface Instructions (δΈζ)
We only train the controlnet, to make it recognize the feature map better. (The fused feature map after injecting IP information is quite hard for controlnet to constrain the pose, so we slightly finetune the controlnet)
We use the retrieval dataset to finetune it. The dataset structure is organized as above.
bash train.sh
This project is based on the work of the following open-source projects and contributors:
7 commits
Python
78.6%
Cuda
19.2%
C++
2.0%