SF3D Docker ControlNet Pipeline

This project wraps Stable Fast 3D with an optional ControlNet style step.

Pipeline:

1. Read the first image from data/.
2. Remove the background with rembg.
3. Generate a styled 512x512 image with Canny ControlNet.
4. Run Stable Fast 3D and export a .glb file.

The project can run with either:

• a local fine-tuned ControlNet in my_custom_controlnet/
• the default lllyasviel/sd-controlnet-canny fallback

Files

• run.py: main image-to-GLB pipeline.
• compare.py: compares default ControlNet and local ControlNet.
• train.py: trains a local ControlNet from final_training_data/.
• download_data.py: downloads Objaverse models for dataset creation.
• blender_script.py: renders downloaded .glb files into training images.
• filter.py: filters rendered images with CLIP.
• preprocess_data.py: creates Canny conditioning pairs and train.jsonl.
• Dockerfile: CUDA runtime image for the main pipeline.

Large Files

The following are intentionally excluded from git:

• my_custom_controlnet/
• data/
• final_training_data/
• blender_bin/
• downloaded archives, videos, PDFs, and generated .glb files
• token.txt

If you want to use the fine-tuned ControlNet, place the model files locally:

my_custom_controlnet/
  config.json
  diffusion_pytorch_model.safetensors

Docker

Build:

docker build -t sf3d-controlnet .

Run on Windows PowerShell:

docker run --gpus all -it --rm `
  -v "${PWD}/data:/app/stable-fast-3d/data" `
  -v "${PWD}/my_custom_controlnet:/app/stable-fast-3d/my_custom_controlnet" `
  sf3d-controlnet

Inside the container:

python run.py

Outputs are written to:

data/output/

Hugging Face Access

Stable Fast 3D is a gated model. Before running inference, request access to:

stabilityai/stable-fast-3d

Then log in inside the environment:

huggingface-cli login

Do not commit Hugging Face tokens.

Training

The training flow is:

download_data.py -> blender_script.py -> filter.py -> preprocess_data.py -> train.py

The final dataset should look like:

final_training_data/
  images/
  conditioning_images/
  train.jsonl