StereoPilot-lite Replica: FullFT + Inference Acceleration

Version status: This branch is v3 / current recommended release. It is the preferred entry point for group members who want to reproduce the latest training pipeline and inference acceleration workflow.

中文说明请见 README.zh-CN.md.

This branch, FullFT-InferAccel-v3, is a public-safe research framework for StereoPilot-lite. It combines the latest full-parameter training replication pipeline with the latest inference acceleration tooling.

The repository contains code, configuration templates, runbooks, and placeholder directories only. It does not include model weights, prepared datasets, caches, checkpoints, exported safetensors, raw movies, or generated videos.

Repository Layout

External Artifacts Required

This repository does not ship model weights. Download the official checkpoints from their original sources and place them under StereoPilot/ckpt/.

Recommended download commands:

pip install "huggingface_hub[cli]"
huggingface-cli download KlingTeam/StereoPilot StereoPilot.safetensors --local-dir StereoPilot/ckpt
huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir StereoPilot/ckpt/Wan2.1-T2V-1.3B

Do not commit downloaded checkpoints to this repository.

Prepared training data should follow this layout:

data/<dataset_name>/
  left/
    clip_000001.mp4
  right/
    clip_000001.mp4
    clip_000001.txt
    cache/wan_stereo/cache_<bucket>/

Processed Dataset Release

Prepared after-caption / before-cache data is available from the shared Google Drive release folder. The Drive folder is authoritative for archive names, checksums, layouts, and QA scope. See docs/model_and_data_downloads.md for full download and extraction instructions.

Expanded 3D Movie Release

The current converged-stereo 3D Movie release contains 31,251 captioned clips in three immutable .tar.zst archives: Batch1 (6,858), Batch2 (6,242), and Batch3 (18,151). In the shared Drive folder, open 3dmovie_release_all_31251clips/; it provides:

• release_manifest.json: archive names, SHA256 values, clip counts, layouts, and QA scope.
• README.md: verification, extraction, QA, cache, and training instructions.
• wan_stereo_dataset_3dmovie_all_31251.toml: a seven-directory dataset template for the current release layout.

Download all three archives and their .sha256 sidecars, verify them, and extract them under a common data parent without flattening left/ or right/ directories. The archives do not include cache files; generate cache locally for the bucket used by the selected training config. The existing fixed-seed 5k/5k runbook remains a smaller recipe and does not automatically consume the expanded 31,251-clip release.

Captioner Source Note

The released data packages already include right/*.txt captions, so normal training reproduction does not need to run the captioner again. If you add new videos and want to regenerate captions, use the repo's ShareCaptioner tools with these external upstream assets:

These external code/model assets are not bundled in this repository. See diffusion-pipe-stereo/docs/sharecaptioner_captioning.md and envs/share4video_caption_optional_5090.yml if you want to run caption generation yourself.

Training Replicate

The current training replicate targets a full-parameter StereoPilot-style pipeline rather than LoRA. It includes:

• ShareCaptioner-based video captions.
• Stereo4D parallel + 3D Movie converged mixed-domain training.
• stereo_loss_mode = 'cycle_recon'.
• cycle_loss_weight = 0.5.
• cycle_detach = false.
• Fixed-seed sample_manifest filtering for reproducible balanced subsets.
• Balanced 5000 Stereo4D + 5000 3D Movie training.
• Public default bucket is 832x480x81; 512x288x81 is kept as a fast variant.
• gradient_accumulation_steps = 16, giving effective batch size 16 on one GPU.
• 10 epochs, with checkpoints saved every 2 epochs.
• Export to official StereoPilot-compatible safetensors.

Start with:

docs/INSTALL.md
docs/training_replicate_runbook.md
docs/data_layout.md
docs/model_and_data_downloads.md

Key example configs:

# Public default, official-aligned bucket
diffusion-pipe-stereo/examples/wan_stereo_dataset.balanced5k_832_ga16.toml
diffusion-pipe-stereo/examples/wan_stereo_fullft_balanced5k_832_ga16_10epoch.toml

# Faster low-cost variant
diffusion-pipe-stereo/examples/wan_stereo_dataset.balanced5k_512_ga16.toml
diffusion-pipe-stereo/examples/wan_stereo_fullft_balanced5k_512_ga16_10epoch.toml

Inference Acceleration

The current practical acceleration path is:

SageAttention + BF16 VAE + channels_last_3d + torch.compile(encoder_decoder)

The inference overlay supports comparisons among:

• Original native StereoPilot inference.
• SageAttention-only transformer acceleration.
• BF16 Wan VAE.
• BF16 Wan VAE + channels_last_3d.
• BF16 Wan VAE + channels_last_3d + VAE encoder/decoder compile.

The evaluation tools record:

• Stage timing: preprocess, VAE encode, text encode, transformer forward, VAE decode, save video.
• End-to-end latency.
• Peak VRAM.
• Optimized-vs-original output fidelity as an acceleration guardrail.
• Output-vs-GT metrics when a right-eye ground truth video is available.
• Multi-sample manifests and summaries.

Start with:

docs/INSTALL.md
docs/inference_acceleration_runbook.md
docs/attention_and_vae_accel_notes.md

Key tools:

inference_accel/tools/attention_backend_patch.py
inference_accel/tools/benchmark_inference_vae_dtype.py
inference_accel/tools/run_vae_accel_multisample_eval.py
inference_accel/tools/compare_video_outputs.py
inference_accel/tools/profile_stereopilot_inference.py
inference_accel/tools/profile_wan_vae_internal.py

What Is Not Included

The following artifacts are intentionally excluded from Git:

*.safetensors
*.pt / *.pth / *.ckpt
training videos (*.mp4 / *.mkv)
cache/
runs/
caption_runs/
accel_outputs/
real files under stereopilot_exports/
raw movie files
prepared datasets
DeepSpeed global_step*

This keeps the repository lightweight, public-safe, and reproducible. To run experiments, prepare the external assets according to the docs and generate caches locally.

Safety Checks

Before committing or pushing, run:

bash scripts/pre_push_safety_check.sh
bash scripts/check_workspace_layout.sh

These scripts check for accidental weights, videos, caches, runs, large files, local paths, and missing essential files.