DIMER-oriented pipeline for Kandinsky 2.2 ControlNet-depth (kandinsky-community/kandinsky-2-2-controlnet-depth,
the 1.25 B-parameter hint-conditioned UNet diffusion model): a photograph becomes a depth hint, and a prompt plus that
hint becomes a 512 × 512 image whose content follows the words and whose layout follows the photograph. The decoder is
pinned to an immutable Hugging Face commit together with the shared kandinsky-community/kandinsky-2-2-prior, the
depth estimator Intel/dpt-large and a CLIP scorer for evaluation. The repository exposes depth-conditioned
generation, a held-out denoising-loss, CLIP and depth-fidelity evaluation, a captioned-image contract with explicit
limits, a bounded LoRA fine-tuning contract with a portable safetensors adapter, a MODEL_CARD.md at DIMER Model
Card Specification 1.2, and a standalone, guided E2E tutorial at DIMER Notebook Specification 2.2.
kandinsky-community/kandinsky-2-2-controlnet-depth08632524a2b7e3bed39a7901d92cdd07e37544d0 — the head of the upstream repository's open, unmerged
pull request #5 (safetensors conversion by patrickvonplaten). The upstream main branch
(4ecd717e8c9086cf4a16ca28b64894f70a42cd08) ships only pickled .bin weights, which this package never loads.
The conversion is bit-identical to main (checked by tools/verify_conversion.py; see docs/WEIGHTS.md).kandinsky-community/kandinsky-2-2-prior at 9fc51ad5732afc5d031724219d22e6c42179c5a8Intel/dpt-large at bc15f29aa3a80d532f2ed650b5e16ac48d8958f9, the transformers 5.17.0
depth-estimation default, loaded explicitly by id and revisionlaion/CLIP-ViT-B-32-laion2B-s34B-b79K at 1a25a446712ba5ee05982a381eed697ef9b435cf (evaluation only)diffusers==0.40.0 + transformers==5.17.0 + peft==0.21.0 + torch==2.14.0 — every loaded weight file
is safetensors, nothing is unpickled and no Hub-hosted code is executedThe hint decides the layout, the prompt decides the content. compute_hints() runs DPT on each 512² centre crop
and min-max normalises the relative depth to 0..255; generate(prompts, hints) takes one hint per prompt. The tutorial's
Section 7 changes only the hint (flat, mismatched or original) to show the split.
Generation has no ground truth, and the numbers say what they are. evaluate() reports the held-out denoising
loss with each record's own hint, at fixed timesteps with seeded noise, so frozen and adapted models see identical
inputs. metrics.score_generations() reports CLIP prompt similarity, label accuracy and similarity to real
photographs, and real_photo_reference() the same on the real photographs, leave-one-out (each photograph compared
with the other photographs of its caption, itself excluded) — a reference line, not a ceiling: generated images can
score above it (real_photo_baseline() is kept as an alias). metrics.score_depth_fidelity()
reports the Pearson correlation and the scale/offset-aligned error between each generated image's estimated depth and
its hint, next to a mismatched-hint baseline. None of these is a human judgement of image quality.
from kandinsky_controlnet_depth_pipeline import KandinskyDepthPipeline, fetch_sample_dataset, sample_prompts
from kandinsky_controlnet_depth_pipeline.metrics import ClipScorer, score_depth_fidelity, score_generations
pipe = KandinskyDepthPipeline.from_pretrained(use_lora=True, allow_download=True) # verifies 3 snapshots, loads UNet + LoRA, MoVQ
splits = fetch_sample_dataset() # 36 / 12 / 12 pinned CC0 iNaturalist bird photographs
records = splits["train"] + splits["validation"] + splits["test"]
pipe.compute_hints(records) # DPT depth hints, cached by pixel digest
pipe.encode_prompts(sample_prompts(records)); pipe.release_prior()
print(pipe.evaluate(splits["test"])["denoising_mse"]) # frozen model, held-out, hint-conditioned
test = splits["test"]
images = pipe.generate([r["caption"] for r in test], [pipe.depth_hint(r) for r in test], seed=1000)["images"]
print(score_generations(ClipScorer(allow_download=True), images, references=test), score_depth_fidelity(pipe, images))
pipe.adapt(splits["train"], splits["validation"]) # bounded LoRA fine-tuning, epoch kept by validation loss
pipe.save_artifact("outputs/adapter")
weights/kandinsky-2-2-controlnet-depth/ README.md model_index.json scheduler/ movq/ unet/ dimer-base-manifest.json
unet/diffusion_pytorch_model.safetensors (git-ignored, 5.01 GB)
movq/diffusion_pytorch_model.safetensors (git-ignored, 271 MB)
weights/kandinsky-2-2-prior/ configs, tokenizer, dimer-base-manifest.json (safetensors git-ignored, 10.57 GB)
weights/dpt-large/ README.md config.json preprocessor_config.json dimer-base-manifest.json
model.safetensors (git-ignored, 1.37 GB)
weights/clip-vit-b-32-laion2b/ config, tokenizer and preprocessor files dimer-base-manifest.json (605 MB model git-ignored)
weights/inat-birds/ the 60 pinned photographs, cached on first fetch (git-ignored)
from_pretrained() calls stage_missing_files(), stage_missing_prior_files() and stage_missing_depth_files()
(fetch only absent manifest entries, only at the pinned 40-character commits, with allow_download=True), then
verify_snapshot(), verify_prior_snapshot() and verify_depth_snapshot() (byte size + SHA-256 of every entry).
docs/WEIGHTS.md records the provenance of all four snapshots, including the conversion pull request.
fetch_sample_dataset() fetches 60 research-grade iNaturalist photographs of six North American birds (10 per species,
CC0 1.0) from the public open-data bucket, each pinned by byte size and SHA-256 in SAMPLE_RECORDS. Captions come from
one template per species. build_sample_dataset draws a seeded 6 / 2 / 2 stratified split per species (36 / 12 / 12)
and check_split_disjoint asserts no image appears twice. write_dataset_csv / load_byod_dataset support BYOD.
save_artifact(dir) writes adapter.safetensors (the 176 LoRA tensors — rank 8 on to_q, to_k, to_v, to_out.0,
1,646,592 parameters) and manifest.json with the decoder, prior and depth-estimator identities, the LoRA
configuration, the training configuration and history, and the file digest.
KandinskyDepthPipeline.from_artifact(dir) re-verifies the snapshots and checks the format, base model, estimator,
tensor scope and digest before deserialising.
python tools/verify_conversion.py downloads main's .bin files and the pinned safetensors at their exact commits,
verifies their SHA-256, and compares every tensor (keys, shapes, dtypes, torch.equal). It needs torch,
safetensors and huggingface-hub, about 11 GB of disk and no GPU. Its 2026-09-29 run on a Kaggle CPU kernel
found both components bit-identical; the record is in docs/release-verification.md.
pip install -e . --no-deps
pytest
The core suite runs offline with only numpy and pillow: temporary manifests, synthetic images and a stub UNet.
tests/test_adaptation.py and tests/test_diffusers_path.py also exercise torch, diffusers and peft on tiny
random models when those libraries are installed, and are skipped otherwise. tests/test_tutorial_stages.py executes
the generated notebook's carrier and run_stage code (always), and runs every tutorial stage on CPU against stubs when
torch is installed.
tutorials/kandinsky_controlnet_depth_colab.ipynb is declared E2E in GUIDED mode and is standalone (DIMER
Notebook Specification 2.2 §4): it is generated by tools/build_notebook.py from tools/notebook_template.py and
carries, byte for byte and SHA-256-verified, the package's four modules, the stage runner tools/tutorial_stages.py,
the hash-locked requirements tutorials/requirements-colab.lock.txt, the four snapshot manifests and the licence. It
carries the §3.5 guided layer (how-to-use, roadmap, glossary, predictions, check-your-reasoning answers, a
change-one-thing activity, troubleshooting and a conclusion scaffold).
The notebook installs nothing into its own kernel. It downloads a pinned uv wheel (checked by size and SHA-256),
builds an isolated CPython 3.12.12 environment, and installs the lock into it with --require-hashes --only-binary :all:; each stage (weights, prepare, encode, frozen, activity, adapt, evaluate, reload) then runs in its
own process and hands results to the next only through files. A hosted runtime's preloaded packages are never
replaced, so Run all needs no restart. The lock is compiled from the pyproject.toml pins for Colab and Kaggle
(uv pip compile pyproject.toml --python-version 3.12 --python-platform x86_64-manylinux_2_28 --generate-hashes --index-url https://pypi.org/simple --only-binary :all: -o tutorials/requirements-colab.lock.txt); recompile it and
regenerate the notebook whenever a pin changes.
Candidate — the tutorial notebook was regenerated after the 2026-10-02 notebook review (explanations, the
leave-one-out real-photo reference and small printed outputs changed), so the hosted records belong to its
previous revision; the current revision has one recorded Colab CLI run on a fresh T4 (2026-10-04, default path, one pass, no
restart). The previous revision runs in an isolated hash-locked environment and passed Run all in one pass on Google
Colab and on a strict Kaggle T4 run, plus the REL12 BYOD journey, at c86e3fe; the conversion check proved the pinned
safetensors bit-identical to main. A strict clean-executor run and the BYOD journey at the current revision are needed before
it is promoted again. See docs/release-verification.md and STATUS.md.
kandinsky-community/kandinsky-2-2-controlnet-depth, kandinsky-community/kandinsky-2-2-prior, Intel/dpt-large); the scorer is MIT.samples.py).LICENSE).This repository’s code and accompanying documentation were developed with generative AI assistance for code development and technical writing under maintainer direction. The maintainer remains responsible for reviewing the implementation, validating results, and making release decisions. AI assistance does not constitute independent verification, provider endorsement, or release approval.
DIMER-oriented pipeline for Kandinsky 2.2 ControlNet-depth (kandinsky-community/kandinsky-2-2-controlnet-depth,
the 1.25 B-parameter hint-conditioned UNet diffusion model): a photograph becomes a depth hint, and a prompt plus that
hint becomes a 512 × 512 image whose content follows the words and whose layout follows the photograph. The decoder is
pinned to an immutable Hugging Face commit together with the shared kandinsky-community/kandinsky-2-2-prior, the
depth estimator Intel/dpt-large and a CLIP scorer for evaluation. The repository exposes depth-conditioned
generation, a held-out denoising-loss, CLIP and depth-fidelity evaluation, a captioned-image contract with explicit
limits, a bounded LoRA fine-tuning contract with a portable safetensors adapter, a MODEL_CARD.md at DIMER Model
Card Specification 1.2, and a standalone, guided E2E tutorial at DIMER Notebook Specification 2.2.
kandinsky-community/kandinsky-2-2-controlnet-depth08632524a2b7e3bed39a7901d92cdd07e37544d0 — the head of the upstream repository's open, unmerged
pull request #5 (safetensors conversion by patrickvonplaten). The upstream main branch
(4ecd717e8c9086cf4a16ca28b64894f70a42cd08) ships only pickled .bin weights, which this package never loads.
The conversion is bit-identical to main (checked by tools/verify_conversion.py; see docs/WEIGHTS.md).kandinsky-community/kandinsky-2-2-prior at 9fc51ad5732afc5d031724219d22e6c42179c5a8Intel/dpt-large at bc15f29aa3a80d532f2ed650b5e16ac48d8958f9, the transformers 5.17.0
depth-estimation default, loaded explicitly by id and revisionlaion/CLIP-ViT-B-32-laion2B-s34B-b79K at 1a25a446712ba5ee05982a381eed697ef9b435cf (evaluation only)diffusers==0.40.0 + transformers==5.17.0 + peft==0.21.0 + torch==2.14.0 — every loaded weight file
is safetensors, nothing is unpickled and no Hub-hosted code is executedThe hint decides the layout, the prompt decides the content. compute_hints() runs DPT on each 512² centre crop
and min-max normalises the relative depth to 0..255; generate(prompts, hints) takes one hint per prompt. The tutorial's
Section 7 changes only the hint (flat, mismatched or original) to show the split.
Generation has no ground truth, and the numbers say what they are. evaluate() reports the held-out denoising
loss with each record's own hint, at fixed timesteps with seeded noise, so frozen and adapted models see identical
inputs. metrics.score_generations() reports CLIP prompt similarity, label accuracy and similarity to real
photographs, and real_photo_reference() the same on the real photographs, leave-one-out (each photograph compared
with the other photographs of its caption, itself excluded) — a reference line, not a ceiling: generated images can
score above it (real_photo_baseline() is kept as an alias). metrics.score_depth_fidelity()
reports the Pearson correlation and the scale/offset-aligned error between each generated image's estimated depth and
its hint, next to a mismatched-hint baseline. None of these is a human judgement of image quality.
from kandinsky_controlnet_depth_pipeline import KandinskyDepthPipeline, fetch_sample_dataset, sample_prompts
from kandinsky_controlnet_depth_pipeline.metrics import ClipScorer, score_depth_fidelity, score_generations
pipe = KandinskyDepthPipeline.from_pretrained(use_lora=True, allow_download=True) # verifies 3 snapshots, loads UNet + LoRA, MoVQ
splits = fetch_sample_dataset() # 36 / 12 / 12 pinned CC0 iNaturalist bird photographs
records = splits["train"] + splits["validation"] + splits["test"]
pipe.compute_hints(records) # DPT depth hints, cached by pixel digest
pipe.encode_prompts(sample_prompts(records)); pipe.release_prior()
print(pipe.evaluate(splits["test"])["denoising_mse"]) # frozen model, held-out, hint-conditioned
test = splits["test"]
images = pipe.generate([r["caption"] for r in test], [pipe.depth_hint(r) for r in test], seed=1000)["images"]
print(score_generations(ClipScorer(allow_download=True), images, references=test), score_depth_fidelity(pipe, images))
pipe.adapt(splits["train"], splits["validation"]) # bounded LoRA fine-tuning, epoch kept by validation loss
pipe.save_artifact("outputs/adapter")
weights/kandinsky-2-2-controlnet-depth/ README.md model_index.json scheduler/ movq/ unet/ dimer-base-manifest.json
unet/diffusion_pytorch_model.safetensors (git-ignored, 5.01 GB)
movq/diffusion_pytorch_model.safetensors (git-ignored, 271 MB)
weights/kandinsky-2-2-prior/ configs, tokenizer, dimer-base-manifest.json (safetensors git-ignored, 10.57 GB)
weights/dpt-large/ README.md config.json preprocessor_config.json dimer-base-manifest.json
model.safetensors (git-ignored, 1.37 GB)
weights/clip-vit-b-32-laion2b/ config, tokenizer and preprocessor files dimer-base-manifest.json (605 MB model git-ignored)
weights/inat-birds/ the 60 pinned photographs, cached on first fetch (git-ignored)
from_pretrained() calls stage_missing_files(), stage_missing_prior_files() and stage_missing_depth_files()
(fetch only absent manifest entries, only at the pinned 40-character commits, with allow_download=True), then
verify_snapshot(), verify_prior_snapshot() and verify_depth_snapshot() (byte size + SHA-256 of every entry).
docs/WEIGHTS.md records the provenance of all four snapshots, including the conversion pull request.
fetch_sample_dataset() fetches 60 research-grade iNaturalist photographs of six North American birds (10 per species,
CC0 1.0) from the public open-data bucket, each pinned by byte size and SHA-256 in SAMPLE_RECORDS. Captions come from
one template per species. build_sample_dataset draws a seeded 6 / 2 / 2 stratified split per species (36 / 12 / 12)
and check_split_disjoint asserts no image appears twice. write_dataset_csv / load_byod_dataset support BYOD.
save_artifact(dir) writes adapter.safetensors (the 176 LoRA tensors — rank 8 on to_q, to_k, to_v, to_out.0,
1,646,592 parameters) and manifest.json with the decoder, prior and depth-estimator identities, the LoRA
configuration, the training configuration and history, and the file digest.
KandinskyDepthPipeline.from_artifact(dir) re-verifies the snapshots and checks the format, base model, estimator,
tensor scope and digest before deserialising.
python tools/verify_conversion.py downloads main's .bin files and the pinned safetensors at their exact commits,
verifies their SHA-256, and compares every tensor (keys, shapes, dtypes, torch.equal). It needs torch,
safetensors and huggingface-hub, about 11 GB of disk and no GPU. Its 2026-09-29 run on a Kaggle CPU kernel
found both components bit-identical; the record is in docs/release-verification.md.
pip install -e . --no-deps
pytest
The core suite runs offline with only numpy and pillow: temporary manifests, synthetic images and a stub UNet.
tests/test_adaptation.py and tests/test_diffusers_path.py also exercise torch, diffusers and peft on tiny
random models when those libraries are installed, and are skipped otherwise. tests/test_tutorial_stages.py executes
the generated notebook's carrier and run_stage code (always), and runs every tutorial stage on CPU against stubs when
torch is installed.
tutorials/kandinsky_controlnet_depth_colab.ipynb is declared E2E in GUIDED mode and is standalone (DIMER
Notebook Specification 2.2 §4): it is generated by tools/build_notebook.py from tools/notebook_template.py and
carries, byte for byte and SHA-256-verified, the package's four modules, the stage runner tools/tutorial_stages.py,
the hash-locked requirements tutorials/requirements-colab.lock.txt, the four snapshot manifests and the licence. It
carries the §3.5 guided layer (how-to-use, roadmap, glossary, predictions, check-your-reasoning answers, a
change-one-thing activity, troubleshooting and a conclusion scaffold).
The notebook installs nothing into its own kernel. It downloads a pinned uv wheel (checked by size and SHA-256),
builds an isolated CPython 3.12.12 environment, and installs the lock into it with --require-hashes --only-binary :all:; each stage (weights, prepare, encode, frozen, activity, adapt, evaluate, reload) then runs in its
own process and hands results to the next only through files. A hosted runtime's preloaded packages are never
replaced, so Run all needs no restart. The lock is compiled from the pyproject.toml pins for Colab and Kaggle
(uv pip compile pyproject.toml --python-version 3.12 --python-platform x86_64-manylinux_2_28 --generate-hashes --index-url https://pypi.org/simple --only-binary :all: -o tutorials/requirements-colab.lock.txt); recompile it and
regenerate the notebook whenever a pin changes.
Candidate — the tutorial notebook was regenerated after the 2026-10-02 notebook review (explanations, the
leave-one-out real-photo reference and small printed outputs changed), so the hosted records belong to its
previous revision; the current revision has one recorded Colab CLI run on a fresh T4 (2026-10-04, default path, one pass, no
restart). The previous revision runs in an isolated hash-locked environment and passed Run all in one pass on Google
Colab and on a strict Kaggle T4 run, plus the REL12 BYOD journey, at c86e3fe; the conversion check proved the pinned
safetensors bit-identical to main. A strict clean-executor run and the BYOD journey at the current revision are needed before
it is promoted again. See docs/release-verification.md and STATUS.md.
kandinsky-community/kandinsky-2-2-controlnet-depth, kandinsky-community/kandinsky-2-2-prior, Intel/dpt-large); the scorer is MIT.samples.py).LICENSE).This repository’s code and accompanying documentation were developed with generative AI assistance for code development and technical writing under maintainer direction. The maintainer remains responsible for reviewing the implementation, validating results, and making release decisions. AI assistance does not constitute independent verification, provider endorsement, or release approval.