Interpreting the Weight Space of Customized Diffusion Models

This repository contains the dataset of model weights and utility files for Interpreting the Weight Space of Customized Diffusion Models (aka weights2weights). The code can be found at https://github.com/snap-research/weights2weights. The files folder contains auxiliary files that our code requires to run. The weights_datasets folder contains the datasets of model weights in addition to metadata and supplemental files. If you are interested in the images used to train models to create our dataset of model weights, visit this link.

Files for creating weights2weights Space

The files/ folder contains the files needed for our code here in order to conduct sampling, inversion, and editing in weights2weights space.

• files/V.pt
  • 99648x10000 dimensional tensor used to project LoRA weights onto a principal component representation in w2w space or to unproject back into the LoRA space.
• files/mean.pt
  • 99648 dimensional tensor of the mean for each parameter in the original LoRA space. Used for projection/unprojection.
• files/std.pt
  • 99648 dimensional tensor of the standard deviation for each parameter in the original LoRA space. Used for projection/unprojection.
• files/proj_1000pc.pt
  • 64974x1000 dimensional tensor where each row is a 1000-dimensional principal component projection for each identity-encoding model in our dataset of model weights.
• files/pinverse.pt
  • Precomputed pseudoinverse of 'proj.pt', used for obtaining the classifier weight space directions given labels.
• files/identity_df.pt
  • A pandas dataframe with binary attribute labels for each identity-encoding model. Used for getting labels for training linear classifiers.
• files/weight_dimensions.pt
  • A dictionary of the dimensionality for each LoRA module in the diffusion UNet. Used to save models in Diffusers pipeline format.
• files/adapter_config.json
  • A configuration file for LoRA. Used to save models in Diffusers pipeline format.

Datasets of Model Weights

The weights_datasets folder contains two datasets of model weights, one set containing LoRA weights encoding different human visual identities, and another for different dog breeds. We also provide metadata and files for handling these datasets. An example on how to open a model from these datasets is available at: https://github.com/snap-research/weights2weights/blob/main/other/datasets.ipynb

• weights_datasets/identities/all_weights.pt
  • 64974x99648 dimensional tensor where each row is a 99648-dimensional vector of flattened LoRAs for each identity-encoding model.
• weights_datasets/identities/identity_df.pt
  • A pandas dataframe with binary attribute labels for each identity-encoding model.
• weights_datasets/dogs/dog_weights.pt
  • 120x99648 dimensional tensor where each row is a 99648-dimensional vector of flattened LoRAs for each dog-encoding model.
• weights_datasets/dogs/dog_labels.pt
  • A list of the dog breeds corresponding to each entry in dog_weights.pt.
• weights_datasets/weight_dimensions.pt
  • A dictionary of the dimensionality for each LoRA module in the diffusion UNet. Used to save models in Diffusers pipeline format.
• weights_datasets/adapter_config.json
  • A configuration file for LoRA. Used to save models in Diffusers pipeline format.

Images for Recreating the Dataset of Weights

The images used to conduct Dreambooth fine-tuning can be found at this link. We provide instructions for how to use these images to train models and recreate our dataset of model weights in our GitHub Repo.