Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models

Official code for the paper, accepted at the ECCV 2026 Workshop on Foundation and Generative Models in Biometrics (FoundGen), "Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models" Fizza Rubab, Yiying Tong, Arun Ross (Michigan State University).

A single linear map, estimated once from paired embeddings, aligns a face-recognition model with an off-the-shelf foundation model. Once aligned, a face embedding can be read with free-form text, rendered into a face image by an unmodified diffusion decoder, and named against a text vocabulary, without training or modifying either model.

Repository structure

ufe/               importable library (pip install -e .): the linear map + model loaders
extraction/        embed_dataset.py (all datasets), crop_lfw.py
experiments/       retrieval, generation, naming, cross-dataset, exposure, ablations
figures/           overview, qualitative montages, result plots

Installation

conda create -n ufe python=3.11 -y
conda activate ufe
pip install -e .
pip install torch==2.9.1 torchvision==0.24.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt

pip install -e . alone is enough to run the evaluations on existing embeddings.

Datasets

Datasets download to the Hugging Face / kagglehub cache under .cache/; the per-dataset loaders in extraction/embed_dataset.py point at those paths.

Models

Face weights come from CVLface (gated: export HF_TOKEN=...); foundation models load from Hugging Face transformers / diffusers (paper Table 1). None of the foundation targets is face-tuned.

Face: ArcFace, AdaFace, AdaFace-ViT, KPRPE. Foundation: CLIP, MetaCLIP, SigLIP, Kandinsky 2.2, Stable unCLIP; DINOv2 (metric only).

Pipeline

Run from the repository root.

export HF_TOKEN=...
python extraction/embed_dataset.py cfp
python extraction/embed_dataset.py utk
python extraction/embed_dataset.py celeba
python extraction/crop_lfw.py
python extraction/embed_dataset.py lfw

python experiments/eval_text_retrieval.py
python experiments/eval_t2i.py
python experiments/eval_naming.py

python figures/make_overview_figure.py

Alignment API

from ufe import LinearAlignment, l2

algo = LinearAlignment()
algo.fit(l2(A_train), C_train)           # source L2-normalised, target raw
C_hat = l2(algo.transform(l2(A_test)))

fit mean-centres both spaces and solves an unregularised least-squares system. It does not normalise anything for you. The experiment scripts apply the following, and results degrade noticeably without them:

• L2-normalise the source before fit and transform. Face templates carry a magnitude that varies with image quality, so leaving it in weights the fit towards high-norm samples.
• Leave the target raw. fit subtracts its training mean and transform adds it back.
• L2-normalise the output before any cosine or retrieval score.
• Embed both models from the same image. The map cannot recover context the source encoder never saw, so pairing a tight crop for the face model with a wider crop for the foundation model degrades every downstream task.
• Fit on data that covers the faces you will apply the map to. Alignment quality for an individual drops sharply when the paired data contains nobody comparable, which shows up as a large reconstruction error for that person while the population average looks healthy.

Scripts and analysis

Citation

@inproceedings{rubab2026unmasking,
  title     = {Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models},
  author    = {Rubab, Fizza and Tong, Yiying and Ross, Arun},
  booktitle = {ECCV Workshop on Foundation and Generative Models in Biometrics (FoundGen)},
  year      = {2026}
}

This work builds on our earlier study of face-embedding compatibility:

@inproceedings{rubab2026compatibility,
  title     = {Compatibility of Face Embeddings Across Deep Neural Networks},
  author    = {Rubab, Fizza and Tong, Yiying and Ross, Arun},
  booktitle = {IEEE International Joint Conference on Biometrics (IJCB)},
  year      = {2026}
}

License

Released under the MIT License (see LICENSE).