Foundation versus Domain-Specific Models: Performance Comparison, Fusion, and Explainability in Face Recognition

Redwan Sony*, Parisa Farmanifard, Arun Ross*, Anil K. Jain
Michigan State University
📧 {sonymd, farmanif, rossarun, jain}@msu.edu

*Corresponding authors

---

📌 Abstract

In this paper, we address the following question: How do generic foundation models (e.g., CLIP, BLIP, GPT-4o, Grok-4) compare against a domain-specific face recognition model (viz., AdaFace or ArcFace) on the face recognition task? Through a series of experiments involving several foundation models and benchmark datasets, we report the following findings: (a) In all face benchmark datasets considered, domain-specific models outperformed zero-shot foundation models. (b) The performance of zero-shot generic foundation models improved on over-segmented face images compared to tightly cropped faces, thereby suggesting the importance of contextual clues. (c) A simple score-level fusion of a foundation model with a domain-specific face recognition model improved the accuracy at low false match rates. (d) Foundation models, such as GPT-4o and Grok-4, are able to provide explainability to the face recognition pipeline. In some instances, foundation models are even able to resolve low-confidence decisions made by AdaFace, thereby reiterating the importance of combining domain-specific face recognition models with generic foundation models in a judicious manner.

---

❓ Research Question

How do generic foundation models (e.g., CLIP, BLIP, GPT-4o, Grok-4) compare against domain-specific face recognition models (e.g., AdaFace, ArcFace) on the face recognition task?

---

🔍 Key Findings

• Domain-specific face recognition models consistently outperform zero-shot foundation models across all evaluated benchmark datasets.
• Zero-shot foundation models show improved performance on over-segmented face images compared to tightly cropped faces, suggesting that contextual information plays an important role.
• A simple score-level fusion between a foundation model and a domain-specific FR model:
  • Improves verification accuracy
  • Is particularly effective at low false match rates (FMRs).
• Foundation models such as GPT-4o and Grok-4 are capable of providing human-interpretable explanations for face recognition decisions.
• In some cases, foundation models help resolve low-confidence decisions made by AdaFace, reinforcing the benefit of combining domain-specific biometric models with generic foundation models.

---

🧠 Models Studied

Domain-Specific Face Recognition Models

• AdaFace
• ArcFace

Foundation Models (Zero-Shot)

• CLIP
• BLIP
• GPT-4o
• Grok-4
• and many more. Look at supplementary materials.

---

📊 Experimental Setup

• Evaluation performed on multiple face recognition benchmark datasets
• Experiments conducted under:
  • Zero-shot foundation model settings
  • Fusion-based configurations
• Analysis includes:
  • Verification performance
  • Effect of face cropping versus contextual information
  • Explainability and confidence analysis

Supplementary Materials: The supplementary materials associated with the publication can be found in the following link. Supplementary Branch

The supplementary materials can also downloaded via the following command.

git clone --depth 1 --branch supplementary [email protected]:redwankarimsony/AFRvsVLM.git

Extract and Evaluate the features

From the root of the repository, run the follwoing command to extract and evalaute the split for the default dataset WebFace42M dataset.

python -m evaluation.extract_and_evaluate \
--model_name OpenClip-huge \
--split 1 \
--device cuda:1