DeepFake Investigation Benchmarking

A comprehensive evaluation pipeline for benchmarking various DeepFake detection models across a curated dataset of real and fake images.

📁 Repository Structure

• dataset/ - Curated images categorized into Real and Fake.
• models/ - Standardized detector modules (model1.py, model2.py, etc.).
• results/ - Consolidated benchmarks (all_results.csv, model_comparison.csv).
• plots/ - Meaningful visualisations (Confusion Matrices, Metric Comparisons).
• main.py - Unified batch-processing evaluation pipeline.
• Dataset metadata.csv - Metadata for the curated dataset.
• evaluate.log - Detailed execution logs.

🚀 Setup & Reproduction

This project is compatible with Python 3.9+.

1. Standard Installation (Recommended for Assessment)

Install all dependencies using pip:

pip install -r requirements.txt

2. Docker Installation (Professional/Isolated)

If you have Docker Desktop installed, you can run the entire pipeline in an isolated container:

docker compose up --build

3. Fast Installation (Optional)

If you have uv installed, you can also use:

uv sync

3. Run the Evaluation

To run the full benchmark across all 4 detectors and generate metrics/plots: If using Pip:

python main.py

(Or uv run main.py if using uv).

📈 Understanding the Metrics

The evaluation results in results/model_comparison.csv use professional forensic metrics:

• Accuracy: The overall percentage of correct predictions.
• AUC (Area Under the Curve): A single score (0 to 1) representing how well the model separates "Real" from "Fake." Higher is better.
• APCER (The "Security Risk"): The % of Fake images the model missed. You want this to be as low as possible.
• BPCER (The "User Annoyance"): The % of Real images the model wrongly flagged as fake. You want this to be as low as possible.

---

🔍 Methodology

This pipeline uses an ensemble of four distinct architectures to ensure architectural diversity:

1. ViT (Vision Transformer) - High recall, excellent for first-pass filtering.
2. EfficientNet-B0 - Balanced performance in a lightweight CNN architecture.
3. Siglip2 - State-of-the-art vision model for high-accuracy binary classification.
4. Smogy-Detector - Specialized detector for catching modern AI-generative artifacts (SDXL/DALL-E).

Performance is measured using Accuracy, Precision, Recall, and F1-Score to provide a holistic view of each model's strengths and weaknesses.