AI Coachella is Real. But is your track a deepfake? (Jerry CS153 Project)

With audio generation models like ElevenLabs and Grok Voice becoming better each day, it is becoming really difficult to tell deepfakes apart from real speech. This creates a serious trust issue, even for something as simple as picking up the phone or scrolling on Instagram reels.

As someone interested in AI research as well as software, I decided to train 4 different ML models with different architectures to detect deepfakes.

1. WavLM-Base+: A waveform-based encoder that analyzes raw audio signals directly.
2. Whisper-Base: A speech-recognition encoder that extracts phonetic and linguistic representations.
3. SSLAM: An audio-specific transformer that evaluates spectrogram-frequency features.
4. DINOv3: A vision transformer that treats the spectrogram as a visual image to detect pattern anomalies.

Each detector is a frozen pretrained backbone with a small trained classification head, so the comparison isolates what each representation actually knows about synthetic speech.

You can try the live demo at https://jerrygu.com/ai-audio-lab.html. The four models run on a Hugging Face Space and analyze any audio clip you upload.

AI Usage Disclosure

I used Claude Code and OpenAI Codex in setting up the experimental and data pipelines. Gemini Deep Research was used for researching AI architectures and literature in the area of audio deepfake detection, and Antigravity CLI (a new Google tool I wanted to explore and learn) was used for frontend UI design.

I also used Claude Code in helping with polishing documentation to be professional.

Results

Dataset v1.0: 9,084 clips (4,542 real, 4,542 fake; train 3,184, validation 452, test 906 per class), about 17 hours, 73 distinct fake generators of 2025 to 2026 vintage. Heads are selected by validation EER; the test split plays no role in selection. Metrics below are the selected head evaluated once per split. Lower EER is better.

Observations. The waveform self-supervised model gives the best test EER, and its learned layer attention concentrates on early Transformer layers (2 and 3), supporting the hypothesis that synthesis artifacts live in low-level acoustic representations rather than top-layer semantics. A pure vision backbone applied to spectrogram images is nearly as strong while being the smallest model. The general-audio spectrogram model trails the speech-specialized representations. Accuracy at the fixed 0.5 threshold is reported for completeness; EER is the threshold-free headline metric.

Use Cases

The most direct use is screening suspicious voice clips. A bank or a call center could run incoming audio through detectors like these and flag likely voice cloning before a human ever reviews it. Social platforms could use the same idea to triage viral audio for fact checking. I see this as a screening signal rather than proof, since even a five percent error rate is far too high to accuse someone on its own.

Future Work

For future work, I want to make the system multimodal, so it can analyze video and audio together instead of just the sound. I also want to build a version that runs in the background and checks audio automatically, so deepfakes get flagged in real time instead of someone having to upload a clip by hand.

Repository Layout

README.md
dataset/
  DATASHEET.md            frozen dataset contract
  SOURCES.md              composition record
  manifest.parquet        one row per clip, produced by the dataset build
  dataset_card.json       build metadata, counts, manifest hash
  audio/{train,val,test}/{real,fake}/<clip_id>.wav
  sources/  build/  common/   dataset-side machinery
training/
  README.md               training protocol, workflow, and results
  specs.py  data.py  engine.py  heads.py  metrics.py  modeling.py
  extract_features.py  train_detector.py  evaluate_detector.py
  waveform_ssl/            WavLM-Base+ method
  asr_logmel_encoder/      Whisper-base encoder method
  audio_spectrogram_vit/   SSLAM ViT-Base method
  vision_spectrogram_vit/  DINOv3 ViT-S/16 method
final_models/
  <method>/detector_head.pt  config.json  metrics.json  evaluation_test.json
backend/
  README.md

Scope Boundaries

dataset/ owns the frozen dataset contract, manifest, metadata, and audio. The dataset contract is documented in dataset/DATASHEET.md.

training/ owns the four detector methods and the shared training engine. It reads dataset/ and never writes back into it.

final_models/ stores the final trained model deliverables: the validation-selected detector head, its configuration, training metrics with train, validation, and test results, and saved-head evaluation records.

backend/ owns the inference API for the public demo. It reads trained artifacts from final_models/, reuses the inference and preprocessing code in training/, and exposes results for the frontend. It does not train models or modify the dataset.

Dataset Handoff

Training consumes:

• dataset/manifest.parquet
• dataset/dataset_card.json
• WAV files referenced by the manifest under dataset/audio/

Every delivered clip is mono 16 kHz WAV PCM16, greater than 0 seconds and at most 30 seconds. Splits are supplied by the dataset side and are never rebuilt by training code. The dataset build is complete at version v1.0.

Methods

Each method trains only the detector head (about 0.2M to 0.5M parameters). No backbone fine-tuning, LoRA, adapters, or backbone updates are part of this project scope. training/README.md documents the full protocol, the feature cache policy, and reproduction commands.