Visual and Auditory DNN Analysis Code

Analysis code for "Visual and auditory deep learning models capture neural representations of naturalistic social interaction in the superior temporal sulcus"

Directory Structure

analysis_code/
├── config.py                              # Centralized paths and parameters
├── utils/                                 # Shared utilities
│   ├── fmri_io.py                        # fMRI data loading/saving
│   ├── stats.py                          # Statistical functions
│   └── visualization.py                  # Plotting utilities
├── 00_preprocessing/
│   └── extract_frames.py                 # Extract video frames by TR
├── 00_rois/
│   ├── ffa_mask.nii                      # Fusiform face area mask
│   ├── ppa_mask.nii.gz                   # Parahippocampal place area mask
│   ├── sts_mask.nii                      # Superior temporal sulcus mask
│   ├── sts_anterior_mask.nii             # Anterior STS mask
│   └── sts_posterior_mask.nii            # Posterior STS mask
├── 01_feature_extraction/
│   ├── extract_clip.py                   # CLIP visual features
│   ├── extract_clap.py                   # CLAP audio features
│   ├── extract_dino.py                   # DINOv2 visual features (control)
│   └── extract_hubert.py                 # HuBERT audio features (control)
├── 02_encoding/
│   ├── voxelwise_encoding_ridge_gpu.py   # GPU-accelerated ridge regression
│   ├── encoding_utils.py                 # Encoding helper functions
│   └── models_config.py                  # Model feature configurations
├── 03_variance_partitioning/
│   └── preference_mapping.py             # Two-stage FDR unique variance
├── 04_pca/
│   └── pca_encoding_weights.py           # PCA on encoding weights
├── 05_stimulus_correlates/
│   ├── annotate_shotvl.py                # Shot scale annotation (ShotVL)
│   ├── annotate_gpt4o.py                 # Social interaction annotation (GPT-4o)
│   └── extract_wavlm_emotion.py          # Emotion extraction (WavLM)
└── 06_geometry/
    └── tsne_predicted_signal.py          # t-SNE visualization

Figure-to-Script Mapping

Usage

1. Setup

Edit config.py to set paths for your system:

• PROJECT_ROOT: Base project directory
• DATA_DIR: Raw data (video, audio)
• FMRI_DATA_DIR: fMRI data location
• ANNOTATIONS_DIR: Feature annotations
• RESULTS_DIR: Output directory

2. Preprocessing

Extract video frames:

python 00_preprocessing/extract_frames.py \
    --video_path /path/to/sherlock.mp4 \
    --output_dir /path/to/frames

3. Feature Extraction

Extract DNN features from movie:

# Visual features (CLIP)
python 01_feature_extraction/extract_clip.py

# Audio features (CLAP)
python 01_feature_extraction/extract_clap.py

# Control models
python 01_feature_extraction/extract_dino.py
python 01_feature_extraction/extract_hubert.py

4. Encoding Models

Run voxelwise encoding:

python 02_encoding/voxelwise_encoding_ridge_gpu.py \
    --model clip_full \
    --roi sts \
    --gpu_id 0

5. Variance Partitioning

Compare unique variance between models:

python 03_variance_partitioning/preference_mapping.py \
    --joint /path/to/clip_clap_results \
    --model1 /path/to/clip_results \
    --model2 /path/to/clap_results \
    --joint_name clip_clap \
    --model1_name clip_full \
    --model2_name clap_full \
    --output preference_map.html

6. PCA Analysis

Analyze encoding weight structure:

python 04_pca/pca_encoding_weights.py --model clip_full

7. Stimulus Correlates

Annotate stimulus features:

# Shot scale (requires ShotVL)
python 05_stimulus_correlates/annotate_shotvl.py

# Emotion from audio
python 05_stimulus_correlates/extract_wavlm_emotion.py

8. t-SNE Visualization

Visualize representational geometry:

python 06_geometry/tsne_predicted_signal.py \
    --model clip_full \
    --roi sts

Requirements

• Python 3.10+
• PyTorch 2.0+
• CUDA 11.8+ (for GPU acceleration)
• See requirements.txt for full dependencies