Large-scale DNN Benchmarking in Dynamic Social Vision

This repository is the official implementation of Dynamic social vision highlights gaps between deep learning and human behavioral and neural responses.

Requirements:

To install requirements:

conda env create -f environment.yml

Additional Requirements:

This code base utilizes and was created in collaboration with an unreleased package - Deepjuice (Model zoology, feature extraction, GPU-accelerated brain + behavioral readout for CogNeuroAI research.)

@article{conwell2023pressures,
 title={What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines},
 author={Conwell, Colin and Prince, Jacob S and Kay, Kendrick N and Alvarez, George A and Konkle, Talia},
 journal={bioRxiv},
 year={2023}
}

Data Requirements:

The neural fMRI, video and preprocessing datasets are currently privately hosted.

Running the benchmarks:

We currently have the following implemented benchmarks defined by individual Classes within the scripts directory:

• VisionNeuralEncoding
• VisionBehaviorEncoding
• VideoNeuralEncoding
• VideoBehaviorEncoding
• LanguageNeuralEncoding
• LanguageBehaviorEncoding
• VisionNeuralRSA
• VideoNeuralRSA

We utilize Rockfish with slurm to run sbatch jobs. In the scripts directory you can find pre-generated shell scripts to run the benchmarks listed above e.g.

sbatch -J {model_uid} batch_vis-neural_encoding.sh {model_uid}

Results

Our benchmark contains over 350+ DNN models across Image, Video and Language models.

Table of the top-10 models on average across all ROIs

Models are arranged in descending order of their overall average performance. Bold scores highlight the model achieves the highest score in each ROI.

Table of the top-10 performing models averaged across features

Models are arranged in descending order of their overall average performance. Bold scores highlight the model that is the top performing model for a given feature. The absence scores of a given feature (e.g., expanse) indicates that the top performing model for that feature was not in the overall top-10 models.

Model Zoo:

A full list of our benchmarked models and associated metadata can be found in the model_zoo directory.

Disclaimers:

Do Not Distribute: This codebase is not fully released and requires private access to run.