The recommended version of Python for Inspect Evals is 3.11 or 3.12. You should be able to run all evals on these versions and also develop the codebase without any issues. You can install and pin a specific Python version by running:

uv python pin 3.11

As for Python 3.13, you should be able to run all evals except sciknoweval (its dependency is gensim which currently does not support 3.13+). Development should work under 3.13, however it's relatively untested — if you run into issues, let us know.

When it comes Python 3.14, at the time of writing this, many packages have yet to release versions for 3.14, so it's unsupported. The major one used by some Inspect Evals is torch. If you find running uv sync succeeding on 3.14, let us know and we'll remove this paragraph.

Below, you can see a workflow for a typical eval. Some of the evaluations require additional dependencies or installation steps. If your eval needs extra dependencies, instructions for installing in the README file in the eval's subdirectory.

Usage

Installation

There are two ways of using Inspect Evals, from pypi as a dependency of your own project and as a standalone checked out GitHub repository.

If you are using it from pypi, install the package and its dependencies via:

pip install inspect-evals

If you are using Inspect Evals in its repository, start by installing the necessary dependencies with:

uv sync

Running evaluations

Now you can start evaluating models. For simplicity's sake, this section assumes you are using Inspect Evals from the standalone repo. If that's not the case and you are not using uv to manage dependencies in your own project, you can use the same commands with uv run dropped.

uv run inspect eval inspect_evals/arc_easy --model openai/gpt-5-nano
uv run inspect eval inspect_evals/arc_challenge --model openai/gpt-5-nano

To run multiple tasks simultaneously use inspect eval-set:

uv run inspect eval-set inspect_evals/arc_easy inspect_evals/arc_challenge

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.arc import arc_easy, arc_challenge
eval(arc_easy)
eval_set([arc_easy, arc_challenge], log_dir='logs-run-42')

After running evaluations, you can view their logs using the inspect view command:

uv run inspect view

For VS Code, you can also download Inspect AI extension for viewing logs.

If you don't want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805
ANTHROPIC_API_KEY=<anthropic-api-key>

Inspect supports many model providers including OpenAI, Anthropic, Google, Mistral, Azure AI, AWS Bedrock, Together AI, Groq, Hugging Face, vLLM, Ollama, and more. See the Model Providers documentation for additional details.

You might also be able to use a newer version of pip (25.1+) to install the project via pip install --group dev . or pip install --group dev '.[swe_bench]'. However this is not officially supported.

Documentation

For details on building the documentation, see the documentation guide.

For information on running tests and CI toggles, see the Technical Contribution Guide in CONTRIBUTING.md.

Hardware recommendations

Disk

We recommend having at least 35 GB of free disk space for Inspect Evals: the full installation takes about 10 GB and you'll also need some space for uv cache and datasets cache (most are small, but some take 13 GB such as MMIU).

Running some evals (e.g., CyBench, GDM capabilities evals) may require extra space beyond this because they pull Docker images. We recommend having at least 65 GB of extra space for running evals that have Dockerfiles in their file tree (though you might get away with less space) on top of the 35 GB suggestion above.

In total, you should be comfortable running evals with 100 GB of free space. If you end up running of out space while having 100+ GB of free space available, please let us know — this might be a bug.

Cache location

Datasets and other large assets are cached under the platform cache directory (~/.cache/inspect_evals on Linux, ~/Library/Caches/inspect_evals on macOS). Set INSPECT_EVALS_CACHE_DIR to put them somewhere else:

export INSPECT_EVALS_CACHE_DIR=/data/inspect-evals-cache

Use it when the default location is not writable (read-only container filesystems, images without a writable HOME), when the cache should live on a larger volume, or to stage assets for a machine with no network access: populate the directory on a connected machine running the same Inspect Evals version, copy it across, and point the variable at it there.

The variable is read when inspect_evals is first imported, so set it in the shell or in a .env file rather than from within Python.

For the same reason it must be an absolute path, or start with ~ for a path under your home directory. A relative path is rejected, because it would point somewhere different depending on where the eval was started from.

RAM

The amount of memory needed for an eval varies significantly with the eval. You'll be able to run most evals with only 0.5 GB of free RAM. However, some evals with larger datasets require 2-3 GB or more. And some evals that use Docker (e.g., some GDM capabilities evals) require up to 32 GB of RAM.

Harbor Framework Evaluations

For running evaluations from the Harbor Framework (e.g. Terminal-Bench 2.0, SWE-Bench Pro), use the Inspect Harbor package, which provides an interface to run Harbor tasks using Inspect AI.