🧫 Single-Cell Omics Arena

Single-Cell Omics Arena (SOAR) is a comprehensive benchmark framework designed to evaluate and improve the performance of instruction-tuned large language models (LLMs) in automated cell type annotation from single-cell omics data.

Updates

• [2025-08-03] 🎉 We are excited to provide a command line interface and the pre-built package soar_benchmark for easier usages.
• [2025-05-12] 🎉 We are excited to announce that Single-Cell Omics Arena (SOAR) is now open-source! We welcome contributions from the community to help advance automated cell type annotation using LLMs.

Installation

1. Create an environment with Python >= 3.11
2. Clone the repo via git clone [email protected]:jhliu17/SOAR.git
3. Install soar_benchmark via pip install -e .

API Key Setup

To execute LLMs provided by OpenAI or hosted on Hugging Face Transformers, an env file (env.toml) should be set up in the project folder. A template env file is provided as env_sample.toml.

Cell Type Annotation with LLMs

Annotation Interface

To print out all supported LLMs, please run:

soar annotate -h

This will print out all available annotation options using LLMs.

soar currently supports the following LLMs for cell type annotation:

• Qwen2 series (1.5B, 7B, 72B)
• Meta Llama-3 70B
• Mixtral-8x7B
• GPT-4o
• GPT-4o-mini

where all LLMs can leverage zero-shot prompting or zero-shot chain-of-thought prompting to finish cell type annotations.

Each model has specific hardware requirements and configurations:

• Smaller models (1.5B-7B): Single GPU with 16GB VRAM
• Larger models (70B-72B): Multi-GPU setup with 4-8 GPUs
• Mixtral-8x7B: 4 GPUs recommended for optimal performance

For example, to reproduce the SOAR-RNA benchmark result on GPT-4o using zero-shot prompting, one can run

soar annotate soar_rna_with_gpt4_o_zero_shot

Custom Dataset

If one would like to leverage a provided LLM annotation configuration to annotate their own dataset, this can be achieved by

soar annotate soar_rna_with_gpt4_o_zero_shot --config.dataset.json-path YOUR_DATASET_PATH

where the custom dataset should follow the same structure as soar_benchmark/datasets/soar_rna.json.

One can further fine-tune the preset configuration by overriding some arguments. For example, increasing the new token number limit to 2048.

soar annotate soar_rna_with_gpt4_o_zero_shot --config.generation.max-new-tokens 2048

# To see more tunable options
soar annotate soar_rna_with_gpt4_o_zero_shot -h

Custom LLM Configuration

If you would like to implement a custom annotation configuration. Please refer to the detailed configuration settings including batch sizes, memory requirements, and hardware specifications in:

• Model configs: soar_benchmark/configs/cell_type_annotation/experiment_soar_rna.py

Once you have implemented a custom configuration, you can use it by calling the built-in annotation function

from soar_benchmark import start_annotation_task

# Your custom configuration
custom_configuration = CellTypeAnnotationTaskConfig(...)

# Start annotation
start_annotation_task(custom_configuration)

Evaluations

To run evaluations on annotated results, please refer to

python -m analysis.cell_type_annotation.squad_eval --chat_results_path outputs/.../qwen2-72b-instruct.json --squad_eval_results_path outputs/.../few_shot_squad_eval_inflect.json

Metrics

To evaluate the free-format cell type annotations generated by LLMs, we employ seven widely used metrics from natural language processing and question answering. These include ROUGE (R-1, R-2, R-L) for measuring n-gram and sequence overlap, METEOR for capturing semantic similarity through surface forms, stems, and synonyms, and BLEU (BLEU-1, BLEU-2, and geometric average) for n-gram overlap, particularly suited for short phrases. Additionally, the Exact Match (EM) and F1 score are used to assess token-level precision and recall, ensuring fair evaluation despite label variability and synonym usage.

SOAR-RNA Benchmark

Zero-shot Cell Type Annotation

Zero-shot Chain-of-thought Cell Type Annotation

SOAR-MultiOmics Benchmark

RNA-seq

ATAC-seq