OpenAudioBench

Introduction

OpenAudioBench is an audio understanding evaluation dataset designed to assess the capabilities of multimodal and audio-focused language models. It spans multiple domains of audio-based tasks, including logical reasoning, general knowledge, and open-ended question answering. The dataset is structured to support the development and benchmarking of advanced models in the research community.

Usage

Here is a step-by-step guide to evaluating the Baichuan-Omni-1.5's performance on OpenAudioBench:

1. Start Model Service

cd server
# Modify MODEL_PATH in web_demo/constants.py to the local model path
MODEL_PATH=""
CUDA_VISIBLE_DEVICES=0 python3 Baichuan-Omni-1.5_server.py --model_path ${MODEL_PATH}

2. Configure Inferrence service

Configure the service name and URL in models. py

# for inferrence
model_dict['BaichuanOmnid5_local_test'] = BaichuanOmnid5(url="http://127.0.0.1:80")

3. Run Evaluation

SERVER_NAME=BaichuanOmnid5_local_test
sh run.sh ${SERVER_NAME}

4. Evaluation Results

In the ./output directory, you will see all the inference, evaluation and final results.