Superhuman Synthesis of Scientific Knowledge with LLM Agents

This project implements the PaperQA2 agent and benchmarks its performance on the LitQA2 dataset using both OpenAI's GPT models and Meta's LLaMA models.

Overview

PaperQA2 is a modular Retrieval-Augmented Generation (RAG) system for answering scientific questions using source-grounded evidence. This repository reproduces the evaluation pipeline described in Skarlinski et al. (2024), specifically focusing on:

• Implementing PaperQA2 to apply LLMs to scientific literature.
• Benchmarking LLMs and compare their performance to humans using LitQA2 dataset.
• Conducting ablation studies (e.g., with/without RCS, varying retrieval top-k) to understand RAG and how it can be used to improve the accuracy of LLMs.

Setup

Install packages

git clone https://github.com/Future-House/paper-qa.git
cd paper-qa
uv sync

Restart the terminal:

source .venv/bin/activate
curl -sS https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install -r requirements.txt

Configure API keys and tokens

• Create a .env file and set the following environment variables for external services
• • OPENAI_API_KEY=sk-xxx
• • OPENREVIEW_USERNAME=your_username
• • OPENREVIEW_PASSWORD=your_password
• • UNPAYWALL_EMAIL=your_email@example.com
• Run huggingface-cli login in terminal, then enter your token to authenticate with Hugging Face

Running Benchmarks

You should prepare PDFs locally to support the benchmark experiments.

GPT

python src/run_litqa_gpt.py

LLaMA (via Ollama)

python src/run_litqa_ollama.py

Notes

Most experiments were run on a personal machine with:

• Chip: Apple M2 Pro
• Memory: 16 GB

LLaMA experiments faced some performance and timeout challenges. Details can be found in record/llama_experiment_record.md

Citation

• [Skarlinski et al., 2024]
  Language agents achieve superhuman synthesis of scientific knowledge
  arXiv:2409.13740

• [Lála et al., 2023]
  PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
  arXiv:2312.07559

• [Narayanan et al., 2024]
  Aviary: training language agents on challenging scientific tasks
  arXiv:2412.21154