Multi-Agent Peer Review Engine

An AI-powered peer review system that simulates the full academic review process using multiple specialized agents. Upload a PDF and receive a structured review with reviewer critiques, independence audits, editorial decisions, and author revisions — all generated by coordinated AI agents.

Supports NIH grant reviews, foundation grant reviews, and journal manuscript peer reviews, with three AI backends: Claude, Gemini, and local open-source models.

---

Table of Contents

• Features
• How It Works
  • Review Workflow
  • Review Types
• Installation
  • Using Conda (Recommended)
  • Using pip
  • Local Backend Setup (Optional)
• Configuration
  • API Keys
  • Local Models
• Usage
  • Web Interface (Streamlit)
  • Command Line
  • Python API
• AI Backends and Models
  • Claude (Anthropic)
  • Gemini (Google)
  • Local (Open-Source)
• Output Format
• Project Structure
• Testing
• License

---

Features

• Multi-agent review simulation — Each review type uses a panel of specialized AI reviewers with distinct roles and expertise
• Independence audit — A challenge pass detects groupthink, confirmation bias, and herd behavior across reviewers
• Iterative revision — The AI author agent revises the document based on feedback, with optional multiple rounds
• Three review types — NIH grant, foundation grant, and journal peer review, each with domain-specific agents, criteria, and scoring rubrics
• Three AI backends — Claude (Anthropic), Gemini (Google), or local open-source models for fully offline review
• Streaming output — Real-time token streaming in both CLI and web interface
• Structured output — Markdown with auto-generated table of contents, review metadata, and optional PDF export
• Model fallback chains — Automatic fallback to alternative models if the primary model is unavailable (Claude and Gemini)

---

How It Works

Review Workflow

Every review follows the same four-stage pipeline, regardless of review type:

 PDF Document
      |
      v
 +--------------------+
 | 1. REVIEW PANEL    |  Multiple specialized reviewers independently
 |    (3-5 agents)    |  critique the document using domain-specific
 +--------------------+  scoring criteria and rubrics
      |
      v
 +--------------------+
 | 2. INDEPENDENCE    |  A challenge agent audits all reviews for
 |    AUDIT           |  groupthink, confirmation bias, and
 +--------------------+  produces addenda where warranted
      |
      v
 +--------------------+
 | 3. SYNTHESIZER     |  An SRO, Editor, or Panel Chair synthesizes
 |    (Decision)      |  the critiques into an official decision
 +--------------------+  (e.g., Fundable, Accept, Revise, Reject)
      |
      v
 +--------------------+
 | 4. AUTHOR          |  The AI author/PI revises the document
 |    REVISION        |  based on all feedback, producing a
 +--------------------+  point-by-point response and revised text
      |
      v
 (Iterate or finish based on decision)

Review Types

---

Installation

Requires Python 3.10+ (3.11 recommended).

Using Conda (Recommended)

git clone https://github.com/mohebial/agentic_writing.git
cd agentic_writing

# Create and activate the environment
conda env create -f environment.yml
conda activate agentic_writing

Using pip

git clone https://github.com/mohebial/agentic_writing.git
cd agentic_writing

pip install -r requirements.txt

Local Backend Setup (Optional)

To use local open-source models instead of cloud APIs:

# Required: PDF-to-markdown conversion
pip install markitdown

# For GGUF quantized models (recommended for most users):
pip install llama-cpp-python

# For standard HuggingFace models:
pip install transformers torch

---

Configuration

API Keys

Cloud backends require API keys set as environment variables:

# For Claude backend
export ANTHROPIC_API_KEY="sk-ant-..."

# For Gemini backend
export GEMINI_API_KEY="AI..."

You can also create a .env file in the project root:

ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AI...

Local Models

Local models require no API keys. Models are automatically downloaded from HuggingFace Hub on first use and cached at ~/.cache/huggingface/hub/.

The default model is Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-GGUF.

To use a model you've already downloaded locally, pass the file path directly:

python -m review_engine nih proposal.pdf --backend local --model ~/models/my-model.gguf

---

Usage

Web Interface (Streamlit)

streamlit run streamlit_app.py

This opens a browser-based GUI where you can:

1. Select a review type (NIH, Foundation, or Journal) from the dropdown
2. Upload a PDF of your document
3. Choose an AI backend (Claude, Gemini, or Local)
4. Select a model (dropdown for cloud backends, text input for local)
5. Set max rounds (for iterative review types)
6. Click "Start Review" to begin

The sidebar shows API key status (cloud backends) or dependency status (local backend). Results are displayed in the main area with download buttons for both Markdown and PDF formats.

Command Line

python -m review_engine <type> <pdf> [options]

Arguments:

Options:

Examples:

# NIH review with Claude
python -m review_engine nih proposal.pdf --backend claude

# Journal review with Gemini
python -m review_engine journal manuscript.pdf --backend gemini

# Foundation review with a specific Claude model
python -m review_engine nih "C:\Users\alimo\OneDrive - UW-Madison\0 - Lab\Grants\2026\BRF\Research_Strategy.pdf" --backend claude --model claude-haiku-4-5-20251001
```bash
# Foundation review with a specific Gemini model
python -m review_engine foundation "C:\Users\alimo\OneDrive - UW-Madison\0 - Lab\Grants\2026\BRF\Research_Strategy.pdf" --backend gemini --model gemini-3.0-flash

NIH review with 3 rounds

python -m review_engine nih proposal.pdf --max-rounds 3

Local model (auto-downloads from HuggingFace)

python -m review_engine nih proposal.pdf --backend local

Local model from a file path

python -m review_engine nih proposal.pdf --backend local --model ~/models/qwen.gguf

Local model on CPU only

python -m review_engine nih proposal.pdf --backend local --n-gpu-layers 0

Local Model

python -m review_engine nih "C:\Users\alimo\OneDrive - UW-Madison\0 - Lab\Grants\2026\BRF\Research_Strategy.pdf" --backend local --model QWEN/Qwen3.5-9B

### Python API

```python
from review_engine.config import ensure_types_loaded, get_config
from review_engine.engine import run_review

ensure_types_loaded()
config = get_config("nih")  # or "foundation", "journal"

result_md = run_review(
    config=config,
    backend="gemini",            # "claude", "gemini", or "local"
    pdf_path="proposal.pdf",
    model="gemini-2.0-flash",
    max_rounds=2,
    output_path="proposal_review.md",
    on_chunk=lambda t: print(t, end=""),   # streaming callback
    on_status=lambda s: print(f"[{s}]"),   # progress callback
)

---

AI Backends and Models

Claude (Anthropic)

Requires ANTHROPIC_API_KEY. Sends the PDF as a base64-encoded document.

Automatic fallback: if the selected model fails, the engine tries each model in the chain above.

Gemini (Google)

Requires GEMINI_API_KEY. Uploads the PDF to Gemini's File API.

Automatic fallback through the chain above.

Local (Open-Source)

No API key required. Runs entirely on your machine. Uses markitdown to convert PDFs to markdown text before sending to the model.

Two inference paths (auto-detected):

Default model: Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-GGUF

GPU support:

• CUDA (NVIDIA): automatically detected
• MPS (Apple Silicon): automatically detected
• CPU: always available, use --n-gpu-layers 0 to force

---

Output Format

Each review produces a Markdown file (and optionally a PDF) with the following structure:

# NIH Grant Review (Claude): proposal.pdf

## Review Metadata
| Field       | Value                    |
|-------------|--------------------------|
| Backend     | Claude                   |
| Model       | claude-haiku-4-5-20251001|
| Review Type | NIH Grant Review         |
| Started     | 2025-03-09 14:30 UTC     |
| Completed   | 2025-03-09 14:45 UTC     |
| Duration    | 15m 23s                  |

## Table of Contents
- [Review Round 1](#review-round-1)
- [Summary Statement](#summary-statement)
- ...

## Review Round 1
### Primary Reviewer
(detailed critique with scoring)

### Secondary Reviewer
...

## Summary Statement (SRO)
...

### SRO Decision: Resubmit — Minor Revisions

## PI Response & Revised Application
...

---

Project Structure

agentic_writing/
├── streamlit_app.py              # Streamlit web interface
├── environment.yml               # Conda environment definition
├── requirements.txt              # pip dependencies
│
├── review_engine/
│   ├── __main__.py               # CLI entry point
│   ├── engine.py                 # Review orchestrator (ReviewSession)
│   ├── config.py                 # ReviewConfig data model & registry
│   ├── helpers.py                # Prompt loading, parsing, PDF export
│   │
│   ├── backends/
│   │   ├── claude.py             # Anthropic Claude backend
│   │   ├── gemini.py             # Google Gemini backend
│   │   └── local.py              # Local LLM backend (GGUF / transformers)
│   │
│   └── review_types/
│       ├── nih/
│       │   ├── config.py         # NIH review configuration
│       │   ├── prompts/          # Agent prompt files
│       │   │   ├── primary_reviewer.txt
│       │   │   ├── secondary_reviewer.txt
│       │   │   ├── tertiary_reviewer.txt
│       │   │   ├── biostatistics.txt
│       │   │   ├── program_officer.txt
│       │   │   ├── challenge_pass.txt
│       │   │   ├── combined_reviewers.txt
│       │   │   ├── sro.txt
│       │   │   └── pi.txt
│       │   └── instructions/     # Scoring & criteria rubrics
│       │       ├── nih_scoring_scale.txt
│       │       ├── nih_criteria.txt
│       │       └── global_rules.txt
│       │
│       ├── foundation/
│       │   ├── config.py
│       │   ├── prompts/          # scientific_reviewer, innovation_reviewer,
│       │   │                     # program_advisor, panel_chair, applicant, ...
│       │   └── instructions/
│       │
│       └── journal/
│           ├── config.py
│           ├── prompts/          # domain_expert, technical_reviewer,
│           │                     # novelty_reviewer, editor, author, ...
│           └── instructions/
│
├── _shared/
│   ├── text.py                   # Text utilities (TOC, slugify, banner)
│   └── pdf.py                    # PDF utilities
│
└── tests/
    ├── test_engine.py            # Engine & workflow tests
    ├── test_config.py            # Config registry tests
    ├── test_helpers.py           # Parsing & helper tests
    ├── test_backends.py          # Backend tests
    ├── test_local_backend.py     # Local backend tests
    └── test_text.py              # Text utility tests

---

Testing

# Run all tests
python -m pytest tests/ -v

# Run a specific test file
python -m pytest tests/test_engine.py -v

# Run with coverage
python -m pytest tests/ --cov=review_engine --cov-report=term-missing

The test suite covers config validation, decision parsing, revision extraction, TOC generation, backend validation, and engine workflow logic. Tests that require optional dependencies (Gemini SDK, markitdown) are automatically skipped if not installed.

---

License

This project is provided as-is for research and educational purposes.