Quick start · Venues · How it works · Panels · Revision loop · Results · Troubleshooting

Give your paper a demanding first audience

Find the weak points in your paper. Revise them. Check whether the changes helped. Paper Review Harness turns a PDF into independent simulated reviews, section scores, and a prioritized revision plan. Thirty personas cover CVPR, ICLR, ICML and NeurIPS. Choose a venue, examine the evidence through distinct scientific lenses, and track each open criticism through later drafts.

Start here: Run a review with an agent · Read the operating rules · Explore the supporting research

This is an independent research tool for simulated feedback. It is not affiliated with these conferences, and its scores are not acceptance predictions.

Finish the useful work, then stop

A review request gets one careful panel review and its report. Revision is a separate requested task: make the changes, verify the evidence, and deliver. Additional panels, retests and score-optimization rounds are optional.

• Resume, do not restart: hx.py status <session> prints launch lines only for missing or invalid jobs.
• Skip duplicates: round creates no new round for byte-identical PDFs without changed, filled author answers. An empty response template does not count; use retest for an intentional repeat.
• Process the current work: aggregate validates and rebuilds only the latest round. Use --round N for an older repair or --all to rebuild history. Progress uses the saved reports.
• Focus follow-ups: changed-section hints direct attention to open points, changed evidence and dependent claims. Reviewers still inspect relevant figures; a text diff is not proof of unchanged visual evidence.
• Keep feedback concise: group duplicate concerns, reference evidence once, and leave resolved points closed unless new contrary evidence appears.

There is no hard cap on explicitly requested iterative work. Each additional pass needs a concrete purpose and new input; missing experiments are reported as blockers, not repeatedly requested through unchanged rounds. Existing sessions keep their frozen protocol text; start a new session to adopt the new concise review instructions.

Choose your venue

python 06_scripts/hx.py new --paper main.pdf --name mypaper --venue iclr
python 06_scripts/hx.py new --paper main.pdf --name mypaper --venue icml
python 06_scripts/hx.py new --paper main.pdf --name mypaper --venue neurips --contribution-type negative_results
python 06_scripts/hx.py personas --venue neurips

These are alternative examples; use a different --name or --tag if running several on the same day. Venue aliases are case-insensitive. Known full labels such as ICML 2026 are accepted; unknown years are rejected rather than silently assigned old rules.

Careful reading is the default for ICLR, ICML and NeurIPS. It checks central claims against specific evidence, inspects relevant appendices and figures, and records what was not checked. It removes forced score offsets, premature first-impression ratings and invented weaknesses. Add --reading careful to use it with CVPR too. The original CVPR behavior simulation remains available by default.

Every careful review includes an evidence_audit in its JSON. ICML and NeurIPS also include four separately justified venue dimensions. Validation checks the output structure and ranges; a human or orchestrator must still examine the scientific observations. Read unchecked coverage as an unresolved limitation, not evidence that the paper is wrong.

New sessions embed their rules, persona cards, rubrics, scales and section assignments so later rounds use the same instruments. Old sessions keep their legacy behavior. Scores from different profiles or reading modes are not directly comparable. Page budgets live in AUTHOR_CONSTRAINTS.md; ICLR 2027's budget requires confirmation before layout revisions.

Read the source-backed design and its limitations. The revision loop is for preparing a draft; it does not reproduce official rebuttal submission rules.

How it works

The Python CLI (cyan) prepares snapshots and prompts, validates outputs, and builds reports. It does not launch agents or call a model API. Your agent environment (violet) runs each reviewer in a separate context; a writer agent or the author revises the paper.

The figure shows the default dialogue loop. With --loop fresh, later rounds repeat full reviews from scratch instead of rechecking earlier points.

Quick start

1. Set up the harness

You need Python 3.9+, PyMuPDF, a paper PDF with extractable text, and an agent environment. A supplement is optional. Supply the paper's source directory if you want a writer agent to revise it.

git clone <repository-url> paper-review-harness
cd paper-review-harness
python -m pip install pymupdf
python 06_scripts/hx.py personas

These examples use python; use python3 if that is your Python command.

python -m venv ../cvpr-review-env
../cvpr-review-env/Scripts/Activate.ps1
python -m pip install pymupdf

python3 -m venv ../cvpr-review-env
source ../cvpr-review-env/bin/activate
python -m pip install pymupdf

[!TIP] Let an agent run steps 2–4. If your agent can read local files and start sub-agents, give it one line:

Follow REVIEW_PAPER.md for PAPER=path/to/main.pdf NAME=mypaper

It creates the session, starts one fresh sub-agent per reviewer, validates, aggregates, and reports the scores. The steps below do the same by hand.

2. Create a review session

python 06_scripts/hx.py new --paper "path/to/main.pdf" --name mypaper

Replace path/to/main.pdf with your PDF path. This uses the defaults: three reviewers, the realistic panel, and full review mode. The command prints a session path such as sessions/YYYY-MM-DD_mypaper and one launch instruction per reviewer.

For dedicated section-review jobs as well, add --mode both. This increases the number of jobs beyond the whole-paper panel.

3. Run independent reviewers

Give each printed prompt to a fresh agent, using exactly:

Read <prompt path> and follow it exactly. Write only the files it names.

Start them in parallel, each in its own context, and add nothing about the paper, your goals, or the other reviewers. Review files appear only after those agents finish. REVIEW_PAPER.md gives the orchestrating agent the full procedure.

4. Validate and read the results

Replace <session> below with the exact session path printed by new, or its full folder name under sessions/. The short --name alone is not the session identifier.

python 06_scripts/hx.py validate "<session>"
python 06_scripts/hx.py aggregate "<session>"

Run aggregate only after every job validates. Then open these files inside the printed session folder:

1. round_01/FEEDBACK.md — what to fix first, why, and what would resolve it.
2. round_01/SCORES.md — overall recommendations and section-level weaknesses.
3. round_01/reviews/<job>/debrief.md — what triggered a reviewer's reaction and the smallest useful change.

If a job fails validation, rerun that job with a fresh agent. Use python 06_scripts/hx.py status "<session>" to recover launch instructions for pending jobs only.

Choose your panel

The presets below are for CVPR. For ICLR, ICML and NeurIPS, use the venue-specific panels above.

A practical starting point: use realistic for a broad first review, writing to focus on presentation, or stress to probe difficult objections.

Use --seed 1 to change a drawn panel or --reviewers R03,R11 to select specific reviewers. The panel stays fixed across revisions within a session.

For a focused pass, add --sections abstract,introduction. The section panel uses its own section-to-persona mapping.

The revision loop

Review → prioritize → revise → check the answers → review again.

1. Prepare the writer. Record your source directory with --tex "path/to/latex" when creating the session, or set tex_dir in its SESSION.json to the absolute source-directory path before starting the writer. Then generate the brief and answer sheet:

   python 06_scripts/hx.py stage "<session>" writer
   

2. Address every open point. Run the printed writer prompt with a writer agent, or follow it yourself. Fill in writer/RESPONSE.md, edit the paper sources, rebuild the PDF, and record changes. Each answer should identify the evidence or exact change that resolves the concern. The stage command only prepares the brief and answer sheet.

3. Create the next round. Supply the revised PDF and, if applicable, the revised supplement:

   python 06_scripts/hx.py round "<session>" --paper "path/to/revised-main.pdf" --supp "path/to/revised-supplement.pdf"
   

   Omit --supp if there is no supplement. Pass it again when there is one; a revised round does not automatically reuse the previous supplement.

4. Run the printed prompts. In the default dialogue loop, fresh instances of the same personas check their own feedback threads against the revised draft. A point can be satisfied, partly, or not_satisfied; new issues can be raised. Held-out reviewers assess the draft from scratch.

5. Validate and aggregate again. Read the updated FEEDBACK.md and session-wide PROGRESS.md, then report completed fixes and remaining blockers. Start another revision only when useful work remains within the requested scope.

Finish once the requested changes are verified. Continue an explicitly requested iterative campaign only with a concrete unresolved issue and new evidence, a changed draft or a substantive answer. There is no hard round cap, but no automatic retesting, cosmetic churn or reopening of resolved points.

Use the session's AUTHOR_CONSTRAINTS.md to record project requirements, permitted experiments, and writing preferences. The writer reads this file; reviewers do not.

Check whether a revision helped

• Held-out reviewers: add --heldout R02,R11 when creating a session. Their reviews stay separate from the writer's feedback. Use a different model from the writer for held-out reviews and blind judging.
• Repeatability: python 06_scripts/hx.py retest "<session>" prepares fresh reviews of an unchanged draft. Run those prompts, validate, and aggregate to estimate score variability.
• Fresh reviews: choose --loop fresh at session creation to repeat the configured review jobs from scratch on later rounds. Unlike dialogue reviewers, these reviewers do not receive the earlier feedback thread or the writer's answers.
• Fix checks and blind comparison: after a revised round, use the commands below and run their printed prompts to check resolved weaknesses and compare changed sections.

python 06_scripts/hx.py stage "<session>" fixcheck
python 06_scripts/hx.py stage "<session>" pairwise

Read scores alongside the underlying evidence. Language-model reviewers can share blind spots, and small score changes can be noise. AGENTS.md defines the independence and revision-acceptance rules.

Reading the results

From criticism to a concrete next step

Every point in FEEDBACK.md names a location, a fix, and a test for resolving it. After a revision, it also carries the reviewer's latest verdict. An invented example, in the exact format the harness writes after one revision:

### R06:W2 - major, efficiency - The Efficiency Pragmatist [partly]
- **Where:** Sec. 5.3, Table 2
- **Complaint:** The 3x speedup is reported without hardware or batch size, so it cannot be checked.
- **Their suggested fix:** State the GPU, batch size, and resolution next to Table 2.
- **What would satisfy them:** Enough setup detail to tell whether the runtimes are comparable.
- **Fixable by (their guess):** writing
- **Their last verdict (partly):** The GPU is now stated, but the baselines were timed at a different batch size.
- **Still asking for:** Re-time the baselines at batch size 1, or explain why that is not possible.

The writer answers each point in RESPONSE.md with a change or an evidence-backed explanation. In the next round, a fresh instance of the same persona rules it satisfied (closed), partly, or not_satisfied (both stay open).

Understand the scores

Read the criticisms before optimizing the numbers. A higher mean can hide one decisive objection or a weaker section.

The headline uses held-out reviewers when present; otherwise it uses feedback reviewers. Section scores are a harness addition. Rating profiles and their source notes live in scales.json.

Output map

01_sources/       Research behind the review design
02_protocol/      Review forms, scoring rubrics, and writer rules
03_personas/      Reviewer persona cards and their registry
04_panels/        Whole-paper and section panel definitions
05_templates/     Output schemas and author constraints
06_scripts/       Python CLI, harness library, and smoke test
assets/           README logo, banner, and workflow figures
sessions/         Local paper snapshots and review campaigns

Troubleshooting

Development and sharing

Run the mechanics check after changes to scripts, personas, panels, or protocol:

python 06_scripts/hx.py personas
python 06_scripts/smoke_test.py

The smoke test uses a synthetic PDF and simulated review records; it makes no model calls.

Sessions contain paper snapshots and reviews and are excluded from Git by default. To package the reusable harness without sessions, caches, or build notes:

python 06_scripts/export_public.py

The exporter writes a folder and ZIP under release/ and scans exported text for identifying strings. Model execution and data handling depend on the agent environment you choose.