Annotation Reading List

A reading list of relevant papers and projects on foundation model annotation.

Consitutional AI and Self-Alignment

• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (May 2023)
• Constitutional AI: Harmlessness from AI Feedback (Dec 2022)

Critique

• LLM Critics Help Catch LLM Bugs (Jun 2024)
• CritiqueLLM: Scaling LLM-as-Critic for Effective and Explainable Evaluation of Large Language Model Generation (Nov 2023)
• Enabling Scalable Oversight via Self-Evolving Critic (Jan 2025)
• Self-critiquing models for assisting human evaluators (Jun 2022)
• Critique-out-Loud Reward Models (Aug 2024)
• Self-Generated Critiques Boost Reward Modeling for Language Models (Nov 2024)

Debate

• AI Safety via Debate (May 2018)
• Scalable AI Safety via Doubly-Efficient Debate (Nov 2023)
• Debating with More Persuasive LLMs Leads to More Truthful Answers (Jul 2024)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (May 2023)
• Scalable AI Safety via Doubly-Efficient Debate (Nov 2023)
• Debate Helps Weak-to-Strong Generalization (Jan 2025)
• Debate helps supervise unreliable experts (Nov 2023)

Iterated Amplification and Weak-to-Strong Generalization

• Supervising Strong Learners by Amplifying Weak Experts (Oct 2018)
• Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning (Feb 2024)
• Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision (Mar 2024)
• Eliciting Strong Capabilities With Weak Supervision (Dec 2023)

LLM-as-judge

• A Survey on LLM-as-a-Judge (Nov 2024)
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges (Jun 2024)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges (Oct 2023)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Jun 2023)

Misc. annotation techniques

• Scalable Oversight by Accounting for Unreliable Feedback (Jun 2024)
• Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback (Jan 2025)
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks (Mar 2023)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators (Mar 2023)
• LLMaAA: Making Large Language Models as Active Annotators (Oct 2023)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (Dec 2023)