Thang1703hrsh/Awesome-Human-AI-Alignment

Python

30 commits

updated Oct 3, 2026

See the code

README

Awesome Human–AI Alignment

A taxonomy-guided collection of research on specifying, supervising, implementing, and assuring Human–AI Alignment.

News

Overview

This survey provides a unified overview of Human–AI Alignment. We propose a lifecycle taxonomy that organizes the field along four dimensions—alignment specification, supervision, mechanisms, and assurance—and use it to structure the literature across 23 non-exclusive research branches.

Software framework

Reference components for 22 cited papers in Task & Assistance, Personalized Alignment, and Uncertainty & Drift are documented in Alignment specification, with explicit implementation limits and an offline demo.

Human Feedback and AI Feedback code, covering the supervision components of 17 cited papers, is documented in Alignment supervision.

The repository also provides a Python framework that mirrors the four lifecycle dimensions while keeping common workflows simple. The core has no runtime dependencies; training libraries are installed only for the methods that need them.

python -m pip install -e .
hai-align catalog validate
python examples/minimal_pipeline.py

To train with DPO:

python -m pip install -e ".[dpo]"
hai-align train dpo --model Qwen/Qwen3-0.6B --dataset trl-lib/ultrafeedback_binarized --output outputs/qwen-dpo

from human_alignment import DPO

run = DPO(
    model="Qwen/Qwen3-0.6B",
    dataset="trl-lib/ultrafeedback_binarized",
    output_dir="outputs/qwen-dpo",
).train()

print(run.generate("What is Human--AI alignment?"))

See Codebase architecture for the public API and extension points. The machine-readable taxonomy lives in src/human_alignment/catalog/data/taxonomy.json.

Preference distillation is available through one consistent API for VPD, PPD, DCKD, TVKD, ADPA, and CTPD:

python -m pip install -e ".[distillation]"

from human_alignment import VPD, PreferenceDistillationExample

data = [
    PreferenceDistillationExample(
        prompt="Explain alignment briefly.",
        responses=("Alignment connects behavior to human targets.", "It is model scaling."),
        teacher_scores=(1.0, 0.0),
    )
]
run = VPD(model="student-model", dataset=data, output_dir="outputs/student-vpd").train()

See Preference distillation for objective-specific dataset schemas, teacher-model use, and migration details.

The benchmarks the survey uses as evidence (RewardBench 2, JudgeBench, AlpacaEval 2) run under their official scoring rules, and three inference-time methods (CAA, best-of-N with a reward model, ARGS) steer a frozen model:

hai-align bench rewardbench2 --reward-model Skywork/Skywork-Reward-V2-Qwen3-0.6B --output-dir eval/rb2

See Benchmarks and inference-time methods.

Safety-alignment training is integrated under the same package, with 25 stages covering SafeRLHF, SafeDPO, BSO, SACPO, CAN, MODPO, CPO, BFPO, MidPO, reward and cost modeling, and multi-objective RLHF:

python -m pip install -e ".[safety]"
hai-align safety list
hai-align safety run recipes/safety_alignment/methods/safedpo.yaml

from human_alignment import SafetyAlignment

run = SafetyAlignment(
    config="recipes/safety_alignment/methods/safedpo.yaml",
    output_dir="outputs/safedpo",
).train()

See Safety alignment for recipe dependencies, paper-faithful implementation choices, evaluation, and H100 workflows.

Additional preference methods are available as IPO, BPO, TDPO, TISDPO, TIDPO, TBPOQ, and TBPOA, with recipes under recipes/preference_optimization/. See Preference optimization for installation, token-weight data formats, objective sources, and checkpoint handling.

The library supports both ends of an alignment experiment:

from human_alignment import DPO, load_checkpoint

# Fine-tune a pretrained model or an existing checkpoint.
trained = DPO(
    model="models/student_sft",
    dataset="data/preferences",
    output_dir="outputs/student_dpo",
).train()

# Load the resulting checkpoint later for inference or assurance.
checkpoint = load_checkpoint(
    "outputs/student_dpo",
    config={"device_map": "auto", "torch_dtype": "bfloat16"},
)
print(checkpoint.generate("Explain alignment."))

Preference-distillation supervision can be prepared without the temporary research scripts:

hai-align prepare distillation dckd \
  --dataset HuggingFaceH4/ultrafeedback_binarized \
  --teacher models/teacher_dpo \
  --tokenizer models/student_sft \
  --output data/ultrafeedback-dckd

Examples

Example	Purpose
`minimal_pipeline.py`	Validate the catalog and run a minimal end-to-end pipeline
`dpo_quickstart.py`	Launch DPO from Python
`dpo.toml`	Configure a DPO run declaratively
`checkpoint_workflow.py`	Train, reload, and use a checkpoint
`preference_distillation.py`	Prepare and run preference distillation

Taxonomy

Survey overview

Overview of the Human–AI Alignment survey framework and application settings

Human–AI alignment is organized as a lifecycle spanning alignment specification, supervision, mechanisms, and assurance across chat, code, mathematical reasoning, multimodal, agentic, robotics, and healthcare settings.

View the high-resolution survey overview (PDF)

Lifecycle taxonomy

Human–AI Alignment taxonomy

The lifecycle taxonomy organizes Human–AI alignment into four dimensions and eight groups; the paper collection below further resolves them into 23 non-exclusive research branches.

View the high-resolution taxonomy (PDF)

Dimension	Guiding question	Groups
Alignment Specification	What should an AI system align to, and whose objectives and values should count?	Alignment Objectives · Values & Stakeholders
Alignment Supervision	Where do alignment signals come from, and how are they expressed and scaled?	Feedback Source · Feedback & Oversight
Alignment Mechanisms	How are alignment signals translated into model behavior during training and inference?	Training-Time Alignment · Inference-Time Alignment
Alignment Assurance	How do we evaluate, stress-test, preserve, interpret, and monitor alignment?	Evaluation & Robustness · System Assurance

Training-time mechanisms at a glance

Five training-time alignment mechanisms

Training-time alignment covers reward and verifier modeling, supervised alignment, preference optimization, reinforcement learning, and alignment distillation.

View the high-resolution training-time diagram (PDF)

Inference-time mechanisms at a glance

Four inference-time alignment mechanisms

Inference-time alignment covers steering, search, iterative refinement, and interactive or agentic control while keeping the model policy fixed.

View the high-resolution inference-time diagram (PDF)

Collection coverage

172 representative assignments covering 167 citation keys in the compact figure, before alias normalization.
598 collection assignments across 23 terminal branches; repeated placement is intentional.
451 unique normalized works are linked from the collection.
6 cross-cutting surveys are listed separately after the branch collection.

Cross-cutting descriptors

System and modality: text, multimodal, agentic, and embodied systems.
Feedback granularity: response, span, reasoning step, token, and action.
Optimization granularity: sequence, segment, token, and trajectory.
Learning regime: offline, online, static, and continual learning.

Feedback granularity and optimization granularity are tracked separately because the unit receiving feedback can differ from the unit optimized by the learning objective.

Paper collection

Papers are sorted by year within each branch. Each entry links to the publication page, DOI, arXiv record, or a clearly labeled Scholar search when the BibTeX record has no direct link.

Alignment Specification

What should an AI system align to, and whose objectives and values should count?

Group	Branch	Papers
Alignment Objectives	Task & Assistance Alignment	35
Alignment Objectives	Safety Alignment	27
Values & Stakeholders	Personalized Alignment	24
Values & Stakeholders	Pluralistic & Societal Alignment	35
Values & Stakeholders	Context, Uncertainty & Drift	23

Alignment Objectives

Task & Assistance Alignment (35)

2026 · Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities — Purpura et al. · purpura2026mosaic · DOI
2026 · ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning — Kwon et al. · kwon2026reasonif · DOI
2026 · Revisiting the Reliability of Language Models in Instruction-Following — Dong et al. · dong2026ifevalpp · DOI
2026 · We Are What We Repeatedly Do: Improving Long Context Instruction Following — Robinette et al. · robinette2026verify · DOI
2026 · SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe — Xiao et al. · xiao2026sftmix · Paper
2025 · FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback — Li et al. · li2025fbbench · Paper
2025 · IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization — Zhang et al. · zhang2025iopo · Paper
2025 · LongReward: Improving Long-context Large Language Models with AI Feedback — Zhang et al. · zhang2025longreward · Paper
2025 · When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following — Harada et al. · harada2025manyifeval · Paper
2025 · A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users — Balepur et al. · balepur2025goodplan · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · UltraIF: Advancing Instruction Following from the Wild — An et al. · an2025ultraif · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · LIONs: An Empirically Optimized Approach to Align Language Models — Yu et al. · yu2024lions · Paper
2024 · Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models — Sun et al. · sun2024parrot · Paper
2024 · RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models — Niu et al. · niu2024ragtruth · Paper
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Direct Preference Optimization: Your Language Model Is Secretly a Reward Model — Rafailov et al. · rafailov2023dpo · Paper
2023 · Enabling Large Language Models to Generate Text with Citations — Gao et al. · gao2023citations · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF — Dong et al. · dong2023steerlm · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Multitask Prompted Training Enables Zero-Shot Task Generalization — Sanh et al. · sanh2022t0 · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2021 · A General Language Assistant as a Laboratory for Alignment — Askell et al. · askell2021assistant · Paper
2021 · WebGPT: Browser-Assisted Question-Answering with Human Feedback — Nakano et al. · nakano2021webgpt · Paper

Safety Alignment (27)

2026 · LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignment — Zhang et al. · zhang2026llmva · Paper
2026 · Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy — Jiang et al. · jiang2026els · Paper
2026 · More Thinking, Less Talking: Internalizing Deliberative Safety into LLM Parameters — Wang et al. · wang2026hiar · Paper
2026 · Reasoning Structure Matters for Safety Alignment of Reasoning Models — In et al. · in2026alttrain · Paper
2026 · Safety Is Not Universal: The Selective Safety Trap in LLM Alignment — Brito et al. · brito2026safetyisnotuniversal · DOI
2026 · AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning — Zhang et al. · zhang2026alphaalign · Paper
2026 · LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety — Yang et al. · yang2026lasa · Paper
2026 · Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization — Niu et al. · niu2026nspo · Paper
2026 · Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance — Zhang et al. · zhang2026safetyoneshot · Paper
2026 · SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety — Kim et al. · kim2026safedpo · Paper
2025 · Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements — Zhang et al. · zhang2025controllablesafetyalignment · Paper
2025 · POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization — Karaman et al. · karaman2025porover · Paper
2025 · Safety Alignment Can Be Not Superficial With Explicit Safety Signals — Li et al. · li2025explicitsafety · Paper
2025 · Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models — Gao et al. · gao2025abd · Paper
2025 · Alignment of Large Language Models with Constrained Learning — Zhang et al. · zhang2025constrainedalignment · Paper
2025 · Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2025antidote · Paper
2025 · Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment — Li et al. · li2025larf · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2024 · Deliberative Alignment: Reasoning Enables Safer Language Models — Guan et al. · guan2024deliberative · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2024 · Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024vaccine · Paper
2023 · BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset — Ji et al. · ji2023beavertails · Paper
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2021 · A General Language Assistant as a Laboratory for Alignment — Askell et al. · askell2021assistant · Paper

Values & Stakeholders

Alignment Supervision

Where do alignment signals come from, and how are they expressed and scaled?

Group	Branch	Papers
Feedback Source	Human Feedback	20
Feedback Source	AI Feedback	23
Feedback Source	Programmatic & Verifiable Feedback	22
Feedback & Oversight	Demonstrations & Preferences	24
Feedback & Oversight	Critique, Process & Trajectory Feedback	19
Feedback & Oversight	Reliable & Scalable Oversight	25

Feedback Source

Human Feedback (20)

2026 · WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback — Shi et al. · shi2026wildfeedback · Paper
2025 · Aligning Language Models with Demonstrated Feedback — Shaikh et al. · shaikh2025ditto · Paper
2025 · Binary Classifier Optimization for Large Language Model Alignment — Jung et al. · jung2025bco · Paper
2025 · Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback — Miranda et al. · miranda2025hybrid · Paper
2025 · MM-RLHF: The Next Step Forward in Multimodal LLM Alignment — Zhang et al. · zhang2025mmrlhf · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · Aligning Large Language Models via Fine-grained Supervision — Xu et al. · xu2024finegrained · Paper
2024 · ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback — Hou et al. · hou2024chatglmrlhf · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · Towards Aligning Language Models with Textual Feedback — Lloret et al. · lloret2024alt · Paper
2023 · HelpSteer: Multi-Attribute Human Preference Dataset for Training and Evaluation of Large Language Models — Wang et al. · wang2023helpsteer · arXiv
2023 · OpenAssistant Conversations – Democratizing Large Language Model Alignment — Köpf et al. · kopf2023openassistant · arXiv
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2019 · Fine-Tuning Language Models from Human Preferences — Ziegler et al. · ziegler2019finetuning · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

AI Feedback (23)

2026 · Aligning Large Language Models via Fully Self-Synthetic Data — Yin et al. · yin2026sao · Paper
2026 · Check Your Work: Structured Checklist Feedback for Improving Large Language Models — Cook et al. · cook2026checklist · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · Diverse AI Feedback For Large Language Model Alignment — Yu et al. · yu2025daif · Paper
2025 · Learning LLM-as-a-Judge for Preference Alignment — Ye et al. · ye2025conj · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2024 · LLM Critics Help Catch LLM Bugs — McAleese et al. · mcaleese2024critics · Paper
2024 · Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation — Liu et al. · liu2024dlma · Paper
2024 · RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Lee et al. · lee2024rlaif · Paper
2024 · Self-Rewarding Language Models — Yuan et al. · yuan2024selfrewarding · Scholar
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2024 · VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment — Li et al. · li2024vlfeedback · Paper
2023 · Enhancing Chat Language Models by Scaling High-quality Instruction Conversations — Ding et al. · ding2023ultrachat · arXiv
2023 · G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — Liu et al. · liu2023geval · arXiv
2023 · JudgeLM: Fine-tuned Large Language Models Are Scalable Judges — Zhu et al. · zhu2023judgelm · arXiv
2023 · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. · zheng2023mtbench · Paper
2023 · Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — Mukherjee et al. · mukherjee2023orca · arXiv
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Self-Alignment: Building an Open-source and Self-Instructive Language Model — Sun et al. · sun2023selfalignment · Scholar
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2023 · WizardLM: Empowering Large Language Models with Evol-Instruct — Xu et al. · xu2023wizardlm · arXiv
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper

Programmatic & Verifiable Feedback (22)

2026 · Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse Domains — Su et al. · su2026rewardbridge · Paper
2026 · Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains — Yuan et al. · yuan2026k2v · Paper
2025 · OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees — Zhang et al. · zhang2025openprm · Paper
2025 · DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI · deepseek2025r1 · Paper
2025 · Generative Verifiers: Reward Modeling as Next-Token Prediction — Zhang et al. · zhang2025genrm · Paper
2025 · Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — Setlur et al. · setlur2025rewarding · Paper
2025 · RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning — Gehring et al. · gehring2025rlef · Paper
2025 · Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models — Dong et al. · dong2025autoif · Paper
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models — Singh et al. · singh2024restem · Paper
2024 · DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search — Xin et al. · xin2024deepseekprover15 · DOI
2024 · DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al. · shao2024deepseekmath · Paper
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · V-STaR: Training Verifiers for Self-Taught Reasoners — Hosseini et al. · hosseini2024vstar · Paper
2023 · CodeT: Code Generation with Generated Tests — Chen et al. · chen2023codet · Paper
2023 · LeanDojo: Theorem Proving with Retrieval-Augmented Language Models — Yang et al. · yang2023leandojo · Paper
2023 · RLTF: Reinforcement Learning from Unit Test Feedback — Liu et al. · liu2023rltf · Paper
2022 · CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — Le et al. · li2022coderl · Paper
2022 · STaR: Bootstrapping Reasoning With Reasoning — Zelikman et al. · zelikman2022star · Paper
2021 · Evaluating Large Language Models Trained on Code — Chen et al. · chen2021humaneval · arXiv
2021 · Measuring Coding Challenge Competence With APPS — Hendrycks et al. · hendrycks2021apps · arXiv

Feedback & Oversight

Demonstrations & Preferences (24)

2026 · WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback — Shi et al. · shi2026wildfeedback · Paper
2025 · Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback — Miranda et al. · miranda2025hybrid · Paper
2025 · Aligning Language Models with Demonstrated Feedback — Shaikh et al. · shaikh2025ditto · Paper
2025 · Aligning Large Language Models with Implicit Preferences from User-Generated Content — Tan et al. · tan2025pugc · Paper
2025 · Binary Classifier Optimization for Large Language Model Alignment — Jung et al. · jung2025bco · Paper
2025 · Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization — Xiao et al. · xiao2025sweetspot · Paper
2025 · Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment — Li et al. · li2025joint · Paper
2024 · Aligning Large Language Models via Fine-grained Supervision — Xu et al. · xu2024finegrained · Paper
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment — Li et al. · li2024demoreward · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Critique, Process & Trajectory Feedback (19)

2026 · Critique-RL: Training Language Models for Critiquing Through Two-Stage Reinforcement Learning — Xi et al. · xi2026critiquerl · Paper
2026 · Exploring Reasoning Reward Model for Agents — Fan et al. · fan2026agentrrm · Paper
2026 · The Bidirectional Process Reward Model — Zhang et al. · zhang2026biprm · Paper
2025 · Dynamic and Generalizable Process Reward Modeling — Yin et al. · yin2025dgprm · Paper
2025 · OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees — Zhang et al. · zhang2025openprm · Paper
2025 · Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — Setlur et al. · setlur2025rewarding · Paper
2025 · Self-Generated Critiques Boost Reward Modeling for Language Models — Yu et al. · yu2025criticrm · Paper
2025 · Teaching Language Models to Critique via Reinforcement Learning — Xie et al. · xie2025ctrl · Paper
2025 · Training Language Model to Critique for Better Refinement — Yu et al. · yu2025rco · Paper
2024 · CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing — Gou et al. · gou2024critic · Scholar
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · LLM Critics Help Catch LLM Bugs — McAleese et al. · mcaleese2024critics · Paper
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper

Reliable & Scalable Oversight (25)

2026 · Agentic Oversight via Dialectic Reasoning — Ranaldi et al. · ranaldi2026agentic · Paper
2026 · Selective Weak-to-Strong Generalization — Lang et al. · lang2026selective · DOI
2026 · Towards Scalable Oversight via Partitioned Human Supervision — Yin et al. · yin2026partitioned · Paper
2026 · Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction — Qiu et al. · qiu2026peerprediction · Paper
2025 · AI Debate Aids Assessment of Controversial Claims — Rahman et al. · rahman2025aidebate · Scholar
2025 · Barriers and Pathways to Human-AI Alignment: A Game-Theoretic Approach — Nayebi · nayebi2025barriers · Paper
2025 · Debate Helps Weak-to-Strong Generalization — Lang et al. · lang2025debate · Paper
2025 · Great Models Think Alike and this Undermines AI Oversight — Goel et al. · goel2025oversight · Paper
2025 · Neural Interactive Proofs — Hammond et al. · hammond2025neuralproofs · Paper
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · Scaling Laws for Scalable Oversight — Engels et al. · engels2025scaling · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · Debating with More Persuasive LLMs Leads to More Truthful Answers — Khan et al. · khan2024debate · Paper
2024 · Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision — Sun et al. · sun2024easyhard · Paper
2024 · On Scalable Oversight with Weak LLMs Judging Strong LLMs — Kenton et al. · kenton2024scalableoversight · Paper
2024 · Prover-Verifier Games Improve Legibility of LLM Outputs — Kirchner et al. · kirchner2024proververifier · Paper
2024 · Reward Modeling Requires Automatic Adjustment Based on Data Quality — Wang et al. · wang2024rmquality · DOI
2024 · RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences — Cheng et al. · cheng2024rime · Scholar
2024 · Self-Exploring Language Models: Active Preference Elicitation for Online Alignment — Zhang et al. · zhang2024selm · Paper
2024 · Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision — Burns et al. · burns2024weakstrong · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · Scaling Laws for Reward Model Overoptimization — Gao et al. · gao2023overoptimization · Paper
2018 · AI Safety via Debate — Irving et al. · irving2018debate · Paper
2018 · Supervising Strong Learners by Amplifying Weak Experts — Christiano et al. · christiano2018amplification · Paper

Alignment Mechanisms

How are alignment signals translated into model behavior during training and inference?

Group	Branch	Papers
Training-Time Alignment	Reward & Verifier Modeling	27
Training-Time Alignment	Supervised Alignment	22
Training-Time Alignment	Preference Optimization	50
Training-Time Alignment	Reinforcement Learning	32
Training-Time Alignment	Alignment Distillation	9
Inference-Time Alignment	Steering, Search & Refinement	31
Inference-Time Alignment	Interactive & Agentic Control	17

Training-Time Alignment

Reward & Verifier Modeling (27)

2026 · OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment — Liu et al. · liu2026openrubrics · Paper
2026 · Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models — Wang et al. · wang2026rationale · Paper
2026 · PRISM: Probabilistic Reward Model with Inherent Structural Modeling — Zhou et al. · zhou2026prismrm · Paper
2026 · RM-R1: Reward Modeling as Reasoning — Chen et al. · chen2026rmr1 · Paper
2026 · The Bidirectional Process Reward Model — Zhang et al. · zhang2026biprm · Paper
2026 · AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling — Miao et al. · miao2026adajudge · Paper
2025 · Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems — Peng et al. · peng2025agenticrm · Paper
2025 · Dynamic and Generalizable Process Reward Modeling — Yin et al. · yin2025dgprm · Paper
2025 · Generative Verifiers: Reward Modeling as Next-Token Prediction — Zhang et al. · zhang2025genrm · Paper
2025 · GRAM: A Generative Foundation Reward Model for Reward Generalization — Wang et al. · wang2025gram · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2025 · Reward Reasoning Models — Guo et al. · guo2025rrm · DOI
2025 · PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment — Chen et al. · chen2025pal · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Prover-Verifier Games Improve Legibility of LLM Outputs — Kirchner et al. · kirchner2024proververifier · Paper
2024 · Reward Model Ensembles Help Mitigate Overoptimization — Coste et al. · coste2024ensembles · Paper
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2019 · Fine-Tuning Language Models from Human Preferences — Ziegler et al. · ziegler2019finetuning · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Supervised Alignment (22)

2026 · OASIS: Online Sample Selection for Continual Instruction Tuning — Lee et al. · lee2026oasis · Paper
2026 · SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe — Xiao et al. · xiao2026sftmix · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · MAIN: Mutual Alignment Is Necessary for Instruction Tuning — Yang et al. · yang2025main · Paper
2025 · Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality — Harada et al. · harada-etal-2025-massive · Paper
2025 · Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process — Hua et al. · hua-etal-2025-intuitive · Paper
2025 · The Best Instruction-Tuning Data are Those That Fit — Zhang et al. · zhang2025grape · Scholar
2024 · LESS: Selecting Influential Data for Targeted Instruction Tuning — Xia et al. · xia2024less · Paper
2024 · MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models — Yang et al. · yang2024metaaligner · DOI
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection — Liu et al. · liu2024selectit · Paper
2024 · Self-Alignment with Instruction Backtranslation — Li et al. · li2024backtranslation · Paper
2024 · What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning — Liu et al. · liu2024deita · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF — Dong et al. · dong2023steerlm · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Multitask Prompted Training Enables Zero-Shot Task Generalization — Sanh et al. · sanh2022t0 · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper

Preference Optimization (50)

2026 · Causal Direct Preference Optimization for Language Model Alignment — Le et al. · le2026causaldpo · Paper
2026 · Keep the Best, Forget the Rest: Reliable Alignment with Order-Aware Preference Optimization — Zhu et al. · zhu2026rappo · Paper
2026 · Mitigating Mismatch within Reference-based Preference Optimization — Yuan et al. · yuan2026hypo · Paper
2026 · Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium — Liu et al. · liu2026statisticalalignment · DOI
2026 · Autoregressive Direct Preference Optimization — Oi et al. · oi2026autoregressive · Paper
2026 · BSO: Safety Alignment Is Density Ratio Matching — Nguyen et al. · nguyen2026bso · DOI
2026 · SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety — Kim et al. · kim2026safedpo · Paper
2026 · Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization — Luo et al. · luo2026sharpness · Paper
2026 · Token-Importance Guided Direct Preference Optimization — Yang et al. · yang2026token · Paper
2026 · TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching — Nguyen et al. · nguyen2026tokenratio · Paper
2025 · Alignment of Large Language Models with Constrained Learning — Zhang et al. · zhang2025constrainedalignment · Paper
2025 · AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization — Wu et al. · wu2025alphadpo · Paper
2025 · Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment — D'Oosterlinck et al. · doosterlinck2025apo · Paper
2025 · Doubly Robust Alignment for Large Language Models — Xu et al. · xu2025doublyrobust · Paper
2025 · KL Penalty Control via Perturbation for Direct Preference Optimization — Lee et al. · lee2025kl · Paper
2025 · Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment — Wang et al. · wang2025mpo · Paper
2025 · MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization — Zhu et al. · zhu2025mmedpo · Paper
2025 · No Preference Left Behind: Group Distributional Preference Optimization — Yao et al. · yao2025gdpo · Paper
2025 · Preference Optimization by Estimating the Ratio of the Data Distribution — Kim et al. · kim2025bregmanpo · Paper
2025 · Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO — Guo et al. · guo2025pro · Paper
2025 · Reward-Aware Preference Optimization: A Unified Mathematical Framework for Model Alignment — Sun et al. · sun2025rpo · Paper
2025 · Risk-aware Direct Preference Optimization under Nested Risk Measure — Zhang et al. · zhang2025risk · Paper
2025 · SDPO: Segment-Level Direct Preference Optimization for Social Agents — Kong et al. · kong2025sdpo · Paper
2025 · Self-Play Preference Optimization for Language Model Alignment — Wu et al. · wu2025sppo · Paper
2025 · SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment — Zhu et al. · zhu2025sgdpo · Paper
2025 · T-REG: Preference Optimization with Token-Level Reward Regularization — Zhou et al. · zhou2025treg · Paper
2025 · TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights — Liu et al. · liu2025tisdpo · Paper
2025 · Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization — Wu et al. · wu2025drdpo · Paper
2025 · Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model — Zhu et al. · zhu2025wspo · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment — Xu et al. · xu2024behaviorbpo · Paper
2024 · Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation — Xu et al. · xu2024cpo · Paper
2024 · Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences — Rosset et al. · rosset2024dno · Paper
2024 · KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh et al. · ethayarajh2024kto · Paper
2024 · MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models — Yang et al. · yang2024metaaligner · DOI
2024 · Multi-turn Reinforcement Learning from Preference Human Feedback — Shani et al. · shani2024multiturn · Scholar
2024 · Nash Learning from Human Feedback — Munos et al. · munos2024nash · Paper
2024 · ORPO: Monolithic Preference Optimization without Reference Model — Hong et al. · hong2024orpo · Paper
2024 · Provably Robust DPO: Aligning Language Models with Noisy Feedback — Ray Chowdhury et al. · chowdhury2024rdpo · Paper
2024 · RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences — Cheng et al. · cheng2024rime · Scholar
2024 · SAIL: Self-Improving Efficient Online Alignment of Large Language Models — Ding et al. · ding2024sail · Paper
2024 · Self-Exploring Language Models: Active Preference Elicitation for Online Alignment — Zhang et al. · zhang2024selm · Paper
2024 · SimPO: Simple Preference Optimization with a Reference-Free Reward — Meng et al. · meng2024simpo · Paper
2024 · Token-Level Direct Preference Optimization — Zeng et al. · zeng2024tdpo · Paper
2024 · β-DPO: Direct Preference Optimization with Dynamic β — Wu et al. · wu2024beta · Paper
2023 · A General Theoretical Paradigm to Understand Learning from Human Preferences — Azar et al. · azar2023psipo · Paper
2023 · Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization — Zhao et al. · zhao2023hadpo · Paper
2023 · Direct Preference Optimization: Your Language Model Is Secretly a Reward Model — Rafailov et al. · rafailov2023dpo · Paper
2023 · RRHF: Rank Responses to Align Language Models with Human Feedback — Yuan et al. · yuan2023rrhf · Paper
2023 · SLiC-HF: Sequence Likelihood Calibration with Human Feedback — Zhao et al. · zhao2023slichf · Paper

Reinforcement Learning (32)

2026 · AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning — Zhang et al. · zhang2026alphaalign · Paper
2026 · From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation — Jiang et al. · jiang2026rlvrr · Paper
2026 · Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization — Niu et al. · niu2026nspo · Paper
2026 · Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models — Zhang et al. · zhang2026corewarding · Paper
2026 · GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning — Chu et al. · chu2026gpg · Paper
2026 · RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments — Zeng et al. · zeng2026rlve · arXiv
2026 · Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning — Deng et al. · deng2026srl · Paper
2025 · Checklists Are Better Than Reward Models for Aligning Language Models — Viswanathan et al. · viswanathan2025rlcf · Paper
2025 · DAPO: Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization — Liu et al. · liu2025dapo_advantage · Paper
2025 · DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI · deepseek2025r1 · Paper
2025 · Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization — Peng et al. · peng2025repo · Paper
2025 · MM-RLHF: The Next Step Forward in Multimodal LLM Alignment — Zhang et al. · zhang2025mmrlhf · Paper
2025 · Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems — Walder & Karkhanis · walder2025passk · Paper
2025 · SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning — Liu et al. · liu2025saturn · Paper
2025 · Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models — Guo et al. · guo2025spo · Paper
2025 · SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data — Fang et al. · fang2025serl · Paper
2025 · Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for LLMs — Le Roux et al. · leroux2025tapered · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs — Ahmadian et al. · ahmadian2024reinforce · Paper
2024 · ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback — Hou et al. · hou2024chatglmrlhf · Paper
2024 · DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al. · shao2024deepseekmath · Paper
2024 · Multi-turn Reinforcement Learning from Preference Human Feedback — Shani et al. · shani2024multiturn · Scholar
2024 · ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models — Li et al. · li2024remax · Paper
2024 · RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Lee et al. · lee2024rlaif · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2024 · SAIL: Self-Improving Efficient Online Alignment of Large Language Models — Ding et al. · ding2024sail · Paper
2023 · A Survey of Reinforcement Learning from Human Feedback — Kaufmann et al. · kaufmann2023survey · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Alignment Distillation (9)

2026 · CTPD: Cross Tokenizer Preference Distillation — Nguyen et al. · nguyen2026ctpd · Paper
2025 · Advantage-Guided Distillation for Preference Alignment in Small Language Models — Gao et al. · gao2025adpa · Paper
2025 · AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation — Zhang et al. · zhang2025aligndistil · Paper
2025 · Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models — Gu et al. · gu2025pad · Paper
2025 · Preference Distillation via Value based Reinforcement Learning — Kwon et al. · kwon2025tvkd · Paper
2024 · CycleAlign: Iterative Distillation from Black-Box LLM to White-Box Models for Better Human Alignment — Hong et al. · hong2024cyclealign · Paper
2024 · Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation — Liu et al. · liu2024dlma · Paper
2024 · Direct Preference Knowledge Distillation for Large Language Models — Li et al. · li2024dpkd · Paper
2024 · PLaD: Preference-Based Large Language Model Distillation with Pseudo-Preference Pairs — Zhang et al. · zhang2024plad · Paper

Inference-Time Alignment

2026 · A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement — Li et al. · li2026trainingfree · arXiv
2026 · Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior — Nguyen et al. · nguyen2026beyondlinear · arXiv
2026 · CyberCorrect: A Cybernetic Framework for Closed-Loop Self-Correction in Large Language Models — Wu et al. · wu2026cybercorrect · arXiv
2026 · Don't Lose Focus: Activation Steering via Key-Orthogonal Projections — Luo et al. · luo2026dontlosefocus · arXiv
2026 · ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment — Zhao et al. · zhao2026odesteer · arXiv
2026 · Spherical Steering: Geometry-Aware Activation Rotation for Language Models — You et al. · you2026spherical · arXiv
2026 · Activation Steering with a Feedback Controller — Nguyen et al. · nguyen2026pidsteering · Paper
2026 · FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models — Weng et al. · weng2026finesteer · Paper
2026 · One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement — Zhou et al. · zhou2026requer · Paper
2025 · Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning — Snell et al. · snell2025scaling · Paper
2025 · SSR: Socratic Self-Refine for Large Language Model Reasoning — Shi et al. · shi2025ssr · arXiv
2025 · Unleashing the True Potential of LLMs: A Feedback-Triggered Self-Correction with Long-Term Multipath Decoding — Li et al. · li2025unleashing · arXiv
2025 · W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search — Ding et al. · ding2025w2s · arXiv
2025 · ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment — Masters et al. · masters2025arcane · Paper
2025 · Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements — Zhang et al. · zhang2025controllablesafetyalignment · Paper
2025 · Improving Instruction-Following in Language Models through Activation Steering — Stolfo et al. · stolfo2025steering · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2025 · Nudging: Inference-time Alignment of LLMs via Guided Decoding — Fei et al. · fei2025nudging · Paper
2025 · Reward-Guided Tree Search for Inference Time Alignment of Large Language Models — Hung et al. · hung2025darwin · Paper
2025 · S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ma et al. · ma2025s2r · Paper
2024 · Controlled Decoding from Language Models — Mudgal et al. · mudgal2023controlled · Paper
2024 · ARGS: Alignment as Reward-Guided Search — Khanov et al. · khanov2024args · Paper
2024 · CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing — Gou et al. · gou2024critic · Scholar
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Steering Language Models with Activation Engineering — Rimsky et al. · rimsky2024caa · DOI
2023 · Activation Addition: Steering Language Models Without Optimization — Turner et al. · turner2023activationaddition · Paper
2023 · Inference-Time Intervention: Eliciting Truthful Answers from a Language Model — Li et al. · li2023iti · DOI
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al. · zou2023repreng · Paper
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2021 · WebGPT: Browser-Assisted Question-Answering with Human Feedback — Nakano et al. · nakano2021webgpt · Paper

Interactive & Agentic Control (17)

2026 · Steering LLMs via Scalable Interactive Oversight — Zhou et al. · zhou2026steering · arXiv
2026 · Designing Meaningful Human Oversight in AI — Zhu et al. · zhu2026oversight · Paper
2026 · Structured Uncertainty guided Clarification for LLM Agents — Suri et al. · suri2026clarification · Paper
2025 · Scalable Human Oversight for Aligned Large Language Models: A Hybrid Framework for Intent Fidelity — Ayankoya et al. · ayankoya2025scalable · DOI
2025 · The Burden of Interactive Alignment with Inconsistent Preferences — Shirali, Ali · shirali2025burden · arXiv
2025 · Corrigibility Transformation: Constructing Goals That Accept Updates — Hudson · hudson2025corrigibility · Paper
2025 · Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance — He et al. · he2025aria · Paper
2025 · Policy Optimization under Imperfect Human Interactions with Agent-Gated Shared Autonomy — Xue et al. · xue2025agsa · Paper
2025 · Position: AI Agents Need Authenticated Delegation — South et al. · south2025delegation · Paper
2025 · The Partially Observable Off-Switch Game — Garber et al. · garber2025offswitch · Paper
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · STaR-GATE: Teaching Language Models to Ask Clarifying Questions — Andukuri et al. · andukuri2024stargate · Scholar
2020 · Consistent Estimators for Learning to Defer to an Expert — Mozannar et al. · mozannar2020defer · Scholar
2017 · The Off-Switch Game — Hadfield-Menell et al. · hadfieldmenell2017offswitch · DOI
2016 · Cooperative Inverse Reinforcement Learning — Hadfield-Menell et al. · hadfieldmenell2016cirl · Scholar
2016 · Safely Interruptible Agents — Orseau et al. · orseau2016interruptible · Scholar
2015 · Corrigibility — Soares et al. · soares2015corrigibility · Paper

Alignment Assurance

How do we evaluate, stress-test, preserve, interpret, and monitor alignment?

Group	Branch	Papers
Evaluation & Robustness	Behavioral & Evaluator Evaluation	26
Evaluation & Robustness	Adversarial & Distribution Robustness	30
Evaluation & Robustness	Alignment Preservation	17
System Assurance	Mechanistic & Theoretical Evidence	31
System Assurance	Monitoring & Auditing	29

Evaluation & Robustness

Behavioral & Evaluator Evaluation (26)

2026 · Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling — Wang et al. · wang2026planrewardbench · DOI
2026 · IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation — Wen et al. · wen2026ifrewardbench · Paper
2026 · LLMs Get Lost In Multi-Turn Conversation — Laban et al. · laban2026lost · Paper
2026 · Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following — Xiong et al. · xiong2026multicrit · Paper
2026 · RewardBench 2: Advancing Reward Model Evaluation — Malik et al. · malik2026rewardbench2 · Paper
2026 · SoSBench: Benchmarking Safety Alignment on Six Scientific Domains — Jiang et al. · jiang2026sosbench · Paper
2025 · ChameleonBench: Quantifying Alignment Faking in Large Language Models — Chaudhury et al. · chaudhury2025chameleonbench · Paper
2025 · JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Tan et al. · tan2025judgebench · Paper
2025 · LiveBench: A Challenging, Contamination-Limited LLM Benchmark — White et al. · white2025livebench · Paper
2025 · PRMBench: A Fine-Grained and Challenging Benchmark for Process-Level Reward Models — Song et al. · song2025prmbench · Paper
2025 · RAG-RewardBench: Benchmarking Reward Models in Retrieval-Augmented Generation — Jin et al. · ragrewardbench2025 · Paper
2025 · Reasoning Models Don't Always Say What They Think — Chen et al. · chen2025cotfaithfulness · Paper
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild — Lin et al. · lin2025wildbench · Scholar
2024 · Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al. · chiang2024arena · Scholar
2024 · HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models — Guan et al. · guan2024hallusionbench · Scholar
2024 · HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal — Mazeika et al. · mazeika2024harmbench · Paper
2024 · RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models — Niu et al. · niu2024ragtruth · Paper
2024 · SafetyBench: Evaluating the Safety of Large Language Models — Zhang et al. · zhang2024safetybench · Paper
2024 · XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models — Rottger et al. · rottger2024xstest · Paper
2023 · AlpacaEval: An Automatic Evaluator of Instruction-Following Models — Li et al. · li2023alpacaeval · Paper
2023 · Enabling Large Language Models to Generate Text with Citations — Gao et al. · gao2023citations · Paper
2023 · FActScore: Fine-Grained Atomic Evaluation of Factual Precision in Long Form Text Generation — Min et al. · min2023factscore · Paper
2023 · Instruction-Following Evaluation for Large Language Models — Zhou et al. · zhou2023ifeval · Paper
2023 · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. · zheng2023mtbench · Paper
2022 · TruthfulQA: Measuring How Models Mimic Human Falsehoods — Lin et al. · lin2022truthfulqa · Paper

Adversarial & Distribution Robustness (30)

2026 · Safety Is Not Universal: The Selective Safety Trap in LLM Alignment — Brito et al. · brito2026safetyisnotuniversal · DOI
2026 · Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases — Hahm et al. · hahm2026alignmenttampering · Paper
2026 · ERTS: Adversarial Robustness Testing of Ethical AI via Semantic Perturbation in a Bounded Consequence Space — Chaudhari · chaudhari2026erts · Paper
2026 · Into the Gray Zone: Domain Contexts Can Blur LLM Safety Boundaries — Hung et al. · hung2026grayzone · Paper
2026 · Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models — Lian et al. · lian2026ethicalvulnerability · DOI
2026 · SafetyMem: Adaptive Jailbreak Defense via Dual-Component Safety Memory — Wang et al. · wang2026safetymem · Paper
2026 · When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models — Choi et al. · choi2026context · Paper
2026 · When Safety Alignment Fails to Generalize: Probing with Language Game Jailbreaks — Long et al. · long2026languagegames · Paper
2025 · Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — Sharma et al. · sharma2025constitutionalclassifiers · Paper
2025 · Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Betley et al. · betley2025emergent · Paper
2025 · Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models — Shen et al. · shen2025antidote · Paper
2025 · Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks — Andriushchenko et al. · andriushchenko2025adaptive · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2025 · Alignment Faking in Large Language Models — Greenblatt et al. · greenblatt2025alignmentfaking · Paper
2024 · CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion — Ren et al. · ren2024codeattack · DOI
2024 · A StrongREJECT for Empty Jailbreaks — Souly et al. · souly2024strongreject · Paper
2024 · BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment — Wang et al. · wang2024backdooralign · Paper
2024 · Fine-Tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Qi et al. · qi2024finetuning · Paper
2024 · HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal — Mazeika et al. · mazeika2024harmbench · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Hubinger et al. · hubinger2024sleeper · Paper
2024 · Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models — Denison et al. · denison2024subterfuge · Paper
2024 · XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models — Rottger et al. · rottger2024xstest · Paper
2023 · Holistic Evaluation of Language Models — Liang et al. · liang2023helm · Scholar
2023 · Model Evaluation for Extreme Risks — Shevlane et al. · shevlane2023extremerisks · Paper
2023 · Universal and Transferable Adversarial Attacks on Aligned Language Models — Zou et al. · zou2023jailbreak · Paper
2022 · Goal Misgeneralization in Deep Reinforcement Learning — Di Langosco et al. · langosco2022goal · Paper
2022 · Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals — Shah et al. · shah2022goal · Paper
2022 · Red Teaming Language Models with Language Models — Perez et al. · perez2022redteaming · DOI
2016 · Concrete Problems in AI Safety — Amodei et al. · amodei2016concrete · Paper

Alignment Preservation (17)

2026 · Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance — Zhang et al. · zhang2026safetyoneshot · Paper
2026 · Alignment Midtraining for Animals — Brazilek et al. · brazilek2026alignment · Paper
2026 · Continual Safety Alignment via Gradient-Based Sample Selection — Bach et al. · bach2026continual · Paper
2026 · Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment — Wang et al. · wang2026sot · Paper
2026 · SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging — Djuhera et al. · djuhera2026safemerge · Paper
2025 · Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2025antidote · Paper
2025 · COPR: Continual Human Preference Learning via Optimal Policy Regularization — Zhang et al. · zhang2025copr · Paper
2025 · Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Betley et al. · betley2025emergent · Paper
2025 · Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment — Li et al. · li2025larf · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2025 · SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation — Li et al. · li2025salora · Scholar
2025 · Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron — Zhao et al. · zhao2025safetyneurons · Paper
2024 · Fine-Tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Qi et al. · qi2024finetuning · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Keeping LLMs Aligned After Fine-Tuning: The Crucial Role of Prompt Templates — Lyu et al. · lyu2024keeping · Paper
2024 · Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024lisa · Paper
2024 · Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024vaccine · Paper

System Assurance

Mechanistic & Theoretical Evidence (31)

2026 · Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions — Naseem · naseem2026mechanistic · arXiv
2026 · Intrinsic Barriers and Practical Pathways for Human–AI Alignment: An Agreement-Based Complexity Analysis — Nayebi · nayebi2026barriers · Paper
2026 · Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models — Lian et al. · lian2026ethicalvulnerability · DOI
2026 · Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training — Yong et al. · yong2026selfjailbreaking · Paper
2026 · Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium — Liu et al. · liu2026statisticalalignment · DOI
2026 · Sustaining AI Safety: Control-Theoretic External Impossibility, Intrinsic Necessity, and Structural Requirements — Mazzu · mazzu2026sustaining · Paper
2026 · The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation — Falahati et al. · falahati2026alignmentgame · Paper
2025 · Open Problems in Mechanistic Interpretability — Sharkey et al. · sharkey2025openproblems · arXiv
2025 · The Alignment Trap: Complexity Barriers — Yao · yao2025alignmenttrap · arXiv
2025 · Corrigibility Transformation: Constructing Goals That Accept Updates — Hudson · hudson2025corrigibility · Paper
2025 · Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? — Gölz et al. · golz2025distortion · Paper
2025 · Reasoning Models Don't Always Say What They Think — Chen et al. · chen2025cotfaithfulness · Paper
2025 · The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence — Wollschläger et al. · wollschlager2025geometry · Paper
2025 · The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions — Pan et al. · pan2025hiddendimensions · Paper
2025 · Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons — Chen et al. · chen2025safetyneurons · Paper
2025 · Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron — Zhao et al. · zhao2025safetyneurons · Paper
2025 · Why Do Some Language Models Fake Alignment While Others Don't? — Sheshadri et al. · sheshadri2025alignmentfaking · DOI
2025 · Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region — Leong et al. · leong2025template · Paper
2024 · Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 — Lieberum et al. · lieberum2024gemmascope · arXiv
2024 · Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Templeton et al. · templeton2024scaling · Paper
2024 · Fundamental Limitations of Alignment in Large Language Models — Wolf et al. · wolf2024limitations · Paper
2024 · Refusal in Language Models Is Mediated by a Single Direction — Arditi et al. · arditi2024refusal · Paper
2024 · Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms — Rafailov et al. · rafailov2024overoptimization · Paper
2024 · Steering Language Models with Activation Engineering — Rimsky et al. · rimsky2024caa · DOI
2024 · Understanding the Learning Dynamics of Alignment with Human Feedback — Im et al. · im2024dynamics · Paper
2024 · What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — Jain et al. · jain2024safetyfinetuning · Paper
2023 · Discovering Latent Knowledge in Language Models Without Supervision — Burns et al. · burns2023latent · Scholar
2023 · Inference-Time Intervention: Eliciting Truthful Answers from a Language Model — Li et al. · li2023iti · DOI
2023 · Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al. · zou2023repreng · Paper
2017 · The Off-Switch Game — Hadfield-Menell et al. · hadfieldmenell2017offswitch · DOI
2015 · Corrigibility — Soares et al. · soares2015corrigibility · Paper

Monitoring & Auditing (29)

2026 · Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs — Feng et al. · feng2026benchmarking · arXiv
2026 · Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage? — Ye et al. · ye2026codingenemy · arXiv
2026 · Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems — Alamdari et al. · alamdari2026formal · DOI
2026 · Sycophancy Towards Researchers Drives Performative Misalignment — Baek et al. · baek2026sycophancy · arXiv
2026 · When Saying ``No'' Is Not Enough: Cognitive-Action Decoupling and the Illusion of Safety in LLM Agents — Yu et al. · yu2026whensayingno · DOI
2026 · AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models — Wang et al. · wang2026agenticeval · Paper
2026 · EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures — Uluirmak et al. · uluirmak2026evalsafetygap · Paper
2026 · Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs — Panfilov et al. · panfilov2026dishonesty · Paper
2026 · Watch the Weights: Unsupervised Monitoring and Control of Fine-Tuned LLMs — Zhong et al. · zhong2026weights · Paper
2025 · Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats — Wen et al. · wen2025adaptivedeployment · Paper
2025 · CALM: Curiosity-Driven Auditing for Large Language Models — Zheng et al. · zheng2025calm · DOI
2025 · An Auditing Test to Detect Behavioral Shift in Language Models — Richter et al. · richter2025auditing · Paper
2025 · Auditing Language Models for Hidden Objectives — Marks et al. · marks2025auditing · Paper
2025 · CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring — Arnav et al. · arnav2025cot · DOI
2025 · Detecting High-Stakes Interactions with Activation Probes — McKenzie et al. · mckenzie2025highstakes · DOI
2025 · From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring — Li et al. · li2025streaming · DOI
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · Why Do Some Language Models Fake Alignment While Others Don't? — Sheshadri et al. · sheshadri2025alignmentfaking · DOI
2025 · SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior — Li et al. · li2025safetyanalyst · Paper
2025 · Alignment Faking in Large Language Models — Greenblatt et al. · greenblatt2025alignmentfaking · Paper
2024 · AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach — Amirizaniani et al. · amirizaniani2024auditllm · DOI
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · Black-Box Access is Insufficient for Rigorous AI Audits — Casper et al. · casper2024blackbox · DOI
2024 · Proving Test Set Contamination in Black-Box Language Models — Oren et al. · oren2024contamination · Scholar
2024 · Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Hubinger et al. · hubinger2024sleeper · Paper
2024 · Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models — Denison et al. · denison2024subterfuge · Paper
2023 · Discovering Language Model Behaviors with Model-Written Evaluations — Perez et al. · perez2023modelwritten · Paper
2023 · Holistic Evaluation of Language Models — Liang et al. · liang2023helm · Scholar
2023 · Model Evaluation for Extreme Risks — Shevlane et al. · shevlane2023extremerisks · Paper

General Survey Context

These surveys span several branches and are kept outside any single technical category.

2025 · AI Alignment: A Contemporary Survey — Ji et al. · ji2025alignment · Paper
2025 · Aligning Multimodal LLM with Human Preference: A Survey — Yu et al. · yu2025multimodalsurvey · Paper
2024 · A Survey on Human Preference Learning for Large Language Models — Jiang et al. · jiang2024preferencesurvey · Paper
2024 · Towards a Unified View of Preference Learning for Large Language Models: A Survey — Gao et al. · gao2024unifiedpreference · Paper
2023 · Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation — Fernandes et al. · fernandes2024feedback · Paper
2023 · Large Language Model Alignment: A Survey — Shen et al. · shen2023alignment · Paper

Contributing

Contributions and bibliographic corrections are welcome:

Include the paper's complete bibliographic record (authors, title, venue, year, and DOI or arXiv ID) in the pull request. Prefer the final venue page and DOI; use arXiv when no archival version is available.
Add papers to every branch they substantively address; assignments do not need to be exclusive.
Explain the proposed placement and include a stable public paper link in the pull request.

Suggested inclusion criteria:

The work contributes to alignment specification, supervision, training, inference-time control, evaluation, robustness, interpretability, or monitoring.
A stable manuscript or archival publication page is publicly available.
The bibliographic metadata can be independently checked.

Scope

This is a curated and evolving research map rather than a claim of exhaustive coverage. Placement indicates relevance to a branch; it does not imply that a paper solves Human–AI Alignment or establishes a deployment guarantee.

Last synchronized with the taxonomy and bibliography on 2026-09-29. Citation aliases are normalized in the collection so that the same work is not counted twice under different BibTeX keys.

Similar repos

icip-cas/awesome-auto-alignment

Collection of papers for scalable automated alignment.

GaryYufei/AlignLLMHumanSurvey

Aligning Large Language Models with Human: A Survey

738

Mr-Tieguigui/LLM-Post-Training

102

huggingface/alignment-handbook

Robust recipes to align language models with human and AI preferences

5,686

PKU-Alignment/safe-rlhf

Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback

1,618

JLZhong23/awesome-reward-models

173

opendilab/awesome-RLHF

A curated list of reinforcement learning with human feedback resources (continually updated)

4,432

liyongqi2002/Awesome-Personalized-Alignment

A curated list of personalized alignment resources (continually updated).

Thang1703hrsh/Awesome-Human-AI-Alignment

Python

30 commits

updated Oct 3, 2026

See the code

README

Awesome Human–AI Alignment

A taxonomy-guided collection of research on specifying, supervising, implementing, and assuring Human–AI Alignment.

News

Overview

Software framework

Human Feedback and AI Feedback code, covering the supervision components of 17 cited papers, is documented in Alignment supervision.

python -m pip install -e .
hai-align catalog validate
python examples/minimal_pipeline.py

To train with DPO:

python -m pip install -e ".[dpo]"
hai-align train dpo --model Qwen/Qwen3-0.6B --dataset trl-lib/ultrafeedback_binarized --output outputs/qwen-dpo

from human_alignment import DPO

run = DPO(
    model="Qwen/Qwen3-0.6B",
    dataset="trl-lib/ultrafeedback_binarized",
    output_dir="outputs/qwen-dpo",
).train()

print(run.generate("What is Human--AI alignment?"))

See Codebase architecture for the public API and extension points. The machine-readable taxonomy lives in src/human_alignment/catalog/data/taxonomy.json.

Preference distillation is available through one consistent API for VPD, PPD, DCKD, TVKD, ADPA, and CTPD:

python -m pip install -e ".[distillation]"

from human_alignment import VPD, PreferenceDistillationExample

data = [
    PreferenceDistillationExample(
        prompt="Explain alignment briefly.",
        responses=("Alignment connects behavior to human targets.", "It is model scaling."),
        teacher_scores=(1.0, 0.0),
    )
]
run = VPD(model="student-model", dataset=data, output_dir="outputs/student-vpd").train()

See Preference distillation for objective-specific dataset schemas, teacher-model use, and migration details.

hai-align bench rewardbench2 --reward-model Skywork/Skywork-Reward-V2-Qwen3-0.6B --output-dir eval/rb2

See Benchmarks and inference-time methods.

python -m pip install -e ".[safety]"
hai-align safety list
hai-align safety run recipes/safety_alignment/methods/safedpo.yaml

from human_alignment import SafetyAlignment

run = SafetyAlignment(
    config="recipes/safety_alignment/methods/safedpo.yaml",
    output_dir="outputs/safedpo",
).train()

See Safety alignment for recipe dependencies, paper-faithful implementation choices, evaluation, and H100 workflows.

The library supports both ends of an alignment experiment:

from human_alignment import DPO, load_checkpoint

# Fine-tune a pretrained model or an existing checkpoint.
trained = DPO(
    model="models/student_sft",
    dataset="data/preferences",
    output_dir="outputs/student_dpo",
).train()

# Load the resulting checkpoint later for inference or assurance.
checkpoint = load_checkpoint(
    "outputs/student_dpo",
    config={"device_map": "auto", "torch_dtype": "bfloat16"},
)
print(checkpoint.generate("Explain alignment."))

Preference-distillation supervision can be prepared without the temporary research scripts:

hai-align prepare distillation dckd \
  --dataset HuggingFaceH4/ultrafeedback_binarized \
  --teacher models/teacher_dpo \
  --tokenizer models/student_sft \
  --output data/ultrafeedback-dckd

Examples

Example	Purpose
`minimal_pipeline.py`	Validate the catalog and run a minimal end-to-end pipeline
`dpo_quickstart.py`	Launch DPO from Python
`dpo.toml`	Configure a DPO run declaratively
`checkpoint_workflow.py`	Train, reload, and use a checkpoint
`preference_distillation.py`	Prepare and run preference distillation

Taxonomy

Survey overview

Overview of the Human–AI Alignment survey framework and application settings

View the high-resolution survey overview (PDF)

Lifecycle taxonomy

Human–AI Alignment taxonomy

The lifecycle taxonomy organizes Human–AI alignment into four dimensions and eight groups; the paper collection below further resolves them into 23 non-exclusive research branches.

View the high-resolution taxonomy (PDF)

Dimension	Guiding question	Groups
Alignment Specification	What should an AI system align to, and whose objectives and values should count?	Alignment Objectives · Values & Stakeholders
Alignment Supervision	Where do alignment signals come from, and how are they expressed and scaled?	Feedback Source · Feedback & Oversight
Alignment Mechanisms	How are alignment signals translated into model behavior during training and inference?	Training-Time Alignment · Inference-Time Alignment
Alignment Assurance	How do we evaluate, stress-test, preserve, interpret, and monitor alignment?	Evaluation & Robustness · System Assurance

Training-time mechanisms at a glance

Five training-time alignment mechanisms

Training-time alignment covers reward and verifier modeling, supervised alignment, preference optimization, reinforcement learning, and alignment distillation.

View the high-resolution training-time diagram (PDF)

Inference-time mechanisms at a glance

Four inference-time alignment mechanisms

Inference-time alignment covers steering, search, iterative refinement, and interactive or agentic control while keeping the model policy fixed.

View the high-resolution inference-time diagram (PDF)

Collection coverage

172 representative assignments covering 167 citation keys in the compact figure, before alias normalization.
598 collection assignments across 23 terminal branches; repeated placement is intentional.
451 unique normalized works are linked from the collection.
6 cross-cutting surveys are listed separately after the branch collection.

Cross-cutting descriptors

System and modality: text, multimodal, agentic, and embodied systems.
Feedback granularity: response, span, reasoning step, token, and action.
Optimization granularity: sequence, segment, token, and trajectory.
Learning regime: offline, online, static, and continual learning.

Feedback granularity and optimization granularity are tracked separately because the unit receiving feedback can differ from the unit optimized by the learning objective.

Paper collection

Papers are sorted by year within each branch. Each entry links to the publication page, DOI, arXiv record, or a clearly labeled Scholar search when the BibTeX record has no direct link.

Alignment Specification

What should an AI system align to, and whose objectives and values should count?

Group	Branch	Papers
Alignment Objectives	Task & Assistance Alignment	35
Alignment Objectives	Safety Alignment	27
Values & Stakeholders	Personalized Alignment	24
Values & Stakeholders	Pluralistic & Societal Alignment	35
Values & Stakeholders	Context, Uncertainty & Drift	23

Alignment Objectives

Task & Assistance Alignment (35)

2026 · Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities — Purpura et al. · purpura2026mosaic · DOI
2026 · ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning — Kwon et al. · kwon2026reasonif · DOI
2026 · Revisiting the Reliability of Language Models in Instruction-Following — Dong et al. · dong2026ifevalpp · DOI
2026 · We Are What We Repeatedly Do: Improving Long Context Instruction Following — Robinette et al. · robinette2026verify · DOI
2026 · SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe — Xiao et al. · xiao2026sftmix · Paper
2025 · FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback — Li et al. · li2025fbbench · Paper
2025 · IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization — Zhang et al. · zhang2025iopo · Paper
2025 · LongReward: Improving Long-context Large Language Models with AI Feedback — Zhang et al. · zhang2025longreward · Paper
2025 · When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following — Harada et al. · harada2025manyifeval · Paper
2025 · A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users — Balepur et al. · balepur2025goodplan · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · UltraIF: Advancing Instruction Following from the Wild — An et al. · an2025ultraif · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · LIONs: An Empirically Optimized Approach to Align Language Models — Yu et al. · yu2024lions · Paper
2024 · Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models — Sun et al. · sun2024parrot · Paper
2024 · RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models — Niu et al. · niu2024ragtruth · Paper
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Direct Preference Optimization: Your Language Model Is Secretly a Reward Model — Rafailov et al. · rafailov2023dpo · Paper
2023 · Enabling Large Language Models to Generate Text with Citations — Gao et al. · gao2023citations · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF — Dong et al. · dong2023steerlm · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Multitask Prompted Training Enables Zero-Shot Task Generalization — Sanh et al. · sanh2022t0 · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2021 · A General Language Assistant as a Laboratory for Alignment — Askell et al. · askell2021assistant · Paper
2021 · WebGPT: Browser-Assisted Question-Answering with Human Feedback — Nakano et al. · nakano2021webgpt · Paper

Safety Alignment (27)

2026 · LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignment — Zhang et al. · zhang2026llmva · Paper
2026 · Mitigating Over-Refusal in Aligned Large Language Models via Inference-Time Activation Energy — Jiang et al. · jiang2026els · Paper
2026 · More Thinking, Less Talking: Internalizing Deliberative Safety into LLM Parameters — Wang et al. · wang2026hiar · Paper
2026 · Reasoning Structure Matters for Safety Alignment of Reasoning Models — In et al. · in2026alttrain · Paper
2026 · Safety Is Not Universal: The Selective Safety Trap in LLM Alignment — Brito et al. · brito2026safetyisnotuniversal · DOI
2026 · AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning — Zhang et al. · zhang2026alphaalign · Paper
2026 · LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety — Yang et al. · yang2026lasa · Paper
2026 · Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization — Niu et al. · niu2026nspo · Paper
2026 · Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance — Zhang et al. · zhang2026safetyoneshot · Paper
2026 · SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety — Kim et al. · kim2026safedpo · Paper
2025 · Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements — Zhang et al. · zhang2025controllablesafetyalignment · Paper
2025 · POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization — Karaman et al. · karaman2025porover · Paper
2025 · Safety Alignment Can Be Not Superficial With Explicit Safety Signals — Li et al. · li2025explicitsafety · Paper
2025 · Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models — Gao et al. · gao2025abd · Paper
2025 · Alignment of Large Language Models with Constrained Learning — Zhang et al. · zhang2025constrainedalignment · Paper
2025 · Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2025antidote · Paper
2025 · Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment — Li et al. · li2025larf · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2024 · Deliberative Alignment: Reasoning Enables Safer Language Models — Guan et al. · guan2024deliberative · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2024 · Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024vaccine · Paper
2023 · BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset — Ji et al. · ji2023beavertails · Paper
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2021 · A General Language Assistant as a Laboratory for Alignment — Askell et al. · askell2021assistant · Paper

Values & Stakeholders

Personalized Alignment (24)

2026 · Personalized Benchmarking: Evaluating LLMs by Individual Preferences — Garbacea et al. · garbacea2026personalizedbenchmark · Paper
2026 · When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs — Sun et al. · sun2026fpps · Paper
2026 · From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment — Li et al. · li2026alignx · Paper
2026 · Instant Personalized Large Language Model Adaptation via Hypernetwork — Tan et al. · tan2026profilepeft · Paper
2026 · Personalizing LLMs with Binary Feedback: A Preference-Calibrated Optimization Framework — Ma et al. · ma2026cbpo · Paper
2025 · Aligning LLMs with Individual Preferences via Interaction — Wu et al. · wu2025interaction · Paper
2025 · Drift: Decoding-time Personalized Alignments with Implicit User Preferences — Kim et al. · kim2025drift · Paper
2025 · FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data — Thonet et al. · thonet2025fast · Paper
2025 · Personalized LLM Decoding via Contrasting Personal Preference — Bu et al. · bu2025cope · Paper
2025 · Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning — Cho et al. · cho2025ticl · Paper
2025 · Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas — Balepur et al. · balepur2025personas · Paper
2025 · A Survey on Personalized Alignment—The Missing Piece for Large Language Models in Real-World Applications — Guan et al. · guan2025personalizedsurvey · Paper
2025 · Aligning LLMs by Predicting Preferences from User Writing Samples — Aroca-Ouellette et al. · aroca2025prose · Paper
2025 · PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment — Chen et al. · chen2025pal · Paper
2025 · Personality Alignment of Large Language Models — Zhu et al. · zhu2025personality · Paper
2025 · Inference-Time Personalized Alignment with a Few User Preference Queries — Pădurean et al. · padurean2025useralign · Paper
2025 · PAD: Personalized Alignment of LLMs at Decoding-time — Chen et al. · chen2025pad · Paper
2025 · Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment — Zhang et al. · zhang2025personajudge · Paper
2025 · Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment — Zhao et al. · zhao2025rlpa · Paper
2024 · Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning — Tan et al. · tan2024oppu · Paper
2024 · Personalized Language Modeling from Personalized Human Feedback — Li et al. · li2024personalized · Paper
2024 · Personalizing Reinforcement Learning from Human Feedback — Poddar et al. · poddar2024personalizing · Paper
2024 · The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models — Kirk et al. · kirk2024prism · Paper
2023 · Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging — Jang et al. · jang2023personalizedsoups · Paper

Pluralistic & Societal Alignment (35)

2026 · Benchmarking Overton Pluralism in LLMs — Poole-Dayan et al. · pooledayan2026overtonbench · Paper
2026 · Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment — Tan et al. · tan2026subgroupvalues · Paper
2026 · Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset — Zhang et al. · zhang2026cultivating · Paper
2026 · CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters — Sun et al. · sun2026cuma · Paper
2026 · Disentangling Consensus and Value-Specific Representations for Controllable Pluralistic Value Alignment of LLMs — Zhou et al. · zhou2026disalign · Paper
2026 · PerSpectra: A Scalable and Configurable Pluralist Benchmark of Perspectives from Arguments — Nie et al. · nie2026perspectra · Paper
2026 · VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models — Kim et al. · kim2026valueflow · Paper
2026 · A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus — Zhang et al. · zhang2026crosscultural · Paper
2026 · ALPHA: Action-Based Learning for Pluralistic Human Alignment in Large Language Models — Bhattacharyya et al. · bhattacharyya2026alpha · DOI
2026 · DVMap: Fine-Grained Pluralistic Value Alignment via High-Consensus Demographic-Value Mapping — Zhu et al. · zhu2026dvmap · Paper
2026 · FGD-Align: Pluralistic Alignment for Large Language Models via Fuzzy Group Decision-Making — Pan et al. · pan2026fgdalign · DOI
2026 · Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models — Zhang et al. · zhang2026culturemanager · Paper
2026 · Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior — Ali et al. · ali2026pluralistic · DOI
2025 · LLMs' Pluralistic Compatibility — Engelmann et al. · engelmann2025pluralisticcompatibility · Paper
2025 · Aligning Trustworthy AI with Democracy: A Dual Taxonomy of Opportunities and Risks — Mentxaka et al. · mentxaka2025democracy · Paper
2025 · Direct Alignment with Heterogeneous Preferences — Shirali et al. · shirali2025heterogeneous · DOI
2025 · Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment — Zhang et al. · zhang2025pluralisticcot · Paper
2025 · Multiple LLM Agents Debate for Equitable Cultural Alignment — Ki et al. · ki2025culturaldebate · Paper
2025 · Pairwise Calibrated Rewards for Pluralistic Alignment — Halpern et al. · halpern2025pairwise · Paper
2025 · PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment — Chen et al. · chen2025pal · Paper
2025 · No Preference Left Behind: Group Distributional Preference Optimization — Yao et al. · yao2025gdpo · Paper
2025 · Self-Pluralising Culture Alignment for Large Language Models — Xu et al. · xu2025culturespa · Paper
2025 · Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression — Adams et al. · adams2025steerable · DOI
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · Collective Constitutional AI: Aligning a Language Model with Public Input — Huang et al. · huang2024collective · Paper
2024 · CultureLLM: Incorporating Cultural Differences into Large Language Models — Li et al. · li2024culturellm · Paper
2024 · Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF — Siththaranjan et al. · siththaranjan2024distributional · Scholar
2024 · Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration — Feng et al. · feng2024modular · Paper
2024 · Position: A Roadmap to Pluralistic Alignment — Sorensen et al. · sorensen2024pluralistic · Scholar
2024 · Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback — Conitzer et al. · conitzer2024socialchoice · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models — Kirk et al. · kirk2024prism · Paper
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper
2022 · Fine-Tuning Language Models to Find Agreement among Humans with Diverse Preferences — Bakker et al. · bakker2022finetuning · Paper
2018 · The Moral Machine Experiment — Awad et al. · awad2018moral · DOI

Context, Uncertainty & Drift (23)

2026 · Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity — Fang et al. · fang2026aam · DOI
2026 · ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment — Lin et al. · lin2026activedpo · Paper
2026 · Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment — Chen et al. · chen2026cfa · DOI
2026 · When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF — Xu et al. · xu2026fadpo · Paper
2026 · Moral Change or Noise? On Problems of Aligning AI with Temporally Unstable Human Feedback — Keswani et al. · keswani2026moralchange · Paper
2026 · Reward Model Routing in Alignment — Wu et al. · wu2026rmrouting · Paper
2025 · Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment — Shen et al. · shen2025activerm · Paper
2025 · PILAF: Optimal Human Preference Sampling for Reward Modeling — Feng et al. · feng2025pilaf · Paper
2025 · Robust LLM Alignment via Distributionally Robust Direct Preference Optimization — Xu et al. · xu2025robustdpo · Paper
2025 · Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning — Li et al. · li2025uipo · Paper
2025 · Ask a Strong LLM Judge when Your Reward Model is Uncertain — Xu et al. · xu2025uncertaintyjudge · Paper
2025 · COPR: Continual Human Preference Learning via Optimal Policy Regularization — Zhang et al. · zhang2025copr · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2025 · Uncertainty and Influence aware Reward Model Refinement for Reinforcement Learning from Human Feedback — Sun et al. · sun2025ugda · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF — Siththaranjan et al. · siththaranjan2024distributional · Scholar
2024 · On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods — Boerstler et al. · boerstler2024stability · Paper
2024 · Perplexity-aware Correction for Robust Alignment with Noisy Preferences — Kong et al. · kong2024perpcorrect · Paper
2017 · Inverse Reward Design — Hadfield-Menell et al. · hadfieldmenell2017ird · Scholar
2017 · The Off-Switch Game — Hadfield-Menell et al. · hadfieldmenell2017offswitch · DOI
2016 · Cooperative Inverse Reinforcement Learning — Hadfield-Menell et al. · hadfieldmenell2016cirl · Scholar
2004 · Apprenticeship Learning via Inverse Reinforcement Learning — Abbeel et al. · abbeel2004apprenticeship · Scholar
2000 · Algorithms for Inverse Reinforcement Learning — Ng et al. · ng2000irl · Scholar

Alignment Supervision

Where do alignment signals come from, and how are they expressed and scaled?

Group	Branch	Papers
Feedback Source	Human Feedback	20
Feedback Source	AI Feedback	23
Feedback Source	Programmatic & Verifiable Feedback	22
Feedback & Oversight	Demonstrations & Preferences	24
Feedback & Oversight	Critique, Process & Trajectory Feedback	19
Feedback & Oversight	Reliable & Scalable Oversight	25

Feedback Source

Human Feedback (20)

2026 · WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback — Shi et al. · shi2026wildfeedback · Paper
2025 · Aligning Language Models with Demonstrated Feedback — Shaikh et al. · shaikh2025ditto · Paper
2025 · Binary Classifier Optimization for Large Language Model Alignment — Jung et al. · jung2025bco · Paper
2025 · Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback — Miranda et al. · miranda2025hybrid · Paper
2025 · MM-RLHF: The Next Step Forward in Multimodal LLM Alignment — Zhang et al. · zhang2025mmrlhf · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · Aligning Large Language Models via Fine-grained Supervision — Xu et al. · xu2024finegrained · Paper
2024 · ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback — Hou et al. · hou2024chatglmrlhf · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · Towards Aligning Language Models with Textual Feedback — Lloret et al. · lloret2024alt · Paper
2023 · HelpSteer: Multi-Attribute Human Preference Dataset for Training and Evaluation of Large Language Models — Wang et al. · wang2023helpsteer · arXiv
2023 · OpenAssistant Conversations – Democratizing Large Language Model Alignment — Köpf et al. · kopf2023openassistant · arXiv
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2019 · Fine-Tuning Language Models from Human Preferences — Ziegler et al. · ziegler2019finetuning · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

AI Feedback (23)

2026 · Aligning Large Language Models via Fully Self-Synthetic Data — Yin et al. · yin2026sao · Paper
2026 · Check Your Work: Structured Checklist Feedback for Improving Large Language Models — Cook et al. · cook2026checklist · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · Diverse AI Feedback For Large Language Model Alignment — Yu et al. · yu2025daif · Paper
2025 · Learning LLM-as-a-Judge for Preference Alignment — Ye et al. · ye2025conj · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2024 · LLM Critics Help Catch LLM Bugs — McAleese et al. · mcaleese2024critics · Paper
2024 · Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation — Liu et al. · liu2024dlma · Paper
2024 · RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Lee et al. · lee2024rlaif · Paper
2024 · Self-Rewarding Language Models — Yuan et al. · yuan2024selfrewarding · Scholar
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2024 · VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment — Li et al. · li2024vlfeedback · Paper
2023 · Enhancing Chat Language Models by Scaling High-quality Instruction Conversations — Ding et al. · ding2023ultrachat · arXiv
2023 · G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — Liu et al. · liu2023geval · arXiv
2023 · JudgeLM: Fine-tuned Large Language Models Are Scalable Judges — Zhu et al. · zhu2023judgelm · arXiv
2023 · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. · zheng2023mtbench · Paper
2023 · Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — Mukherjee et al. · mukherjee2023orca · arXiv
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Self-Alignment: Building an Open-source and Self-Instructive Language Model — Sun et al. · sun2023selfalignment · Scholar
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2023 · WizardLM: Empowering Large Language Models with Evol-Instruct — Xu et al. · xu2023wizardlm · arXiv
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper

Programmatic & Verifiable Feedback (22)

2026 · Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse Domains — Su et al. · su2026rewardbridge · Paper
2026 · Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains — Yuan et al. · yuan2026k2v · Paper
2025 · OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees — Zhang et al. · zhang2025openprm · Paper
2025 · DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI · deepseek2025r1 · Paper
2025 · Generative Verifiers: Reward Modeling as Next-Token Prediction — Zhang et al. · zhang2025genrm · Paper
2025 · Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — Setlur et al. · setlur2025rewarding · Paper
2025 · RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning — Gehring et al. · gehring2025rlef · Paper
2025 · Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models — Dong et al. · dong2025autoif · Paper
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models — Singh et al. · singh2024restem · Paper
2024 · DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search — Xin et al. · xin2024deepseekprover15 · DOI
2024 · DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al. · shao2024deepseekmath · Paper
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · V-STaR: Training Verifiers for Self-Taught Reasoners — Hosseini et al. · hosseini2024vstar · Paper
2023 · CodeT: Code Generation with Generated Tests — Chen et al. · chen2023codet · Paper
2023 · LeanDojo: Theorem Proving with Retrieval-Augmented Language Models — Yang et al. · yang2023leandojo · Paper
2023 · RLTF: Reinforcement Learning from Unit Test Feedback — Liu et al. · liu2023rltf · Paper
2022 · CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — Le et al. · li2022coderl · Paper
2022 · STaR: Bootstrapping Reasoning With Reasoning — Zelikman et al. · zelikman2022star · Paper
2021 · Evaluating Large Language Models Trained on Code — Chen et al. · chen2021humaneval · arXiv
2021 · Measuring Coding Challenge Competence With APPS — Hendrycks et al. · hendrycks2021apps · arXiv

Feedback & Oversight

Demonstrations & Preferences (24)

2026 · WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback — Shi et al. · shi2026wildfeedback · Paper
2025 · Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback — Miranda et al. · miranda2025hybrid · Paper
2025 · Aligning Language Models with Demonstrated Feedback — Shaikh et al. · shaikh2025ditto · Paper
2025 · Aligning Large Language Models with Implicit Preferences from User-Generated Content — Tan et al. · tan2025pugc · Paper
2025 · Binary Classifier Optimization for Large Language Model Alignment — Jung et al. · jung2025bco · Paper
2025 · Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization — Xiao et al. · xiao2025sweetspot · Paper
2025 · Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment — Li et al. · li2025joint · Paper
2024 · Aligning Large Language Models via Fine-grained Supervision — Xu et al. · xu2024finegrained · Paper
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment — Li et al. · li2024demoreward · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Critique, Process & Trajectory Feedback (19)

2026 · Critique-RL: Training Language Models for Critiquing Through Two-Stage Reinforcement Learning — Xi et al. · xi2026critiquerl · Paper
2026 · Exploring Reasoning Reward Model for Agents — Fan et al. · fan2026agentrrm · Paper
2026 · The Bidirectional Process Reward Model — Zhang et al. · zhang2026biprm · Paper
2025 · Dynamic and Generalizable Process Reward Modeling — Yin et al. · yin2025dgprm · Paper
2025 · OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees — Zhang et al. · zhang2025openprm · Paper
2025 · Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — Setlur et al. · setlur2025rewarding · Paper
2025 · Self-Generated Critiques Boost Reward Modeling for Language Models — Yu et al. · yu2025criticrm · Paper
2025 · Teaching Language Models to Critique via Reinforcement Learning — Xie et al. · xie2025ctrl · Paper
2025 · Training Language Model to Critique for Better Refinement — Yu et al. · yu2025rco · Paper
2024 · CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing — Gou et al. · gou2024critic · Scholar
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · LLM Critics Help Catch LLM Bugs — McAleese et al. · mcaleese2024critics · Paper
2024 · RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback — Yu et al. · yu2024rlhfv · Paper
2024 · UltraFeedback: Boosting Language Models with Scaled AI Feedback — Cui et al. · cui2024ultrafeedback · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2022 · Constitutional AI: Harmlessness from AI Feedback — Bai et al. · bai2022constitutional · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper

Reliable & Scalable Oversight (25)

2026 · Agentic Oversight via Dialectic Reasoning — Ranaldi et al. · ranaldi2026agentic · Paper
2026 · Selective Weak-to-Strong Generalization — Lang et al. · lang2026selective · DOI
2026 · Towards Scalable Oversight via Partitioned Human Supervision — Yin et al. · yin2026partitioned · Paper
2026 · Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction — Qiu et al. · qiu2026peerprediction · Paper
2025 · AI Debate Aids Assessment of Controversial Claims — Rahman et al. · rahman2025aidebate · Scholar
2025 · Barriers and Pathways to Human-AI Alignment: A Game-Theoretic Approach — Nayebi · nayebi2025barriers · Paper
2025 · Debate Helps Weak-to-Strong Generalization — Lang et al. · lang2025debate · Paper
2025 · Great Models Think Alike and this Undermines AI Oversight — Goel et al. · goel2025oversight · Paper
2025 · Neural Interactive Proofs — Hammond et al. · hammond2025neuralproofs · Paper
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · Scaling Laws for Scalable Oversight — Engels et al. · engels2025scaling · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · Debating with More Persuasive LLMs Leads to More Truthful Answers — Khan et al. · khan2024debate · Paper
2024 · Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision — Sun et al. · sun2024easyhard · Paper
2024 · On Scalable Oversight with Weak LLMs Judging Strong LLMs — Kenton et al. · kenton2024scalableoversight · Paper
2024 · Prover-Verifier Games Improve Legibility of LLM Outputs — Kirchner et al. · kirchner2024proververifier · Paper
2024 · Reward Modeling Requires Automatic Adjustment Based on Data Quality — Wang et al. · wang2024rmquality · DOI
2024 · RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences — Cheng et al. · cheng2024rime · Scholar
2024 · Self-Exploring Language Models: Active Preference Elicitation for Online Alignment — Zhang et al. · zhang2024selm · Paper
2024 · Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision — Burns et al. · burns2024weakstrong · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2023 · Scaling Laws for Reward Model Overoptimization — Gao et al. · gao2023overoptimization · Paper
2018 · AI Safety via Debate — Irving et al. · irving2018debate · Paper
2018 · Supervising Strong Learners by Amplifying Weak Experts — Christiano et al. · christiano2018amplification · Paper

Alignment Mechanisms

How are alignment signals translated into model behavior during training and inference?

Group	Branch	Papers
Training-Time Alignment	Reward & Verifier Modeling	27
Training-Time Alignment	Supervised Alignment	22
Training-Time Alignment	Preference Optimization	50
Training-Time Alignment	Reinforcement Learning	32
Training-Time Alignment	Alignment Distillation	9
Inference-Time Alignment	Steering, Search & Refinement	31
Inference-Time Alignment	Interactive & Agentic Control	17

Training-Time Alignment

Reward & Verifier Modeling (27)

2026 · OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment — Liu et al. · liu2026openrubrics · Paper
2026 · Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models — Wang et al. · wang2026rationale · Paper
2026 · PRISM: Probabilistic Reward Model with Inherent Structural Modeling — Zhou et al. · zhou2026prismrm · Paper
2026 · RM-R1: Reward Modeling as Reasoning — Chen et al. · chen2026rmr1 · Paper
2026 · The Bidirectional Process Reward Model — Zhang et al. · zhang2026biprm · Paper
2026 · AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling — Miao et al. · miao2026adajudge · Paper
2025 · Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems — Peng et al. · peng2025agenticrm · Paper
2025 · Dynamic and Generalizable Process Reward Modeling — Yin et al. · yin2025dgprm · Paper
2025 · Generative Verifiers: Reward Modeling as Next-Token Prediction — Zhang et al. · zhang2025genrm · Paper
2025 · GRAM: A Generative Foundation Reward Model for Reward Generalization — Wang et al. · wang2025gram · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2025 · Reward Reasoning Models — Guo et al. · guo2025rrm · DOI
2025 · PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment — Chen et al. · chen2025pal · Paper
2024 · HelpSteer 2: Open-source Dataset for Training Top-Performing Reward Models — Wang et al. · wang2024helpsteer2 · DOI
2024 · Let's Verify Step by Step — Lightman et al. · lightman2024verify · Paper
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Prover-Verifier Games Improve Legibility of LLM Outputs — Kirchner et al. · kirchner2024proververifier · Paper
2024 · Reward Model Ensembles Help Mitigate Overoptimization — Coste et al. · coste2024ensembles · Paper
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2023 · Fine-Grained Human Feedback Gives Better Rewards for Language Model Training — Wu et al. · wu2023finegrained · Paper
2022 · Improving Alignment of Dialogue Agents via Targeted Human Judgements — Glaese et al. · glaese2022sparrow · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper
2022 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. · bai2022hh · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2019 · Fine-Tuning Language Models from Human Preferences — Ziegler et al. · ziegler2019finetuning · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Supervised Alignment (22)

2026 · OASIS: Online Sample Selection for Continual Instruction Tuning — Lee et al. · lee2026oasis · Paper
2026 · SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe — Xiao et al. · xiao2026sftmix · Paper
2025 · Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing — Xu et al. · xu2025magpie · Paper
2025 · MAIN: Mutual Alignment Is Necessary for Instruction Tuning — Yang et al. · yang2025main · Paper
2025 · Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality — Harada et al. · harada-etal-2025-massive · Paper
2025 · Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process — Hua et al. · hua-etal-2025-intuitive · Paper
2025 · The Best Instruction-Tuning Data are Those That Fit — Zhang et al. · zhang2025grape · Scholar
2024 · LESS: Selecting Influential Data for Targeted Instruction Tuning — Xia et al. · xia2024less · Paper
2024 · MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models — Yang et al. · yang2024metaaligner · DOI
2024 · Scaling Instruction-Finetuned Language Models — Chung et al. · chung2024flan · Paper
2024 · SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection — Liu et al. · liu2024selectit · Paper
2024 · Self-Alignment with Instruction Backtranslation — Li et al. · li2024backtranslation · Paper
2024 · What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning — Liu et al. · liu2024deita · Paper
2023 · LIMA: Less Is More for Alignment — Zhou et al. · zhou2023lima · Paper
2023 · Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · wang2023selfinstruct · Paper
2023 · SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF — Dong et al. · dong2023steerlm · Paper
2023 · The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. · longpre2023flancollection · Paper
2022 · Cross-Task Generalization via Natural Language Crowdsourcing Instructions — Mishra et al. · mishra2022natural · Paper
2022 · Finetuned Language Models Are Zero-Shot Learners — Wei et al. · wei2022flan · Paper
2022 · Multitask Prompted Training Enables Zero-Shot Task Generalization — Sanh et al. · sanh2022t0 · Paper
2022 · Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — Wang et al. · wang2022supernatural · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper

Preference Optimization (50)

2026 · Causal Direct Preference Optimization for Language Model Alignment — Le et al. · le2026causaldpo · Paper
2026 · Keep the Best, Forget the Rest: Reliable Alignment with Order-Aware Preference Optimization — Zhu et al. · zhu2026rappo · Paper
2026 · Mitigating Mismatch within Reference-based Preference Optimization — Yuan et al. · yuan2026hypo · Paper
2026 · Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium — Liu et al. · liu2026statisticalalignment · DOI
2026 · Autoregressive Direct Preference Optimization — Oi et al. · oi2026autoregressive · Paper
2026 · BSO: Safety Alignment Is Density Ratio Matching — Nguyen et al. · nguyen2026bso · DOI
2026 · SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety — Kim et al. · kim2026safedpo · Paper
2026 · Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization — Luo et al. · luo2026sharpness · Paper
2026 · Token-Importance Guided Direct Preference Optimization — Yang et al. · yang2026token · Paper
2026 · TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching — Nguyen et al. · nguyen2026tokenratio · Paper
2025 · Alignment of Large Language Models with Constrained Learning — Zhang et al. · zhang2025constrainedalignment · Paper
2025 · AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization — Wu et al. · wu2025alphadpo · Paper
2025 · Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment — D'Oosterlinck et al. · doosterlinck2025apo · Paper
2025 · Doubly Robust Alignment for Large Language Models — Xu et al. · xu2025doublyrobust · Paper
2025 · KL Penalty Control via Perturbation for Direct Preference Optimization — Lee et al. · lee2025kl · Paper
2025 · Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment — Wang et al. · wang2025mpo · Paper
2025 · MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization — Zhu et al. · zhu2025mmedpo · Paper
2025 · No Preference Left Behind: Group Distributional Preference Optimization — Yao et al. · yao2025gdpo · Paper
2025 · Preference Optimization by Estimating the Ratio of the Data Distribution — Kim et al. · kim2025bregmanpo · Paper
2025 · Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO — Guo et al. · guo2025pro · Paper
2025 · Reward-Aware Preference Optimization: A Unified Mathematical Framework for Model Alignment — Sun et al. · sun2025rpo · Paper
2025 · Risk-aware Direct Preference Optimization under Nested Risk Measure — Zhang et al. · zhang2025risk · Paper
2025 · SDPO: Segment-Level Direct Preference Optimization for Social Agents — Kong et al. · kong2025sdpo · Paper
2025 · Self-Play Preference Optimization for Language Model Alignment — Wu et al. · wu2025sppo · Paper
2025 · SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment — Zhu et al. · zhu2025sgdpo · Paper
2025 · T-REG: Preference Optimization with Token-Level Reward Regularization — Zhou et al. · zhou2025treg · Paper
2025 · TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights — Liu et al. · liu2025tisdpo · Paper
2025 · Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization — Wu et al. · wu2025drdpo · Paper
2025 · Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model — Zhu et al. · zhu2025wspo · Paper
2024 · Active Preference Learning for Large Language Models — Muldrew et al. · muldrew2024active · Scholar
2024 · BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment — Xu et al. · xu2024behaviorbpo · Paper
2024 · Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation — Xu et al. · xu2024cpo · Paper
2024 · Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences — Rosset et al. · rosset2024dno · Paper
2024 · KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh et al. · ethayarajh2024kto · Paper
2024 · MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models — Yang et al. · yang2024metaaligner · DOI
2024 · Multi-turn Reinforcement Learning from Preference Human Feedback — Shani et al. · shani2024multiturn · Scholar
2024 · Nash Learning from Human Feedback — Munos et al. · munos2024nash · Paper
2024 · ORPO: Monolithic Preference Optimization without Reference Model — Hong et al. · hong2024orpo · Paper
2024 · Provably Robust DPO: Aligning Language Models with Noisy Feedback — Ray Chowdhury et al. · chowdhury2024rdpo · Paper
2024 · RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences — Cheng et al. · cheng2024rime · Scholar
2024 · SAIL: Self-Improving Efficient Online Alignment of Large Language Models — Ding et al. · ding2024sail · Paper
2024 · Self-Exploring Language Models: Active Preference Elicitation for Online Alignment — Zhang et al. · zhang2024selm · Paper
2024 · SimPO: Simple Preference Optimization with a Reference-Free Reward — Meng et al. · meng2024simpo · Paper
2024 · Token-Level Direct Preference Optimization — Zeng et al. · zeng2024tdpo · Paper
2024 · β-DPO: Direct Preference Optimization with Dynamic β — Wu et al. · wu2024beta · Paper
2023 · A General Theoretical Paradigm to Understand Learning from Human Preferences — Azar et al. · azar2023psipo · Paper
2023 · Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization — Zhao et al. · zhao2023hadpo · Paper
2023 · Direct Preference Optimization: Your Language Model Is Secretly a Reward Model — Rafailov et al. · rafailov2023dpo · Paper
2023 · RRHF: Rank Responses to Align Language Models with Human Feedback — Yuan et al. · yuan2023rrhf · Paper
2023 · SLiC-HF: Sequence Likelihood Calibration with Human Feedback — Zhao et al. · zhao2023slichf · Paper

Reinforcement Learning (32)

2026 · AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning — Zhang et al. · zhang2026alphaalign · Paper
2026 · From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation — Jiang et al. · jiang2026rlvrr · Paper
2026 · Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization — Niu et al. · niu2026nspo · Paper
2026 · Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models — Zhang et al. · zhang2026corewarding · Paper
2026 · GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning — Chu et al. · chu2026gpg · Paper
2026 · RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments — Zeng et al. · zeng2026rlve · arXiv
2026 · Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning — Deng et al. · deng2026srl · Paper
2025 · Checklists Are Better Than Reward Models for Aligning Language Models — Viswanathan et al. · viswanathan2025rlcf · Paper
2025 · DAPO: Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization — Liu et al. · liu2025dapo_advantage · Paper
2025 · DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI · deepseek2025r1 · Paper
2025 · Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization — Peng et al. · peng2025repo · Paper
2025 · MM-RLHF: The Next Step Forward in Multimodal LLM Alignment — Zhang et al. · zhang2025mmrlhf · Paper
2025 · Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems — Walder & Karkhanis · walder2025passk · Paper
2025 · SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning — Liu et al. · liu2025saturn · Paper
2025 · Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models — Guo et al. · guo2025spo · Paper
2025 · SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data — Fang et al. · fang2025serl · Paper
2025 · Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for LLMs — Le Roux et al. · leroux2025tapered · Paper
2024 · Aligning Large Multimodal Models with Factually Augmented RLHF — Sun et al. · liu2024llavarlhf · Paper
2024 · Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs — Ahmadian et al. · ahmadian2024reinforce · Paper
2024 · ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback — Hou et al. · hou2024chatglmrlhf · Paper
2024 · DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Shao et al. · shao2024deepseekmath · Paper
2024 · Multi-turn Reinforcement Learning from Preference Human Feedback — Shani et al. · shani2024multiturn · Scholar
2024 · ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models — Li et al. · li2024remax · Paper
2024 · RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Lee et al. · lee2024rlaif · Paper
2024 · Rule Based Rewards for Language Model Safety — Mu et al. · mu2024rulebased · Scholar
2024 · Safe RLHF: Safe Reinforcement Learning from Human Feedback — Dai et al. · dai2024saferlhf · Paper
2024 · SAIL: Self-Improving Efficient Online Alignment of Large Language Models — Ding et al. · ding2024sail · Paper
2023 · A Survey of Reinforcement Learning from Human Feedback — Kaufmann et al. · kaufmann2023survey · Paper
2022 · Solving Math Word Problems with Process- and Outcome-Based Feedback — Uesato et al. · uesato2022process · Paper
2022 · Training Language Models to Follow Instructions with Human Feedback — Ouyang et al. · ouyang2022instructgpt · Paper
2020 · Learning to Summarize with Human Feedback — Stiennon et al. · stiennon2020summarize · Paper
2017 · Deep Reinforcement Learning from Human Preferences — Christiano et al. · christiano2017preferences · Paper

Alignment Distillation (9)

2026 · CTPD: Cross Tokenizer Preference Distillation — Nguyen et al. · nguyen2026ctpd · Paper
2025 · Advantage-Guided Distillation for Preference Alignment in Small Language Models — Gao et al. · gao2025adpa · Paper
2025 · AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation — Zhang et al. · zhang2025aligndistil · Paper
2025 · Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models — Gu et al. · gu2025pad · Paper
2025 · Preference Distillation via Value based Reinforcement Learning — Kwon et al. · kwon2025tvkd · Paper
2024 · CycleAlign: Iterative Distillation from Black-Box LLM to White-Box Models for Better Human Alignment — Hong et al. · hong2024cyclealign · Paper
2024 · Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation — Liu et al. · liu2024dlma · Paper
2024 · Direct Preference Knowledge Distillation for Large Language Models — Li et al. · li2024dpkd · Paper
2024 · PLaD: Preference-Based Large Language Model Distillation with Pseudo-Preference Pairs — Zhang et al. · zhang2024plad · Paper

Inference-Time Alignment

2026 · A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement — Li et al. · li2026trainingfree · arXiv
2026 · Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior — Nguyen et al. · nguyen2026beyondlinear · arXiv
2026 · CyberCorrect: A Cybernetic Framework for Closed-Loop Self-Correction in Large Language Models — Wu et al. · wu2026cybercorrect · arXiv
2026 · Don't Lose Focus: Activation Steering via Key-Orthogonal Projections — Luo et al. · luo2026dontlosefocus · arXiv
2026 · ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment — Zhao et al. · zhao2026odesteer · arXiv
2026 · Spherical Steering: Geometry-Aware Activation Rotation for Language Models — You et al. · you2026spherical · arXiv
2026 · Activation Steering with a Feedback Controller — Nguyen et al. · nguyen2026pidsteering · Paper
2026 · FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models — Weng et al. · weng2026finesteer · Paper
2026 · One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement — Zhou et al. · zhou2026requer · Paper
2025 · Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning — Snell et al. · snell2025scaling · Paper
2025 · SSR: Socratic Self-Refine for Large Language Model Reasoning — Shi et al. · shi2025ssr · arXiv
2025 · Unleashing the True Potential of LLMs: A Feedback-Triggered Self-Correction with Long-Term Multipath Decoding — Li et al. · li2025unleashing · arXiv
2025 · W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search — Ding et al. · ding2025w2s · arXiv
2025 · ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment — Masters et al. · masters2025arcane · Paper
2025 · Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements — Zhang et al. · zhang2025controllablesafetyalignment · Paper
2025 · Improving Instruction-Following in Language Models through Activation Steering — Stolfo et al. · stolfo2025steering · Paper
2025 · LLaVA-Critic: Learning to Evaluate Multimodal Models — Xiong et al. · xiong2025llavacritic · DOI
2025 · Nudging: Inference-time Alignment of LLMs via Guided Decoding — Fei et al. · fei2025nudging · Paper
2025 · Reward-Guided Tree Search for Inference Time Alignment of Large Language Models — Hung et al. · hung2025darwin · Paper
2025 · S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning — Ma et al. · ma2025s2r · Paper
2024 · Controlled Decoding from Language Models — Mudgal et al. · mudgal2023controlled · Paper
2024 · ARGS: Alignment as Reward-Guided Search — Khanov et al. · khanov2024args · Paper
2024 · CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing — Gou et al. · gou2024critic · Scholar
2024 · Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations — Wang et al. · wang2024mathshepherd · Paper
2024 · Steering Language Models with Activation Engineering — Rimsky et al. · rimsky2024caa · DOI
2023 · Activation Addition: Steering Language Models Without Optimization — Turner et al. · turner2023activationaddition · Paper
2023 · Inference-Time Intervention: Eliciting Truthful Answers from a Language Model — Li et al. · li2023iti · DOI
2023 · Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. · shinn2023reflexion · Scholar
2023 · Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al. · zou2023repreng · Paper
2023 · Self-Refine: Iterative Refinement with Self-Feedback — Madaan et al. · madaan2023selfrefine · Paper
2021 · WebGPT: Browser-Assisted Question-Answering with Human Feedback — Nakano et al. · nakano2021webgpt · Paper

Interactive & Agentic Control (17)

2026 · Steering LLMs via Scalable Interactive Oversight — Zhou et al. · zhou2026steering · arXiv
2026 · Designing Meaningful Human Oversight in AI — Zhu et al. · zhu2026oversight · Paper
2026 · Structured Uncertainty guided Clarification for LLM Agents — Suri et al. · suri2026clarification · Paper
2025 · Scalable Human Oversight for Aligned Large Language Models: A Hybrid Framework for Intent Fidelity — Ayankoya et al. · ayankoya2025scalable · DOI
2025 · The Burden of Interactive Alignment with Inconsistent Preferences — Shirali, Ali · shirali2025burden · arXiv
2025 · Corrigibility Transformation: Constructing Goals That Accept Updates — Hudson · hudson2025corrigibility · Paper
2025 · Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance — He et al. · he2025aria · Paper
2025 · Policy Optimization under Imperfect Human Interactions with Agent-Gated Shared Autonomy — Xue et al. · xue2025agsa · Paper
2025 · Position: AI Agents Need Authenticated Delegation — South et al. · south2025delegation · Paper
2025 · The Partially Observable Off-Switch Game — Garber et al. · garber2025offswitch · Paper
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · STaR-GATE: Teaching Language Models to Ask Clarifying Questions — Andukuri et al. · andukuri2024stargate · Scholar
2020 · Consistent Estimators for Learning to Defer to an Expert — Mozannar et al. · mozannar2020defer · Scholar
2017 · The Off-Switch Game — Hadfield-Menell et al. · hadfieldmenell2017offswitch · DOI
2016 · Cooperative Inverse Reinforcement Learning — Hadfield-Menell et al. · hadfieldmenell2016cirl · Scholar
2016 · Safely Interruptible Agents — Orseau et al. · orseau2016interruptible · Scholar
2015 · Corrigibility — Soares et al. · soares2015corrigibility · Paper

Alignment Assurance

How do we evaluate, stress-test, preserve, interpret, and monitor alignment?

Group	Branch	Papers
Evaluation & Robustness	Behavioral & Evaluator Evaluation	26
Evaluation & Robustness	Adversarial & Distribution Robustness	30
Evaluation & Robustness	Alignment Preservation	17
System Assurance	Mechanistic & Theoretical Evidence	31
System Assurance	Monitoring & Auditing	29

Evaluation & Robustness

Behavioral & Evaluator Evaluation (26)

2026 · Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling — Wang et al. · wang2026planrewardbench · DOI
2026 · IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation — Wen et al. · wen2026ifrewardbench · Paper
2026 · LLMs Get Lost In Multi-Turn Conversation — Laban et al. · laban2026lost · Paper
2026 · Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following — Xiong et al. · xiong2026multicrit · Paper
2026 · RewardBench 2: Advancing Reward Model Evaluation — Malik et al. · malik2026rewardbench2 · Paper
2026 · SoSBench: Benchmarking Safety Alignment on Six Scientific Domains — Jiang et al. · jiang2026sosbench · Paper
2025 · ChameleonBench: Quantifying Alignment Faking in Large Language Models — Chaudhury et al. · chaudhury2025chameleonbench · Paper
2025 · JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Tan et al. · tan2025judgebench · Paper
2025 · LiveBench: A Challenging, Contamination-Limited LLM Benchmark — White et al. · white2025livebench · Paper
2025 · PRMBench: A Fine-Grained and Challenging Benchmark for Process-Level Reward Models — Song et al. · song2025prmbench · Paper
2025 · RAG-RewardBench: Benchmarking Reward Models in Retrieval-Augmented Generation — Jin et al. · ragrewardbench2025 · Paper
2025 · Reasoning Models Don't Always Say What They Think — Chen et al. · chen2025cotfaithfulness · Paper
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild — Lin et al. · lin2025wildbench · Scholar
2024 · Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al. · chiang2024arena · Scholar
2024 · HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models — Guan et al. · guan2024hallusionbench · Scholar
2024 · HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal — Mazeika et al. · mazeika2024harmbench · Paper
2024 · RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models — Niu et al. · niu2024ragtruth · Paper
2024 · SafetyBench: Evaluating the Safety of Large Language Models — Zhang et al. · zhang2024safetybench · Paper
2024 · XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models — Rottger et al. · rottger2024xstest · Paper
2023 · AlpacaEval: An Automatic Evaluator of Instruction-Following Models — Li et al. · li2023alpacaeval · Paper
2023 · Enabling Large Language Models to Generate Text with Citations — Gao et al. · gao2023citations · Paper
2023 · FActScore: Fine-Grained Atomic Evaluation of Factual Precision in Long Form Text Generation — Min et al. · min2023factscore · Paper
2023 · Instruction-Following Evaluation for Large Language Models — Zhou et al. · zhou2023ifeval · Paper
2023 · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. · zheng2023mtbench · Paper
2022 · TruthfulQA: Measuring How Models Mimic Human Falsehoods — Lin et al. · lin2022truthfulqa · Paper

Adversarial & Distribution Robustness (30)

2026 · Safety Is Not Universal: The Selective Safety Trap in LLM Alignment — Brito et al. · brito2026safetyisnotuniversal · DOI
2026 · Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases — Hahm et al. · hahm2026alignmenttampering · Paper
2026 · ERTS: Adversarial Robustness Testing of Ethical AI via Semantic Perturbation in a Bounded Consequence Space — Chaudhari · chaudhari2026erts · Paper
2026 · Into the Gray Zone: Domain Contexts Can Blur LLM Safety Boundaries — Hung et al. · hung2026grayzone · Paper
2026 · Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models — Lian et al. · lian2026ethicalvulnerability · DOI
2026 · SafetyMem: Adaptive Jailbreak Defense via Dual-Component Safety Memory — Wang et al. · wang2026safetymem · Paper
2026 · When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models — Choi et al. · choi2026context · Paper
2026 · When Safety Alignment Fails to Generalize: Probing with Language Game Jailbreaks — Long et al. · long2026languagegames · Paper
2025 · Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — Sharma et al. · sharma2025constitutionalclassifiers · Paper
2025 · Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Betley et al. · betley2025emergent · Paper
2025 · Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models — Shen et al. · shen2025antidote · Paper
2025 · Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks — Andriushchenko et al. · andriushchenko2025adaptive · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2025 · Alignment Faking in Large Language Models — Greenblatt et al. · greenblatt2025alignmentfaking · Paper
2024 · CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion — Ren et al. · ren2024codeattack · DOI
2024 · A StrongREJECT for Empty Jailbreaks — Souly et al. · souly2024strongreject · Paper
2024 · BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment — Wang et al. · wang2024backdooralign · Paper
2024 · Fine-Tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Qi et al. · qi2024finetuning · Paper
2024 · HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal — Mazeika et al. · mazeika2024harmbench · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Hubinger et al. · hubinger2024sleeper · Paper
2024 · Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models — Denison et al. · denison2024subterfuge · Paper
2024 · XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models — Rottger et al. · rottger2024xstest · Paper
2023 · Holistic Evaluation of Language Models — Liang et al. · liang2023helm · Scholar
2023 · Model Evaluation for Extreme Risks — Shevlane et al. · shevlane2023extremerisks · Paper
2023 · Universal and Transferable Adversarial Attacks on Aligned Language Models — Zou et al. · zou2023jailbreak · Paper
2022 · Goal Misgeneralization in Deep Reinforcement Learning — Di Langosco et al. · langosco2022goal · Paper
2022 · Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals — Shah et al. · shah2022goal · Paper
2022 · Red Teaming Language Models with Language Models — Perez et al. · perez2022redteaming · DOI
2016 · Concrete Problems in AI Safety — Amodei et al. · amodei2016concrete · Paper

Alignment Preservation (17)

2026 · Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance — Zhang et al. · zhang2026safetyoneshot · Paper
2026 · Alignment Midtraining for Animals — Brazilek et al. · brazilek2026alignment · Paper
2026 · Continual Safety Alignment via Gradient-Based Sample Selection — Bach et al. · bach2026continual · Paper
2026 · Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment — Wang et al. · wang2026sot · Paper
2026 · SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging — Djuhera et al. · djuhera2026safemerge · Paper
2025 · Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2025antidote · Paper
2025 · COPR: Continual Human Preference Learning via Optimal Policy Regularization — Zhang et al. · zhang2025copr · Paper
2025 · Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Betley et al. · betley2025emergent · Paper
2025 · Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment — Li et al. · li2025larf · Paper
2025 · Lifelong Safety Alignment for Language Models — Wang et al. · wang2025lifelong · Paper
2025 · SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation — Li et al. · li2025salora · Scholar
2025 · Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron — Zhao et al. · zhao2025safetyneurons · Paper
2024 · Fine-Tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Qi et al. · qi2024finetuning · Paper
2024 · Improving Alignment and Robustness with Circuit Breakers — Zou et al. · zou2024circuitbreakers · Paper
2024 · Keeping LLMs Aligned After Fine-Tuning: The Crucial Role of Prompt Templates — Lyu et al. · lyu2024keeping · Paper
2024 · Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024lisa · Paper
2024 · Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack — Huang et al. · huang2024vaccine · Paper

System Assurance

Mechanistic & Theoretical Evidence (31)

2026 · Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions — Naseem · naseem2026mechanistic · arXiv
2026 · Intrinsic Barriers and Practical Pathways for Human–AI Alignment: An Agreement-Based Complexity Analysis — Nayebi · nayebi2026barriers · Paper
2026 · Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models — Lian et al. · lian2026ethicalvulnerability · DOI
2026 · Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training — Yong et al. · yong2026selfjailbreaking · Paper
2026 · Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium — Liu et al. · liu2026statisticalalignment · DOI
2026 · Sustaining AI Safety: Control-Theoretic External Impossibility, Intrinsic Necessity, and Structural Requirements — Mazzu · mazzu2026sustaining · Paper
2026 · The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation — Falahati et al. · falahati2026alignmentgame · Paper
2025 · Open Problems in Mechanistic Interpretability — Sharkey et al. · sharkey2025openproblems · arXiv
2025 · The Alignment Trap: Complexity Barriers — Yao · yao2025alignmenttrap · arXiv
2025 · Corrigibility Transformation: Constructing Goals That Accept Updates — Hudson · hudson2025corrigibility · Paper
2025 · Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? — Gölz et al. · golz2025distortion · Paper
2025 · Reasoning Models Don't Always Say What They Think — Chen et al. · chen2025cotfaithfulness · Paper
2025 · The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence — Wollschläger et al. · wollschlager2025geometry · Paper
2025 · The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions — Pan et al. · pan2025hiddendimensions · Paper
2025 · Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons — Chen et al. · chen2025safetyneurons · Paper
2025 · Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron — Zhao et al. · zhao2025safetyneurons · Paper
2025 · Why Do Some Language Models Fake Alignment While Others Don't? — Sheshadri et al. · sheshadri2025alignmentfaking · DOI
2025 · Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region — Leong et al. · leong2025template · Paper
2024 · Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 — Lieberum et al. · lieberum2024gemmascope · arXiv
2024 · Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Templeton et al. · templeton2024scaling · Paper
2024 · Fundamental Limitations of Alignment in Large Language Models — Wolf et al. · wolf2024limitations · Paper
2024 · Refusal in Language Models Is Mediated by a Single Direction — Arditi et al. · arditi2024refusal · Paper
2024 · Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms — Rafailov et al. · rafailov2024overoptimization · Paper
2024 · Steering Language Models with Activation Engineering — Rimsky et al. · rimsky2024caa · DOI
2024 · Understanding the Learning Dynamics of Alignment with Human Feedback — Im et al. · im2024dynamics · Paper
2024 · What Makes and Breaks Safety Fine-tuning? A Mechanistic Study — Jain et al. · jain2024safetyfinetuning · Paper
2023 · Discovering Latent Knowledge in Language Models Without Supervision — Burns et al. · burns2023latent · Scholar
2023 · Inference-Time Intervention: Eliciting Truthful Answers from a Language Model — Li et al. · li2023iti · DOI
2023 · Representation Engineering: A Top-Down Approach to AI Transparency — Zou et al. · zou2023repreng · Paper
2017 · The Off-Switch Game — Hadfield-Menell et al. · hadfieldmenell2017offswitch · DOI
2015 · Corrigibility — Soares et al. · soares2015corrigibility · Paper

Monitoring & Auditing (29)

2026 · Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs — Feng et al. · feng2026benchmarking · arXiv
2026 · Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage? — Ye et al. · ye2026codingenemy · arXiv
2026 · Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems — Alamdari et al. · alamdari2026formal · DOI
2026 · Sycophancy Towards Researchers Drives Performative Misalignment — Baek et al. · baek2026sycophancy · arXiv
2026 · When Saying ``No'' Is Not Enough: Cognitive-Action Decoupling and the Illusion of Safety in LLM Agents — Yu et al. · yu2026whensayingno · DOI
2026 · AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models — Wang et al. · wang2026agenticeval · Paper
2026 · EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures — Uluirmak et al. · uluirmak2026evalsafetygap · Paper
2026 · Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs — Panfilov et al. · panfilov2026dishonesty · Paper
2026 · Watch the Weights: Unsupervised Monitoring and Control of Fine-Tuned LLMs — Zhong et al. · zhong2026weights · Paper
2025 · Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats — Wen et al. · wen2025adaptivedeployment · Paper
2025 · CALM: Curiosity-Driven Auditing for Large Language Models — Zheng et al. · zheng2025calm · DOI
2025 · An Auditing Test to Detect Behavioral Shift in Language Models — Richter et al. · richter2025auditing · Paper
2025 · Auditing Language Models for Hidden Objectives — Marks et al. · marks2025auditing · Paper
2025 · CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring — Arnav et al. · arnav2025cot · DOI
2025 · Detecting High-Stakes Interactions with Activation Probes — McKenzie et al. · mckenzie2025highstakes · DOI
2025 · From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring — Li et al. · li2025streaming · DOI
2025 · RewardBench: Evaluating Reward Models for Language Modeling — Lambert et al. · lambert2025rewardbench · Paper
2025 · Why Do Some Language Models Fake Alignment While Others Don't? — Sheshadri et al. · sheshadri2025alignmentfaking · DOI
2025 · SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior — Li et al. · li2025safetyanalyst · Paper
2025 · Alignment Faking in Large Language Models — Greenblatt et al. · greenblatt2025alignmentfaking · Paper
2024 · AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach — Amirizaniani et al. · amirizaniani2024auditllm · DOI
2024 · AI Control: Improving Safety Despite Intentional Subversion — Greenblatt et al. · greenblatt2024control · Paper
2024 · Black-Box Access is Insufficient for Rigorous AI Audits — Casper et al. · casper2024blackbox · DOI
2024 · Proving Test Set Contamination in Black-Box Language Models — Oren et al. · oren2024contamination · Scholar
2024 · Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Hubinger et al. · hubinger2024sleeper · Paper
2024 · Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models — Denison et al. · denison2024subterfuge · Paper
2023 · Discovering Language Model Behaviors with Model-Written Evaluations — Perez et al. · perez2023modelwritten · Paper
2023 · Holistic Evaluation of Language Models — Liang et al. · liang2023helm · Scholar
2023 · Model Evaluation for Extreme Risks — Shevlane et al. · shevlane2023extremerisks · Paper

General Survey Context

These surveys span several branches and are kept outside any single technical category.

2025 · AI Alignment: A Contemporary Survey — Ji et al. · ji2025alignment · Paper
2025 · Aligning Multimodal LLM with Human Preference: A Survey — Yu et al. · yu2025multimodalsurvey · Paper
2024 · A Survey on Human Preference Learning for Large Language Models — Jiang et al. · jiang2024preferencesurvey · Paper
2024 · Towards a Unified View of Preference Learning for Large Language Models: A Survey — Gao et al. · gao2024unifiedpreference · Paper
2023 · Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation — Fernandes et al. · fernandes2024feedback · Paper
2023 · Large Language Model Alignment: A Survey — Shen et al. · shen2023alignment · Paper

Contributing

Contributions and bibliographic corrections are welcome:

Include the paper's complete bibliographic record (authors, title, venue, year, and DOI or arXiv ID) in the pull request. Prefer the final venue page and DOI; use arXiv when no archival version is available.
Add papers to every branch they substantively address; assignments do not need to be exclusive.
Explain the proposed placement and include a stable public paper link in the pull request.

Suggested inclusion criteria:

The work contributes to alignment specification, supervision, training, inference-time control, evaluation, robustness, interpretability, or monitoring.
A stable manuscript or archival publication page is publicly available.
The bibliographic metadata can be independently checked.

Scope

Last synchronized with the taxonomy and bibliography on 2026-09-29. Citation aliases are normalized in the collection so that the same work is not counted twice under different BibTeX keys.

Similar repos

icip-cas/awesome-auto-alignment

Collection of papers for scalable automated alignment.

GaryYufei/AlignLLMHumanSurvey

Aligning Large Language Models with Human: A Survey

738

Mr-Tieguigui/LLM-Post-Training

102

huggingface/alignment-handbook

Robust recipes to align language models with human and AI preferences

5,686

PKU-Alignment/safe-rlhf

Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback

1,618

JLZhong23/awesome-reward-models

173

opendilab/awesome-RLHF

A curated list of reinforcement learning with human feedback resources (continually updated)

4,432

liyongqi2002/Awesome-Personalized-Alignment

A curated list of personalized alignment resources (continually updated).

Thang1703hrsh/Awesome-Human-AI-Alignment

README

Awesome Human–AI Alignment

News

Overview

Software framework

Examples

Taxonomy

Survey overview

Lifecycle taxonomy

Training-time mechanisms at a glance

Inference-time mechanisms at a glance

Collection coverage

Cross-cutting descriptors

Paper collection

Alignment Specification

Alignment Objectives

Task & Assistance Alignment (35)

Safety Alignment (27)

Values & Stakeholders

Personalized Alignment (24)

Pluralistic & Societal Alignment (35)

Context, Uncertainty & Drift (23)

Alignment Supervision

Feedback Source

Human Feedback (20)

AI Feedback (23)

Programmatic & Verifiable Feedback (22)

Feedback & Oversight

Demonstrations & Preferences (24)

Critique, Process & Trajectory Feedback (19)

Reliable & Scalable Oversight (25)

Alignment Mechanisms

Training-Time Alignment

Reward & Verifier Modeling (27)

Supervised Alignment (22)

Preference Optimization (50)

Reinforcement Learning (32)

Alignment Distillation (9)

Inference-Time Alignment

Steering, Search & Refinement (31)

Interactive & Agentic Control (17)

Alignment Assurance

Evaluation & Robustness

Behavioral & Evaluator Evaluation (26)

Adversarial & Distribution Robustness (30)

Alignment Preservation (17)

System Assurance

Mechanistic & Theoretical Evidence (31)

Monitoring & Auditing (29)

General Survey Context

Contributing

Scope

Similar repos

Thang1703hrsh/Awesome-Human-AI-Alignment

README

Awesome Human–AI Alignment

News

Overview

Software framework

Examples

Taxonomy

Survey overview

Lifecycle taxonomy

Training-time mechanisms at a glance

Inference-time mechanisms at a glance

Collection coverage

Cross-cutting descriptors

Paper collection

Alignment Specification

Alignment Objectives

Task & Assistance Alignment (35)

Safety Alignment (27)

Values & Stakeholders

Personalized Alignment (24)

Pluralistic & Societal Alignment (35)

Context, Uncertainty & Drift (23)

Alignment Supervision

Feedback Source

Human Feedback (20)