NLP Paper

natural language processing paper list

Contents

• Bert Series
• Transformer Series
• Transfer Learning
• Text Summarization
• Sentiment Analysis
• Question Answering
• Machine Translation
• Surver paper
• Downstream task
  • QA MC Dialogue
  • Slot filling
  • Analysis
  • Word segmentation parsing NER
  • Pronoun coreference resolution
  • Word sense disambiguation
  • Sentiment analysis
  • Relation extraction
  • Knowledge base
  • Text classification
  • WSC WNLI NLI
  • Commonsense
  • Extractive summarization
  • IR
• Generation
• Quality evaluator
• Modification (multi-task, masking strategy, etc.)
• Probe
• Multi-lingual
• Other than English models
• Domain specific
• Multi-modal
• Model compression
• LLM
• Reasoning and Chain of Thought
• Alignment and RLHF
• Retrieval Augmented Generation
• LLM Agents and Tool Use
• Long Context and Efficient Architectures
• Efficient Training Inference and Serving
• Pretraining Data Scaling Laws and Model Merging
• Prompting and In Context Learning
• Code LLM
• Evaluation and Benchmark
• Safety Hallucination and Interpretability
• Misc

Bert Series

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - NAACL 2019)
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding - arXiv 2019)
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding - arXiv 2019)
• RoBERTa: A Robustly Optimized BERT Pretraining Approach - arXiv 2019)
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations - arXiv 2019)
• Multi-Task Deep Neural Networks for Natural Language Understanding - arXiv 2019)
• What does BERT learn about the structure of language? (ACL2019)
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (ACL2019) [github]
• Open Sesame: Getting Inside BERT's Linguistic Knowledge (ACL2019 WS)
• Analyzing the Structure of Attention in a Transformer Language Model (ACL2019 WS)
• What Does BERT Look At? An Analysis of BERT's Attention (ACL2019 WS)
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains (ACL2019 WS)
• Inducing Syntactic Trees from BERT Representations (ACL2019 WS)
• A Multiscale Visualization of Attention in the Transformer Model (ACL2019 Demo)
• Visualizing and Measuring the Geometry of BERT
• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP2019)
• Are Sixteen Heads Really Better than One? (NeurIPS2019)
• On the Validity of Self-Attention as Explanation in Transformer Models
• Visualizing and Understanding the Effectiveness of BERT (EMNLP2019)
• Attention Interpretability Across NLP Tasks
• Revealing the Dark Secrets of BERT (EMNLP2019)
• Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs (EMNLP2019)
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (EMNLP2019)
• A Primer in BERTology: What we know about how BERT works
• Do NLP Models Know Numbers? Probing Numeracy in Embeddings (EMNLP2019)
• How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations (CIKM2019)
• Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
• What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
• Calibration of Pre-trained Transformers
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models [github]
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices [github]
• Measuring and Reducing Gendered Correlations in Pre-trained Models
• DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION [github] (ACL2021)
• STRUCTBERT: INCORPORATING LANGUAGE STRUCTURES INTO PRE-TRAINING FOR DEEP LANGUAGE UNDERSTANDING (ACL2021)
• SpanBERT: Improving Pre-training by Representing and Predicting Spans [github] (ACL2021)

Transformer Series

• Attention Is All You Need - arXiv 2017)
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context - arXiv 2019)
• Universal Transformers - ICLR 2019)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer - arXiv 2019)
• Reformer: The Efficient Transformer - ICLR 2020)
• Adaptive Attention Span in Transformers (ACL2019)
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (ACL2019) [github]
• Generating Long Sequences with Sparse Transformers
• Adaptively Sparse Transformers (EMNLP2019)
• Compressive Transformers for Long-Range Sequence Modelling
• The Evolved Transformer (ICML2019)
• Reformer: The Efficient Transformer (ICLR2020) [github]
• GRET: Global Representation Enhanced Transformer (AAAI2020)
• Transformer on a Diet [github]
• Efficient Content-Based Sparse Attention with Routing Transformers
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Recipes for building an open-domain chatbot
• Longformer: The Long-Document Transformer
• UnifiedQA: Crossing Format Boundaries With a Single QA System [github]
• Big Bird: Transformers for Longer Sequences
• Longformer: The Long-Document Transformer [github] (ACL2021)
• REFORMER: THE EFFICIENT TRANSFORMER (ACL2021)
• Linformer: Self-Attention with Linear Complexity (ACL2021)
• RETHINKING ATTENTION WITH PERFORMERS [github] (ICLR2021)
• Big Bird: Transformers for Longer Sequences (ACL2021)

Transfer Learning

• Deep contextualized word representations - NAACL 2018)
• Universal Language Model Fine-tuning for Text Classification - ACL 2018)
• Improving Language Understanding by Generative Pre-Training - Alec Radford)
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - NAACL 2019)
• Cloze-driven Pretraining of Self-attention Networks - arXiv 2019)
• Unified Language Model Pre-training for Natural Language Understanding and Generation - arXiv 2019)
• MASS: Masked Sequence to Sequence Pre-training for Language Generation - ICML 2019)
• MPNet: Masked and Permuted Pre-training for Language Understanding)[github]
• UNILMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training)[github]
• AdapterFusion:Non-Destructive Task Composition for Transfer Learning) (ACL2022)
• Prefix-Tuning: Optimizing Continuous Prompts for Generation (ACL2022)
• LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS (ACL2022)
• Self-Instruct: Aligning Language Models with Self-Generated Instructions (ACL2023)
• WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR2024)
• LIMA: Less Is More for Alignment (NeurIPS2023)
• QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS2023)
• ReLoRA: High-Rank Training Through Low-Rank Updates (arXiv 2023)
• AlpaGasus: Training A Better Alpaca with Fewer Data (ICLR2024)
• Instruction Tuning for Large Language Models: A Survey (arXiv 2023)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR2024)
• VeRA: Vector-based Random Matrix Adaptation (ICLR2024)
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 (arXiv 2023)
• Orca 2: Teaching Small Language Models How to Reason (arXiv 2023)
• DoRA: Weight-Decomposed Low-Rank Adaptation [github] (ICML2024)
• LoRA+: Efficient Low Rank Adaptation of Large Models (ICML2024)
• A Survey on Knowledge Distillation of Large Language Models (arXiv 2024)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection [github] (ICML2024)
• LoRA Learns Less and Forgets Less (TMLR)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence (arXiv 2024)
• OpenThoughts: Data Recipes for Reasoning Models (arXiv 2025)
• Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting (arXiv 2026)
• On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters (arXiv 2026)
• Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA (arXiv 2026)

Text Summarization

• Positional Encoding to Control Output Sequence Length - Sho Takase(2019)
• Fine-tune BERT for Extractive Summarization - Yang Liu(2019)
• Language Models are Unsupervised Multitask Learners - Alec Radford(2019)
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss - Wan-Ting Hsu(2018)
• A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents - Arman Cohan(2018)
• GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES - Peter J. Liu(2018)
• Get To The Point: Summarization with Pointer-Generator Networks - Abigail See(2017)
• A Neural Attention Model for Sentence Summarization - Alexander M. Rush(2015)
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (ACL2021)
• Abstractive Text Summarization Using BART (ACL2021)
• CTRLSUM: TOWARDS GENERIC CONTROLLABLE TEXT SUMMARIZATION (ACL2021)

Sentiment Analysis

• Multi-Task Deep Neural Networks for Natural Language Understanding - Xiaodong Liu(2019)
• Aspect-level Sentiment Analysis using AS-Capsules - Yequan Wang(2019)
• On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis - Jose Camacho-Collados(2018)
• Learned in Translation: Contextualized Word Vectors - Bryan McCann(2018)
• Universal Language Model Fine-tuning for Text Classification - Jeremy Howard(2018)
• Convolutional Neural Networks with Recurrent Neural Filters - Yi Yang(2018)
• Information Aggregation via Dynamic Routing for Sequence Encoding - Jingjing Gong(2018)
• Learning to Generate Reviews and Discovering Sentiment - Alec Radford(2017)
• A Structured Self-attentive Sentence Embedding - Zhouhan Lin(2017)

Question Answering

• Language Models are Unsupervised Multitask Learners - Alec Radford(2019)
• Improving Language Understanding by Generative Pre-Training - Alec Radford(2018)
• Bidirectional Attention Flow for Machine Comprehension - Minjoon Seo(2018)
• Reinforced Mnemonic Reader for Machine Reading Comprehension - Minghao Hu(2017)
• Neural Variational Inference for Text Processing - Yishu Miao(2015)
• UnifiedQA: Crossing Format Boundaries with a Single QA System [github] (ACL2021)
• REALM: Retrieval-Augmented Language Model Pre-Training (ACL2021)

Machine Translation

• The Evolved Transformer - David R. So(2019)

Surver paper

• Evolution of transfer learning in natural language processing
• Pre-trained Models for Natural Language Processing: A Survey
• A Survey on Contextual Embeddings
• A Survey of Large Language Models (FrontComputSci2026)
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (arXiv 2023)
• A Survey on Model Compression for Large Language Models (TACL)
• Efficient Large Language Models: A Survey (arXiv 2023)
• Large Language Models: A Survey (arXiv 2024)
• A Survey on Post-training of Large Language Models (arXiv 2025)
• The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook (arXiv 2026)
• World Action Models: A Survey (arXiv 2026)

Downstream task

QA MC Dialogue

• A BERT Baseline for the Natural Questions
• MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (ACL2019)
• Unsupervised Domain Adaptation on Reading Comprehension
• BERTQA -- Attention on Steroids
• A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (EMNLP2019)
• SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
• Multi-hop Question Answering via Reasoning Chains
• Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
• Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (EMNLP2019 WS)
• End-to-End Open-Domain Question Answering with BERTserini (NAALC2019)
• Latent Retrieval for Weakly Supervised Open Domain Question Answering (ACL2019)
• Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (EMNLP2019)
• Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (ICLR2020)
• Learning to Ask Unanswerable Questions for Machine Reading Comprehension (ACL2019)
• Unsupervised Question Answering by Cloze Translation (ACL2019)
• Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation
• A Recurrent BERT-based Model for Question Generation (EMNLP2019 WS)
• Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds
• Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (ACL2019)
• Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning (CIKM2019)
• SG-Net: Syntax-Guided Machine Reading Comprehension
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
• Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (EMNLP2019)
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning (ICLR2020)
• Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
• BAS: An Answer Selection Method Using BERT Language Model
• Beat the AI: Investigating Adversarial Human Annotations for Reading Comprehension
• A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehension (ACL2019 WS)
• FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension (ACL2019 WS)
• BERT with History Answer Embedding for Conversational Question Answering (SIGIR2019)
• GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension (ICML2019 WS)
• Beyond English-only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian (RANLP2019)
• XQA: A Cross-lingual Open-domain Question Answering Dataset (ACL2019)
• Cross-Lingual Machine Reading Comprehension (EMNLP2019)
• Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
• Multilingual Question Answering from Formatted Text applied to Conversational Agents
• BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels (EMNLP2019)
• MLQA: Evaluating Cross-lingual Extractive Question Answering
• Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension (TACL)
• SberQuAD - Russian Reading Comprehension Dataset: Description and Analysis
• Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (EMNLP2019)
• BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer (Interspeech2019)
• Dialog State Tracking: A Neural Reading Comprehension Approach
• A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems (ICASSP2020)
• Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking
• Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker
• Domain Adaptive Training BERT for Response Selection
• BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding
• A BERT Baseline for the Natural Questions
• Wizard of Wikipedia
• BlenderBot 3: a deployed conversational agent that continually∗ learns to responsibly engage (ACL2022)

Slot filling

• BERT for Joint Intent Classification and Slot Filling
• Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model
• A Comparison of Deep Learning Methods for Language Understanding (Interspeech2019)

Analysis

• Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention
• Neural Aspect and Opinion Term Extraction with Mined Rules as Weak Supervision (ACL2019)
• BERT-based Lexical Substitution (ACL2019)
• Assessing BERT’s Syntactic Abilities
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
• Simple BERT Models for Relation Extraction and Semantic Role Labeling
• LIMIT-BERT : Linguistic Informed Multi-Task BERT
• A Simple BERT-Based Approach for Lexical Simplification
• Multi-headed Architecture Based on BERT for Grammatical Errors Correction (ACL2019 WS)
• Towards Minimal Supervision BERT-based Grammar Error Correction
• BERT-Based Arabic Social Media Author Profiling
• Sentence-Level BERT and Multi-Task Learning of Age and Gender in Social Media
• Evaluating the Factual Consistency of Abstractive Text Summarization
• NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution
• xSLUE: A Benchmark and Analysis Platform for Cross-Style Language Understanding and Evaluation
• TabFact: A Large-scale Dataset for Table-based Fact Verification
• Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents
• LAMBERT: Layout-Aware language Modeling using BERT for information extraction
• Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings (ECIR2020) [github]
• Keyphrase Extraction with Span-based Feature Representations
• What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

Word segmentation parsing NER

• BERT Meets Chinese Word Segmentation
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT
• Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisited (EMNLP2019)
• Is POS Tagging Necessary or Even Helpful for Neural Dependency Parsing?
• Parsing as Pretraining (AAAI2020)
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
• Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement
• Named Entity Recognition -- Is there a glass ceiling? (CoNLL2019)
• A Unified MRC Framework for Named Entity Recognition
• Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models
• Robust Named Entity Recognition with Truecasing Pretraining (AAAI2020)
• LTP: A New Active Learning Strategy for Bert-CRF Based Named Entity Recognition
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• Portuguese Named Entity Recognition using BERT-CRF
• Towards Lingua Franca Named Entity Recognition with BERT

Pronoun coreference resolution

• Resolving Gendered Ambiguous Pronouns with BERT (ACL2019 WS)
• Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution Challenge (ACL2019 WS)
• Gendered Pronoun Resolution using BERT and an extractive question answering formulation (ACL2019 WS)
• MSnet: A BERT-based Network for Gendered Pronoun Resolution (ACL2019 WS)
• Fill the GAP: Exploiting BERT for Pronoun Resolution (ACL2019 WS)
• On GAP Coreference Resolution Shared Task: Insights from the 3rd Place Solution (ACL2019 WS)
• Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution (ACL2019 WS)
• BERT Masked Language Modeling for Co-reference Resolution (ACL2019 WS)
• Coreference Resolution with Entity Equalization (ACL2019)
• BERT for Coreference Resolution: Baselines and Analysis (EMNLP2019) [github]
• WikiCREM: A Large Unsupervised Corpus for Coreference Resolution (EMNLP2019)
• Ellipsis and Coreference Resolution as Question Answering
• Coreference Resolution as Query-based Span Prediction
• Multi-task Learning Based Neural Bridging Reference Resolution

Word sense disambiguation

• GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge (EMNLP2019)
• Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations (EMNLP2019)
• Using BERT for Word Sense Disambiguation
• Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation (ACL2019)
• Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings (KONVENS2019)

Sentiment analysis

• Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (NAACL2019)
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (NAACL2019)
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (EMNLP2019 WS)
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
• An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese (ACL2019)
• "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT
• Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference
• Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

Relation extraction

• Matching the Blanks: Distributional Similarity for Relation Learning (ACL2019)
• BERT-Based Multi-Head Selection for Joint Entity-Relation Extraction (NLPCC2019)
• Enriching Pre-trained Language Model with Entity Information for Relation Classification
• Span-based Joint Entity and Relation Extraction with Transformer Pre-training
• Fine-tune Bert for DocRED with Two-step Process
• Entity, Relation, and Event Extraction with Contextualized Span Representations (EMNLP2019)

Knowledge base

• KG-BERT: BERT for Knowledge Graph Completion
• Language Models as Knowledge Bases? (EMNLP2019) [github]
• BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA
• Inducing Relational Knowledge from BERT (AAAI2020)
• Latent Relation Language Models (AAAI2020)
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (ICLR2020)
• Zero-shot Entity Linking with Dense Entity Retrieval
• Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNLL2019)
• Improving Entity Linking by Modeling Latent Entity Type Information (AAAI2020)
• PEL-BERT: A Joint Model for Protocol Entity Linking
• How Can We Know What Language Models Know?
• REALM: Retrieval-Augmented Language Model Pre-Training

Text classification

• How to Fine-Tune BERT for Text Classification?
• X-BERT: eXtreme Multi-label Text Classification with BERT
• DocBERT: BERT for Document Classification
• Enriching BERT with Knowledge Graph Embeddings for Document Classification
• Classification and Clustering of Arguments with Contextualized Word Embeddings (ACL2019)
• BERT for Evidence Retrieval and Claim Verification
• Stacked DeBERT: All Attention in Incomplete Data for Text Classification
• Cost-Sensitive BERT for Generalisable Sentence Classification with Imbalanced Data

WSC WNLI NLI

• Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
• A Surprisingly Robust Trick for the Winograd Schema Challenge
• WinoGrande: An Adversarial Winograd Schema Challenge at Scale (AAAI2020)
• Improving Natural Language Inference with a Pretrained Parser
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Adversarial Analysis of Natural Language Inference Systems (ICSC2020)
• HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference (LREC2020)
• Evaluating BERT for natural language inference: A case study on the CommitmentBank (EMNLP2019)

Commonsense

• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (NAACL2019)
• HellaSwag: Can a Machine Really Finish Your Sentence? (ACL2019) [website]
• Story Ending Prediction by Transferable BERT (IJCAI2019)
• Explain Yourself! Leveraging Language Models for Commonsense Reasoning (ACL2019)
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• Informing Unsupervised Pretraining with External Linguistic Knowledge
• Commonsense Knowledge + BERT for Level 2 Reading Comprehension Ability Test
• BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge
• Commonsense Knowledge Mining from Pretrained Models (EMNLP2019)
• KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (EMNLP2019)
• Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (EMNLP2019 WS)
• Do Massively Pretrained Language Models Make Better Storytellers? (CoNLL2019)
• PIQA: Reasoning about Physical Commonsense in Natural Language (AAAI2020)
• Evaluating Commonsense in Pre-trained Language Models (AAAI2020)
• Why Do Masked Neural Language Models Still Need Common Sense Knowledge?
• Do Neural Language Representations Learn Physical Commonsense? (CogSci2019)

Extractive summarization

• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (ACL2019)
• Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression
• Discourse-Aware Neural Extractive Model for Text Summarization
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization[github]
• Discourse-Aware Neural Extractive Text Summarization[github]

IR

• Passage Re-ranking with BERT
• Investigating the Successes and Failures of BERT for Passage Re-Ranking
• Understanding the Behaviors of BERT in Ranking
• Document Expansion by Query Prediction
• CEDR: Contextualized Embeddings for Document Ranking (SIGIR2019)
• Deeper Text Understanding for IR with Contextual Neural Language Modeling (SIGIR2019)
• FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance (SIGIR2019)
• Multi-Stage Document Ranking with BERT
• REALM: Retrieval-Augmented Language Model Pre-Training
• How Much Knowledge Can You Pack Into the Parameters of a Language Model? [github]
• Dense Passage Retrieval for Open-Domain Question Answering [github]

Generation

• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (NAACL2019 WS)
• Pretraining-Based Natural Language Generation for Text Summarization
• Text Summarization with Pretrained Encoders (EMNLP2019) [github (original)] [github (huggingface)]
• Multi-stage Pretraining for Abstractive Summarization
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• MASS: Masked Sequence to Sequence Pre-training for Language Generation (ICML2019) [github], [github]
• Unified Language Model Pre-training for Natural Language Understanding and Generation [github] (NeurIPS2019)
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training [github]
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Towards Making the Most of BERT in Neural Machine Translation
• Improving Neural Machine Translation with Pre-trained Representation
• On the use of BERT for Neural Machine Translation (EMNLP2019 WS)
• Incorporating BERT into Neural Machine Translation (ICLR2020)
• Recycling a Pre-trained BERT Encoder for Neural Machine Translation
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• Mask-Predict: Parallel Decoding of Conditional Masked Language Models (EMNLP2019)
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
• Cross-Lingual Natural Language Generation via Pre-Training (AAAI2020) [github]
• Multilingual Denoising Pre-training for Neural Machine Translation
• PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• Unsupervised Pre-training for Natural Language Generation: A Literature Review
• Improving Language Understanding by Generative Pre-Training
• Language Models are Unsupervised Multitask Learners
• Language Models are Few-Shot Learners

Quality evaluator

• BERTScore: Evaluating Text Generation with BERT (ICLR2020)
• Machine Translation Evaluation with BERT Regressor
• SumQE: a BERT-based Summary Quality Estimation Model (EMNLP2019)
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (EMNLP2019) [github]
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward

Modification (multi-task, masking strategy, etc.)

• Multi-Task Deep Neural Networks for Natural Language Understanding (ACL2019)
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (ICML2019)
• Unifying Question Answering and Text Classification via Span Extraction
• ERNIE: Enhanced Language Representation with Informative Entities (ACL2019)
• ERNIE: Enhanced Representation through Knowledge Integration
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (AAAI2020)
• Pre-Training with Whole Word Masking for Chinese BERT
• SpanBERT: Improving Pre-training by Representing and Predicting Spans [github]
• Blank Language Models
• Efficient Training of BERT by Progressively Stacking (ICML2019) [github]
• RoBERTa: A Robustly Optimized BERT Pretraining Approach [github]
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (ICLR2020)
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (ICLR2020) [github] [blog]
• FreeLB: Enhanced Adversarial Training for Language Understanding (ICLR2020)
• KERMIT: Generative Insertion-Based Modeling for Sequences
• DisSent: Sentence Representation Learning from Explicit Discourse Relations (ACL2019)
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (ICLR2020)
• Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding
• SenseBERT: Driving Some Sense into BERT
• Semantics-aware BERT for Language Understanding (AAAI2020)
• K-BERT: Enabling Language Representation with Knowledge Graph
• Knowledge Enhanced Contextual Word Representations (EMNLP2019)
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP2019)
• SBERT-WK: A Sentence Embedding Method By Dissecting BERT-based Word Models
• Universal Text Representation from BERT: An Empirical Study
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• Transfer Fine-Tuning: A BERT Case Study (EMNLP2019)
• Improving Pre-Trained Multilingual Models with Vocabulary Expansion (CoNLL2019)
• SesameBERT: Attention for Anywhere
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [github]
• SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Probe

• A Structural Probe for Finding Syntax in Word Representations (NAACL2019)
• Linguistic Knowledge and Transferability of Contextual Representations (NAACL2019) [github]
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (*SEM2019)
• BERT Rediscovers the Classical NLP Pipeline (ACL2019)
• Probing Neural Network Comprehension of Natural Language Arguments (ACL2019)
• Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (EMNLP2019 WS)
• What do you mean, BERT? Assessing BERT as a Distributional Semantics Model
• Quantity doesn't buy quality syntax with neural language models (EMNLP2019)
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (ICLR2020)
• oLMpics -- On what Language Model Pre-training Captures
• How Much Knowledge Can You Pack Into the Parameters of a Language Model?
• What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge
• Attention is not Explanation (ACL2021)

Multi-lingual

• Multilingual Constituency Parsing with Self-Attention and Pre-Training (ACL2019)
• Language Model Pretraining (NeurIPS2019) [github]
• 75 Languages, 1 Model: Parsing Universal Dependencies Universally (EMNLP2019) [github]
• Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations (EMNLP2019 WS)
• Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (EMNLP2019)
• How multilingual is Multilingual BERT? (ACL2019)
• How Language-Neutral is Multilingual BERT?
• Is Multilingual BERT Fluent in Language Generation?
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (EMNLP2019)
• BERT is Not an Interlingua and the Bias of Tokenization (EMNLP2019 WS)
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study (ICLR2020)
• Multilingual Alignment of Contextual Word Representations (ICLR2020)
• On the Cross-lingual Transferability of Monolingual Representations
• Unsupervised Cross-lingual Representation Learning at Scale
• Emerging Cross-lingual Structure in Pretrained Language Models
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• Fully Unsupervised Crosslingual Semantic Textual Similarity Metric Based on BERT for Identifying Parallel Data (CoNLL2019)
• What the [MASK]? Making Sense of Language-Specific BERT Models
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (ACL2021) [github]
• KLUE: Korean Language Understanding Evaluation (arXiv 2021)
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (EMNLP2021)
• Crosslingual Generalization through Multitask Finetuning (ACL2023)
• MEGA: Multilingual Evaluation of Generative AI (EMNLP2023)
• Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS2023)
• A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models (arXiv 2023)
• KoBBQ: Korean Bias Benchmark for Question Answering (TACL)
• Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models (arXiv 2023)
• HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (LREC-COLING2024)
• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR2024)
• AceGPT, Localizing Large Language Models in Arabic (NAACL2024)
• Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model (arXiv 2023)
• SeaLLMs -- Large Language Models for Southeast Asia (ACL2024)
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (ACL2024)
• KMMLU: Measuring Massive Multitask Language Understanding in Korean (NAACL2025)
• KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge (ACL2024 Findings)
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks (arXiv 2024)
• CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean (LREC-COLING2024)
• Latxa: An Open Language Model and Evaluation Suite for Basque (ACL2024)
• HyperCLOVA X Technical Report (arXiv 2024)
• Sailor: Open Language Models for South-East Asia (arXiv 2024)
• Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities (arXiv 2024)
• Aya 23: Open Weight Releases to Further Multilingual Progress (arXiv 2024)
• Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark (ACL2024)
• LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs (arXiv 2024)
• SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages (arXiv 2024)
• EXAONE 3.0 7.8B Instruction Tuned Language Model (arXiv 2024)
• EuroLLM: Multilingual Language Models for Europe (arXiv 2024)
• Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs (arXiv 2024)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (arXiv 2024)
• Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (ACL2025)
• Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement (arXiv 2024)
• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier (arXiv 2024)
• EXAONE 3.5: Series of Large Language Models for Real-world Use Cases (arXiv 2024)
• Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap (arXiv 2025)
• Kanana: Compute-efficient Bilingual Language Models (arXiv 2025)
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (EMNLP2025)
• EXAONE Deep: Reasoning Enhanced Language Models (arXiv 2025)
• SEA-LION: Southeast Asian Languages in One Network (AACL2025)
• Trillion 7B Technical Report (arXiv 2025)
• Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs (arXiv 2025)
• HyperCLOVA X THINK Technical Report (arXiv 2025)
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes (arXiv 2025)
• T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playground (arXiv 2025)
• K-EXAONE Technical Report (arXiv 2026)
• Solar Open Technical Report (arXiv 2026)
• EXAONE 4.5 Technical Report (arXiv 2026)
• K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts (arXiv 2026)
• Motif 3: Technical Report (arXiv 2026)

Other than English models

• CamemBERT: a Tasty French Language Model
• FlauBERT: Unsupervised Language Model Pre-training for French
• Multilingual is not enough: BERT for Finnish
• BERTje: A Dutch BERT Model
• RobBERT: a Dutch RoBERTa-based Language Model
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• AraBERT: Transformer-based Model for Arabic Language Understanding
• PhoBERT: Pre-trained language models for Vietnamese
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

Domain specific

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (ACL2019 WS)
• BERT-based Ranking for Biomedical Entity Normalization
• PubMedQA: A Dataset for Biomedical Research Question Answering (EMNLP2019)
• Pre-trained Language Model for Biomedical Question Answering
• How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Publicly Available Clinical BERT Embeddings (NAACL2019 WS)
• Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT
• SciBERT: Pretrained Contextualized Embeddings for Scientific Text [github]
• PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

Multi-modal

• VideoBERT: A Joint Model for Video and Language Representation Learning (ICCV2019)
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (NeurIPS2019)
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Selfie: Self-supervised Pretraining for Image Embedding
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Contrastive Bidirectional Transformer for Temporal Representation Learning
• M-BERT: Injecting Multimodal Information in the BERT Structure
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers (EMNLP2019)
• Fusion of Detected Objects in Text for Visual Question Answering (EMNLP2019)
• BERT representations for Video Question Answering (WACV2020)
• Unified Vision-Language Pre-Training for Image Captioning and VQA [github]
• Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations (ICLR2020)
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• UNITER: Learning UNiversal Image-TExt Representations
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks
• BERT Can See Out of the Box: On the Cross-modal Transferability of Text Representations
• BERT for Large-scale Video Segment Classification with Test-time Augmentation (ICCV2019WS)
• SpeechBERT: Cross-Modal Pre-trained Language Model for End-to-end Spoken Question Answering
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Effectiveness of self-supervised pre-training for speech recognition
• Understanding Semantics from Speech Through Pre-training
• Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models
• Robust Speech Recognition via Large-Scale Weak Supervision (arXiv 2022)
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (arXiv 2023)
• Sigmoid Loss for Language Image Pre-Training (ICCV2023)
• EVA-CLIP: Improved Training Techniques for CLIP at Scale (arXiv 2023)
• Visual Instruction Tuning (NeurIPS2023)
• Evaluating Object Hallucination in Large Vision-Language Models (EMNLP2023)
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (ACL2024)
• AudioPaLM: A Large Language Model That Can Speak and Listen (arXiv 2023)
• A Survey on Multimodal Large Language Models (arXiv 2023)
• MMBench: Is Your Multi-modal Model an All-around Player? (arXiv 2023)
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (arXiv 2023)
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (ICLR2024)
• Improved Baselines with Visual Instruction Tuning (CVPR2024)
• CogVLM: Visual Expert for Pretrained Language Models (arXiv 2023)
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (arXiv 2023)
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (EMNLP2024)
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (CVPR2024)
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (arXiv 2023)
• MM-LLMs: Recent Advances in MultiModal Large Language Models (ACL2024 Findings)
• mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding (EMNLP2024 Findings)
• BLINK: Multimodal Large Language Models Can See but Not Perceive (ECCV2024)
• What matters when building vision-language models? (arXiv 2024)
• Chameleon: Mixed-Modal Early-Fusion Foundation Models (arXiv 2024)
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (arXiv 2024)
• Long Context Transfer from Language to Vision (arXiv 2024)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS2024)
• ColPali: Efficient Document Retrieval with Vision Language Models (ICLR2025)
• PaliGemma: A versatile 3B VLM for transfer (arXiv 2024)
• Qwen2-Audio Technical Report (arXiv 2024)
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone (arXiv 2024)
• LLaVA-OneVision: Easy Visual Task Transfer (arXiv 2024)
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (arXiv 2024)
• Building and better understanding vision-language models: insights and future directions (arXiv 2024)
• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (arXiv 2024)
• General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model (arXiv 2024)
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (ACL2025)
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution (arXiv 2024)
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models (arXiv 2024)
• Emu3: Next-Token Prediction is All You Need (arXiv 2024)
• Moshi: a speech-text foundation model for real-time dialogue (arXiv 2024)
• Pixtral 12B (arXiv 2024)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (arXiv 2024)
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (arXiv 2024)
• PaliGemma 2: A Family of Versatile VLMs for Transfer (arXiv 2024)
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (arXiv 2024)
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (arXiv 2025)
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling (arXiv 2025)
• Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction (arXiv 2025)
• Qwen2.5-VL Technical Report (arXiv 2025)
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (arXiv 2025)
• Visual-RFT: Visual Reinforcement Fine-Tuning (arXiv 2025)
• Qwen2.5-Omni Technical Report (arXiv 2025)
• Kimi-VL Technical Report (arXiv 2025)
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (arXiv 2025)
• Kimi-Audio Technical Report (arXiv 2025)
• Seed1.5-VL Technical Report (arXiv 2025)
• Aya Vision: A Multilingual Multimodal Foundation Model and Benchmark (arXiv 2025)
• Emerging Properties in Unified Multimodal Pretraining (arXiv 2025)
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (arXiv 2025)
• Voxtral (arXiv 2025)
• TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models (arXiv 2025)
• Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch (arXiv 2025)
• STEP3-VL-10B Technical Report (arXiv 2026)
• Innovator-VL: A Multimodal Large Language Model for Scientific Discovery (arXiv 2026)
• DeepSeek-OCR 2: Visual Causal Flow (arXiv 2026)
• Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders (arXiv 2026)
• InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing (arXiv 2026)
• Qianfan-OCR: A Unified End-to-End Model for Document Intelligence (arXiv 2026)
• Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale (arXiv 2026)
• LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model (arXiv 2026)
• GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents (arXiv 2026)
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction (arXiv 2026)
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture (arXiv 2026)
• Kwai Keye-VL-2.0 Technical Report (arXiv 2026)
• MonkeyOCRv2: A Visual-Text Foundation Model for Document AI (arXiv 2026)
• VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding (arXiv 2026)
• Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning (arXiv 2026)

Model compression

• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Patient Knowledge Distillation for BERT Model Compression (EMNLP2019)
• Small and Practical BERT Models for Sequence Labeling (EMNLP2019)
• Pruning a BERT-based Question Answering Model
• TinyBERT: Distilling BERT for Natural Language Understanding [github]
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (NeurIPS2019 WS) [github]
• Knowledge Distillation from Internal Representations (AAAI2020)
• PoWER-BERT: Accelerating BERT inference for Classification Tasks
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Extreme Language Model Compression with Optimal Subwords and Shared Projections
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• MobileBERT: Task-Agnostic Compression of BERT by Progressive Knowledge Transfer
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Q8BERT: Quantized 8Bit BERT (NeurIPS2019 WS)

LLM

• Attention Is All You Need
• Improving Language Understanding by Generative Pre-Training
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Language Models are Unsupervised Multitask Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Scaling Laws for Neural Language Models
• Language models are few-shot learners
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Evaluating Large Language Models Trained on Code
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• WebGPT: Browser-assisted question-answering with human feedback
• Improving language models by retrieving from trillions of tokens
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LaMDA: Language Models for Dialog Applications
• Solving Quantitative Reasoning Problems with Language Models
• Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Training language models to follow instructions with human feedback
• PaLM: Scaling Language Modeling with Pathways
• An empirical analysis of compute-optimal large language model training
• OPT: Open Pre-trained Transformer Language Models
• Unifying Language Learning Paradigms
• Emergent Abilities of Large Language Models
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Language Models are General-Purpose Interfaces
• Improving alignment of dialogue agents via targeted human judgements
• Scaling Instruction-Finetuned Language Models
• GLM-130B: An Open Bilingual Pre-trained Model
• Holistic Evaluation of Language Models
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• Galactica: A Large Language Model for Science
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• LLaMA: Open and Efficient Foundation Language Models
• Language Is Not All You Need: Aligning Perception with Language Models
• PaLM-E: An Embodied Multimodal Language Model
• GPT-4 Technical Report
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
• PaLM 2 Technical Report
• RWKV: Reinventing RNNs for the Transformer Era
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• TinyLlama: An Open-Source Small Language Model
• Scaling Instruction-Finetuned Language Models [github] (ACL2023)
• SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling (NAACL2024)
• Mixtral of Experts (arXiv 2024)
• OLMo: Accelerating the Science of Language Models (ACL2024)
• Yi: Open Foundation Models by 01.AI (arXiv 2024)
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (arXiv 2024)
• Gemma: Open Models Based on Gemini Research and Technology (arXiv 2024)
• InternLM2 Technical Report (arXiv 2024)
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv 2024)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv 2024)
• Nemotron-4 340B Technical Report (arXiv 2024)
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (arXiv 2024)
• Qwen2 Technical Report (arXiv 2024)
• Falcon2-11B Technical Report (arXiv 2024)
• Apple Intelligence Foundation Language Models (arXiv 2024)
• The Llama 3 Herd of Models (arXiv 2024)
• Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2024)
• OLMoE: Open Mixture-of-Experts Language Models (arXiv 2024)
• GPT-4o System Card (arXiv 2024)
• Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent (arXiv 2024)
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv 2024)
• Phi-4 Technical Report (arXiv 2024)
• Qwen2.5 Technical Report (arXiv 2024)
• OpenAI o1 System Card (arXiv 2024)
• DeepSeek-V3 Technical Report (arXiv 2024)
• 2 OLMo 2 Furious (COLM2025)
• MiniMax-01: Scaling Foundation Models with Lightning Attention (arXiv 2025)
• Kimi k1.5: Scaling Reinforcement Learning with LLMs (arXiv 2025)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [github] (Nature2025)
• Qwen2.5-1M Technical Report (arXiv 2025)
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model (arXiv 2025)
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (arXiv 2025)
• Gemma 3 Technical Report (arXiv 2025)
• Command A: An Enterprise-Ready Large Language Model (arXiv 2025)
• Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models (arXiv 2025)
• Phi-4-reasoning Technical Report (arXiv 2025)
• Llama-Nemotron: Efficient Reasoning Models (arXiv 2025)
• Qwen3 Technical Report (arXiv 2025)
• Magistral (arXiv 2025)
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (arXiv 2025)
• Apple Intelligence Foundation Language Models: Tech Report 2025 (arXiv 2025)
• Kimi K2: Open Agentic Intelligence (arXiv 2025)
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (arXiv 2025)
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv 2025)
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning (arXiv 2025)
• NVIDIA Nemotron 3: Efficient and Open Intelligence (arXiv 2025)
• Recursive Language Models (arXiv 2025)
• Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models (arXiv 2025)
• Ministral 3 (arXiv 2026)
• LongCat-Flash-Thinking-2601 Technical Report (arXiv 2026)
• Kimi K2.5: Visual Agentic Intelligence (arXiv 2026)
• ERNIE 5.0 Technical Report (arXiv 2026)
• Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters (arXiv 2026)
• GLM-5: from Vibe Coding to Agentic Engineering (arXiv 2026)
• Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale (arXiv 2026)
• VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models (arXiv 2026)
• Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent (arXiv 2026)
• Gemma 4 Technical Report (arXiv 2026)
• Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning (arXiv 2026)
• Kimi K3: Open Frontier Intelligence (arXiv 2026)

Reasoning and Chain of Thought

• The Impact of Reasoning Step Length on Large Language Models (ACL2024 Findings)
• Self-Rewarding Language Models (ICML2024)
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models [github] (arXiv 2024)
• Chain-of-Thought Reasoning Without Prompting (NeurIPS2024)
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (arXiv 2024)
• AlphaMath Almost Zero: Process Supervision without Process (NeurIPS2024)
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS2024)
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision (arXiv 2024)
• Recursive Introspection: Teaching Language Model Agents How to Self-Improve (arXiv 2024)
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (arXiv 2024)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (arXiv 2024)
• Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR2025)
• Training Language Models to Self-Correct via Reinforcement Learning (arXiv 2024)
• Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions (arXiv 2024)
• Free Process Rewards without Process Labels (arXiv 2024)
• Training Large Language Models to Reason in a Continuous Latent Space [github] (COLM2025)
• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (arXiv 2025)
• Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models (arXiv 2025)
• s1: Simple test-time scaling [github] (EMNLP2025)
• Process Reinforcement through Implicit Rewards [github] (arXiv 2025)
• Demystifying Long Chain-of-Thought Reasoning in LLMs (arXiv 2025)
• LIMO: Less is More for Reasoning (COLM2025)
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling (arXiv 2025)
• LIMR: Less is More for RL Scaling (arXiv 2025)
• Chain of Draft: Thinking Faster by Writing Less (arXiv 2025)
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning (COLM2025)
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2025)
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (TMLR)
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (COLM2025)
• Understanding R1-Zero-Like Training: A Critical Perspective (arXiv 2025)
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well? (arXiv 2025)
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (arXiv 2025)
• Sleep-time Compute: Beyond Inference Scaling at Test-time (arXiv 2025)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS2025)
• TTRL: Test-Time Reinforcement Learning (arXiv 2025)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (arXiv 2025)
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data (arXiv 2025)
• Reasoning Models Don't Always Say What They Think (arXiv 2025)
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models (NeurIPS2025)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (arXiv 2025)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (arXiv 2025)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS2025)
• Reinforcement Pre-Training (arXiv 2025)
• Self-Adapting Language Models (arXiv 2025)
• Spurious Rewards: Rethinking Training Signals in RLVR (arXiv 2025)
• Inverse Scaling in Test-Time Compute (TMLR)
• DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (arXiv 2025)
• Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning (arXiv 2025)
• Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving (arXiv 2025)
• Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models (arXiv 2025)
• Universal Reasoning Model (arXiv 2025)
• When Reasoning Meets Its Laws (arXiv 2025)
• PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (arXiv 2026)
• The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning (arXiv 2026)
• Agentic Reasoning for Large Language Models (arXiv 2026)
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (arXiv 2026)
• Reinforcement Learning via Self-Distillation (arXiv 2026)
• Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning (arXiv 2026)
• No Global Plan in Chain-of-Thought: Uncover the Latent Planning Horizon of LLMs (arXiv 2026)
• Does Your Reasoning Model Implicitly Know When to Stop Thinking? (arXiv 2026)
• Experiential Reinforcement Learning (arXiv 2026)
• How Far Can Unsupervised RLVR Scale LLM Training? (arXiv 2026)
• Efficient Reasoning with Balanced Thinking (arXiv 2026)
• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (arXiv 2026)
• Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability (arXiv 2026)
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe (arXiv 2026)
• Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling (arXiv 2026)
• Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs (arXiv 2026)
• Weak-to-Strong Generalization via Direct On-Policy Distillation (arXiv 2026)
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement (arXiv 2026)
• On-Policy Self-Distillation without Any Supervision (arXiv 2026)
• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL (arXiv 2026)

Alignment and RLHF

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (arXiv 2022)
• Constitutional AI: Harmlessness from AI Feedback (arXiv 2022)
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears (NeurIPS2023)
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (TMLR)
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS2023)
• Secrets of RLHF in Large Language Models Part I: PPO (arXiv 2023)
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML2024)
• Statistical Rejection Sampling Improves Preference Optimization (ICLR2024)
• Reward Model Ensembles Help Mitigate Overoptimization (ICLR2024)
• A General Theoretical Paradigm to Understand Learning from Human Preferences (arXiv 2023)
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (arXiv 2023)
• Zephyr: Direct Distillation of LM Alignment (arXiv 2023)
• Controlled Decoding from Language Models (ICML2024)
• Nash Learning from Human Feedback (arXiv 2023)
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (arXiv 2023)
• A Survey of Reinforcement Learning from Human Feedback (TMLR)
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (ICML2024)
• West-of-N: Synthetic Preferences for Self-Improving Reward Models (arXiv 2024)
• WARM: On the Benefits of Weight Averaged Reward Models (arXiv 2024)
• KTO: Model Alignment as Prospect Theoretic Optimization (ICML2024)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS2024)
• A Roadmap to Pluralistic Alignment (ICML2024)
• ORPO: Monolithic Preference Optimization without Reference Model (EMNLP2024)
• RewardBench: Evaluating Reward Models for Language Modeling (NAACL2025 Findings)
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML2024)
• Token-level Direct Preference Optimization (arXiv 2024)
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (arXiv 2024)
• Iterative Reasoning Preference Optimization (arXiv 2024)
• SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS2024)
• BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (arXiv 2024)
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep (arXiv 2024)
• HybridFlow: A Flexible and Efficient RLHF Framework (arXiv 2024)
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (arXiv 2024)
• Reinforcement Learning Enhanced LLMs: A Survey (arXiv 2024)
• Alignment faking in large language models (arXiv 2024)
• Deliberative Alignment: Reasoning Enables Safer Language Models (arXiv 2024)
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs (ICML2025)
• Auditing language models for hidden objectives (arXiv 2025)
• Inference-Time Scaling for Generalist Reward Modeling (arXiv 2025)
• RewardBench 2: Advancing Reward Model Evaluation (ICLR2026)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices (arXiv 2025)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (arXiv 2026)
• Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR (arXiv 2026)
• F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare (arXiv 2026)
• VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training (arXiv 2026)
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation (arXiv 2026)
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization (arXiv 2026)
• Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (arXiv 2026)
• DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards (arXiv 2026)
• The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning (arXiv 2026)
• Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (arXiv 2026)

Retrieval Augmented Generation

• Towards General Text Embeddings with Multi-stage Contrastive Learning (arXiv 2023)
• Ragas: Automated Evaluation of Retrieval Augmented Generation (EACL2024)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR2024)
• Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents (arXiv 2023)
• Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models (EMNLP2024)
• RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! (arXiv 2023)
• Dense X Retrieval: What Retrieval Granularity Should We Use? (EMNLP2024)
• Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv 2023)
• Improving Text Embeddings with Large Language Models (ACL2024)
• RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture (arXiv 2024)
• Corrective Retrieval Augmented Generation (arXiv 2024)
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (ICLR2024)
• Nomic Embed: Training a Reproducible Long Context Text Embedder (TMLR)
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (ACL2024 Findings)
• Multilingual E5 Text Embeddings: A Technical Report (arXiv 2024)
• Repetition Improves Language Model Embeddings (arXiv 2024)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (NAACL2024)
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv 2024)
• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (NeurIPS2024)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR2025)
• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs (arXiv 2024)
• Searching for Best Practices in Retrieval-Augmented Generation (EMNLP2024)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (arXiv 2024)
• Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (EMNLP2024)
• LightRAG: Simple and Fast Retrieval-Augmented Generation [github] (EMNLP2025 Findings)
• KBLaM: Knowledge Base augmented Language Model (arXiv 2024)
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (arXiv 2024)
• Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks (WWW2025)
• Retrieval-Augmented Generation with Graphs (GraphRAG) (arXiv 2025)
• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (arXiv 2025)
• MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR2025)
• Rank1: Test-Time Compute for Reranking in Information Retrieval (COLM2025)
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning (arXiv 2025)
• Gemini Embedding: Generalizable Embeddings from Gemini (arXiv 2025)
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (arXiv 2025)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS2025)
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability (NeurIPS2025)
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (arXiv 2025)
• How Far Are We from Genuinely Useful Deep Research Agents? (arXiv 2025)
• Deep Research: A Systematic Survey (arXiv 2025)
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (arXiv 2025)
• Step-DeepResearch Technical Report (arXiv 2025)
• Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking (arXiv 2026)
• DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation (arXiv 2026)
• WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora (ACL2026 Findings)
• OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data (arXiv 2026)
• OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis (arXiv 2026)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (arXiv 2026)
• GrepSeek: Training Search Agents for Direct Corpus Interaction (arXiv 2026)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories (arXiv 2026)
• SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research (arXiv 2026)
• RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM (arXiv 2026)
• A New Role for Relevance: Guiding Corpus Interaction in Agentic Search (arXiv 2026)

LLM Agents and Tool Use

• ReAct: Synergizing Reasoning and Acting in Language Models (ICLR2023)
• Toolformer: Language Models Can Teach Themselves to Use Tools (NeurIPS2023)
• Reflexion: Language Agents with Verbal Reinforcement Learning (NeurIPS2023)
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face (NeurIPS2023)
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (NeurIPS2023)
• Generative Agents: Interactive Simulacra of Human Behavior (UIST2023)
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (EMNLP2023)
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency (arXiv 2023)
• Gorilla: Large Language Model Connected with Massive APIs (NeurIPS2024)
• Voyager: An Open-Ended Embodied Agent with Large Language Models (arXiv 2023)
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (EMNLP2024)
• Mind2Web: Towards a Generalist Agent for the Web (NeurIPS2023)
• WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR2024)
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR2024)
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (ICLR2024)
• AgentBench: Evaluating LLMs as Agents (ICLR2024)
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (COLM2024)
• A Survey on Large Language Model based Autonomous Agents (arXiv 2023)
• Cognitive Architectures for Language Agents (TMLR)
• The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv 2023)
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (ICLR2024)
• Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML2024)
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (ICLR2024)
• MemGPT: Towards LLMs as Operating Systems (arXiv 2023)
• GAIA: a benchmark for General AI Assistants (arXiv 2023)
• CogAgent: A Visual Language Model for GUI Agents (CVPR2024)
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (ACL2024)
• Executable Code Actions Elicit Better LLM Agents (ICML2024)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS2024)
• A Survey on the Memory Mechanism of Large Language Model based Agents (arXiv 2024)
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (arXiv 2024)
• Tool Learning with Large Language Models: A Survey (arXiv 2024)
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2024)
• Agentless: Demystifying LLM-based Software Engineering Agents (arXiv 2024)
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents (ICLR2025)
• Agent-as-a-Judge: Evaluate Agents with Agents (arXiv 2024)
• Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks (arXiv 2024)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction (ICML2025)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents (arXiv 2025)
• Why Do Multi-Agent LLM Systems Fail? (arXiv 2025)
• Survey on Evaluation of LLM-based Agents (ACL2026 Findings)
• Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions (arXiv 2025)
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems (arXiv 2025)
• DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments (EMNLP2025)
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (arXiv 2025)
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (arXiv 2025)
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (InformationFusion2025)
• WebDancer: Towards Autonomous Information Seeking Agency (arXiv 2025)
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (arXiv 2025)
• Establishing Best Practices for Building Rigorous Agentic Benchmarks (arXiv 2025)
• Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction (arXiv 2025)
• Memory in the Age of AI Agents (arXiv 2025)
• Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills (arXiv 2025)
• Reinforcement Learning for Self-Improving Agent with Skill Library (arXiv 2025)
• Toward Efficient Agents: Memory, Tool learning, and Planning (arXiv 2026)
• EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience (arXiv 2026)
• Computer Environments Elicit General Agentic Intelligence in LLMs (arXiv 2026)
• MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (arXiv 2026)
• AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (arXiv 2026)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning (arXiv 2026)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv 2026)
• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents (arXiv 2026)
• Memento-Skills: Let Agents Design Agents (arXiv 2026)
• From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents (arXiv 2026)
• Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (arXiv 2026)
• ClawBench: Can AI Agents Complete Everyday Online Tasks? (arXiv 2026)
• Recursive Multi-Agent Systems (arXiv 2026)
• Agents' Last Exam (arXiv 2026)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents (arXiv 2026)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces (arXiv 2026)
• PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems (arXiv 2026)
• Are We Ready For An Agent-Native Memory System? (arXiv 2026)
• Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable (arXiv 2026)
• Metis: Memory Foundation Model (arXiv 2026)
• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents (arXiv 2026)
• Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design (arXiv 2026)
• Demystifying Agent Skills: Why They Work-Until They Don't (arXiv 2026)

Long Context and Efficient Architectures

• Hyena Hierarchy: Towards Larger Convolutional Language Models (arXiv 2023)
• MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (arXiv 2023)
• Extending Context Window of Large Language Models via Positional Interpolation (arXiv 2023)
• Lost in the Middle: How Language Models Use Long Contexts (TACL)
• Retentive Network: A Successor to Transformer for Large Language Models (arXiv 2023)
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (ACL2024)
• YaRN: Efficient Context Window Extension of Large Language Models (arXiv 2023)
• Ring Attention with Blockwise Transformers for Near-Infinite Context (arXiv 2023)
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv 2023)
• Gated Linear Attention Transformers with Hardware-Efficient Training (arXiv 2023)
• LongAlign: A Recipe for Long Context Alignment of Large Language Models (EMNLP2024 Findings)
• World Model on Million-Length Video And Language With Blockwise RingAttention (arXiv 2024)
• ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (arXiv 2024)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (arXiv 2024)
• Simple linear attention language models balance the recall-throughput tradeoff (arXiv 2024)
• Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (arXiv 2024)
• Jamba: A Hybrid Transformer-Mamba Language Model (arXiv 2024)
• Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence (arXiv 2024)
• RULER: What's the Real Context Size of Your Long-Context Language Models? (COLM2024)
• Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (arXiv 2024)
• xLSTM: Extended Long Short-Term Memory (arXiv 2024)
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML2024)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (arXiv 2024)
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR2025)
• An Empirical Study of Mamba-based Language Models (arXiv 2024)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS2024)
• How to Train Long-Context Language Models (Effectively) (ACL2025)
• Differential Transformer (ICLR2025)
• Hymba: A Hybrid-head Architecture for Small Language Models (arXiv 2024)
• Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR2025)
• Memory Layers at Scale (arXiv 2024)
• Byte Latent Transformer: Patches Scale Better Than Tokens (ACL2025)
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (ACL2025)
• Titans: Learning to Memorize at Test Time (arXiv 2025)
• NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML2025)
• Large Language Diffusion Models (arXiv 2025)
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (ACL2025)
• MoBA: Mixture of Block Attention for Long-Context LLMs (arXiv 2025)
• Forgetting Transformer: Softmax Attention with a Forget Gate (arXiv 2025)
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (arXiv 2025)
• RWKV-7 "Goose" with Expressive Dynamic State Evolution (arXiv 2025)
• A Comprehensive Survey on Long Context Language Modeling (arXiv 2025)
• Multi-Token Attention (arXiv 2025)
• Log-Linear Attention (arXiv 2025)
• Mercury: Ultra-Fast Language Models Based on Diffusion (arXiv 2025)
• QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management (arXiv 2025)
• mHC: Manifold-Constrained Hyper-Connections (arXiv 2025)
• Fast-weight Product Key Memory (arXiv 2026)
• The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models (arXiv 2026)
• Scaling Embeddings Outperforms Scaling Experts in Language Models (arXiv 2026)
• FASA: Frequency-aware Sparse Attention (arXiv 2026)
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing (arXiv 2026)
• LLaDA2.1: Speeding Up Text Diffusion via Token Editing (arXiv 2026)
• SLA2: Sparse-Linear Attention with Learnable Routing and QAT (arXiv 2026)
• SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning (arXiv 2026)
• dLLM: Simple Diffusion Language Modeling (arXiv 2026)
• Attention Residuals (arXiv 2026)
• Mixture-of-Depths Attention (arXiv 2026)
• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens (arXiv 2026)
• HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention (arXiv 2026)
• Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation (arXiv 2026)
• δ-mem: Efficient Online Memory for Large Language Models (arXiv 2026)
• GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment (arXiv 2026)
• HRM-Text: Efficient Pretraining Beyond Scaling (arXiv 2026)
• MiniMax Sparse Attention (arXiv 2026)
• Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling (arXiv 2026)
• LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget (arXiv 2026)
• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling (arXiv 2026)

Efficient Training Inference and Serving

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [github] (ICLR2023)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML2023)
• Fast Inference from Transformers via Speculative Decoding (ICML2023)
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot [github] (ICML2023)
• LLM-Pruner: On the Structural Pruning of Large Language Models [github] (NeurIPS2023)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys2024)
• MiniLLM: On-Policy Distillation of Large Language Models (ICLR2024)
• A Simple and Effective Pruning Approach for Large Language Models [github] (ICLR2024)
• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (NeurIPS2023)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (arXiv 2023)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP2023)
• Efficient Streaming Language Models with Attention Sinks [github] (ICLR2024)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [github] (ICLR2024)
• BitNet: Scaling 1-bit Transformers for Large Language Models (arXiv 2023)
• FP8-LM: Training FP8 Large Language Models (arXiv 2023)
• SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS2024)
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems (arXiv 2023)
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (ACL2024 Findings)
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI2024)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads [github] (ICML2024)
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (ICML2024)
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding [github] (ICML2024)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [github] (ICML2024)
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks [github] (ICML2024)
• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models (ICLR2025)
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (arXiv 2024)
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2024)
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (arXiv 2024)
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (arXiv 2024)
• A Survey on Efficient Inference for Large Language Models (arXiv 2024)
• SnapKV: LLM Knows What You are Looking for Before Generation (NeurIPS2024)
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (ACL2024)
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving (arXiv 2024)
• SpinQuant: LLM quantization with learned rotations (ICLR2025)
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (arXiv 2024)
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML2024)
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (EMNLP2024)
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv 2024)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS2024)
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (arXiv 2024)
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (arXiv 2024)
• Star Attention: Efficient LLM Inference over Long Sequences (ICML2025)
• Optimizing Large Language Model Training Using FP4 Quantization (ICML2025)
• Muon is Scalable for LLM Training (arXiv 2025)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (arXiv 2025)
• BitNet b1.58 2B4T Technical Report (arXiv 2025)
• Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures (ISCA2025)
• Fast and Accurate Causal Parallel Decoding using Jacobi Forcing (arXiv 2025)
• Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation (arXiv 2026)
• DFlash: Block Diffusion for Flash Speculative Decoding (arXiv 2026)
• DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference (arXiv 2026)
• IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (arXiv 2026)
• TAPS: Task Aware Proposal Distributions for Speculative Sampling (arXiv 2026)
• MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU (arXiv 2026)
• MinT: Managed Infrastructure for Training and Serving Millions of LLMs (arXiv 2026)
• Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding (arXiv 2026)
• LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers (arXiv 2026)

Pretraining Data Scaling Laws and Model Merging

• Scaling Data-Constrained Language Models (NeurIPS2023)
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (NeurIPS2023)
• TIES-Merging: Resolving Interference When Merging Models (NeurIPS2023)
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning (arXiv 2023)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (ICML2024)
• Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (ICML2024)
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (ACL2024)
• Datasets for Large Language Models: A Comprehensive Survey (arXiv 2024)
• Simple and Scalable Strategies to Continually Pre-train Large Language Models (arXiv 2024)
• Evolutionary Optimization of Model Merging Recipes (NatMachIntell2025)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR2025)
• Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws (arXiv 2024)
• DataComp-LM: In search of the next generation of training sets for language models (arXiv 2024)
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (arXiv 2024)
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities (arXiv 2024)
• Scaling Laws for Precision (arXiv 2024)
• Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset (ACL2025)
• Distillation Scaling Laws (ICML2025)
• Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data (arXiv 2025)
• Parallel Scaling Law for Language Models (arXiv 2025)
• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (arXiv 2025)
• DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI (arXiv 2025)
• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs (arXiv 2026)
• OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration (arXiv 2026)
• Less is Enough: Synthesizing Diverse Data in Feature Space of LLMs (arXiv 2026)
• On Data Engineering for Scaling LLM Terminal Capabilities (arXiv 2026)
• DataFlex: A Unified Data-Centric Dynamic Training Framework for Large Language Models (arXiv 2026)
• Adam's Law: Textual Frequency Law on Large Language Models (arXiv 2026)
• Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora (arXiv 2026)
• DataPrep-Bench: Benchmarking LLMs as Training Data Preparators (arXiv 2026)

Prompting and In Context Learning

• A Survey on In-context Learning (EMNLP2024)
• Batch Prompting: Efficient Inference with Large Language Model APIs (EMNLP2023)
• Larger language models do in-context learning differently (arXiv 2023)
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search (EMNLP2023)
• Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (EMNLP2023)
• Efficient Guided Generation for Large Language Models (arXiv 2023)
• Large Language Models as Optimizers (ICLR2024)
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (ICLR2024)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR2024)
• Function Vectors in Large Language Models (ICLR2024)
• In-Context Learning Creates Task Vectors (EMNLP2023 Findings)
• Universal Self-Consistency for Large Language Model Generation (arXiv 2023)
• PromptBench: A Unified Library for Evaluation of Large Language Models (JMLR)
• Many-Shot In-Context Learning (NeurIPS2024)
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (arXiv 2024)
• TextGrad: Automatic "Differentiation" via Text (arXiv 2024)
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (EMNLP2024)
• Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (arXiv 2024)
• XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models (MLSys2025)
• A Survey of Context Engineering for Large Language Models (arXiv 2025)
• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (ICLR2026)

Code LLM

• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (EMNLP2023)
• Teaching Large Language Models to Self-Debug (ICLR2024)
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (NeurIPS2023)
• StarCoder: may the source be with you! (arXiv 2023)
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR2024)
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct (arXiv 2023)
• Code Llama: Open Foundation Models for Code (arXiv 2023)
• CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS2023)
• Magicoder: Empowering Code Generation with OSS-Instruct (ICML2024)
• Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models (arXiv 2023)
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution (ICML2024)
• Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering (arXiv 2024)
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence (arXiv 2024)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback (arXiv 2024)
• OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (ACL2024 Findings)
• StarCoder 2 and The Stack v2: The Next Generation (arXiv 2024)
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (arXiv 2024)
• Granite Code Models: A Family of Open Foundation Models for Code Intelligence (arXiv 2024)
• DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data (arXiv 2024)
• CodeGemma: Open Code Models Based on Gemma (arXiv 2024)
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (arXiv 2024)
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR2025)
• Lean-STaR: Learning to Interleave Thinking and Proving (arXiv 2024)
• DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search (arXiv 2024)
• Qwen2.5-Coder Technical Report (arXiv 2024)
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (arXiv 2024)
• OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (ACL2025)
• CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (arXiv 2025)
• Competitive Programming with Large Reasoning Models (arXiv 2025)
• SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (arXiv 2025)
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS2025)
• OpenCodeReasoning: Advancing Data Distillation for Competitive Coding (COLM2025)
• Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning (arXiv 2025)
• SWE-smith: Scaling Data for Software Engineering Agents (arXiv 2025)
• DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition (arXiv 2025)
• Seed-Coder: Let the Code Model Curate Data for Itself (arXiv 2025)
• LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (arXiv 2025)
• From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence (arXiv 2025)
• X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests (arXiv 2026)
• Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey (arXiv 2026)
• SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents (arXiv 2026)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (arXiv 2026)
• SWE-Universe: Scale Real-World Verifiable Environments to Millions (arXiv 2026)
• Qwen3-Coder-Next Technical Report (arXiv 2026)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? (arXiv 2026)
• InCoder-32B: Code Foundation Model for Industrial Scenarios (arXiv 2026)
• LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning (arXiv 2026)
• Embarrassingly Simple Self-Distillation Improves Code Generation (arXiv 2026)
• GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning (arXiv 2026)
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories (arXiv 2026)
• MaxProof: Scaling Mathematical Proof with Generative-Verifier RL and Population-Level Test-Time Scaling (arXiv 2026)
• FastContext: Training Efficient Repository Explorer for Coding Agents (arXiv 2026)
• The Verification Horizon: No Silver Bullet for Coding Agent Rewards (arXiv 2026)
• Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models (arXiv 2026)
• SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring (arXiv 2026)
• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? (arXiv 2026)

Evaluation and Benchmark

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS2023)
• A Survey on Evaluation of Large Language Models (arXiv 2023)
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR2024)
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark (COLM2024)
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (ACL2024)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML2024)
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (COLM2024)
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (NeurIPS2024)
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (arXiv 2024)
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR2025)
• Measuring short-form factuality in large language models (arXiv 2024)
• FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (arXiv 2024)
• Humanity's Last Exam (arXiv 2025)
• SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (arXiv 2025)
• BIG-Bench Extra Hard (ACL2025)
• Measuring AI Ability to Complete Long Software Tasks (NeurIPS2025)
• The Leaderboard Illusion (arXiv 2025)
• HealthBench: Evaluating Large Language Models Towards Improved Human Health (arXiv 2025)
• ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems (arXiv 2025)
• DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (arXiv 2025)
• OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions (arXiv 2026)
• GENIUS: Generative Fluid Intelligence Evaluation Suite (arXiv 2026)
• RubricBench: Aligning Model-Generated Rubrics with Human Standards (arXiv 2026)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models (arXiv 2026)
• MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models (arXiv 2026)
• MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations (arXiv 2026)
• ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv 2026)

Safety Hallucination and Interpretability

• A Watermark for Large Language Models (ICML2023)
• DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (ICML2023)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2023)
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (EMNLP2023)
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (NeurIPS2023)
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (EMNLP2023)
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (EMNLP2023)
• Jailbroken: How Does LLM Safety Training Fail? (NeurIPS2023)
• Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv 2023)
• Studying Large Language Model Generalization with Influence Functions (arXiv 2023)
• SafetyBench: Evaluating the Safety of Large Language Models (ACL2024)
• Language Models Represent Space and Time (ICLR2024)
• Jailbreaking Black Box Large Language Models in Twenty Queries (arXiv 2023)
• Towards Understanding Sycophancy in Language Models (ICLR2024)
• Knowledge Editing for Large Language Models: A Survey (arXiv 2023)
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (TOIS)
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (NeurIPS2024)
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv 2024)
• Security and Privacy Challenges of Large Language Models: A Survey (arXiv 2024)
• Long-form factuality in large language models (NeurIPS2024)
• The Geometry of Categorical and Hierarchical Concepts in Large Language Models (ICLR2025)
• Scaling and evaluating sparse autoencoders (arXiv 2024)
• Refusal in Language Models Is Mediated by a Single Direction (arXiv 2024)
• Transcoders Find Interpretable LLM Feature Circuits (NeurIPS2024)
• Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (EMNLP2024)
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (arXiv 2024)
• Open Problems in Mechanistic Interpretability (arXiv 2025)
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (arXiv 2025)
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (arXiv 2025)
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (arXiv 2025)
• Defeating Prompt Injections by Design (arXiv 2025)
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (arXiv 2025)
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data (arXiv 2025)
• Persona Vectors: Monitoring and Controlling Character Traits in Language Models (arXiv 2025)
• Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits (arXiv 2025)
• Probabilistic Guarantees for Reducing Contextual Hallucinations in LLMs (arXiv 2026)
• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security (arXiv 2026)
• Spider-Sense: Intrinsic Risk Sensing for Efficient Agent Defense with Hierarchical Adaptive Screening (arXiv 2026)
• Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (arXiv 2026)
• Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding (arXiv 2026)
• Alignment Makes Language Models Normative, Not Descriptive (arXiv 2026)
• Emergent Social Intelligence Risks in Generative Multi-Agent Systems (arXiv 2026)
• Agentic Abstention: Do Agents Know When to Stop Instead of Act? (arXiv 2026)
• AISPA: User-Centric System Prompt Auditing for Large Language Model Applications (arXiv 2026)
• OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution (arXiv 2026)
• The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads (arXiv 2026)
• Stealing Reasoning Traces from Proprietary LLM APIs (arXiv 2026)

Misc

• jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models [github]
• Cloze-driven Pretraining of Self-attention Networks
• Learning and Evaluating General Linguistic Intelligence
• To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (ACL2019 WS)
• Learning to Speak and Act in a Fantasy Text Adventure Game (EMNLP2019)
• Conditional BERT Contextual Augmentation
• Data Augmentation using Pre-trained Transformer Models
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (ICLR2020)
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models (ICLR2020)
• A Mutual Information Maximization Perspective of Language Representation Learning (ICLR2020)
• Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment (AAAI2020)
• Thieves on Sesame Street! Model Extraction of BERT-based APIs (ICLR2020)
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Extending Machine Language Models toward Human-Level Language Understanding
• Glyce: Glyph-vectors for Chinese Character Representations
• Back to the Future -- Sequential Alignment of Text Representations
• Improving Cuneiform Language Identification with BERT (NAACL2019 WS)
• BERT has a Moral Compass: Improvements of ethical and moral values of machines
• SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction (ACM-BCB2019)
• On the comparability of Pre-trained Language Models
• Transformers: State-of-the-art Natural Language Processing
• Jukebox: A Generative Model for Music
• WT5?! Training Text-to-Text Models to Explain their Predictions
• TAPAS: Weakly Supervised Table Parsing via Pre-training [github]
• TABERT: Pretraining for Joint Understanding of Textual and Tabular Data

Author

ChangWookJun / @changwookjun (changwookjun@gmail.com)