chanmuzi/Papers

Paper list and short/long summaries I've read for my research or interests

23

148 commits

updated May 29, 2024

See the code

README

LLM (Large Language Model)

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04OpenELM: An Efficient Language Model Family with Open-source Training and Inference Frameworklayer-wise scalingAppleMakrdown
2024.04Rho-1: Not All Tokens Are What You NeedRoh-1, Selective Language ModelingTsinghua University, MicrosoftMakrdown
2024.03Jamba: A Hybrid Transformer-Mamba Language ModelJamaba, Transformer, Mamba, MoEAI21labsMakrdown
2024.02Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language ModelsGriffin, HawkGoogle DeepMindBlog
2024.02OLMo: Accelerating the Science of Language ModelsOpen Source, OLMoAI2Makrdown
2024.01Orion-14B: Open-source Multilingual Large Language ModelsOpen Source, MLLMOrionStar Inc.Markdown
2023.05PaLM2 Technical ReportPaLM2GoogleBlog
2023.04Scaling Transformer to 1M tokens and beyond with RMTAttention, RMTNeural Networks and Deep Learning Lab, MIPT, Dolgoprudny, RussiaBlog
2023.03GPT-4 Technical ReportGPT-4OpenAIBlog
2019.05BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingBERTGoogle AI LanguageBlogNAACL-HLT 2019

Reasoning

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04Self-Explore to Avoid the Pit: Improving the Reasoning Capabilities of Language Models with Fine-grained RewardsSelf-Explore, Granular Learning SignalsKAISTMakrdown
2024.02Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language ModelsBoosting of Thoughts (BoT)University of TorontoMarkdownICLR 2024
2024.02Chain-of-Thought Reasoning Without PromptingCoT-decodingGoogle DeepMindBlog
2024.02Self-Discover: Large Language Models Self-Compose Reasoning StructuresDecompose, Reasoning StructureUniversity of Southern CaliforniaBlog
2024.02A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning ChainsBenchmark, REVEALGoogleBlog
2024.01LLMs cannot find reasoning errors, but can correct them!CoTGoogle ResearchMarkdown
2023.10Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex ReasoningPrompt DecompositionIIT Delhi, IndiaMarkdownEMNLP 2023
2023.05Learning to Reason and Memorize with Self-NotesSelf-NotesMeta AI, NVIDIABlogNeurIPS 2023 Poster
2023.04Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory SystemSelf-Controlled Memory (SCM)ByteDance AI LabBlog
2023.04Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelsleast-to-most promptingGoogle ResearchBlogICLR 2023
2023.04PINTO: Faithful Language Reasoning Using Prompt-Generated RationalesRationaleUniversity of Southern CaliforniaBlogICLR 2023
2023.04Answering Questions by Meta-Reasoning over Multiple Chains of ThoughtCoT, Meta-ReasoningAI2BlogEMNLP 2023
2023.01Specializing Smaller Language Models towards Multi-Step ReasoningKnowledge Distillation, ReasoningAI2Makrdown

RAG (Retrieval Augmented Generation)


Training

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04The Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsInstruction HierarchyOpenAIMarkdown
2024.01Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text GenerationRELC, RL, PPOMcGill, Toronto, Mila, Google ResearchBlogACL 2024 under review
2024.01Can AI Assistants Know What They Don't Know?HallucinationFudan UniversityBlog
2024.01Tuning Language Models by ProxyProxy-tuningAllen Institue of AIBlog
2024.01Self-Play Fine-Tuning Converts Weak Language Models to Strong Language ModelsSelf-Play...Markdown
2024.01Knowledge Fusion of Large Language ModelsKnowledge FusionTencent AI LabBlog
2023.12Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisionsuperhuman modelOpenAIBlog
2023.12Select, Prompt, Filter: Distilling Large Language Models for Summarizing ConversationsKnowledge DistillationZoom Video CommunicationsMarkdownEMNLP 2023
2023.12Beyond Human Data: Scaling Self-Training for Problem-Solving with Language ModelsSelf-Training, ReSTGoogle DeepMindMarkdown
2023.10SELF: Self-Evolution with Language FeedbackSELFThe University of Hong KongBlog
2023.03oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimesRoBERTa, Knowledge DistillationDepartment of Computer Science, the University of Illinois Urbana-ChampaignBlog

Prompting

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.03Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational SystemsDynamic ContextsUniversity of Illinois, ChicagoMakrdownWWW`2024 workshop PromptEng
2024.02Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded KnowledgeHSP, Summarization (not mentioned though)National University of SingaporeMarkdown
2024.02A Human-Inspired Reading Agent with Gist Memory of Very Long ContextsGist Memory, ReadAgentGoogle DeepMindMarkdown
2024.01Meta-Prompting: Enhancing Language Models with Task-Agnostic ScaffoldingMeta-PromptingStanford University, OpenAIMarkdown

PEFT

DateTitle(arxiv)KeywordAffiliationNoteConference
2024.03PERL: Parameter Efficient Reinforcement Learning from Human FeedbackPERL, LoRA, Reward ModelGoogle ResearchMarkdown
2024.02ResLoRA: Identity Residual Mapping in Low-Rank AdaptionResLoRA, MergeMicrosoftBlog
2024.02LoRA+: Efficient Low Rank Adaptation of Large ModelsLoRA+UC BerkeleyMarkdown
2023.04LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelLLaMA, Visual InstructionShanghai Artificial Intelligence LaboratoryBlog

Compression

DateTitle(arxiv)KeywordAffiliationNoteConference
2024.01SliceGPT: Compress Large Language Models by Deleting Rows and ColumnsModel CompressionMicrosoft ResearchMarkdown
2023.04Learning to Compress Prompts with Gist TokensGist Token, CompressionStanford UniversityBlogNeurIPS 2023
2022.05Matryoshka Representation LearningMatryoshka, AdaptiveGoogle ResearchMarkdownNeurIPS 2022

Tool, Agent

DateTitle(arxiv)KeywordAffiliationNoteConference
2023.04Chameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsTools, ChameleonUniversity of California, Los Angeles, Microsoft ResearchBlogNeurIPS 2023

Evaluation / Benchmark / Dataset

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.05Observational Scaling Laws and the Predictability of Language Model PerformanceObservational Scaling LawStanford, TorontoMakrdown
2024.03RewardBench: Evaluating Reward Models for Language ModelingReward Model, DPOAI2Markdown
2024.02Learning to Generate Instruction Tuning Datasets for Zero-Shot Task AdaptationBonito, Instruction Tuning DatasetBrown UniversityMarkdownACL 2024 submission
2024.02LLM-based NLG Evaluation: Current Status and ChallengesNLG, EvaluationPeking UniversityMarkdown
2024.02Can Large Language Models Understand Context?Benchmark, In-Context LearningAppleMarkdownFindings of EACL 2024
2024.01Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining ResearchDolma, Dataset, PretrainingAI2Markdown
2023.05Do Large Language Models Know What They Don't Know?Self-Knowledge, SelfAware (Dataset)School of Computer Science, Fudan UniversityMarkdown
2023.04Are Emergent Abilities of Large Language Models a Mirage?Emergent AbilityStanford UniversityBlogNeurIPS 2023 oral
2023.04Evaluating Verifiability in Generative Search EnginesGenerative Search EngineStanford UniversityBlogFindings of EMNLP 2023
2016.10SQuAD: 100,000+ Questions for Machine Comprehension of TextSQUAD, QA, BenchmarkStanford UniversityBlog

Others

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.02The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits1-bit, BitNet b1.58Microsoft ResearchBlog
2024.02LongRoPE: Extending LLM Context Window Beyond 2 Million TokensLongRoPE, 2048k tokensMicrosoft ResearchMakrdown
2023.04ResiDual: Transformer with Dual Residual ConnectionsLayer NormalizationMicrosoft ResearchBlogICLR 2024 rejected

Multimodal

DateTitle (arxiv)KeywordAffiliationNoteConference
2023.03Visual Instruction TuningLLaVA, Visual Instruction TuningMicrosoft ResearchBlogNeurIPS 2023 oral
2021.08BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from DocumentsBROS, AMLM, TMLMNAVER CLOVAMarkdownAAAI 2022

Contributors

chanmuzi

148 commits

chanmuzi/Papers

Paper list and short/long summaries I've read for my research or interests

23

148 commits

updated May 29, 2024

See the code

README

LLM (Large Language Model)

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04OpenELM: An Efficient Language Model Family with Open-source Training and Inference Frameworklayer-wise scalingAppleMakrdown
2024.04Rho-1: Not All Tokens Are What You NeedRoh-1, Selective Language ModelingTsinghua University, MicrosoftMakrdown
2024.03Jamba: A Hybrid Transformer-Mamba Language ModelJamaba, Transformer, Mamba, MoEAI21labsMakrdown
2024.02Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language ModelsGriffin, HawkGoogle DeepMindBlog
2024.02OLMo: Accelerating the Science of Language ModelsOpen Source, OLMoAI2Makrdown
2024.01Orion-14B: Open-source Multilingual Large Language ModelsOpen Source, MLLMOrionStar Inc.Markdown
2023.05PaLM2 Technical ReportPaLM2GoogleBlog
2023.04Scaling Transformer to 1M tokens and beyond with RMTAttention, RMTNeural Networks and Deep Learning Lab, MIPT, Dolgoprudny, RussiaBlog
2023.03GPT-4 Technical ReportGPT-4OpenAIBlog
2019.05BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingBERTGoogle AI LanguageBlogNAACL-HLT 2019

Reasoning

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04Self-Explore to Avoid the Pit: Improving the Reasoning Capabilities of Language Models with Fine-grained RewardsSelf-Explore, Granular Learning SignalsKAISTMakrdown
2024.02Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language ModelsBoosting of Thoughts (BoT)University of TorontoMarkdownICLR 2024
2024.02Chain-of-Thought Reasoning Without PromptingCoT-decodingGoogle DeepMindBlog
2024.02Self-Discover: Large Language Models Self-Compose Reasoning StructuresDecompose, Reasoning StructureUniversity of Southern CaliforniaBlog
2024.02A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning ChainsBenchmark, REVEALGoogleBlog
2024.01LLMs cannot find reasoning errors, but can correct them!CoTGoogle ResearchMarkdown
2023.10Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex ReasoningPrompt DecompositionIIT Delhi, IndiaMarkdownEMNLP 2023
2023.05Learning to Reason and Memorize with Self-NotesSelf-NotesMeta AI, NVIDIABlogNeurIPS 2023 Poster
2023.04Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory SystemSelf-Controlled Memory (SCM)ByteDance AI LabBlog
2023.04Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelsleast-to-most promptingGoogle ResearchBlogICLR 2023
2023.04PINTO: Faithful Language Reasoning Using Prompt-Generated RationalesRationaleUniversity of Southern CaliforniaBlogICLR 2023
2023.04Answering Questions by Meta-Reasoning over Multiple Chains of ThoughtCoT, Meta-ReasoningAI2BlogEMNLP 2023
2023.01Specializing Smaller Language Models towards Multi-Step ReasoningKnowledge Distillation, ReasoningAI2Makrdown

RAG (Retrieval Augmented Generation)


Training

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.04The Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsInstruction HierarchyOpenAIMarkdown
2024.01Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text GenerationRELC, RL, PPOMcGill, Toronto, Mila, Google ResearchBlogACL 2024 under review
2024.01Can AI Assistants Know What They Don't Know?HallucinationFudan UniversityBlog
2024.01Tuning Language Models by ProxyProxy-tuningAllen Institue of AIBlog
2024.01Self-Play Fine-Tuning Converts Weak Language Models to Strong Language ModelsSelf-Play...Markdown
2024.01Knowledge Fusion of Large Language ModelsKnowledge FusionTencent AI LabBlog
2023.12Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisionsuperhuman modelOpenAIBlog
2023.12Select, Prompt, Filter: Distilling Large Language Models for Summarizing ConversationsKnowledge DistillationZoom Video CommunicationsMarkdownEMNLP 2023
2023.12Beyond Human Data: Scaling Self-Training for Problem-Solving with Language ModelsSelf-Training, ReSTGoogle DeepMindMarkdown
2023.10SELF: Self-Evolution with Language FeedbackSELFThe University of Hong KongBlog
2023.03oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimesRoBERTa, Knowledge DistillationDepartment of Computer Science, the University of Illinois Urbana-ChampaignBlog

Prompting

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.03Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational SystemsDynamic ContextsUniversity of Illinois, ChicagoMakrdownWWW`2024 workshop PromptEng
2024.02Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded KnowledgeHSP, Summarization (not mentioned though)National University of SingaporeMarkdown
2024.02A Human-Inspired Reading Agent with Gist Memory of Very Long ContextsGist Memory, ReadAgentGoogle DeepMindMarkdown
2024.01Meta-Prompting: Enhancing Language Models with Task-Agnostic ScaffoldingMeta-PromptingStanford University, OpenAIMarkdown

PEFT

DateTitle(arxiv)KeywordAffiliationNoteConference
2024.03PERL: Parameter Efficient Reinforcement Learning from Human FeedbackPERL, LoRA, Reward ModelGoogle ResearchMarkdown
2024.02ResLoRA: Identity Residual Mapping in Low-Rank AdaptionResLoRA, MergeMicrosoftBlog
2024.02LoRA+: Efficient Low Rank Adaptation of Large ModelsLoRA+UC BerkeleyMarkdown
2023.04LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelLLaMA, Visual InstructionShanghai Artificial Intelligence LaboratoryBlog

Compression

DateTitle(arxiv)KeywordAffiliationNoteConference
2024.01SliceGPT: Compress Large Language Models by Deleting Rows and ColumnsModel CompressionMicrosoft ResearchMarkdown
2023.04Learning to Compress Prompts with Gist TokensGist Token, CompressionStanford UniversityBlogNeurIPS 2023
2022.05Matryoshka Representation LearningMatryoshka, AdaptiveGoogle ResearchMarkdownNeurIPS 2022

Tool, Agent

DateTitle(arxiv)KeywordAffiliationNoteConference
2023.04Chameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsTools, ChameleonUniversity of California, Los Angeles, Microsoft ResearchBlogNeurIPS 2023

Evaluation / Benchmark / Dataset

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.05Observational Scaling Laws and the Predictability of Language Model PerformanceObservational Scaling LawStanford, TorontoMakrdown
2024.03RewardBench: Evaluating Reward Models for Language ModelingReward Model, DPOAI2Markdown
2024.02Learning to Generate Instruction Tuning Datasets for Zero-Shot Task AdaptationBonito, Instruction Tuning DatasetBrown UniversityMarkdownACL 2024 submission
2024.02LLM-based NLG Evaluation: Current Status and ChallengesNLG, EvaluationPeking UniversityMarkdown
2024.02Can Large Language Models Understand Context?Benchmark, In-Context LearningAppleMarkdownFindings of EACL 2024
2024.01Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining ResearchDolma, Dataset, PretrainingAI2Markdown
2023.05Do Large Language Models Know What They Don't Know?Self-Knowledge, SelfAware (Dataset)School of Computer Science, Fudan UniversityMarkdown
2023.04Are Emergent Abilities of Large Language Models a Mirage?Emergent AbilityStanford UniversityBlogNeurIPS 2023 oral
2023.04Evaluating Verifiability in Generative Search EnginesGenerative Search EngineStanford UniversityBlogFindings of EMNLP 2023
2016.10SQuAD: 100,000+ Questions for Machine Comprehension of TextSQUAD, QA, BenchmarkStanford UniversityBlog

Others

DateTitle (arxiv)KeywordAffiliationNoteConference
2024.02The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits1-bit, BitNet b1.58Microsoft ResearchBlog
2024.02LongRoPE: Extending LLM Context Window Beyond 2 Million TokensLongRoPE, 2048k tokensMicrosoft ResearchMakrdown
2023.04ResiDual: Transformer with Dual Residual ConnectionsLayer NormalizationMicrosoft ResearchBlogICLR 2024 rejected

Multimodal

DateTitle (arxiv)KeywordAffiliationNoteConference
2023.03Visual Instruction TuningLLaVA, Visual Instruction TuningMicrosoft ResearchBlogNeurIPS 2023 oral
2021.08BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from DocumentsBROS, AMLM, TMLMNAVER CLOVAMarkdownAAAI 2022

Contributors

chanmuzi

148 commits