Shadowgraph
TrendingAbout
Sign in

safr-ai-lab/survey-llm

A survey of privacy problems in Large Language Models (LLMs). Contains summary of the corresponding paper along with relevant code

Python

69

18 commits

updated May 30, 2024

See the code

README

Privacy Issues in Large Language Models

This repository is a collection of links to papers and code repositories relevant in implementing LLMs with reduced privacy risks. These correspond to papers discussed in our survey available at: https://arxiv.org/abs/2312.06717

This repository will be periodically updated with relevant papers scraped from Arxiv. The survey paper itself will be updated on a slightly less frequent basis. Papers that have been added to this repository but not the paper will be marked with asterisks.

If you have a paper relevant to LLM privacy, please nominate them for inclusion

Repo last updated 5/30/2024

Paper last updated 5/30/2024

Table of Contents

Citation

@misc{neel2023privacy,
      title={Privacy Issues in Large Language Models: A Survey}, 
      author={Seth Neel and Peter Chang},
      year={2023},
      eprint={2312.06717},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Memorization

drawing

Image from Carlini 2020

Paper TitleYearAuthorCode
Emergent and Predictable Memorization in Large Language Models2023Biderman et al.[Code]
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks2019Carlini et al.
Quantifying Memorization Across Neural Language Models2023Carlini et al.
Do Localization Methods Actually Localize Memorized Data in LLMs?2023Chang et al.[Code]
Does Learning Require Memorization? A Short Tale about a Long Tail2020Feldman et al.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy2023Ippolito et al.
Measuring Forgetting of Memorized Training Examples2023Jagielski et al.
Deduplicating Training Data Mitigates Privacy Risks in Language Models2022Kandpal et al.
How BPE Affects Memorization in Transformers2021Kharitonov et al.
Deduplicating Training Data Makes Language Models Better2022Lee et al.[Code]
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN2021McCoy et al.[Code]
Training Production Language Models without Memorizing User Data2020Ramaswamy et al.
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks2022Raunak et al.
Understanding Unintended Memorization in Federated Learning2020Thakkar et al.
Investigating the Impact of Pre-trained Word Embeddings on Memorization in Neural Networks2020Thomas et al.
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models2022Tirumala et al.
Counterfactual Memorization in Neural Language Models2021Zhang et al.
Provably Confidential Language Modelling2022Zhao et al.[Code]
Quantifying and Analyzing Entity-level Memorization in Large Language Models2023Zhou et al.

Privacy Attacks

image info

Image from Tindall

Paper TitleYearAuthorCode
Detecting Pretraining Data from Large Language Models2023Shi et al.
TMI! Finetuned Models Leak Private Information from their Pretraining Data2023Abascal et al.
Extracting Training Data from Large Language Models2020Carlini et al.[Code]
Membership Inference Attacks From First Principles2022Carlini et al.[Code]
Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration2023Fu et al.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?2020Hisamoto et al.[Code]
Membership Inference Attacks on Machine Learning: A Survey2021Hu et al.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?2021Lehman et al.[Code]
MoPe: Model Perturbation-based Privacy Attacks on Language Models2023Li et al.
When Machine Learning Meets Privacy: A Survey and Outlook2021Liu et al.
Data Portraits: Recording Foundation Model Training Data2023Marone et al.[Code]
Membership Inference Attacks against Language Models via Neighbourhood Comparison2023Mattern et al.
Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models2023Meeus et al.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks2022Mireshghallah et al.Contact fmireshg@eng.ucsd.edu
Scalable Extraction of Training Data from Production Language Models2023Nasr et al.
Detecting Pretraining Data from Large Language Models2023Shi et al.[Code]
Membership Inference Attacks against Machine Learning Models2017Shokri et al.
Information Leakage in Embedding Models2020Song and Raghunathan
Auditing Data Provenance in Text-Generation Models2019Song and Shmatikov
Beyond Memorization: Violating Privacy Via Inference with Large Language Models2023Staab et al.[Code]
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting2018Yeom et al.
Bag of Tricks for Training Data Extraction from Language Models2023Yu et al.[Code]
Analyzing Information Leakage of Updates to Natural Language Models2020Zanella-Béguelin et al.
Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation2023Zhang et al.[Code]

Furthermore, see [Google Training Data Extraction Challenge]

Private LLMs

drawing

Image from Google AI Blog

Paper TitleYearAuthorCode
Deep Learning with Differential Privacy2016Abadi et al.
Large-Scale Differentially Private BERT2021Anil et al.
Sanitizing Sentence Embeddings (and Labels) for Local Differential Privacy2023Du et al.[Code]
DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass2023Du et al.[Code]
An Efficient DP-SGD Mechanism for Large Scale NLP Models2022Dupuy et al.
The Algorithmic Foundations of Differential Privacy2006Dwork and Roth
Submix: Practical Private Prediction for Large-Scale Language Models2022Ginart et al.
Federated Learning for Mobile Keyboard Prediction2019Hard et al.
Learning and Evaluating a Differentially Private Pre-trained Language Model2021Hoory et al.
Knowledge Sanitization of Large Language Models2023Ishibashi and Shimodaira
Differentially Private Language Models Benefit from Public Pre-training2020Kerrigan et al.[Code]
Large Language Models Can Be Strong Differentially Private Learners2022Li et al.
Differentially Private Decoding in Large Language Models2022Majmudar et al.
Communication-Efficient Learning of Deep Networks from Decentralized Data2016McMahan et al.
Learning Differentially Private Recurrent Language Models2018McMahan et al.
Selective Differential Privacy for Language Modeling2022Shi et al.[Code]
Training Production Language Models without Memorizing User Data2020Ramaswamy et al.
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation2023Tang et al.[Code]
Understanding Unintended Memorization in Federated Learning2020Thakkar et al.
Differentially Private Fine-tuning of Language Models2022Yu et al.[Code]
Provably Confidential Language Modelling2022Zhao et al.[Code]
***Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory2023Mireshghallah et al.[Code]
***Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models2023Zhou et al.

Unlearning

drawing

Image from Felps 2020

Paper TitleYearAuthorCode
Machine Unlearning2020Bourtoule et al.[Code]
Unlearn What You Want to Forget: Efficient Unlearning for LLMs2023Chen et al.
Who's Harry Potter? Approximate Unlearning in LLMs2023Eldan et al.
Amnesiac Machine Learning2020Graves et al.
Adaptive Machine Unlearning2021Gupta et al.[Code]
Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy2024Hayes et al.
Preserving Privacy Through DeMemorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models2023Kassem et al.
Privacy Adhering Machine Un-learning in NLP2022Kumar et al.
Knowledge Unlearning for Mitigating Privacy Risks in Language Models2022Jang et al.[Code]
Unlearnable Algorithms for In-context Learning2024Muresanu et al.
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning2020Neel et al.
A Survey of Machine Unlearning2022Nguyen et al.
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks2023Patil et al.[Code]
In-Context Unlearning: Language Models as Few Shot Unlearners2023Pawelczyk et al.
Large Language Model Unlearning2023Yao et al.[Code]

Copyright

image info

Custom Image

Paper TitleYearAuthorCode
Can Copyright be Reduced to Privacy?2023Elkin-Koren et al.
DeepCreativity: Measuring Creativity with Deep Learning Techniques2022Franceschelli et al.
Foundation Models and Fair Use2023Henderson et al.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy2023Ippolito et al.
Copyright Violations and Large Language Models2023Karamolegkou et al.[Code]
Formalizing Human Ingenuity: A Quantitative Framework for Copyright Law's Substantial Similarity2022Scheffler et al.
On Provable Copyright Protection for Generative Models2023Vyas et al.

Additional Related Surveys

Paper TitleYearAuthorCode
Membership Inference Attacks on Machine Learning: A Survey2022Hu et al.
When Machine Learning Meets Privacy: A Survey and Outlook2021Liu et al.
Rethinking Machine Unlearning for Large Language Models2024Liu et al.
A Survey of Machine Unlearning2022Nguyen et al.
***A Survey of Large Language Models2023Zhao et al.

Contact Info

Repository maintained by Peter Chang (pchang@hbs.edu)

Contributors

pwchang

17 commits

sethneel

1 commits

Similar repos

michele17284/Awesome-Privacy-Preserving-LLMs

Collection of all the papers talking about/relevant to the topic of privacy-preserving LLMs

41

chrisliu298/awesome-llm-unlearning

A resource repository for machine unlearning in large language models

626

ydyjya/Awesome-LLM-Safety

A curated list of safety-related papers, articles, and resources focused on Large Language Models…

1,914

MinghuiChen43/awesome-trustworthy-deep-learning

A curated list of trustworthy deep learning papers. Continually updating...

392

chawins/llm-sp

Papers and resources related to the security and privacy of LLMs 🤖

583

byerose/Awesome-Foundation-Model-Security

A curated list of trustworthy Generative AI papers. Daily updating...

77

NY1024/Awesome-Trustworthy-GenAI

8

Zhou-Zi7/Awesome-AI-Security-BIG4

This Github repository summarizes a list of research papers on AI security from the four top…

186

Languages

Python

100.0%

safr-ai-lab/survey-llm

A survey of privacy problems in Large Language Models (LLMs). Contains summary of the corresponding paper along with relevant code

Python

69

18 commits

updated May 30, 2024

See the code

README

Privacy Issues in Large Language Models

This repository is a collection of links to papers and code repositories relevant in implementing LLMs with reduced privacy risks. These correspond to papers discussed in our survey available at: https://arxiv.org/abs/2312.06717

This repository will be periodically updated with relevant papers scraped from Arxiv. The survey paper itself will be updated on a slightly less frequent basis. Papers that have been added to this repository but not the paper will be marked with asterisks.

If you have a paper relevant to LLM privacy, please nominate them for inclusion

Repo last updated 5/30/2024

Paper last updated 5/30/2024

Table of Contents

Citation

@misc{neel2023privacy,
      title={Privacy Issues in Large Language Models: A Survey}, 
      author={Seth Neel and Peter Chang},
      year={2023},
      eprint={2312.06717},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Memorization

drawing

Image from Carlini 2020

Paper TitleYearAuthorCode
Emergent and Predictable Memorization in Large Language Models2023Biderman et al.[Code]
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks2019Carlini et al.
Quantifying Memorization Across Neural Language Models2023Carlini et al.
Do Localization Methods Actually Localize Memorized Data in LLMs?2023Chang et al.[Code]
Does Learning Require Memorization? A Short Tale about a Long Tail2020Feldman et al.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy2023Ippolito et al.
Measuring Forgetting of Memorized Training Examples2023Jagielski et al.
Deduplicating Training Data Mitigates Privacy Risks in Language Models2022Kandpal et al.
How BPE Affects Memorization in Transformers2021Kharitonov et al.
Deduplicating Training Data Makes Language Models Better2022Lee et al.[Code]
How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN2021McCoy et al.[Code]
Training Production Language Models without Memorizing User Data2020Ramaswamy et al.
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks2022Raunak et al.
Understanding Unintended Memorization in Federated Learning2020Thakkar et al.
Investigating the Impact of Pre-trained Word Embeddings on Memorization in Neural Networks2020Thomas et al.
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models2022Tirumala et al.
Counterfactual Memorization in Neural Language Models2021Zhang et al.
Provably Confidential Language Modelling2022Zhao et al.[Code]
Quantifying and Analyzing Entity-level Memorization in Large Language Models2023Zhou et al.

Privacy Attacks

image info

Image from Tindall

Paper TitleYearAuthorCode
Detecting Pretraining Data from Large Language Models2023Shi et al.
TMI! Finetuned Models Leak Private Information from their Pretraining Data2023Abascal et al.
Extracting Training Data from Large Language Models2020Carlini et al.[Code]
Membership Inference Attacks From First Principles2022Carlini et al.[Code]
Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration2023Fu et al.
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?2020Hisamoto et al.[Code]
Membership Inference Attacks on Machine Learning: A Survey2021Hu et al.
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?2021Lehman et al.[Code]
MoPe: Model Perturbation-based Privacy Attacks on Language Models2023Li et al.
When Machine Learning Meets Privacy: A Survey and Outlook2021Liu et al.
Data Portraits: Recording Foundation Model Training Data2023Marone et al.[Code]
Membership Inference Attacks against Language Models via Neighbourhood Comparison2023Mattern et al.
Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models2023Meeus et al.
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks2022Mireshghallah et al.Contact fmireshg@eng.ucsd.edu
Scalable Extraction of Training Data from Production Language Models2023Nasr et al.
Detecting Pretraining Data from Large Language Models2023Shi et al.[Code]
Membership Inference Attacks against Machine Learning Models2017Shokri et al.
Information Leakage in Embedding Models2020Song and Raghunathan
Auditing Data Provenance in Text-Generation Models2019Song and Shmatikov
Beyond Memorization: Violating Privacy Via Inference with Large Language Models2023Staab et al.[Code]
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting2018Yeom et al.
Bag of Tricks for Training Data Extraction from Language Models2023Yu et al.[Code]
Analyzing Information Leakage of Updates to Natural Language Models2020Zanella-Béguelin et al.
Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation2023Zhang et al.[Code]

Furthermore, see [Google Training Data Extraction Challenge]

Private LLMs

drawing

Image from Google AI Blog

Paper TitleYearAuthorCode
Deep Learning with Differential Privacy2016Abadi et al.
Large-Scale Differentially Private BERT2021Anil et al.
Sanitizing Sentence Embeddings (and Labels) for Local Differential Privacy2023Du et al.[Code]
DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass2023Du et al.[Code]
An Efficient DP-SGD Mechanism for Large Scale NLP Models2022Dupuy et al.
The Algorithmic Foundations of Differential Privacy2006Dwork and Roth
Submix: Practical Private Prediction for Large-Scale Language Models2022Ginart et al.
Federated Learning for Mobile Keyboard Prediction2019Hard et al.
Learning and Evaluating a Differentially Private Pre-trained Language Model2021Hoory et al.
Knowledge Sanitization of Large Language Models2023Ishibashi and Shimodaira
Differentially Private Language Models Benefit from Public Pre-training2020Kerrigan et al.[Code]
Large Language Models Can Be Strong Differentially Private Learners2022Li et al.
Differentially Private Decoding in Large Language Models2022Majmudar et al.
Communication-Efficient Learning of Deep Networks from Decentralized Data2016McMahan et al.
Learning Differentially Private Recurrent Language Models2018McMahan et al.
Selective Differential Privacy for Language Modeling2022Shi et al.[Code]
Training Production Language Models without Memorizing User Data2020Ramaswamy et al.
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation2023Tang et al.[Code]
Understanding Unintended Memorization in Federated Learning2020Thakkar et al.
Differentially Private Fine-tuning of Language Models2022Yu et al.[Code]
Provably Confidential Language Modelling2022Zhao et al.[Code]
***Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory2023Mireshghallah et al.[Code]
***Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models2023Zhou et al.

Unlearning

drawing

Image from Felps 2020

Paper TitleYearAuthorCode
Machine Unlearning2020Bourtoule et al.[Code]
Unlearn What You Want to Forget: Efficient Unlearning for LLMs2023Chen et al.
Who's Harry Potter? Approximate Unlearning in LLMs2023Eldan et al.
Amnesiac Machine Learning2020Graves et al.
Adaptive Machine Unlearning2021Gupta et al.[Code]
Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy2024Hayes et al.
Preserving Privacy Through DeMemorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models2023Kassem et al.
Privacy Adhering Machine Un-learning in NLP2022Kumar et al.
Knowledge Unlearning for Mitigating Privacy Risks in Language Models2022Jang et al.[Code]
Unlearnable Algorithms for In-context Learning2024Muresanu et al.
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning2020Neel et al.
A Survey of Machine Unlearning2022Nguyen et al.
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks2023Patil et al.[Code]
In-Context Unlearning: Language Models as Few Shot Unlearners2023Pawelczyk et al.
Large Language Model Unlearning2023Yao et al.[Code]

Copyright

image info

Custom Image

Paper TitleYearAuthorCode
Can Copyright be Reduced to Privacy?2023Elkin-Koren et al.
DeepCreativity: Measuring Creativity with Deep Learning Techniques2022Franceschelli et al.
Foundation Models and Fair Use2023Henderson et al.
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy2023Ippolito et al.
Copyright Violations and Large Language Models2023Karamolegkou et al.[Code]
Formalizing Human Ingenuity: A Quantitative Framework for Copyright Law's Substantial Similarity2022Scheffler et al.
On Provable Copyright Protection for Generative Models2023Vyas et al.

Additional Related Surveys

Paper TitleYearAuthorCode
Membership Inference Attacks on Machine Learning: A Survey2022Hu et al.
When Machine Learning Meets Privacy: A Survey and Outlook2021Liu et al.
Rethinking Machine Unlearning for Large Language Models2024Liu et al.
A Survey of Machine Unlearning2022Nguyen et al.
***A Survey of Large Language Models2023Zhao et al.

Contact Info

Repository maintained by Peter Chang (pchang@hbs.edu)

Similar repos

michele17284/Awesome-Privacy-Preserving-LLMs

Collection of all the papers talking about/relevant to the topic of privacy-preserving LLMs

41

chrisliu298/awesome-llm-unlearning

A resource repository for machine unlearning in large language models

626

ydyjya/Awesome-LLM-Safety

A curated list of safety-related papers, articles, and resources focused on Large Language Models…

1,914

MinghuiChen43/awesome-trustworthy-deep-learning

A curated list of trustworthy deep learning papers. Continually updating...

392

chawins/llm-sp

Papers and resources related to the security and privacy of LLMs 🤖

583

byerose/Awesome-Foundation-Model-Security

A curated list of trustworthy Generative AI papers. Daily updating...

77

NY1024/Awesome-Trustworthy-GenAI

8

Zhou-Zi7/Awesome-AI-Security-BIG4

This Github repository summarizes a list of research papers on AI security from the four top…

186

Contributors

pwchang

17 commits

sethneel

1 commits

Languages

Python

100.0%