Knowledge-Distillation-Paper

This resposity maintains a collection of important papers on knowledge distillation.

• Knowledge-Distillation-Paper
  • Pioneering Papers
  • Survey Papers
  • Distillation Accelerates Diffusion Models
  • Feature Distillation
  • Online Knowledge Distillation
  • Multi-Teacher Knowledge Distillation
  • Data-Free Knowledge Distillation
  • Distillation for Segmentation
  • Useful Resources

Pioneering Papers

• Model Compression, KDD 2006

  • https://dl.acm.org/doi/abs/10.1145/1150402.1150464
  • Cristian Buciluǎ, Rich Caruana, Alexandru Niculescu-Mizil.

• Do Deep Nets Really Need to be Deep?, NeurIPS 2014

  • https://arxiv.org/abs/1312.6184
  • Lei Jimmy Ba, Rich Caruana.

• Distilling the Knowledge in a Neural Network, NeurIPS-workshop 2014

  • https://arxiv.org/abs/1503.02531
  • https://neurips.cc/virtual/2014/workshop/4294
  • Geoffrey Hinton, Oriol Vinyals, Jeff Dean.

Survey Papers

• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks, TPAMI 2022

  • https://arxiv.org/abs/2004.05937
  • Lin Wang, Kuk-Jin Yoon.

• Knowledge Distillation: A Survey, IJCV 2021

  • https://arxiv.org/abs/2006.05525
  • Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao.

• A Comprehensive Survey on Knowledge Distillation

  • https://arxiv.org/abs/2503.12067
  • Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, Shohreh Kasaei.

Distillation Meets Diffusion Models

Extremely Promising !!!!!

• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed [Tensorflow]

  • https://arxiv.org/abs/2101.02388
  • Eric Luhman, Troy Luhman

• Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022 [Tensorflow]

  • https://arxiv.org/abs/2202.00512
  • Tim Salimans, Jonathan Ho

• Accelerating Diffusion Sampling with Classifier-based Feature Distillation, ICME 2023 [PyTorch]

  • https://arxiv.org/abs/2211.12039
  • Wujie Sun, Defang Chen, Can Wang, Deshi Ye, Yan Feng, Chun Chen

• Fast Sampling of Diffusion Models via Operator Learning, ICML 2023 [PyTorch]

  • https://arxiv.org/abs/2211.13449
  • Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, Anima Anandkumar

• Consistency Models, ICML 2023 [PyTorch]

  • https://arxiv.org/abs/2303.01469
  • Yang Song, Prafulla Dhariwal, Mark Chen, Ilya Sutskever

• TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation [PyTorch]

  • https://arxiv.org/abs/2303.04248
  • David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, Eric Gu

• A Geometric Perspective on Diffusion Models [PyTorch]

  • https://arxiv.org/abs/2305.19947
  • Defang Chen, Zhenyu Zhou, Jian-Ping Mei, Chunhua Shen, Chun Chen, Can Wang

• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping, ICML 2024 [PyTorch]

  • https://arxiv.org/abs/2306.05544
  • Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, Josh Susskind

• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion, ICLR 2024 [PyTorch]

  • https://arxiv.org/abs/2310.02279
  • Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, Stefano Ermon

• One-step Diffusion with Distribution Matching Distillation, CVPR 2024 [PyTorch]

  • https://arxiv.org/abs/2311.18828
  • Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, Taesung Park

• Fast ODE-based Sampling for Diffusion Models in Around 5 Steps, CVPR 2024 [PyTorch]

  • https://arxiv.org/abs/2312.00094
  • Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen

• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation, ICML 2024 [PyTorch]

  • https://arxiv.org/abs/2404.04057
  • Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, Hai Huang

• On the Trajectory Regularity of ODE-based Diffusion Sampling, ICML 2024 [PyTorch]

  • https://arxiv.org/abs/2405.11326
  • Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, Siwei Lyu

• Improved Distribution Matching Distillation for Fast Image Synthesis, NeurIPS 2024 [PyTorch]

  • https://arxiv.org/abs/2405.14867
  • Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman

• Simple and Fast Distillation of Diffusion Models, NeurIPS 2024 [PyTorch]

  • https://arxiv.org/abs/2409.19681
  • Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, Siwei Lyu

• DICE: Distilling Classifier-Free Guidance into Text Embeddings

  • https://arxiv.org/abs/2502.03726
  • Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, Siwei Lyu

Feature Distillation

• FitNets: Hints for Thin Deep Nets, ICLR 2015 [Theano]

  • https://arxiv.org/abs/1412.6550
  • Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio.

• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, ICLR 2017 [PyTorch]

  • https://arxiv.org/abs/1612.03928
  • Sergey Zagoruyko, Nikos Komodakis.

• Learning Deep Representations with Probabilistic Knowledge Transfer, ECCV 2018 [Pytorch]

  • https://arxiv.org/abs/1803.10837
  • Nikolaos Passalis, Anastasios Tefas.

• Relational Knowledge Distillation, CVPR 2019 [Pytorch]

  • https://arxiv.org/abs/1904.05068
  • Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho.

• Variational Information Distillation for Knowledge Transfer, CVPR 2019

  • https://arxiv.org/abs/1904.05835
  • Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai.

• Similarity-Preserving Knowledge Distillation, CVPR 2019

  • https://arxiv.org/abs/1907.09682
  • Frederick Tung, Greg Mori.

• Contrastive Representation Distillation, ICLR 2020 [Pytorch]

  • https://arxiv.org/abs/1910.10699
  • Yonglong Tian, Dilip Krishnan, Phillip Isola.

• Heterogeneous Knowledge Distillation using Information Flow Modeling, CVPR 2020 [Pytorch]

  • https://arxiv.org/abs/2005.00727
  • Nikolaos Passalis, Maria Tzelepi, Anastasios Tefas.

• Cross-Layer Distillation with Semantic Calibration, AAAI 2021 [Pytorch][TKDE]

  • https://arxiv.org/abs/2012.03236
  • Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, Chun Chen.

• Distilling Knowledge via Knowledge Review, CVPR 2021 [Pytorch]

  • https://arxiv.org/abs/2104.09044
  • Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia.

• Distilling Holistic Knowledge with Graph Neural Networks, ICCV 2021 [Pytorch]

  • https://arxiv.org/abs/2108.05507
  • Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, Jiajun Bu.

• Decoupled Knowledge Distillation, CVPR 2022 [Pytorch]

  • https://arxiv.org/abs/2203.08679
  • Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang

• Knowledge Distillation with the Reused Teacher Classifier, CVPR 2022 [Pytorch]

  • https://arxiv.org/abs/2203.14001
  • Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, Chun Chen.

Online Knowledge Distillation

• Deep Mutual Learning, CVPR 2018 [TensorFlow]

  • https://arxiv.org/abs/1706.00384
  • Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu.

• Large scale distributed neural network training through online distillation, ICLR 2018

  • https://arxiv.org/abs/1804.03235
  • Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl and Geoffrey E. Hinton.

• Knowledge Distillation by On-the-Fly Native Ensemble, NeurIPS 2018 [PyTorch]

  • https://arxiv.org/abs/1806.04606
  • Xu Lan, Xiatian Zhu, Shaogang Gong.

• Online Knowledge Distillation with Diverse Peers, AAAI 2020 [Pytorch]

  • https://arxiv.org/abs/1912.00350
  • Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng and Chun Chen.

• Feature-map-level Online Adversarial Knowledge Distillation, ICML 2020

  • https://arxiv.org/abs/2002.01775
  • Inseop Chung, SeongUk Park, Jangho Kim, Nojun Kwak.

• Peer collaborative learning for online knowledge distillation, AAAI 2021

  • https://arxiv.org/abs/2006.04147
  • Guile Wu, Shaogang Gong.

Multi-Teacher Knowledge Distillation

• Distilling knowledge from ensembles of neural networks for speech recognition, INTERSPEECH 2016

  • https://www.isca-speech.org/archive_v0/Interspeech_2016/pdfs/1190.PDF
  • Austin Waters, Yevgen Chebotar.

• Efficient Knowledge Distillation from an Ensemble of Teachers, INTERSPEECH 2017

  • https://isca-speech.org/archive_v0/Interspeech_2017/pdfs/0614.PDF
  • Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, Bhuvana Ramabhadran.

• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space, NeurIPS 2020 [Pytorch]

  • https://proceedings.neurips.cc/paper/2020/hash/91c77393975889bd08f301c9e13a44b7-Abstract.html
  • Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, Changshui Zhang.

• Reinforced Multi-Teacher Selection for Knowledge Distillation, AAAI 2021

  • https://arxiv.org/abs/2012.06048
  • Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, Daxin Jiang

• Confidence-Aware Multi-Teacher Knowledge Distillation, ICASSP 2022 [Pytorch]

  • https://arxiv.org/abs/2201.00007
  • Hailin Zhang, Defang Chen, Can Wang.

• Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning, ICME 2023 [Pytorch]

  • https://arxiv.org/abs/2306.06634
  • Hailin Zhang, Defang Chen, Can Wang.

Data-Free Knowledge Distillation

• Data-Free Knowledge Distillation for Deep Neural Networks, NeurIPS-workshop 2017 [Tensorflow]

  • https://arxiv.org/abs/1710.07535
  • Raphael Gontijo Lopes, Stefano Fenu, Thad Starner

• DAFL: Data-Free Learning of Student Networks, ICCV 2019 [PyTorch]

  • https://arxiv.org/abs/1904.01186
  • Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian

• Zero-Shot Knowledge Distillation in Deep Networks, ICML 2019 [Tensorflow]

  • https://arxiv.org/abs/1905.08114
  • Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, Anirban Chakraborty

• Zero-shot Knowledge Transfer via Adversarial Belief Matching, NeurIPS 2019 [Pytorch]

  • https://arxiv.org/abs/1905.09768
  • Paul Micaelli, Amos Storkey

• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion, CVPR 2020 [Pytorch]

  • https://arxiv.org/abs/1912.08795
  • Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha, Jan Kautz

• The Knowledge Within: Methods for Data-Free Model Compression, CVPR 2020

  • https://arxiv.org/abs/1912.01274
  • Matan Haroush, Itay Hubara, Elad Hoffer, Daniel Soudry

• Contrastive Model Inversion for Data-Free Knowledge Distillation, IJCAI 2021 [Pytorch]

  • https://arxiv.org/abs/2105.08584
  • Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, Mingli Song

• Customizing Synthetic Data for Data-Free Student Learning, ICME 2023 [Pytorch]

  • https://arxiv.org/abs/2307.04542
  • Shiya Luo, Defang Chen, Can Wang

Distillation for Segmentation

• Structured Knowledge Distillation for Dense Prediction, CVPR 2019, TPAMI 2020 [Pytorch]

  • https://arxiv.org/abs/1903.04197
  • Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen.

• Channel-wise Knowledge Distillation for Dense Prediction, ICCV 2021 [Pytorch]

  • https://arxiv.org/abs/2011.13256
  • Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, Chunhua Shen.

• Cross-Image Relational Knowledge Distillation for Semantic Segmentation, CVPR 2022 [Pytorch]

  • https://arxiv.org/abs/2204.06986
  • Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, Qian Zhang.

• Holistic Weighted Distillation for Semantic Segmentation, ICME 2023 [Pytorch]

  • Wujie Sun, Defang Chen, Can Wang, Deshi Ye, Yan Feng, Chun Chen.

Useful Resources

• Acceptance rates of the main AI conferences [Link]
• AI conference deadlines [Link]
• CCF conference deadlines [Link]