D-X-Y/Awesome-AutoDL

Automated Deep Learning: Neural Architecture Search Is Not the End (a curated list of AutoDL resources and an in-depth analysis)

Python

2,344

195 commits

updated Sep 26, 2022

See the code

README

Awesome AutoDL

A curated list of automated deep learning related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search.

Please feel free to pull requests or open an issue to add papers.

Table of Contents

Awesome Blogs
Awesome AutoDL Libraies
Awesome Benchmarks
Deep Learning-based NAS and HPO
Awesome Surveys

Awesome Blogs

Awesome AutoDL Libraies

Awesome Benchmarks

Title	Venue	Code
NAS-Bench-101: Towards Reproducible Neural Architecture Search	ICML 2019	GitHub
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search	ICLR 2020	Github
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search	arXiv 2020	GitHub
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search	ICLR 2020	GitHub
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size	TPAMI 2021	GitHub
NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition	ICLR 2021	GitHub
HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark	ICLR 2021	GitHub
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing	arXiv 2020	GitHub
NAS-Bench-x11 and the Power of Learning Curves	NeurIPS 2021	GitHub

Deep Learning-based NAS and HPO

Type	G	RL	EA	PD	Other
Explanation	gradient-based	reinforcement learning	evolutionary algorithm	performance prediction	other types

2021 Venues

Title	Venue	Type	Code
CATE: Computation-aware Neural Architecture Encoding with Transformers	ICML	O	GitHub
Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator	CVPR	G	Github
Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition	ICCV	EA	Github
AutoFormer: Searching Transformers for Visual Recognition	ICCV	EA	GitHub
LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search	CVPR	EA	GitHub
One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking	CVPR	EA	GitHub
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators	ICLR	G	GitHub
Zero-Cost Proxies for Lightweight NAS	ICLR	O	GitHub
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective	ICLR	-	GitHub
DrNAS: Dirichlet Neural Architecture Search	ICLR	G	GitHub
Rethinking Architecture Selection in Differentiable NAS	ICLR	O	GitHub
Evolving Reinforcement Learning Algorithms	ICLR	EA	GitHub
AutoHAS: Differentiable Hyper-parameter and Architecture Search	ICLR-W	G	-
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function	CVPR	PD	github

2020 Venues

Title	Venue	Type	Code
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search	NeurIPS	-	GitHub
PyGlove: Symbolic Programming for Automated Machine Learning	NeurIPS	library	-
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search	NeurIPS	PD	GitHub
RandAugment: Practical Automated Data Augmentation with a Reduced Search Space	NeurIPS		GitHub
Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians	NeurIPS	G	GitHub
A Study on Encodings for Neural Architecture Search	NeurIPS		GitHub
AutoBSS: An Efficient Algorithm for Block Stacking Style Search	NeurIPS
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS	NeurIPS	G	GitHub
Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding	NeurIPS
Revisiting Parameter Sharing for Automatic Neural Channel Number Search	NeurIPS
Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search	NeurIPS	MCTS	GitHub
Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search	AAAI	MCTS	GitHub
Representation Sharing for Fast Object Detector Search and Beyond	ECCV	G	GitHub
Are Labels Necessary for Neural Architecture Search?	ECCV	G	-
Single Path One-Shot Neural Architecture Search with Uniform Sampling	ECCV	EA	-
Neural Predictor for Neural Architecture Search	ECCV	O	-
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models	ECCV	G	-
BATS: Binary ArchitecTure Search	ECCV	-	-
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification	ECCV	-	-
Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization	ECCV	-	-
Angle-based Search Space Shrinking for Neural Architecture Search	ECCV	-	-
Anti-Bandit Neural Architecture Search for Model Defense	ECCV	-	-
TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search	ECCV	G	GitHub
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search	ECCV	G	GitHub
Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search	ECCV	RL	-
DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search	ECCV	G	-
Optimizing Millions of Hyperparameters by Implicit Differentiation	AISTATS	G	-
Evolving Machine Learning Algorithms From Scratch	ICML	EA	-
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization	ICML	G	GitHub
NADS: Neural Architecture Distribution Search for Uncertainty Awareness	ICML	-	-
Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data	ICML	-	-
Neural Architecture Search in a Proxy Validation Loss Landscape	ICML	-	-
Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection	CVPR	-	GitHub
Designing Network Design Spaces	CVPR	-	GitHub
UNAS: Differentiable Architecture Search Meets Reinforcement Learning	CVPR	G/RL	GitHub
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation	CVPR	G	GitHub
A Semi-Supervised Assessor of Neural Architectures	CVPR	PD	-
Binarizing MobileNet via Evolution-based Searching	CVPR	EA	-
Rethinking Performance Estimation in Neural Architecture Search	CVPR	-	GitHub
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy	CVPR	G	GitHub
SGAS: Sequential Greedy Architecture Search	CVPR	G	Github
Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS	CVPR	RL	-
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions	CVPR	G	Github
AdversarialNAS: Adversarial Neural Architecture Search for GANs	CVPR	G	Github
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks	CVPR	G	Github
Block-wisely Supervised Neural Architecture Search with Knowledge Distillation	CVPR	G	Github
Overcoming Multi-Model Forgetting in One-Shot NAS with Diversity Maximization	CVPR	G	Github
Densely Connected Search Space for More Flexible Neural Architecture Search	CVPR	G	Github
EfficientDet: Scalable and Efficient Object Detection	CVPR	RL	-
NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search	ICLR	-	Github
Understanding Architectures Learnt by Cell-based Neural Architecture Search	ICLR	G	GitHub
Evaluating The Search Phase of Neural Architecture Search	ICLR	-
AtomNAS: Fine-Grained End-to-End Neural Architecture Search	ICLR		GitHub
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search	ICLR	-	GitHub
Once for All: Train One Network and Specialize it for Efficient Deployment	ICLR	G	GitHub
Efficient Transformer for Mobile Applications	ICLR	-	-
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search	ICLR	G	GitHub
Adversarial AutoAugment	ICLR	-	-
NAS evaluation is frustratingly hard	ICLR	-	GitHub
FasterSeg: Searching for Faster Real-time Semantic Segmentation	ICLR	G	GitHub
Computation Reallocation for Object Detection	ICLR	-	-
Towards Fast Adaptation of Neural Architectures with Meta Learning	ICLR	-	GitHub
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures	ICLR	EA	-
How to Own the NAS in Your Spare Time	ICLR	-	-
Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search	ICPR	G	github

2019 Venues

Title	Venue	Type	Code
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions	ICLR	-	-
DATA: Differentiable ArchiTecture Approximation	NeurIPS	-	-
Random Search and Reproducibility for Neural Architecture Search	UAI	G	GitHub
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition	EMNLP	G	-
Continual and Multi-Task Architecture Search	ACL	RL	-
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation	ICCV	G	GitHub
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	GitHub
Searching for MobileNetV3	ICCV	EA	-
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	GitHub
Fast and Practical Neural Architecture Search	ICCV
Teacher Guided Architecture Search	ICCV		-
AutoDispNet: Improving Disparity Estimation With AutoML	ICCV	G	-
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?	ICCV	EA	-
One-Shot Neural Architecture Search via Self-Evaluated Template Network	ICCV	G	Github
Evolving Space-Time Neural Architectures for Videos	ICCV	EA	GitHub
AutoGAN: Neural Architecture Search for Generative Adversarial Networks	ICCV	RL	github
Discovering Neural Wirings	NeurIPS	G	Github
Towards modular and programmable architecture search	NeurIPS	Other	Github
Network Pruning via Transformable Architecture Search	NeurIPS	G	Github
Deep Active Learning with a NeuralArchitecture Search	NeurIPS	-	-
DetNAS: Backbone Search for Object Detection	NeurIPS	EA	GitHub
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers	NeurIPS	-	-
Efficient Forward Architecture Search	NeurIPS	G	Github
Efficient Neural ArchitectureTransformation Search in Channel-Level for Object Detection	NeurIPS	G	-
XNAS: Neural Architecture Search with Expert Advice	NeurIPS	G	GitHub
DARTS: Differentiable Architecture Search	ICLR	G	github
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware	ICLR	RL/G	github
Graph HyperNetworks for Neural Architecture Search	ICLR	G	-
Learnable Embedding Space for Efficient Neural Architecture Compression	ICLR	Other	github
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution	ICLR	EA	-
SNAS: stochastic neural architecture search	ICLR	G	-
NetTailor: Tuning the Architecture, Not Just the Weights	CVPR	G	Github
Searching for A Robust Neural Architecture in Four GPU Hours	CVPR	G	Github
ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation	CVPR	-	-
Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search	CVPR	EA	github
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search	CVPR	G	-
RENAS: Reinforced Evolutionary Neural Architecture Search	CVPR	G	-
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation	CVPR	G	GitHub
MnasNet: Platform-Aware Neural Architecture Search for Mobile	CVPR	RL	Github
MFAS: Multimodal Fusion Architecture Search	CVPR	EA	-
A Neurobiological Evaluation Metric for Neural Network Model Search	CVPR	Other	-
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells	CVPR	RL	-
Customizable Architecture Search for Semantic Segmentation	CVPR	-	-
Regularized Evolution for Image Classifier Architecture Search	AAAI	EA	-
AutoAugment: Learning Augmentation Policies from Data	CVPR	RL	-
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules	ICML	EA	-
The Evolved Transformer	ICML	EA	Github
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks	ICML	RL	-
NAS-Bench-101: Towards Reproducible Neural Architecture Search	ICML	Other	Github
On Network Design Spaces for Visual Recognition	ICCV	G	Github

2018 Venues

Title	Venue	Type	Code
Towards Automatically-Tuned Deep Neural Networks	BOOK	-	GitHub
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications	ECCV	-	github
Efficient Architecture Search by Network Transformation	AAAI	RL	github
Learning Transferable Architectures for Scalable Image Recognition	CVPR	RL	github
N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning	ICLR	RL	-
A Flexible Approach to Automated RNN Architecture Generation	ICLR	RL/PD	-
Practical Block-wise Neural Network Architecture Generation	CVPR	RL	-
Path-Level Network Transformation for Efficient Architecture Search	ICML	RL	github
Hierarchical Representations for Efficient Architecture Search	ICLR	EA	-
Understanding and Simplifying One-Shot Architecture Search	ICML	G	-
SMASH: One-Shot Model Architecture Search through HyperNetworks	ICLR	G	github
Neural Architecture Optimization	NeurIPS	G	github
Searching for efficient multi-scale architectures for dense image prediction	NeurIPS	Other	-
Progressive Neural Architecture Search	ECCV	PD	github
Neural Architecture Search with Bayesian Optimisation and Optimal Transport	NeurIPS	Other	github
Differentiable Neural Network Architecture Search	ICLR-W	G	-
Accelerating Neural Architecture Search using Performance Prediction	ICLR-W	PD	-

2017 Venues

Title	Venue	Type	Code
Neural Architecture Search with Reinforcement Learning	ICLR	RL	-
Designing Neural Network Architectures using Reinforcement Learning	ICLR	RL	-
Neural Optimizer Search with Reinforcement Learning	ICML	RL	-
Learning Curve Prediction with Bayesian Neural Networks	ICLR	PD	-
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization	ICLR	PD	-
Hyperparameter Optimization: A Spectral Approach	NeurIPS-W	Other	github
Learning to Compose Domain-Specific Transformations for Data Augmentation	NeurIPS	-	-

Previous Venues

2012-2016

Title	Venue	Type	Code
Speeding up Automatic Hyperparameter Optimization of Deep Neural Networksby Extrapolation of Learning Curves	IJCAI	PD	github

arXiv

Title	Date	Type	Code
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search	2018.10	EA	-
Training Frankenstein’s Creature to Stack: HyperTree Architecture Search	2018.10	G	-
Population Based Training of Neural Networks	2017.11	EA	GitHub
EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition	2022.3	G	-
U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search	2022.3	G	Github

Awesome Surveys

Title	Venue	Year	Code
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions	ACM Computing Surveys	2021	-
Automated Machine Learning on Graphs: A Survey	ICLR-W	2021	GitHub
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice	Neurocomputing	2020	github
AutonoML: Towards an Integrated Framework for Autonomous Machine Learning	arXiv	2020	-
Automated Machine Learning	Springer Book	2019	-
Neural architecture search: A survey	JMLR	2019	-
AutoML: A Survey of the State-of-the-Art	arXiv	2019	GitHub
A Survey on Neural Architecture Search	arXiv	2019	-
Taking human out of learning applications: A survey on automated machine learning	arXiv	2018	-
IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective	Engineering Applications of Artificial Intelligence	2022	github

autodl

automl

awesome

deep-learning

hyper-parameter-optimization

nas

neural-architecture-search

Project health

Overall

3.0/10

Maintained

0/10

Code review

2/10

OpenSSF Scorecard

PyPI

awesome-autodl

Latest release

1.2.0 · Mar 20, 2022

Releases this year

Significant stargazers

LoveSy

5,710 followers · starred Aug 2019

沈云航 Yunhang Shen

153 followers · starred Sep 2019

Kunchang Li

385 followers · starred Feb 2021

Qingyou Meng

22 followers · starred Mar 2023

Similar repos

windmaple/awesome-AutoML

Curating a list of AutoML-related research, tools, projects and other resources

945

markdtw/awesome-architecture-search

A curated list of awesome architecture search resources

1,191

hibayesian/awesome-automl-papers

A curated list of automated machine learning papers, articles, tutorials, slides and projects

4,163

Engineer1999/A-Curated-List-of-Must-Read-ML-Research-Papers

A comprehensive, categorized collection of hundreds of research papers and surveys in Machine…

he-y/Awesome-Pruning

A curated list of neural network pruning resources.

2,497

MathiasGruber/AI-Paper-References

References for read papers + relevant links

hrcheng1066/awesome-pruning

310

ahkarami/Great-Deep-Learning-Tutorials

A Great Collection of Deep Learning Tutorials and Repositories

422