Deep learning reading list from Ilya Sutskever

深度学习精炼秘笈

til, Ilya sutskever gave john carmack this reading list of approx 30 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’

[Twitter Post] [Arc.net Link]

• The Annotated Transformer. Sasha Rush, et al. [Blog] [Code]
• The First Law of Complexodynamics. Scott Aaronson. [Blog]
• The Unreasonable Effectiveness of Recurrent Neural Networks. Andrej Karpathy. [Blog] [Code]
• Understanding LSTM Networks. Christopher Olah. [Blog]
• Recurrent Neural Network Regularization. Wojciech Zaremba, et al. [ArXiv] [pdf] [Code]
• Keeping Neural Networks Simple by Minimizing the Description Length of the Weights. Geoffrey E. Hinton and Drew van Camp. [Paper] [pdf]
• Pointer Networks. Oriol Vinyals, et al. [Paper] [pdf]
• ImageNet Classification with Deep Convolutional Neural Networks. Alex Krizhevsky, et al. [Paper] [pdf]
• Order Matters: Sequence to sequence for sets. Oriol Vinyals, et al. [ArXiv] [pdf]
• GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism. Yanping Huang, et al. [ArXiv] [pdf]
• Deep Residual Learning for Image Recognition. Kaiming He, et al.
• Multi-Scale Context Aggregation by Dilated Convolutions. Fisher Yu and Vladlen Koltun.
• Neural Message Passing for Quantum Chemistry. Justin Gilmer, et al.
• Attention Is All You Need. Ashish Vaswani, et al.
• Neural Machine Translation by Jointly Learning to Align and Translate. Dzmitry Bahdanau, et al.
• Identity Mappings in Deep Residual Networks. Kaiming He, et al.
• A simple neural network module for relational reasoning. Adam Santoro, et al.
• Variational Lossy Autoencoder. Xi Chen, et al.
• Relational recurrent neural networks. Adam Santoro, et al.
• Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton. Scott Aaronson, et al.
• Neural Turing Machines. Alex Graves, et al.
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. Dario Amodei, et al.
• Scaling Laws for Neural Language Models. Jared Kaplan, et al.
• A Tutorial Introduction to the Minimum Description Length Principle. Peter Grunwald.
• Machine Super Intelligence. Shane Legg.
• Kolmogorov Complexity and Algorithmic Randomness. A.Shen, V. A. Uspensky, and N. Vereshchagin.
• CS231n: Convolutional Neural Networks for Visual Recognition.