The 30 fundamental papers and resources that represent 90% of what matters in modern AI
"If you really learn all of these, you'll know 90% of what matters today."
— Ilya Sutskever (Co-founder of OpenAI, Former Chief Scientist)
In a conversation with John Carmack (legendary game developer and AI researcher), Ilya Sutskever shared a curated reading list of approximately 30 foundational papers, blog posts, and resources that form the backbone of modern deep learning and artificial intelligence.
This list is exceptional because it:
This repository organizes Sutskever's recommendations into a structured learning path, provides context for each resource, and maintains an active community of learners working through the material together.
Unlike most AI reading lists that grow indefinitely, Sutskever's list is deliberately constrained. The "90% of what matters" philosophy means these papers provide maximum insight with minimal redundancy. Master these, and you'll have a foundation that enables you to:
This roadmap organizes the list by difficulty and learning objectives. Each path builds on previous knowledge, so follow the progression unless you already have the prerequisites.
Prerequisites: Basic programming, calculus, and linear algebra
Goal: Understand what neural networks are and how they work
Time commitment: 4-6 weeks
CS231n: Convolutional Neural Networks for Visual Recognition
Stanford's legendary course. Start here for the best introduction to deep learning fundamentals, covering everything from backpropagation to CNNs.
Course Website
The Unreasonable Effectiveness of RNNs — Andrej Karpathy
An intuitive, code-driven introduction to recurrent networks. Shows what RNNs can do before diving into how they work.
Blog Post
Understanding LSTM Networks — Christopher Olah
The clearest visual explanation of LSTMs ever written. Essential for understanding sequence modeling.
Blog Post
The Annotated Transformer — Harvard NLP
A line-by-line implementation of "Attention is All You Need" with full working code. Learn by building.
Blog Post
ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)
The 2012 paper that started the deep learning revolution. Understanding this gives context for everything that followed.
Paper
Prerequisites: Completed Beginner Path or equivalent knowledge
Goal: Master core architectures and attention mechanisms
Time commitment: 6-8 weeks
Deep Residual Learning for Image Recognition (ResNet)
Skip connections revolutionized deep learning by enabling networks with 100+ layers. This is the paper that made modern deep learning possible.
arXiv
Attention Is All You Need
The transformer architecture that powers GPT, BERT, and most modern language models. One of the most influential ML papers ever written.
arXiv
Neural Machine Translation by Jointly Learning to Align and Translate
The paper that introduced attention mechanisms. Essential for understanding transformers.
arXiv
Pointer Networks
How to handle variable-length outputs and combinatorial problems with neural networks.
arXiv
Recurrent Neural Network Regularization
The right way to apply dropout to RNNs. Short but crucial for practical implementations.
arXiv
Generating Sequences With Recurrent Neural Networks
Alex Graves' masterpiece on sequence generation with RNNs. Introduces mixture density networks and shows how to generate handwriting and text.
arXiv
On the difficulty of training Recurrent Neural Networks
Explains the vanishing/exploding gradient problem in RNNs and introduces gradient clipping. Essential for understanding RNN training challenges.
arXiv
Identity Mappings in Deep Residual Networks
Improves on ResNet by analyzing exactly how skip connections work. Important for architecture design.
arXiv
Order Matters: Sequence to Sequence for Sets
Handling unordered data with sequence models. Clever solutions to important problems.
arXiv
Multi-Scale Context Aggregation by Dilated Convolutions
Efficient receptive fields without pooling. Key technique for semantic segmentation and dense prediction.
arXiv
Prerequisites: Strong understanding of architectures from Intermediate Path
Goal: Explore novel architectures and advanced training techniques
Time commitment: 8-10 weeks
Neural Turing Machines
Combining neural networks with external memory. Pioneering work in differentiable programming.
arXiv
A Simple Neural Network Module for Relational Reasoning
How to reason about relationships between objects. Enables compositional understanding.
arXiv
Relational Recurrent Neural Networks
Enhancing RNNs with relational inductive biases for improved reasoning.
arXiv
Variational Lossy Autoencoder
Advanced generative modeling with hierarchical latent variables.
arXiv
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
How to train models that don't fit on a single GPU. Essential for understanding modern large-scale training.
arXiv
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
End-to-end speech recognition with RNNs. Shows how to apply deep learning to real-world audio problems.
arXiv
Neural Quantum Chemistry
Deep learning for molecular property prediction. Example of deep learning beyond traditional ML domains.
arXiv
Machine Learning: The High-Interest Credit Card of Technical Debt
Essential reading on ML systems engineering. Addresses the hidden costs and long-term maintenance challenges of ML systems in production.
PDF
Prerequisites: Strong mathematical background, comfort with information theory
Goal: Understand the theoretical underpinnings of why deep learning works
Time commitment: 10-12 weeks (can be done in parallel with other paths)
The First Law of Complexodynamics — Scott Aaronson
Theoretical framework for understanding complexity in physical systems. Provides deep insights into learning and compression.
Blog Post
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
Formalizing intuitions about complexity. Important for understanding what neural networks can and cannot learn.
arXiv
Scaling Laws for Neural Language Models
How model performance scales with compute, data, and parameters. Essential for understanding modern AI capabilities and limitations.
arXiv
A Tutorial Introduction to the Minimum Description Length Principle
Compression and learning are two sides of the same coin. Fundamental principle underlying many ML algorithms.
arXiv
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Applying MDL to neural networks. Occam's razor formalized.
PDF
Prerequisites: All of the above, plus serious dedication
Goal: Achieve deep theoretical understanding
Time commitment: Several months
Machine Super Intelligence — Shane Legg
PhD thesis on formal theories of intelligence and artificial general intelligence. Dense but rewarding.
PDF
An Introduction to Kolmogorov Complexity and Its Applications (Chapter 7, page 434 onwards)
The mathematical foundations of information, randomness, and learning. Challenging but fundamental.
PDF
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| ImageNet Classification with Deep CNNs | Krizhevsky, Sutskever, Hinton | 2012 | Deep CNNs + GPU training + ReLU + Dropout | Proved deep learning works at scale, sparked the modern AI revolution |
| Deep Residual Learning | He et al. | 2015 | Skip connections enabling very deep networks | Made 100+ layer networks trainable, foundation of most modern architectures |
| Identity Mappings in Deep Residual Networks | He et al. | 2016 | Analysis and improvement of ResNet design | Clarified why skip connections work so well |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| The Unreasonable Effectiveness of RNNs | Karpathy | 2015 | Accessible introduction to RNN capabilities | Best first introduction to sequence modeling |
| Understanding LSTM Networks | Olah | 2015 | Clear visual explanation of LSTM architecture | The definitive LSTM explainer |
| Recurrent Neural Network Regularization | Zaremba et al. | 2014 | Proper dropout for RNNs | Essential technique for training RNNs |
| Generating Sequences With Recurrent Neural Networks | Graves | 2013 | Sequence generation with mixture density networks | Masterpiece on generating handwriting and text |
| On the difficulty of training RNNs | Pascanu et al. | 2012 | Vanishing/exploding gradients and gradient clipping | Essential for understanding RNN training challenges |
| Deep Speech 2 | Amodei et al. | 2015 | End-to-end speech recognition | Shows how to apply deep learning to real audio |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Neural Machine Translation by Jointly Learning to Align and Translate | Bahdanau et al. | 2014 | Attention mechanism for seq2seq | Introduced the attention mechanism used everywhere today |
| Attention Is All You Need | Vaswani et al. | 2017 | Transformer architecture | Powers GPT, BERT, and nearly all modern NLP |
| The Annotated Transformer | Rush et al. | 2018 | Line-by-line implementation | Learn transformers by building one |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Pointer Networks | Vinyals et al. | 2015 | Output sequences of discrete tokens pointing to inputs | Handles combinatorial optimization problems |
| Neural Turing Machines | Graves et al. | 2014 | Differentiable external memory | Pioneered neural network + data structure combinations |
| Order Matters: Sequence to Sequence for Sets | Vinyals et al. | 2015 | Read/process phase for set inputs | Elegant solution to permutation invariance |
| Multi-Scale Context Aggregation by Dilated Convolutions | Yu & Koltun | 2016 | Dilated/atrous convolutions | Exponentially growing receptive fields |
| A Simple Neural Network Module for Relational Reasoning | Santoro et al. | 2017 | Relation networks | First-class support for reasoning about relationships |
| Relational Recurrent Neural Networks | Santoro et al. | 2018 | Multi-head dot-product attention in RNNs | Combines recurrence with relational reasoning |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Variational Lossy Autoencoder | Chen et al. | 2016 | Hierarchical VAE with learned inference | Advanced generative modeling techniques |
| Paper | Authors | Year | Domain | Why It Matters |
|---|---|---|---|---|
| Neural Quantum Chemistry | Gilmer et al. | 2017 | Chemistry | Shows deep learning's reach beyond traditional ML |
| ML: The High-Interest Credit Card of Technical Debt | Sculley et al. | 2015 | ML Systems | Hidden costs and maintenance challenges of production ML |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| GPipe | Huang et al. | 2018 | Pipeline parallelism for large models | Essential for training models that don't fit on one GPU |
| Scaling Laws for Neural Language Models | Kaplan et al. | 2020 | Power law relationships in model scaling | Predicts performance based on compute/data/parameters |
| Paper | Authors | Year | Key Concept | Why It Matters |
|---|---|---|---|---|
| Keeping Neural Networks Simple by Minimizing Description Length | Hinton & Van Camp | 1993 | MDL principle for neural networks | Foundational principle: learning is compression |
| A Tutorial Introduction to the MDL Principle | Grünwald | 2004 | Minimum Description Length | Unifies learning and compression theoretically |
| Quantifying Complexity in Closed Systems | Aaronson et al. | 2013 | Complexity measures in physical systems | Theoretical foundations for understanding learning |
| The First Law of Complexodynamics | Aaronson | 2013 | Entropy and complexity in physical systems | Deep connections between physics and computation |
| Resource | Author | Year | Topic | Why It Matters |
|---|---|---|---|---|
| Machine Super Intelligence | Shane Legg | 2008 | Formal theories of AGI | Rigorous treatment of intelligence from first principles |
| Kolmogorov Complexity (p.434+) | Li & Vitányi | 2013 | Information theory | Mathematical foundations of information and randomness |
| Resource | Institution | Topic | Why It Matters |
|---|---|---|---|
| CS231n: CNNs for Visual Recognition | Stanford | Deep learning fundamentals | The best introduction to deep learning, period |
We welcome contributions from the community! This repository improves when people share their insights and implementations.
Paper summaries: Write clear, concise summaries (300-500 words) explaining the key ideas in accessible language.
Code implementations: Share clean, well-documented implementations with links to your repos.
Learning notes: Contribute your notes, diagrams, or explanations that helped you understand difficult concepts.
Fix issues: Report broken links, errors in descriptions, or suggest improvements.
Translations: Help make this resource accessible to non-English speakers.
Create your own checklist by copying this into a GitHub issue or personal document:
## Beginner Path
- [ ] CS231n Course
- [ ] The Unreasonable Effectiveness of RNNs
- [ ] Understanding LSTM Networks
- [ ] The Annotated Transformer
- [ ] ImageNet Classification with Deep CNNs
## Intermediate Path
- [ ] Deep Residual Learning (ResNet)
- [ ] Attention Is All You Need
- [ ] Neural Machine Translation (Attention)
- [ ] Pointer Networks
- [ ] RNN Regularization
- [ ] Generating Sequences With RNNs
- [ ] On the difficulty of training RNNs
- [ ] Identity Mappings in ResNets
- [ ] Order Matters (Sets)
- [ ] Dilated Convolutions
## Advanced Path
- [ ] Neural Turing Machines
- [ ] Relational Reasoning Module
- [ ] Relational RNNs
- [ ] Variational Lossy Autoencoder
- [ ] GPipe
- [ ] Deep Speech 2
- [ ] Neural Quantum Chemistry
- [ ] ML: The High-Interest Credit Card of Technical Debt
## Theory & Foundations
- [ ] First Law of Complexodynamics
- [ ] Coffee Automaton (Complexity)
- [ ] Scaling Laws for LLMs
- [ ] MDL Tutorial
- [ ] MDL for Neural Networks
## Doctoral Level
- [ ] Machine Super Intelligence
- [ ] Kolmogorov Complexity (Chapter 7)
Q: Do I need to read every paper cover-to-cover?
A: No. Read for understanding, not completion. Some papers you'll skim, others you'll implement. Let your learning goals guide you.
Q: What math background do I need?
A: Calculus, linear algebra, probability. CS231n has good review materials if you're rusty.
Q: How long does it take to complete?
A: 6-12 months working steadily. The goal isn't speed—it's deep understanding.
Q: Can I skip the theory papers?
A: You can, but you shouldn't. They provide crucial intuitions for why techniques work.
Q: Are there video lectures?
A: CS231n has excellent videos. For most papers, you'll need to read the actual paper.
This curated list is available under the MIT License.
Individual papers and resources are subject to their respective licenses and copyrights.
The 30 fundamental papers and resources that represent 90% of what matters in modern AI
"If you really learn all of these, you'll know 90% of what matters today."
— Ilya Sutskever (Co-founder of OpenAI, Former Chief Scientist)
In a conversation with John Carmack (legendary game developer and AI researcher), Ilya Sutskever shared a curated reading list of approximately 30 foundational papers, blog posts, and resources that form the backbone of modern deep learning and artificial intelligence.
This list is exceptional because it:
This repository organizes Sutskever's recommendations into a structured learning path, provides context for each resource, and maintains an active community of learners working through the material together.
Unlike most AI reading lists that grow indefinitely, Sutskever's list is deliberately constrained. The "90% of what matters" philosophy means these papers provide maximum insight with minimal redundancy. Master these, and you'll have a foundation that enables you to:
This roadmap organizes the list by difficulty and learning objectives. Each path builds on previous knowledge, so follow the progression unless you already have the prerequisites.
Prerequisites: Basic programming, calculus, and linear algebra
Goal: Understand what neural networks are and how they work
Time commitment: 4-6 weeks
CS231n: Convolutional Neural Networks for Visual Recognition
Stanford's legendary course. Start here for the best introduction to deep learning fundamentals, covering everything from backpropagation to CNNs.
Course Website
The Unreasonable Effectiveness of RNNs — Andrej Karpathy
An intuitive, code-driven introduction to recurrent networks. Shows what RNNs can do before diving into how they work.
Blog Post
Understanding LSTM Networks — Christopher Olah
The clearest visual explanation of LSTMs ever written. Essential for understanding sequence modeling.
Blog Post
The Annotated Transformer — Harvard NLP
A line-by-line implementation of "Attention is All You Need" with full working code. Learn by building.
Blog Post
ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)
The 2012 paper that started the deep learning revolution. Understanding this gives context for everything that followed.
Paper
Prerequisites: Completed Beginner Path or equivalent knowledge
Goal: Master core architectures and attention mechanisms
Time commitment: 6-8 weeks
Deep Residual Learning for Image Recognition (ResNet)
Skip connections revolutionized deep learning by enabling networks with 100+ layers. This is the paper that made modern deep learning possible.
arXiv
Attention Is All You Need
The transformer architecture that powers GPT, BERT, and most modern language models. One of the most influential ML papers ever written.
arXiv
Neural Machine Translation by Jointly Learning to Align and Translate
The paper that introduced attention mechanisms. Essential for understanding transformers.
arXiv
Pointer Networks
How to handle variable-length outputs and combinatorial problems with neural networks.
arXiv
Recurrent Neural Network Regularization
The right way to apply dropout to RNNs. Short but crucial for practical implementations.
arXiv
Generating Sequences With Recurrent Neural Networks
Alex Graves' masterpiece on sequence generation with RNNs. Introduces mixture density networks and shows how to generate handwriting and text.
arXiv
On the difficulty of training Recurrent Neural Networks
Explains the vanishing/exploding gradient problem in RNNs and introduces gradient clipping. Essential for understanding RNN training challenges.
arXiv
Identity Mappings in Deep Residual Networks
Improves on ResNet by analyzing exactly how skip connections work. Important for architecture design.
arXiv
Order Matters: Sequence to Sequence for Sets
Handling unordered data with sequence models. Clever solutions to important problems.
arXiv
Multi-Scale Context Aggregation by Dilated Convolutions
Efficient receptive fields without pooling. Key technique for semantic segmentation and dense prediction.
arXiv
Prerequisites: Strong understanding of architectures from Intermediate Path
Goal: Explore novel architectures and advanced training techniques
Time commitment: 8-10 weeks
Neural Turing Machines
Combining neural networks with external memory. Pioneering work in differentiable programming.
arXiv
A Simple Neural Network Module for Relational Reasoning
How to reason about relationships between objects. Enables compositional understanding.
arXiv
Relational Recurrent Neural Networks
Enhancing RNNs with relational inductive biases for improved reasoning.
arXiv
Variational Lossy Autoencoder
Advanced generative modeling with hierarchical latent variables.
arXiv
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
How to train models that don't fit on a single GPU. Essential for understanding modern large-scale training.
arXiv
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
End-to-end speech recognition with RNNs. Shows how to apply deep learning to real-world audio problems.
arXiv
Neural Quantum Chemistry
Deep learning for molecular property prediction. Example of deep learning beyond traditional ML domains.
arXiv
Machine Learning: The High-Interest Credit Card of Technical Debt
Essential reading on ML systems engineering. Addresses the hidden costs and long-term maintenance challenges of ML systems in production.
PDF
Prerequisites: Strong mathematical background, comfort with information theory
Goal: Understand the theoretical underpinnings of why deep learning works
Time commitment: 10-12 weeks (can be done in parallel with other paths)
The First Law of Complexodynamics — Scott Aaronson
Theoretical framework for understanding complexity in physical systems. Provides deep insights into learning and compression.
Blog Post
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
Formalizing intuitions about complexity. Important for understanding what neural networks can and cannot learn.
arXiv
Scaling Laws for Neural Language Models
How model performance scales with compute, data, and parameters. Essential for understanding modern AI capabilities and limitations.
arXiv
A Tutorial Introduction to the Minimum Description Length Principle
Compression and learning are two sides of the same coin. Fundamental principle underlying many ML algorithms.
arXiv
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Applying MDL to neural networks. Occam's razor formalized.
PDF
Prerequisites: All of the above, plus serious dedication
Goal: Achieve deep theoretical understanding
Time commitment: Several months
Machine Super Intelligence — Shane Legg
PhD thesis on formal theories of intelligence and artificial general intelligence. Dense but rewarding.
PDF
An Introduction to Kolmogorov Complexity and Its Applications (Chapter 7, page 434 onwards)
The mathematical foundations of information, randomness, and learning. Challenging but fundamental.
PDF
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| ImageNet Classification with Deep CNNs | Krizhevsky, Sutskever, Hinton | 2012 | Deep CNNs + GPU training + ReLU + Dropout | Proved deep learning works at scale, sparked the modern AI revolution |
| Deep Residual Learning | He et al. | 2015 | Skip connections enabling very deep networks | Made 100+ layer networks trainable, foundation of most modern architectures |
| Identity Mappings in Deep Residual Networks | He et al. | 2016 | Analysis and improvement of ResNet design | Clarified why skip connections work so well |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| The Unreasonable Effectiveness of RNNs | Karpathy | 2015 | Accessible introduction to RNN capabilities | Best first introduction to sequence modeling |
| Understanding LSTM Networks | Olah | 2015 | Clear visual explanation of LSTM architecture | The definitive LSTM explainer |
| Recurrent Neural Network Regularization | Zaremba et al. | 2014 | Proper dropout for RNNs | Essential technique for training RNNs |
| Generating Sequences With Recurrent Neural Networks | Graves | 2013 | Sequence generation with mixture density networks | Masterpiece on generating handwriting and text |
| On the difficulty of training RNNs | Pascanu et al. | 2012 | Vanishing/exploding gradients and gradient clipping | Essential for understanding RNN training challenges |
| Deep Speech 2 | Amodei et al. | 2015 | End-to-end speech recognition | Shows how to apply deep learning to real audio |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Neural Machine Translation by Jointly Learning to Align and Translate | Bahdanau et al. | 2014 | Attention mechanism for seq2seq | Introduced the attention mechanism used everywhere today |
| Attention Is All You Need | Vaswani et al. | 2017 | Transformer architecture | Powers GPT, BERT, and nearly all modern NLP |
| The Annotated Transformer | Rush et al. | 2018 | Line-by-line implementation | Learn transformers by building one |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Pointer Networks | Vinyals et al. | 2015 | Output sequences of discrete tokens pointing to inputs | Handles combinatorial optimization problems |
| Neural Turing Machines | Graves et al. | 2014 | Differentiable external memory | Pioneered neural network + data structure combinations |
| Order Matters: Sequence to Sequence for Sets | Vinyals et al. | 2015 | Read/process phase for set inputs | Elegant solution to permutation invariance |
| Multi-Scale Context Aggregation by Dilated Convolutions | Yu & Koltun | 2016 | Dilated/atrous convolutions | Exponentially growing receptive fields |
| A Simple Neural Network Module for Relational Reasoning | Santoro et al. | 2017 | Relation networks | First-class support for reasoning about relationships |
| Relational Recurrent Neural Networks | Santoro et al. | 2018 | Multi-head dot-product attention in RNNs | Combines recurrence with relational reasoning |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| Variational Lossy Autoencoder | Chen et al. | 2016 | Hierarchical VAE with learned inference | Advanced generative modeling techniques |
| Paper | Authors | Year | Domain | Why It Matters |
|---|---|---|---|---|
| Neural Quantum Chemistry | Gilmer et al. | 2017 | Chemistry | Shows deep learning's reach beyond traditional ML |
| ML: The High-Interest Credit Card of Technical Debt | Sculley et al. | 2015 | ML Systems | Hidden costs and maintenance challenges of production ML |
| Paper | Authors | Year | Key Innovation | Why It Matters |
|---|---|---|---|---|
| GPipe | Huang et al. | 2018 | Pipeline parallelism for large models | Essential for training models that don't fit on one GPU |
| Scaling Laws for Neural Language Models | Kaplan et al. | 2020 | Power law relationships in model scaling | Predicts performance based on compute/data/parameters |
| Paper | Authors | Year | Key Concept | Why It Matters |
|---|---|---|---|---|
| Keeping Neural Networks Simple by Minimizing Description Length | Hinton & Van Camp | 1993 | MDL principle for neural networks | Foundational principle: learning is compression |
| A Tutorial Introduction to the MDL Principle | Grünwald | 2004 | Minimum Description Length | Unifies learning and compression theoretically |
| Quantifying Complexity in Closed Systems | Aaronson et al. | 2013 | Complexity measures in physical systems | Theoretical foundations for understanding learning |
| The First Law of Complexodynamics | Aaronson | 2013 | Entropy and complexity in physical systems | Deep connections between physics and computation |
| Resource | Author | Year | Topic | Why It Matters |
|---|---|---|---|---|
| Machine Super Intelligence | Shane Legg | 2008 | Formal theories of AGI | Rigorous treatment of intelligence from first principles |
| Kolmogorov Complexity (p.434+) | Li & Vitányi | 2013 | Information theory | Mathematical foundations of information and randomness |
| Resource | Institution | Topic | Why It Matters |
|---|---|---|---|
| CS231n: CNNs for Visual Recognition | Stanford | Deep learning fundamentals | The best introduction to deep learning, period |
We welcome contributions from the community! This repository improves when people share their insights and implementations.
Paper summaries: Write clear, concise summaries (300-500 words) explaining the key ideas in accessible language.
Code implementations: Share clean, well-documented implementations with links to your repos.
Learning notes: Contribute your notes, diagrams, or explanations that helped you understand difficult concepts.
Fix issues: Report broken links, errors in descriptions, or suggest improvements.
Translations: Help make this resource accessible to non-English speakers.
Create your own checklist by copying this into a GitHub issue or personal document:
## Beginner Path
- [ ] CS231n Course
- [ ] The Unreasonable Effectiveness of RNNs
- [ ] Understanding LSTM Networks
- [ ] The Annotated Transformer
- [ ] ImageNet Classification with Deep CNNs
## Intermediate Path
- [ ] Deep Residual Learning (ResNet)
- [ ] Attention Is All You Need
- [ ] Neural Machine Translation (Attention)
- [ ] Pointer Networks
- [ ] RNN Regularization
- [ ] Generating Sequences With RNNs
- [ ] On the difficulty of training RNNs
- [ ] Identity Mappings in ResNets
- [ ] Order Matters (Sets)
- [ ] Dilated Convolutions
## Advanced Path
- [ ] Neural Turing Machines
- [ ] Relational Reasoning Module
- [ ] Relational RNNs
- [ ] Variational Lossy Autoencoder
- [ ] GPipe
- [ ] Deep Speech 2
- [ ] Neural Quantum Chemistry
- [ ] ML: The High-Interest Credit Card of Technical Debt
## Theory & Foundations
- [ ] First Law of Complexodynamics
- [ ] Coffee Automaton (Complexity)
- [ ] Scaling Laws for LLMs
- [ ] MDL Tutorial
- [ ] MDL for Neural Networks
## Doctoral Level
- [ ] Machine Super Intelligence
- [ ] Kolmogorov Complexity (Chapter 7)
Q: Do I need to read every paper cover-to-cover?
A: No. Read for understanding, not completion. Some papers you'll skim, others you'll implement. Let your learning goals guide you.
Q: What math background do I need?
A: Calculus, linear algebra, probability. CS231n has good review materials if you're rusty.
Q: How long does it take to complete?
A: 6-12 months working steadily. The goal isn't speed—it's deep understanding.
Q: Can I skip the theory papers?
A: You can, but you shouldn't. They provide crucial intuitions for why techniques work.
Q: Are there video lectures?
A: CS231n has excellent videos. For most papers, you'll need to read the actual paper.
This curated list is available under the MIT License.
Individual papers and resources are subject to their respective licenses and copyrights.