Reading Notes 2017/2018

Author: Robert T. Lange

This repository contains simple reading notes, thoughts, questions and summaries of the papers/book chapters, which I (Robert T. Lange) have read in the second half of 2017 and 2018. This includes the summer break, where I attended a Free-Energy Principle summer school organised by Prof. Blankenburg and Prof. Ostwald (both BCCN and FU Berlin), the DS^3 Summer School as well as the European Summer School in Information Retrieval (ESSIR) and the time of my Computing (ML) Master's at Imperial College London.

The documents are grouped by overarching topic. First of all I hope that this way I am able to structure my knowledge gains and have a quick read to remind myself of keypoints. Second, I hope that you are able to follow my current interests and research progress.

Notes:

1. If there is a tick in the box, there exists a summary markdown file in the subdirectory. Otherwise, I have only read the document. Furthermore, I list below the chapters of books, while the markdown file contains the summary for the full (or the parts that I have worked in) book.
2. Most papers focus on Deep RL and Hierarchical RL (due to thesis and general interest).

2019-03

• Finn et al (2018) - Probabilistic Model-Agnostic Meta-Learning
• Andrychowicz et al (2016) - Learning to learn by gradient descent by gradient descent
• Finn et al (2017) - Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Multi-Agent RL

2018-01

• Das et al (2019) - TarMAC: Targeted Multi-Agent Communication
• Hausknecht and Stone (2015) - Deep Recurrent Q-Learning for POMDPs

2018-11

• Strouse et al (2018) - Learning to Share and Hide Intentions using Info Regularization
• Foerster et al (2016) - Learning to Communicate with Deep MARL

Biologically-Plausible Deep Learning

2019-02

• Whittington & Bogacz (2019) - Theories of Error Propagation in the Brain

2018-12

• Sacramento et al (2018) - Dendritic cortical microcircuits approximate the backpropagation algorithm
• Bartunov et al (2018) - Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Lillicrap et al (2016) - Random synaptic feedback weights support error backpropagation for deep learning

2018-11

• Garnelo et al (2018b) - Neural Processes

Hierarchical Reinforcement Learning

2018-08

• Pastra and Aloimonos (2012) - The minimalist grammar of action
• Bacon et al (2017) - The Option-Critic Architecture
• Daniel et al (2016) - Probabilistic inference for determining options in reinforcement learning
• Smith et al (2018) - An Inference-Based Policy Gradient Method for Learning Options

2018-07

• McGovern & Sutton (1998) - Macro-Actions in Reinforcement Learning: An Empiricial Analysis
• McGovern et al (1997) - Roles of Macro-Actions in Accelerating Reinforcement Learning

2018-06

• Yao et al (2014) - Universal Option Models
• Levy et al (2018) - Hierarchical Reinforcement Learning with Hindsight
• Bakker and Schmidhuber (2004) - Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization
• Mannor et al (2004) - Dynamic Abstraction in Reinforcement Learning via Clustering
• Menache et al (2002) - Q-Cut - Dynamic Discovery of Sub-Goals in Reinforcement Learning
• Stolle and Barto (2002) - Learning Options in Reinforcement Learning
• McGovern and Barto (2001) - Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density

2018-05

• Frans et al (2018) - Meta Learning Shared Hierarchies
• Florensa et al (2017) - Stochastic Neural Networks for Hierarchical Reinforcement Learning

Formal Grammars, Grammatical Inference and Surprisal

2018-07

• Siyari et al (2016) - Lexis: An Optimization Framework

2018-06

• Stout et al (2018) - Grammars of action in human behavior and evolution

2018-05

• Hale (2014) - Automaton theories of human sentence comprehension
• Schoenhense & Faisal (2017) - Data-efficient inference of hierarchical structure in sequential data by information-greedy grammar inference
• Hale (2001) - A Probabilistic Earley Parser as a Psycholinguistic Model

(Deep) Reinforcement Learning

2018-06

• Schaul et al (2015) - Universal Value Function Approximators
• Gershman and Daw (2017) - Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework
• Rusu et al (2016) - Policy Distillation
• Andrychowicz et al (2018) - Hindsight Experience Replay
• Choshen, Fox, Loewenstein (2018) - DORA - Directed Outreaching Reinforcement Action-Selection

2018-05

• Li (2017) - Deep Reinforcement Learning: An Overview
• Arulkumaran (2017) - A Brief Survey of Deep Reinforcement Learning
• Dayan (1993b) - Improving Generalisation for Temporal Difference Learning: The Successor Representation

2017-08

• Barto & Sutton (2016 - draft) - Ch. 1: The Reinforcement Learning Problem

Free-Energy Principle

2017-07

• Friston (2010) - The free-energy principle: a unified brain theory?
• Limanowski, Blankenburg (2013) - Minimal self-models and the free energy principle

2017-08

• Ostwald (2015) - The Free Energy Principle for Perception: An Introduction
• Bogacz (2017) - A tutorial on the free-energy framework for modelling perception and learning

Variational Inference

2019-02

• Zhang et al. (2018) - Advances in Variational Inference

2017-07

• Ostwald et al. (2014) - A tutorial on variational Bayes for latent linear stochastic time-series models

Deep Learning (Application + Theory)

2019-06

• Morcos et al (2018) - Insights on representational similarity in neural networks with canonical correlation

2019-03

• Spoerer et al (2017) - Recurrent Convolutional NNs: A Better Model of Biological Object Recognition

2017-07

• Goodfellow et al. (2016) - Ch. 6: Deep Feedforward Networks
• Gal, Ghahramani (2015) - Dropout as a Bayesian Approximation: Insights and Applications

2017-08

• Gal & Ghahramani (2016) - On Modern Deep Learning and Variational Inference

Optimization

2017-07

• Ruder (2016) - An overview of gradient descent optimization algorithms

Bayesian Optimization & Gaussian Processes

2017-07

• Shahriari et al. (2015) - Taking the Human Out of the Loop: A Review of Bayesian Optimization

Real Intelligence

2017-08

• Jeff Hawkins (2003) - On Intelligence
• Hassabis (2017) - Neuroscience-Inspired Artificial Intelligence

Machine Learning Reading Group ICL

2017-10

• #1: Daume (2004) - From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less
• #2: Zhang et al (2017) - Theory of Deep Learning III: Generalization Properties of SGD
• #3: Marco et al (2017) - Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization

2017-11

• #5: Lee et al (2017) - Deep Neural Networks as Gaussian Processes