Course contents

The course covers the theoretical basis of a wide range of different network types and their implementation in TensorFlow. To get an overview about the course content, you can have a look at the following list.

Session 1: Introduction

Part 1: Brains and Neurons
Part 2: The Perceptron
Part 3: Deep Learning and Big Data
Downloads: Slides

Session 2: Multilayer Perceptron

Part 1: Improving the Model
Part 2: Linear and Non-linear Transformations
Part 3: Multilayer Perceptron
Part 4: Types of Learning
Part 5: Output Activation Functions
Downloads: Slides

Session 3: Gradient Descent and Backpropagation

Part 1: The Crucial Insight
Part 2: The Error Surface
Part 3: Numerical Optimization
Part 4: Derivatives, Partial Derivatives and Gradients
Part 5: The Backpropagation Algorithm
Part 6: Gradient Descent
Part 7: Non-Convex Functions
Part 8: Batch, Mini-Batch and Stochastic Learning
Part 9: Optimizing Standard Gradient Descent
Downloads: Slides, Homework Assignment, Solution

Session 4: TensorFlow

Part 1: Deep Learning Frameworks
Part 2: GPU Acceleration
Part 3: Data Flow Graphs
Part 4: Writing Code with TensorFlow
Part 5: TensorFlow and Automatic Differentiation
Part 6: Advanced TensorFlow Code
Downloads: Slides, Homework Assignment, Solution

Session 5: Convolutional Neural Networks

Part 1: Visual Information Processing in the Human Brain
Part 2: Cross Corelation
Part 3: Convolutional Neural Networks
Part 4: Forward Propagation in CNNs
Part 5: Efficiency of CNNs
Part 6: CNN Applications
Downloads: Slides, Homework Assignment, Solution

Session 6: Training Artificial Neural Networks

Part 1: Improving the Network Structure
Part 2: Vanishing and Exploding Gradients
Part 3: Activation Functions
Part 4: Weight Initialization
Part 5: Data Preparation and Batch Normalization
Part 6: Learning Rates
Part 7: Tools Against Overfitting

Session 7: Advanced Convolutional Neural Networks

Part 1: Overview
Part 2: Comparing Networks
Part 3: AlexNet and VGG
Part 4: 1x1 Convolutions
Part 5: Inception and Residual Networks
Part 6: Densely Connected Networks
Downloads: Slides, Homework Assignment, Solution

Session 8: Word Embeddings

Part 1: Statistical Language Models
Part 2: Vector Space Models
Part 3: Distributional Hypothesis
Part 4: Continuous Bag-of-Words Model
Part 5: Skip-Gram Model
Part 6: Negative Sampling
Part 7: Noise-Contrastive Estimation
Part 8: Evaluation and Visualization of Word Embeddings
Downloads: Slides, Homework Assignment, Solution

Session 9: Data and Training Visualization

Part 1: Deep Dream
Part 2: Visualization of Higher Order Features
Part 3: Adversarial Examples
Part 4: Dimensionality Reduction with t-SNE
Part 5: TensorBoard

Session 10: Recurrent Neural Networks

Part 1: Sequences of Data
Part 2: Recurrent Neural Networks
Part 3: Generative Recurrent Neural Networks
Part 4: Backpropagation through Time
Part 5: Batch Generation and Training
Part 6: Long Short-Term Memory

Downloads: Slides, Homework Assignment, Solution

Session 11: Advanced Recurrent Neural Networks

Session 12: Reinforcement Learning 1

Part 1: What is Reinforcement Learning?
Part 2: Agent and Environment
Part 3: Rewards and Returns
Part 4: State Transition Matrix
Part 5: Markov Decision Process
Part 6: Components of an RL Agent
Part 7: Bellman Equation
Part 8: Optimality and Learning

Session 13: Deep Reinforcement Learning 2

Part 1: Iterative Policy Evaluation
Part 2: Q-Learning
Part 3: Deep Q-Learning
Part 4: Policy Gradients
Part 5: Human vs. Reinforcement Learning

Downloads: Slides, Homework Assignment, Solution

Session 14: Generative Adversarial Networks

Part 1: Generative Models
Part 2: Generative Adversarial Networks
Part 3: Training Generative Adversarial Networks
Part 4: Advanced Generative Adversarial Networks

Downloads: Slides, Homework Assignment, Solution

Session 15: Echo State Networks 1

Session 16: Echo State Networks 2

Downloads: Slides, Homework Assignment, Solution

Session 17: Spiking Neural Networks 1

Part 1: Rate vs. Temporal Coding
Part 2: Electricity and Neurons
Part 3: Neurodynamics
Part 4: Differential Equations
Part 5: Leaky Integrate and Fire Neuron
Downloads: Slides, Homework Assignment, Solution

Session 18: Spiking Neural Networks 2

Part 1: Liquid State Machines
Part 2: Plasticity
Part 3: Backpropagating Errors in Spiking Neural Networks
Downloads: Slides, Homework Assignment, Solution

General

Part 1: Further Reading
Part 2: Machine Learning and Ethics
Part 3: The Small Artificial Neural Networks Dictionary