Neural Networks for Machine Learning

About this course: Learn about artificial neural networks and how they're being used for machine learning, as applied to speech and object recognition, image segmentation, modeling language and human motion, etc. We'll emphasize both the basic algorithms and the practical tricks needed to get them to work well. This course contains the same content presented on Coursera beginning in 2013. It is not a continuation or update of the original course. It has been adapted for the new platform. Please be advised that the course is suited for an intermediate level learner - comfortable with calculus and with experience programming (Python).

Created by:  University of Toronto

• Taught by:  Geoffrey Hinton, Professor

  Department of Computer Science

Language English How To Pass Pass all graded assignments to complete the course. User Ratings 4.5 stars Average User Rating 4.5See what learners said Coursework

Each course is like an interactive textbook, featuring pre-recorded videos, quizzes and projects.

Help from your peers

Connect with thousands of other learners and debate ideas, discuss course material, and get help mastering concepts.

Certificates

Earn official recognition for your work, and share your success with friends, colleagues, and employers.

University of Toronto Established in 1827, the University of Toronto has one of the strongest research and teaching faculties in North America, presenting top students at all levels with an intellectual environment unmatched in depth and breadth on any other Canadian campus.

Syllabus


WEEK 1


Introduction
Introduction to the course - machine learning and neural nets


5 videos, 8 readings expand

1. Reading: Syllabus and Course Logistics
2. Reading: Lecture Slides (and resources)
3. Video: Why do we need machine learning? [13 min]
4. Video: What are neural networks? [8 min]
5. Video: Some simple models of neurons [8 min]
6. Video: A simple example of learning [6 min]
7. Video: Three types of learning [8 min]
8. Reading: Setting Up Your Programming Assignment Environment
9. Reading: Installing Octave on Windows
10. Reading: Installing Octave on Mac OS X (10.10 Yosemite and 10.9 Mavericks)
11. Reading: Installing Octave on Mac OS X (10.8 Mountain Lion and Earlier)
12. Reading: Installing Octave on GNU/Linux
13. Reading: Octave

Graded: Lecture 1 Quiz

WEEK 2


The Perceptron learning procedure
An overview of the main types of neural network architecture


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Types of neural network architectures [7 min]
3. Video: Perceptrons: The first generation of neural networks [8 min]
4. Video: A geometrical view of perceptrons [6 min]
5. Video: Why the learning works [5 min]
6. Video: What perceptrons can't do [15 min]

Graded: Lecture 2 Quiz

WEEK 3


The backpropagation learning proccedure
Learning the weights of a linear neuron


5 videos, 2 readings expand

1. Reading: Lecture Slides (and resources)
2. Video: Learning the weights of a linear neuron [12 min]
3. Video: The error surface for a linear neuron [5 min]
4. Video: Learning the weights of a logistic output neuron [4 min]
5. Video: The backpropagation algorithm [12 min]
6. Video: Using the derivatives computed by backpropagation [10 min]
7. Reading: Forward Propagation in Neural Networks

Graded: Lecture 3 Quiz
Graded: Programming Assignment 1: The perceptron learning algorithm.

WEEK 4


Learning feature vectors for words
Learning to predict the next word


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Learning to predict the next word [13 min]
3. Video: A brief diversion into cognitive science [4 min]
4. Video: Another diversion: The softmax output function [7 min]
5. Video: Neuro-probabilistic language models [8 min]
6. Video: Ways to deal with the large number of possible outputs [15 min]

Graded: Lecture 4 Quiz

WEEK 5


Object recognition with neural nets
In this module we look at why object recognition is difficult.


4 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Why object recognition is difficult [5 min]
3. Video: Achieving viewpoint invariance [6 min]
4. Video: Convolutional nets for digit recognition [16 min]
5. Video: Convolutional nets for object recognition [17min]

Graded: Lecture 5 Quiz
Graded: Programming Assignment 2: Learning Word Representations.

WEEK 6


Optimization: How to make the learning go faster
We delve into mini-batch gradient descent as well as discuss adaptive learning rates.


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Overview of mini-batch gradient descent
3. Video: A bag of tricks for mini-batch gradient descent
4. Video: The momentum method
5. Video: Adaptive learning rates for each connection
6. Video: Rmsprop: Divide the gradient by a running average of its recent magnitude

Graded: Lecture 6 Quiz

WEEK 7


Recurrent neural networks
This module explores training recurrent neural networks


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Modeling sequences: A brief overview
3. Video: Training RNNs with back propagation
4. Video: A toy example of training an RNN
5. Video: Why it is difficult to train an RNN
6. Video: Long-term Short-term-memory

Graded: Lecture 7 Quiz

WEEK 8


recurrent neural networks
We continue our look at recurrent neural networks


3 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Modeling character strings with multiplicative connections [14 mins]
3. Video: Learning to predict the next character using HF [12 mins]
4. Video: Echo State Networks [9 min]

Graded: Lecture 8 Quiz

WEEK 9


Ways to make neural networks generalize better
We discuss strategies to make neural networks generalize better


6 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Overview of ways to improve generalization [12 min]
3. Video: Limiting the size of the weights [6 min]
4. Video: Using noise as a regularizer [7 min]
5. Video: Introduction to the full Bayesian approach [12 min]
6. Video: The Bayesian interpretation of weight decay [11 min]
7. Video: MacKay's quick and dirty method of setting weight costs [4 min]

Graded: Lecture 9 Quiz
Graded: Programming assignment 3: Optimization and generalization

WEEK 10


Combining multiple neural networks to improve generalization
This module we look at why it helps to combine multiple neural networks to improve generalization


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Why it helps to combine models [13 min]
3. Video: Mixtures of Experts [13 min]
4. Video: The idea of full Bayesian learning [7 min]
5. Video: Making full Bayesian learning practical [7 min]
6. Video: Dropout [9 min]

Graded: Lecture 10 Quiz

WEEK 11


Hopfield nets and Boltzmann machines



5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Hopfield Nets [13 min]
3. Video: Dealing with spurious minima [11 min]
4. Video: Hopfield nets with hidden units [10 min]
5. Video: Using stochastic units to improv search [11 min]
6. Video: How a Boltzmann machine models data [12 min]

Graded: Lecture 11 Quiz

WEEK 12


Restricted Boltzmann machines (RBMs)
This module deals with Boltzmann machine learning


5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Boltzmann machine learning [12 min]
3. Video: OPTIONAL VIDEO: efficient ways to get the statistics [15 mins]
4. Video: Restricted Boltzmann Machines [11 min]
5. Video: An example of RBM learning [7 mins]
6. Video: RBMs for collaborative filtering [8 mins]

Graded: Lecture 12 Quiz

WEEK 13


Stacking RBMs to make Deep Belief Nets



3 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: The ups and downs of back propagation [10 min]
3. Video: Belief Nets [13 min]
4. Video: The wake-sleep algorithm [13 min]

Graded: Programming Assignment 4: Restricted Boltzmann Machines
Graded: Lecture 13 Quiz

WEEK 14


Deep neural nets with generative pre-training



5 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: Learning layers of features by stacking RBMs [17 min]
3. Video: Discriminative learning for DBNs [9 mins]
4. Video: What happens during discriminative fine-tuning? [8 mins]
5. Video: Modeling real-valued data with an RBM [10 mins]
6. Video: OPTIONAL VIDEO: RBMs are infinite sigmoid belief nets [17 mins]

Graded: Lecture 14 Quiz

WEEK 15


Modeling hierarchical structure with neural nets



6 videos, 1 reading expand

1. Reading: Lecture Slides (and resources)
2. Video: From PCA to autoencoders [5 mins]
3. Video: Deep auto encoders [4 mins]
4. Video: Deep auto encoders for document retrieval [8 mins]
5. Video: Semantic Hashing [9 mins]
6. Video: Learning binary codes for image retrieval [9 mins]
7. Video: Shallow autoencoders for pre-training [7 mins]

Graded: Lecture 15 Quiz
Graded: Final Exam

WEEK 16


Recent applications of deep neural nets



3 videos expand

1. Video: OPTIONAL: Learning a joint model of images and captions [10 min]
2. Video: OPTIONAL: Hierarchical Coordinate Frames [10 mins]
3. Video: OPTIONAL: Bayesian optimization of hyper-parameters [13 min]