CSC2515 Fall 2007
Introduction to Machine
Learning
Lecture 4: Backpropagation
Learning by perturbing weights
The idea behind backpropagation
Non-linear neurons with smooth derivatives
Sketch of the
backpropagation algorithm
on a single training case
Overview of the applications in this lecture
An example of relational information
Another way to express the same information
The structure of the neural net
How to show the weights of hidden units
The features it learned for person 1
Another way to see that it works
A basic problem in speech recognition
Why the trigram model is silly
Bengio’s neural net for predicting the next word
2-D display of some of the 100-D feature vectors learned by another language model
Applying backpropagation to shape recognition
The replicated feature approach
Backpropagation with weight constraints
Combining the outputs of replicated features
An advantage of modeling sequential data
The equivalence between layered, feedforward nets and recurrent nets
Teaching signals for recurrent networks
A good problem for a recurrent network
The algorithm for binary addition
A recurrent net for binary addition
The connectivity of the network
Preventing overfitting by early stopping
How to deal with the fact that the space of all possible parameters vectors is huge