CSC321: Neural Networks

Lecture 10: Speeding up the Learning

The error surface for a linear neuron

Convergence speed

How the learning goes wrong

Five ways to speed up learning

The momentum method

Adaptive learning rates on each connection

Online versus batch learning

Stochastic gradient descent

Newton’s method

Covariance Matrices

Fixing up the error surface

Curvature Matrices (optional material!)

Another, more general way to fix up the error surface

Extra problems that occur in multilayer non-linear networks