CSC321: Neural Networks

Lecture 7:  Ways to prevent overfitting

Overfitting

Preventing overfitting

Limiting the size of the weights

Weight-decay via noisy inputs

Other kinds of weight penalty

The effect of weight-decay

Deciding how much to restrict the capacity

Using a validation set

Preventing overfitting by early stopping

Why early stopping works

Combining networks

Combining networks reduces variance

A picture

How the combined predictor compares with the individual predictors

Ways to make predictors differ