A picture of the effect of the regularizer
•
The overall cost function
is the sum of two
parabolic bowls.
•
The sum is also a
parabolic bowl.
•
The combined minimum
lies on the line between
the minimum of the
squared error and the
origin.
•
The regularizer just
shrinks the weights.