Mode averaging
If we generate from the model,
half the instances of a 1 at the
data layer will be caused by a
(1,0) at the hidden layer and half
will be caused by a (0,1).
So the recognition weights
will learn to produce (0.5,0.5)
This represents a distribution
that puts half its mass on
very improbable hidden
configurations.
Its much better to just pick one
mode and pay one bit.
-10 -10
+20 +20
-20
minimum of
KL(Q||P)
minimum of
KL(P||Q)
P