Independence relationships of hidden variables  in three types of model that have one hidden layer

Causal          Product of           Square model         experts (RBM)          ICA

independent (generation is easy)

dependent (rejecting away)

Hidden states unconditional on data Hidden states conditional on data

independent (by definition)

independent (the posterior collapses to a single point)

independent (inference is easy)

dependent (explaining away)

We can use an almost complementary prior to reduce this dependency so that variational inference works