A coding view of FA, PPCA and PCA
Factor analysis pays to communicate the hidden factor
values:
log p(value|gaussian)
It also pays to communicate the residual errors in each
observed value:
log p(residual|noise model for that dimension)
PPCA pays both costs but uses the same noise model
for all data dimensions (suboptimal)
PCA ignores the cost of communicating the factor
values. It also uses the same noise model for all input
dimensions.