Multimodal Learning with Deep Boltzmann Machines
Nitish Srivastava, Ruslan Salakhutdinov
University of Toronto
[paper]
[supplementary material]
[poster]
[video]
[Presentation]
Extended JMLR version [paper][bibtex]
Code
Code for training deep models Deepnet
Data
- Description of the features txt
- Preprocessed Data tar.gz [5.6 GB]
Results
Reconstruction of multimodal queries
These were made by taking a multimodal query and reconstructing it after doing mean-field inference in the model.
Reconstructions/Retrieval from individual pathways
These reconstructions were made by going up and down the stack of RBMs used for pretraining the DBM.
Text
- Replicated softmax model (Text model 1)
Text RBM
- 2 hidden layer DBN (Binary RBM on top of Text model 1) (Text model 2).
Text DBN
Image
- Gaussian RBM (Image model 1).
Image RBM
- 2 hidden layer DBN (Binary RBM on top of Image model 1) (Image model 2).
Image DBN
Samples from conditional models
DeepNet model files and trainers
DBN Model files can be found here or
on GitHub
Model files for DBMs