latest news

July 1, 2015

MovieBook and BookCorpus data released.

Abstract

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This work aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we propose a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

Paper


     
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Yukun Zhu*, Ryan Kiros*, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler

Arxiv, June 2015

* denotes equal contribution

Data

  • MovieBook data: ground-truth alignments for 11 movie/book pairs [coming soon...]
  • BookCorpus:
  • Original books [txt format] [mat format]

    * txt format contains the original txt files, organized in the genre subfolders. Note that a book can appear in multiple subfolders.

    * mat format contains our processed books files in .mat format. The processed files have info on sentences, paragraphs, chapters and dialogs. Note that due to various formats that the books can appear in, there may be occasional mistakes in our parsing. If you find them, please email mblist@cs.toronto.edu.

    All sentences in 11,038 books (pre-processed) [part1 (2.5GB)] [part2 (2.1GB)]

    * Note that only 7,087 out of 11,038 books in BookCorpus are unique. Among them 2089 books have one duplicate, 733 books have two and 95 books have more than two duplicates. We don't remove them in training skip-thoughts.

Please cite our paper if you use this data.

Citation

@inproceedings{moviebook,
title = {Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books},
author = {Yukun Zhu and Ryan Kiros and Richard Zemel and Ruslan Salakhutdinov and Raquel Urtasun and Antonio Torralba and Sanja Fidler},
booktitle = {arXiv preprint arXiv:1506.06724},
year = {2015}
}