Dataset Introduction

Movie4D dataset proposes the problem of creating a more immersive and interactive experience for the audience. Specifically, we are interested in the effects that people can normally feel. This natually leads to our 9 effect categories: shake, splash, gravity, light, wind, weather, temperature, liquid surrounding, and physical interaction. To better describe the effects happen within the scene, we also include detailed attributes such as intensity, direction



Annotation Structure

Each annotation is made with the structure illustrated to the right. In addition to basic class label and time stamps, we also include effect-specific details and POV information for better analysis. The explanation can be found below.

Overall, for each 5-min clip, we would have a sequence of effect being exist or not. The below plot demonstrates how the annotation labels exist in the clip. Notice, we do allow multiple effects occur at the same time.



Dataset Statistics and Analysis

Here are some visualizations and stats of our dataset. For more information and in depth analysis, please refer to our paper. Explanation of each effect type can be found in Explore section of the website.

The below stacked bar plot shows the density of each effect within selected movies. Notice with more action-intense and sci-fi movies, there are more effects happening given the same duration.

The left chart shows the number of annotations being made on differenct effects with same time period. We can see that effects that are more character-centric (eg. shake, physical interaction) tend to have more annotations made with short time span. Whereas for ambient environmental effects (eg. temperature, light), more annotations were made in a longer time span.

The below 2 pie charts show the distribution of effects in annotation number (left) and effect duration (right) in the dataset. It is expected to see that everyday life effects such as light, physical interact are mostly seen in movies. While all the surrounding effects such as temperature, weather, despite having fewer annotations, they take up longer time for the characters to experience.



Effect Showcase

We show examples of effects with different variations of intensity in the table below. Due to space constraint, the effect detail attributes are suppressed.

Effect Mild Medium Strong
Shake
Splash
Gravity
Light
Wind
Weather
Temperature
Liquid
Physical
Interaction



Label Explained

Toggle the text on specific effect to view the details.
The details of each GIF animation example are suppressed.

Shake

Splash

Gravity

Light

Wind

Weather

Temperature

Liquid Surrounding

Physical Interaction

Credits: University of Toronto: Yuhao Zhou, Makarand Tapaswi, Sanja Fidler