Dataset Introduction
|
|
Movie4D dataset proposes the problem of creating a more immersive and interactive experience for the audience.
Specifically, we are interested in the effects that people can normally feel.
This natually leads to our 9 effect categories: shake, splash, gravity, light, wind, weather, temperature, liquid surrounding, and physical interaction.
To better describe the effects happen within the scene, we also include detailed attributes such as intensity, direction
|
|
|
Annotation Structure
|
|
Each annotation is made with the structure illustrated to the right.
In addition to basic class label and time stamps, we also include effect-specific details and POV information for better analysis.
The explanation can be found below.
Overall, for each 5-min clip, we would have a sequence of effect being exist or not.
The below plot demonstrates how the annotation labels exist in the clip.
Notice, we do allow multiple effects occur at the same time.
|
|
|
|
|
Dataset Statistics and Analysis
|
|
Here are some visualizations and stats of our dataset.
For more information and in depth analysis, please refer to our paper.
Explanation of each effect type can be found in Explore section of the website.
The below stacked bar plot shows the density of each effect within selected movies.
Notice with more action-intense and sci-fi movies, there are more effects happening given the same duration.
|
|
|
The left chart shows the number of annotations being made on differenct effects with same time period.
We can see that effects that are more character-centric (eg. shake, physical interaction) tend to have more annotations made with short time span.
Whereas for ambient environmental effects (eg. temperature, light), more annotations were made in a longer time span.
|
|
The below 2 pie charts show the distribution of effects in annotation number (left) and effect duration (right) in the dataset.
It is expected to see that everyday life effects such as light, physical interact are mostly seen in movies.
While all the surrounding effects such as temperature, weather, despite having fewer annotations, they take up longer time for the characters to experience.
|
|
|
|
Effect Showcase
|
|
We show examples of effects with different variations of intensity in the table below. Due to space constraint, the effect detail attributes are suppressed.
|
| Effect |
Mild |
Medium |
Strong |
| Shake |
 |
 |
 |
| Splash |
 |
 |
 |
| Gravity |
 |
 |
 |
| Light |
 |
 |
 |
| Wind |
 |
 |
 |
| Weather |
 |
 |
 |
| Temperature |
 |
 |
 |
| Liquid |
 |
 |
 |
Physical
Interaction |
 |
 |
 |
|
|
Label Explained
|
|
Toggle the text on specific effect to view the details.
The details of each GIF animation example are suppressed.
Shake
Shaking:
- This effect should be annotated when a force is imposed on the POV (character or camera) such that it is felt physically.
- Type: N/A.
- Direction: is divided into 5 categories.
- Left-to-right (e.g. a person shaking the character from the side)
- Front-to-back (e.g. a sudden break in the vehicle)
- Up-and-down (e.g. trampoline)
- All directions (e.g. rocket launch, or when the direction is not recognizable)
- Not clear (e.g. not visible in the frame, can't say)
Splash
Water Splashing:
- Water splashing is not limited to water. It can be any liquid in general. Whenever some liquid splashes onto the POV, the effect should be annotated.
- Type: N/A.
- Direction: is divided into 8 categories.
- From left
- From right
- From front
- From back
- From top (e.g. shower, diving headlong)
- From bottom (e.g. diving feet first in)
- All around (e.g. a blast of water from all sides)
- Not clear (e.g. not visible in the frame, can't say)
Gravity
Gravity:
- Gravity can be quite hard to detect. Usually gravity effects exist in sci-fi movies.
An annotation should be made only when the annotator is confident he understands the concept of gravity.
For example, a person sitting in rocket which is starting to take off, or, a person in an accelerating elevator that is going down will experience slightly lower than usual gravity (remember the feeling in your stomach?).
It can also be seen as a make-up effect to better describe the general motion experience of a character.
- Type: The types of gravity cover some basic effects.
- Zero gravity (e.g. free fall motion)
- Low gravity (e.g. accelerating towards the earth, video clips from the moon)
- High gravity (e.g. character in an elevator accelerating upwards)
- Direction: N/A
Light
Extra Lighting Effect:
- This effect is usually accompanied with extra lighting effects (possibly with color) in the clip.
- Type: The type annotates the color of the specific light, and a special one for lightning.
- White
- Red
- Yellow
- Orange
- Green
- Blue
- Purple
- Lightning (e.g. use when lightning occurs, instead of saying white light)
- Others
- Direction: is divided into 8 categories.
- From left
- From right
- From front
- From back
- From top
- From bottom
- All around
- Not clear (e.g. not visible in the frame, can't say)
Wind
Wind:
- Wind can be the result of natural weather phenomena, the result of traveling very fast, or the boom caused by an explosion.
- Type: Wind can be cold, hot or normal (when there is no obvious temperature difference compared to the environment of the wind).
- Hot (e.g. a tiger roaring in the character's face)
- Cold (e.g. in a sudden wind blows the camp away in the snowstorm)
- Normal (e.g. character puts his head out while in a fast car)
- Direction: Direction is divided into 8 categories.
- From left
- From right
- From front
- From back
- From top
- From bottom
- All around (e.g. walking in a really windy day)
- Not clear (e.g. not visible in the frame, can't say)
Weather
Weather:
- Weather effects are subject to all characters in the scene, and typically, POV should be camera. Whenever there is an interesting or sudden change in the weather, an annotation is needed. We are not interested in the weather effect if the character is not experiencing it and is for example, in a room.
- Type: The type includes some basic weather conditions.
- Sunny (i.e. annotate only extreme sunlight, like in a desert)
- Rain
- Snow
- Fog
- Windy
- Snowstorm
- Others
- Direction: N/A
Temperature
Ambient temperature:
- This effect indicates the temperature in the scene. Please annotate only when this is something special.
- Type: We are only interested in extreme hot and cold temperatures. If the ambient temperature doesn't have a dramatic change, you don't need to annotate this effect.
- High temperature (e.g. entering a bakery (hot room) from outside)
- Low temperature (e.g. sudden drop in room temperature due to a ghost :))
- Direction: N/A
Liquid Surrounding
Liquid surroundings:
- This effect indicates that the person is in a large body of liquid. For example, this could be a diving expedition.
- Type: We only consider two types
- Water (e.g. diver in a sea, swimming pool)
- Other liquid
- Direction: N/A
Physical Interaction
Physical Interaction:
- Physical interaction is different from the shaking effect (punching, twisting, etc.). This is usually another source of force that is applied more locally to the POV (typically character). The character is usually aware of which part of his body is experiencing the force.
- Type: There are multiple types for physical interactions. Try to distinguish the specific type for each effect. If there is no perfect choice, please use "Others" and fill in the comment box.
- Hit (e.g. punching)
- Pinch
- Twist (e.g. twist leg after jumping)
- Sting
- Rub
- Drag (e.g. drag a person by his feet)
- Massage
- Impact (i.e. sudden change in speed)
- Gunshot
- Others
- Direction: Direction is divided into 8 categories. If you can clearly observe the direction, please try to be specific. If it is not recognizable, you can choose the "Others" option.
- Back
- Chest
- Arm
- Leg
- Head
- Hands
- Feet
- Others
|
|
Credits: University of Toronto: Yuhao Zhou, Makarand Tapaswi, Sanja Fidler
|