Automated detection of anomalous events on stairs

Description

Stairs have always been dangerous and risky for people to maneuver. The U.S. National Electronic Injury Surveillance System estimates that over one million people received hosipital treatment due to stair or step related injuries in the U.S. in 2005. The ultimate goal of this project is to build a system that can watch stairs with only a camera and automatically detect any anomalous (unsafe) events that may occur. The real-world motivation for this is to help further the understanding of the causes and catalysts of injuries on stairs and prevent accidents on stairs. Furthermore, such a system can help promote independence for older adults as their inability to safely traverse stairs greatly impedes their ability to live independently at home. The system makes use of advanced computer vision and machine learning algorithms to distinguish normal stair traversals from anomalous ones. This research is being done at the Intelligent Assistive Technology and Systems Lab at the University of Toronto in conjunction with the Machine Learning Group.

The system uses a number of computer vision algorithms to identify and represent motion of people walking down the stairs. Optical flow and background subtraction are used to compute the overall motion of the person's body. The person's feet are automatically tracked as they descend the stairs using a mixed state particle filter. Machine learning techniques (Hidden Markov Models) are then applied to this data to classify any motion which is anomalous (and therefore potentially dangerous). This has given us strong anomalous event detection results for multiple test subjects.

Research Team

Jasper Snoek - University of Toronto
Jesse Hoey - University of Dundee
Liam Stewart - University of Toronto
Richard S. Zemel - University of Toronto
Alex Mihailidis - University of Toronto

Videos

One very interesting aspect of the system is tracking feet as they decend the stairs. Features derived from the motion of the feet are used to identify anomalous events. We created a model of the appearance of the feet using histograms of the orientations of gradients around the feet. Then we tracked the feet using a particle filtering (importance sampling) algorithm with mixed state dynamics. Here are a number of videos displaying the performance of the system on some test data. All of these videos use the Microsoft MPEG4 V2 video codec.

1: A video showing the positions of a person's feet as tracked by our foot tracker during a number of normal stair descents. The positions of the right and left feet are shown as a white ‘+’ and ‘X’ respectively.

2: The positions of a person's feet as tracked by our foot tracker during a number of anomalous stair descents. Tracking the feet in these sequences is particularly challenging due to occlusions of the feet and irregular motions. The positions of the right and left feet are shown as a white ‘+’ and ‘X’ respectively.

3: The progression of the sample set for the right foot as the foot descends the stairs. This descent is interesting because the foot becomes fully occluded by the right knee during a large misstep. As a result the sample set diverges and briefly becomes bimodal before reconverging on the foot. Each sample is shown as a white dot.

4: The progression of the sample sets for both the right and left feet as the person descends the stairs. This descent is interesting because right foot misses a step causing the left foot to be occluded. A random one third of each of the sample sets is shown.

5: A video showing how the feet are initially found for a number of different stair descents. This video shows how the sparse set of samples (hypothesized locations of the feet) over the top of the staircase quickly converge on the feet as the feet are found. Each red dot indicates a sample for the right foot and each blue dot indicates a sample for the left foot. The video shows the first 5 frames of each descent.

6: This video shows how the feet are initially found under more difficult circumstances for a number of different stair descents. In these cases the feet start at various positions around the top of the stairs. This video shows how the sparse set of samples (hypothesized locations of the feet) over the top of the staircase quickly converge on the feet as the feet are found. Each red dot indicates a sample for the right foot and each blue dot indicates a sample for the left foot. The video shows the first 5 frames of each descent.

7: A heel slip that the automated system missed (a missed anomaly) from our experiments. This is a very difficult anomalous event to detect even by a human.