Modeling Appearance Change in Image Sequences

Research Overview

Most methods for estimating optical flow assume that image appearance of the moving objects remains more or less constant. The most common assumption, referred to as brightness constancy, posits that the light reflected from a moving surface toward the camera (surface irradiance) remains constant from on frame to the next. Unfortunately there are many cases where this assumption is violated, causing poor motion estimates; examples include occlusions, changing orientations of surfaces with respect to light sources, specularities, and shadows.

To deal with complex forms of appearance changes, Michael Black, Yaser Yacoob and I proposed a formulation in which appearance changes were modeled by a linear mixture of causes. These included photometric causes such as lighting variations and specularities, as well as object-specific variations in appearance caused by complex local patterns of motion and occlusion. This research represents an encouraging first step toward the inference of the different physical causes of intensity variations in image sequences, but there exist many open problems in this area.

In subsequent work, Horst Haussecker and I derived a linear formulation of a more restricted class of appearance changes based on physical models of image formation and photometry. Such physical models include changes in surface reflection caused by changes in surface orientation with respect to a direction light source, and thermal diffusion in infrared (IR) imaging. The latter formulation was used for tracking unmarked paper in a copier with laser-induced thermal emittance patterns on the paper surface.

Related Publications

Return to David Fleet's home page.