A framework for learning robust, adaptive appearance models to be used for motion-based tracking of natural objects is proposed. The approach involves a mixture of stable image structure, learned over long time courses, along with 2-frame transient motion information and an outlier process. This class of appearance models is shown to have low storage requirements and that the model parameters can be learned efficiently with an on-line variant of the expectation-maximization (EM) algorithm. Two implementations of appearance models based on the framework presented here are developed. The first is based on the filter responses from a steerable pyramid. This model is used in a motion-based tracking algorithm to provide robustness in the face of image outliers, such as those caused by occlusions. It also provides the ability to adapt to natural changes in appearance, such as those due to facial expressions or variations in 3D pose. The second implementation is based on a robust representation of the color of the target object. Experimental results on a variety of natural image sequences of people moving within cluttered environments are shown, both for the wavelet- and color-based appearance models individually and in combination with one another. A quantitative analysis of the performance of the tracker is provided for one of the test sequences.

Algorithms for the estimation of binocular disparity are formulated using low-level visual processing. Typically, these approaches do not model discontinuities in disparity and thus perform poorly at occlusion boundaries and depth discontinuities. We seek to improve the accuracy of binocular stereo disparity estimates by explicitly modeling disparity discontinuities and occlusion. We focus on intermediate-level visual processing, and exploit the fact that many depth discontinuities correspond to monocular changes in intensity. Thus, we propose a segment-based approach to disparity estimation that combines parametric disparity models with gray-scale image segmentation. The segments represent hypothetical surface facets. The models of the disparity within each segment are used to: (1) find occlusion, and (2) merge neighbouring segments with coherent disparity. We begin by obtaining coarse disparity estimates using a gradient-based SSD technique. An intensity segmentation is obtained using a smoothing process based on robust statistics and then locating spatial outliers. Then, we fit variable-order parametric models to the coarse disparity estimates within each segment. The disparity models are used to locate occlusions by testing for pixels that are not binocularly visible. Next, we refine the parameters of the disparity models using robust estimation. This calculation is performed on a segment-by-segment basis, allowing for discontinuities at segment boundaries and for the presence of occluded pixels. We introduce a process to merge adjacent segments that can be described by a single disparity model. The merging process reduces the number of segments that correspond only to textural changes or intensity variation by joining segments likely to belong to the same surface. We evaluate the performance of our algorithm on a number of natural and synthetic image pairs.

We propose a framework for learning robust, adaptive, appearance models to be used for motion-based tracking of natural objects. The model adapts to slowly changing appearnce, and it maintains a natural measure of the stability of the observed image structure during tracking. By identifying stable properties of appearance, we can weight them more heavilty for motion estimation, while less stable properties can be proportionally downweighted. The appearance model involves a mixture of stable image structure, learned over long time courses, along with two-frame information and an outlier process. An online EM-algorithm is used to adapt the appearance model paramters over time. An implementation of this approach is developed for an appearance model based on the filter responses from a steerable pyramid. The model is used in a motion-based tracking algorithm to provide robustness in the face of image outliers, such as those caused by occlusions, while adapting to natural changes in appearance such as those due to facial expressions of variations in 3D pose.

Image alignment should be based on features that are robust with respect to phenomena such as global illumination changes, shading variations, and local highlights. While hue provides a natural degree of invariance to these phenomena, we show that it has several deficiencies in terms of gradient-based image alignment. To overcome these limitations, we introduce a 2D representation for hue that maintains a highly compressed representation for saturation. This allows the representation to model gray without sacrificing the desirable properties of hue. We show that our approach has a more consistent local domain of convergence when used for gradient-based alignment, and demonstrate its use in the context of a probabilistic motion-based tracker.

We consider the segmentation of a trajectory into piecewise polynomial parts, or possibly other forms. Segmentation is formulated as an optimization problem which trades off model fitting error versus the cost of introducing new segments. We present a novel dynamic programming approach that finds the optimal segmentation for trajectories. The approach is easily extended to handle different segment types and top down information about segment boundaries, when available. We show segmentation results for video sequences of a basketball undergoing gravitational and nongravitational motion.

We propose a framework for learning robust, adaptive, appearance models to be used for motion-based tracking of natural objects. The approach involves a mixture of stable image structure, learned over long time courses, along with 2-frame motion information and an outlier process. An on-line EM-algorithm is used to adapt the appearance model parameters over time. An implementation of this approach is developed for an appearance model based on the filter responses from a steerable pyramid. This model is used in a motion-based tracking algorithm to provide robustness in the face of image outliers, such as those caused by occlusions. It is also provides the ability to adapt to natural changes in appearance, such as those due to facial expressions or variations in 3D pose. We show experimental results on a variety of natural image sequences of people moving within cluttered environments.