Shape Perception

A useful shape representation should capture what is structurally important while being invariant to image translation, image rotation, image scale, rotation in depth (viewpoint), articulation, appearance, and within-class shape deformation. Early in my career, this led to qualitative volumetric part models and other part-based descriptions for generic object recognition. Later, this led to other symmetry-based representations, including shock graphs, bone graphs, blob and ridge graphs, medial surface graphs, and top-point graphs, all of which encoded topology and part structure rather than surface appearance. These representations may capture both topological structure—parts and their relations—and metric geometric structure, including the 3-D information needed to reason about physical objects. More recently, we have explored generative latent shape spaces and probabilistic directed distance fields as learnable geometric representations.

Our work develops structural shape representations, from part and symmetry descriptions to learned latent spaces, that preserve topology and part relations while supporting recognition across changes in appearance, viewpoint, articulation, and within-class shape variation.

Before a visual agent can act on (or react to) a sensed image, it must understand the contents of the image, i.e., recognize the objects. Recognition, in turn, requires separating the objects from the background (and from each other). This is the problem of segmentation and grouping: grouping pixels, edges and contours, identifying symmetry and closure, inferring part structure, and building larger units from smaller ones. Much of our work in this area has explored how non-accidental relationships, such as symmetry, can guide the grouping of local image measurements into representations of parts and objects.

Our work shows how non-accidental geometric relations, especially symmetry, can constrain bottom-up grouping and support the recovery of meaningful object parts from local image evidence.

Our early work in generic object recognition (or object categorization) involved matching shape representations recovered from the image to a priori object models. Many of these recovered shape representations could be represented as graphs, thereby formulating shape matching as graph matching. However, graph matching in this setting differed from traditional graph matching in two important ways: 1) since our recovered shape representations were often multi-scale or coarse-to-fine, our graphs were often directed acyclic graphs (DAGs); and 2) due to noise, occlusion, and within-class structural variation, correspondence was often many-to-many, rather than one-to-one. This led us to develop novel embeddings of graphs into geometric spaces where the complexity of inexact graph indexing and matching could be better managed through geometric algorithms.

Our graph-embedding methods recast inexact structural matching as a geometric problem, enabling efficient comparison of hierarchical shape descriptions while accommodating the many-to-many correspondences produced by occlusion, noise, and within-class variation.

Human observers can quickly recover meaningful shape and scene structure from remarkably impoverished images. This observation has brought our work toward the intersection of computer vision, psychophysics, and neuroscience. We have studied whether geometric regularities such as contour relationships, symmetry, local parallelism, and medial structure help explain rapid human scene perception, and whether computational measures of shape predict human behavioural and neural responses.

Our studies show that computational measures of contour organization, symmetry, local parallelism, and medial structure predict aspects of human scene categorization and neural response, providing testable links between geometric shape structure and human perception.

Visual structure is useful not only for recognition, but also for deciding how an agent should interact with the world. Physical interaction requires recovering aspects of 3-D geometry: where surfaces lie in space, how parts are arranged, and where contact can occur. This has motivated our work on affordances, dexterous grasping, navigation, and other problems in which geometry is tightly coupled to action. In these settings, a shape representation is valuable precisely because it captures what matters for reasoning, contact, manipulation, or movement.

Our work uses recovered geometric and functional structure to support action, from selecting visual landmarks and identifying tool affordances to generating contact-rich dexterous grasps.

Modern foundation models have changed the landscape of computer vision, but their success raises a question that is especially interesting to us: what have they actually learned about shape?

Mid-level human vision has evolved to exploit how the regularities in our physical world project to nonaccidental features in images of our world, e.g., the spatial coherence of solid objects projects to nonaccidental closure, the continuity of surfaces projects to nonaccidental proximity and smoothness, and the ubiquitous symmetry of objects and their parts projects to symmetry. Have vision transformers learned to exploit these same regularities, and if so, to what extent does mid-level vision in today’s transformers align with mid-level vision in humans?

If vision transformers are not aligned with human mid-level vision, can we identify and explain their failure modes? Can these failure modes be addressed through targeted training? And if we can achieve alignment with human vision, what can vision transformers tell us about human mid-level vision? These questions connect many of the themes that have run through our research for decades. They also suggest a new role for modern vision models: not only as engineering systems to be improved, but as computational objects that can be probed to understand the representations underlying visual shape perception.

Our current work probes what vision transformers have learned about shape, whether their behaviour on controlled shape-perception tasks aligns with human behaviour, and how agreements and discrepancies can help us understand both artificial and biological shape perception.