Sven Dickinson
Research

Shape Perception

How do visual systems represent, recover, and recognize shape?

Karel Nepraš sculpture Great Dialogue, showing two human-like figures facing one another on pedestals
Figure 1. Great Dialogue, Karel Nepraš (from the cover of Object Categorization: Computer and Human Vision Perspectives, S. Dickinson, A. Leonardis, B. Schiele, and M. Tarr (eds.), Cambridge University Press, 2009.)

The Challenge of Shape Perception

Despite the unusual configuration of local geometric structure in the scene above, your visual system has the remarkable ability to recover an abstract representation of shape from this geometric “noise” and recognize the sculpture as depicting two human figures engaged in dialogue.

Contemporary AI systems can still struggle to recover the same abstract shape interpretation from complex visual structure, underscoring the gap between successful recognition and human-like shape perception.

Understanding how biological and artificial visual systems accomplish this remarkable feat has been the central goal of our research for over 40 years.

Overview

My research focuses on shape perception in computer and human vision: how visual systems organize image structure into meaningful shapes, recover 2-D and 3-D shape structure, represent those shapes, and use them to recognize and interact with objects.

The work spans shape representation, perceptual grouping and shape recovery, graph-based matching and recognition, human shape perception, and shape-based reasoning for action. A recurring theme is that shape is inherently multiscale and compositional, and that effective representations must capture both local structure and the organization of parts into larger wholes.

More recently, I have become interested in what modern learned vision systems have discovered about shape and perceptual organization, and in the relationship between computational and human shape perception.

Enduring questions about shape

Across our work, a small set of fundamental questions about shape has persisted even as the representations, methods, and applications have changed. These questions can be viewed both as stages in a computational account of shape perception—from grouping local image evidence, to representing and matching shape, to using it for recognition and action—and as enduring research problems that have motivated different representations and methods over time.

01

How should shape be represented?

Directed Distance Fields map an oriented point (position and direction) to visibility and, when visible, distance to the first surface intersection. From Fig. 1 of Aumentado-Armstrong, Tsogkas, Dickinson, and Jepson, Probabilistic Directed Distance Fields for Ray-Based Shape Representations, PAMI, 2025.

A useful shape representation should capture what is structurally important while being invariant to image translation, image rotation, image scale, rotation in depth (viewpoint), articulation, appearance, and within-class shape deformation. Early in my career, this led to qualitative volumetric part models and other part-based descriptions for generic object recognition. Later, this led to other symmetry-based representations, including shock graphs, bone graphs, blob and ridge graphs, medial surface graphs, and top-point graphs, all of which encoded topology and part structure rather than surface appearance. These representations may capture both topological structure—parts and their relations—and metric geometric structure, including the 3-D information needed to reason about physical objects. More recently, we have explored generative latent shape spaces and probabilistic directed distance fields as learnable geometric representations.

Our work develops structural shape representations, from part and symmetry descriptions to learned latent spaces, that preserve topology and part relations while supporting recognition across changes in appearance, viewpoint, articulation, and within-class shape variation.

Representative publications

Full publications list →
T. Aumentado-Armstrong, S. Tsogkas, S. Dickinson, and A. Jepson
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 47, November 2025, pp 10243-10261.
T. Aumentado-Armstrong, S. Tsogkas, S. Dickinson, and A. Jepson
International Journal of Computer Vision (IJCV), Volume 131, 2023, pp 1611–1641.
K. Siddiqi, A. Shokoufandeh, S. Dickinson, and S. Zucker
International Journal of Computer Vision (IJCV), Volume 30, 1999, pp 1–24.
S. Dickinson and D. Metaxas
International Journal of Computer Vision (IJCV), Vol. 13, No. 3, 1994, pp 1–20.
More →Less ↑
T. Aumentado-Armstrong, S. Tsogkas, S. Dickinson, and A. Jepson
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
D. Macrini, S. Dickinson, D. Fleet, and K. Siddiqi
Computer Vision and Image Understanding (CVIU), Vol. 115, No. 7, July 2011, pp 1044–1061.
F. Demirci, B. Platel, A. Shokoufandeh, L. Florack, and S. Dickinson
Journal of Mathematical Imaging and Vision (JMIV), Vol. 35, No. 2, 2009, pp 103–116.
A. Shokoufandeh, L. Bretzner, D. Macrini, M.F. Demirci, C. Jonsson, and S. Dickinson
Computer Vision and Image Understanding (CVIU), Volume 103, 2006, pp 139–154.
S. Dickinson, A. Pentland, and A. Rosenfeld
In K. Leibovic (ed.), Vision: A Convergence of Disciplines, Springer Verlag, NY, 1990, pp 398–421.
02

How does shape emerge from local image structure?

Spatial context flux encodes the relationship between image pixels and nearby skeletal points, supporting precise recovery of object skeletons from natural images. Detail from Fig. 1b of Xu, Wang, Tsogkas, Wan, Bai, Dickinson, and Siddiqi, DeepFlux for Skeleton Detection in the Wild, IJCV, 2021.

Before a visual agent can act on (or react to) a sensed image, it must understand the contents of the image, i.e., recognize the objects. Recognition, in turn, requires separating the objects from the background (and from each other). This is the problem of segmentation and grouping: grouping pixels, edges and contours, identifying symmetry and closure, inferring part structure, and building larger units from smaller ones. Much of our work in this area has explored how non-accidental relationships, such as symmetry, can guide the grouping of local image measurements into representations of parts and objects.

Our work shows how non-accidental geometric relations, especially symmetry, can constrain bottom-up grouping and support the recovery of meaningful object parts from local image evidence.

Representative publications

Full publications list →
Y. Xu, Y. Wang, S. Tsogkas, J. Wan, X. Bai, S. Dickinson, and K. Siddiqi
International Journal of Computer Vision (IJCV), Issue 4, 2021.
A. Levinshtein, C. Sminchisescu, and S. Dickinson
International Journal of Computer Vision (IJCV), Volume 104, Number 2, 2013, pp 117–134.
A. Levinshtein, A. Stere, K. Kutulakos, D. Fleet, S. Dickinson, and K. Siddiqi
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 31, No. 12, December 2009, pp 2290–2297.
K. Siddiqi, J. Zhang, D. Macrini, A. Shokoufandeh, S. Bioux, and S. Dickinson
Machine Vision and Applications (MVA), Vol. 19, No. 4, July 2008, pp 261–275.
More →Less ↑
S. Tsogkas and S. Dickinson
International Conference on Computer Vision (ICCV), 2017.
T. Lee, S. Fidler, and S. Dickinson
Proceedings, International Conference on Computer Vision (ICCV), Santiago, Chile, December 2015.
S. Dickinson, A. Levinshtein, P. Sala, and C. Sminchisescu
In Shape Perception in Human and Computer Vision: An Interdisciplinary Perspective, S. Dickinson and Z. Pizlo (Eds.), Springer Verlag, 2013.
T. Lee, S. Fidler, and S. Dickinson
Proceedings, International Conference on Computer Vision (ICCV), Sydney, Australia, December 2013.
A. Levinshtein, C. Sminchisescu, and S. Dickinson
International Journal of Computer Vision (IJCV), Volume 100, Number 1, 2012, pp 99–119.
M. Jamieson, A. Fazly, S. Stevenson, S. Dickinson, and S. Wachsmuth
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 32, No. 1, January 2010, pp 148–164.
03

How can shape be matched and recognized?

Two graphs are embedded into a common normed space, matched many-to-many using Earth Mover's Distance, and mapped back to explicit graph correspondences. From Fig. 2 of Demirci, Shokoufandeh, Keselman, Bretzner, and Dickinson, Object Recognition as Many-to-Many Feature Matching, IJCV, 2006.

Our early work in generic object recognition (or object categorization) involved matching shape representations recovered from the image to a priori object models. Many of these recovered shape representations could be represented as graphs, thereby formulating shape matching as graph matching. However, graph matching in this setting differed from traditional graph matching in two important ways: 1) since our recovered shape representations were often multi-scale or coarse-to-fine, our graphs were often directed acyclic graphs (DAGs); and 2) due to noise, occlusion, and within-class structural variation, correspondence was often many-to-many, rather than one-to-one. This led us to develop novel embeddings of graphs into geometric spaces where the complexity of inexact graph indexing and matching could be better managed through geometric algorithms.

Our graph-embedding methods recast inexact structural matching as a geometric problem, enabling efficient comparison of hierarchical shape descriptions while accommodating the many-to-many correspondences produced by occlusion, noise, and within-class variation.

Representative publications

Full publications list →
S. Dickinson
In S. Dickinson, A. Leonardis, B. Schiele, and M. Tarr (eds.), Object Categorization: Computer and Human Vision Perspectives, Cambridge University Press, 2009, pp 1–37.
F. Demirci, A. Shokoufandeh, Y. Keselman, L. Bretzner, and S. Dickinson
International Journal of Computer Vision (IJCV), Volume 69, Number 2, August 2006, pp 203–222.
A. Shokoufandeh, D. Macrini, S. Dickinson, K. Siddiqi, and S. Zucker
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Special Issue on Syntactic and Structural Pattern Recognition, Volume 27, Number 7, July 2005, pp 1125–1140.
S. Dickinson, A. Pentland, and A. Rosenfeld
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 14, No. 2, 1992, pp 174–198.
More →Less ↑
M. Demirci, Y. Osmanlioglu, A. Shokoufandeh, and S. Dickinson
Computer Vision and Image Understanding (CVIU), Special Issue on Graph-Based Representations, Vol. 115, No. 7, July 2011, pp 976–983.
D. Macrini, S. Dickinson, D. Fleet, and K. Siddiqi
Computer Vision and Image Understanding (CVIU), Vol. 115, No. 8, August 2011, pp 1187–1206.
F. Demirci, A. Shokoufandeh, and S. Dickinson
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 31, No. 5, May 2009, pp 944–952.
Y. Keselman and S. Dickinson
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Volume 27, Number 7, July 2005, pp 1141–1156.
Y. Keselman, A. Shokoufandeh, M. F. Demirci, and S. Dickinson
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2003.
A. Shokoufandeh, I. Marsic, and S. Dickinson
Image and Vision Computing (IVC), Volume 17, 1999, pp 445–460.
From Volumes to Views: An Approach to 3-D Object Recognition
S. Dickinson, A. Pentland, and A. Rosenfeld
Computer Vision and Image Understanding (CVIU), Vol. 55, No. 2, March 1992, pp 130–154.
04

How does shape support human visual perception?

An office line drawing scored by local ribbon symmetry (top), with the most-symmetric (middle) and least-symmetric (bottom) contour subsets used to study the role of local symmetry in human scene perception. Human performance in scene categorization is significantly better on the most-symmetric contour subsets. From Fig. 1 of Wilder, Rezanejad, Dickinson, Siddiqi, Jepson, and Walther, Neural Correlates of Local Parallelism During Naturalistic Vision, PLOS ONE, 2022.

Human observers can quickly recover meaningful shape and scene structure from remarkably impoverished images. This observation has brought our work toward the intersection of computer vision, psychophysics, and neuroscience. We have studied whether geometric regularities such as contour relationships, symmetry, local parallelism, and medial structure help explain rapid human scene perception, and whether computational measures of shape predict human behavioural and neural responses.

Our studies show that computational measures of contour organization, symmetry, local parallelism, and medial structure predict aspects of human scene categorization and neural response, providing testable links between geometric shape structure and human perception.

Representative publications

Full publications list →
M. Rezanejad, J. Wilder, D. B. Walther, A. D. Jepson, S. Dickinson, and K. Siddiqi
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 46, No. 4, April 2024, pp 2041-2053.
J. Wilder, M. Rezanejad, S. Dickinson, K. Siddiqi, A. Jepson, D. B. Walther
PLOS ONE, January 2022, pp 1–17.
J. Wilder, M. Rezanejad, S. Dickinson, K. Siddiqi, A. Jepson, and D. B. Walther
Cognition, Volume 182, 2019, pp 307–317.
J. Wilder, S. Dickinson, A. Jepson, and D. B. Walther
Journal of Vision, Volume 18, Number 8, 2018, pp 1–15.
More →Less ↑
M. Rezanejad, G. Downs, J. Wilder, D. B. Walther, A. Jepson, S. Dickinson, and K. Siddiqi
Proceedings, Conference on Cognitive Computational Neuroscience (CCN), Berlin, September 2019.
J. Wilder, M. Rezanejad, K. Siddiqi, A. Jepson, S. Dickinson, and D. B. Walther
Proceedings, Conference on Cognitive Computational Neuroscience (CCN), Berlin, September 2019.
M. Rezanejad, G. Downs, J. Wilder, D. B. Walther, A. Jepson, S. Dickinson, and K. Siddiqi
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Shape Perception in Human and Computer Vision: An Interdisciplinary Perspective
S. Dickinson and Z. Pizlo (Eds.)
Advances in Computer Vision and Pattern Recognition Series, Springer Verlag, 2013.
05

How can recovered shape support action?

Fast-Grasp’D generates contact-rich grasps by optimizing over the full degrees of freedom of the hand, while GraspIt! primarily produces fingertip grasps with fewer contacts and reduced stability. From Fig. 4 of Turpin, Zhong, Zhang, Zhu, Heiden, Macklin, Tsogkas, Dickinson, and Garg, Fast-Grasp’D: Dexterous Multi-finger Grasp Generation Through Differentiable Simulation, ICRA, 2023.

Visual structure is useful not only for recognition, but also for deciding how an agent should interact with the world. Physical interaction requires recovering aspects of 3-D geometry: where surfaces lie in space, how parts are arranged, and where contact can occur. This has motivated our work on affordances, dexterous grasping, navigation, and other problems in which geometry is tightly coupled to action. In these settings, a shape representation is valuable precisely because it captures what matters for reasoning, contact, manipulation, or movement.

Our work uses recovered geometric and functional structure to support action, from selecting visual landmarks and identifying tool affordances to generating contact-rich dexterous grasps.

Representative publications

Full publications list →
D. Turpin, T. Zhong, S. Zhang, G. Zhu, E. Heiden, M. Macklin, S. Tsogkas, S. Dickinson, A. Garg
International Conference on Robotics and Automation (ICRA), 2023.
D. Turpin, L. Wang, S. Tsogkas, S. Dickinson, and A. Garg
Robotics: Science and Systems (RSS), 2021.
P. Sala, R. Sim, A. Shokoufandeh, and S. Dickinson
IEEE Transactions on Robotics (RO), Vol. 22, No. 2, April 2006, pp 334–349.
J. Tsotsos, G. Verghese, S. Dickinson, M. Jenkin, A. Jepson, E. Milios, F. Nuflo, S. Stevenson, M. Black, D. Metaxas, S. Culhane, Y. Ye, and R. Mann
Image and Vision Computing (IVC), Special Issue on Vision for the Disabled, Vol. 16, 1998, pp 275–292.
More →Less ↑
D. Turpin, L. Wang, E. Heiden, Y.-C. Chen, M. Macklin, S. Tsogkas, S. Dickinson, and A. Garg
European Conference on Computer Vision (ECCV), 2022.
S. Dickinson, H. Christensen, J. Tsotsos, and G. Olofsson
Computer Vision and Image Understanding (CVIU), Vol. 67, No. 3, September 1997, pp 239–260.
E. Rivlin, S. Dickinson, and A. Rosenfeld
Computer Vision and Image Understanding (CVIU), Special Issue on Function-based Object Recognition, Vol. 62, No. 2, September 1995, pp 164–176.
S. Dickinson and L. Davis
IEEE Transactions on Robotics and Automation (RA), Vol. 6, No. 2, 1990, pp 232–242.
06 · Current directions

What have modern vision systems learned about shape?

We extract patch representations from frozen, pre-trained ViTs and fit linear probes to predict figure-ground assignment on natural images, excluding patches that span both foreground and background (shown in black). We evaluate probes on both natural images and cue-controlled stimuli where textures are randomly replaced. From Fig. 1 of Tangemann, Lo, Pizlo, Siddiqi, Walther, and Dickinson, Shape-based Figure-Ground Organization in Vision Transformers, NeurIPS 2026.

Modern foundation models have changed the landscape of computer vision, but their success raises a question that is especially interesting to us: what have they actually learned about shape?

Mid-level human vision has evolved to exploit how the regularities in our physical world project to nonaccidental features in images of our world, e.g., the spatial coherence of solid objects projects to nonaccidental closure, the continuity of surfaces projects to nonaccidental proximity and smoothness, and the ubiquitous symmetry of objects and their parts projects to symmetry. Have vision transformers learned to exploit these same regularities, and if so, to what extent does mid-level vision in today’s transformers align with mid-level vision in humans?

If vision transformers are not aligned with human mid-level vision, can we identify and explain their failure modes? Can these failure modes be addressed through targeted training? And if we can achieve alignment with human vision, what can vision transformers tell us about human mid-level vision? These questions connect many of the themes that have run through our research for decades. They also suggest a new role for modern vision models: not only as engineering systems to be improved, but as computational objects that can be probed to understand the representations underlying visual shape perception.

Our current work probes what vision transformers have learned about shape, whether their behaviour on controlled shape-perception tasks aligns with human behaviour, and how agreements and discrepancies can help us understand both artificial and biological shape perception.

Representative publications

Full publications list →
M. Tangemann, B. Lo, Z. Pizlo, K. Siddiqi, D. B. Walther, and S. Dickinson
Proceedings, Neural Information Processing Systems (NeurIPS), Sydney, Australia, December 2026.