Interactive Annotation of 3D Object Geometry
using 2D Scribbles

Tianchang Shen*
Jun Gao*
Amlan Kar
Sanja Fidler

ECCV, 2020

Interactive 3D Annotation Framework: User selects an object by indicating it with a mask. Our framework consists of two interaction modules: 1) large errors: user provides feedback by drawing scribbles in desired 2D views -- we exploit this to predict a better shape, 2) small errors: user drags & drops mesh vertices, and we repredict the mesh locally. Invisible object regions are up to user's priors/imagination.

Inferring detailed 3D geometry of the scene is crucial for robotics applications, simulation, and 3D content creation. However, such information is hard to obtain, and thus very few datasets support it. In this paper, we propose an interactive framework for annotating 3D object geometry from both point cloud data and RGB imagery. The key idea behind our approach is to exploit strong priors that humans have about the 3D world in order to interactively annotate complete 3D shapes. Our framework targets a wide pool of annotators, \ie naive users without artistic or graphics expertise. In particular, we introduce two simple-to-use interaction modules. First, we make an automatic guess of the 3D shape and allow the user to provide feedback about large errors by drawing scribbles in desired 2D views. Next, we aim to correct minor errors, in which users drag and drop 3D mesh vertices, assisted by a neural interactive module implemented as a Graph Convolutional Network. Experimentally, we show that only a few user interactions are needed to produce good quality 3D shapes on popular benchmarks such as ShapeNet, Pix3D and ScanNet. We implement our framework as a web service and conduct a user study, where we show that user annotated data using our method effectively facilitates real-world learning tasks.


Tianchang Shen*, Jun Gao*, Amlan Kar, Sanja Fidler

Interactive Annotation of 3D Object Geometry using 2D Scribbles


Please address correspondence regarding this paper to Sanja Fidler.


Demo Video


Examples of ShapeNet objects annotated (in simulation) using our framework. First two rows are from seen classes while others are from unseen classes.

Full-scene 3D annotation (top) and CAD alignment (bottom) on ScanNet.
Example shapes (Pix3D) annotated by real human using our web tool. Each example is from a different object category which is unseen to our base model.

We thank David Acuna for his website template.