Non-Rigid Structure from Locally-Rigid Motion

 
NOTE: New code is below, but it is advised to check back over the next month as I verify things all compile and work as they should.
 
We introduce locally-rigid motion, a general framework for solving the M-point, N-view structure-from-motion problem for unknown bodies deforming under orthography. The key idea is to first solve many local 3-point, N-view rigid problems independently, providing a "soup" of specific, plausibly rigid, 3D triangles. The main advantage here is that the extraction of 3D triangles requires only very weak assumptions: (1) deformations can be locally approximated by near-rigid motion of three points (i.e., stretching not dominant) and (2) local motions involve some generic rotation in depth. Triangles from this soup are then grouped into bodies, and their depth flips and instantaneous relative depths are determined. Results on several sequences, both our own and from related work, suggest these conditions apply in diverse settings---including very challenging ones (e.g., multiple deforming bodies). Our starting point is a novel linear solution to 3-point structure from motion, a problem for which no general algorithms currently exist.
 
People
 
Allan D. Jepson (University of Toronto)
Jonathan Taylor (University of Toronto)
Kiriakos N. Kutulakos (University of Toronto)
 
Related Publications
  • Jonathan Taylor, Allan D. Jepson and Kiriakos N. Kutulakos, Non-Rigid Structure from Locally-Rigid Motion. Proc. Computer Vision and Pattern Recognition Conf., San Francisco, CA, 2010. [pdf]
  • Derivations and details on legacy edge functional and alignment costs (see below). [pdf]
  • Local Rigidity Code Documentation including details of deviations from the publication above. [pdf]
Code
 
A new version (July 2012) of this code lrigid-0.2.0.tar.gz is available with accompanying documentation explaining how to control the parameters. Please see README.txt in the package for a list of dependencies and howto run on Ubuntu (although other platforms may work). To use QPBO (-q option), you must install pyqpbo-0.0.1.tar.gz.
 
Image Sequences
 
tear.tar.gz
paper.tar.gz Sequence originally used by M. Salzmann, R. Hartley, P. Fua, "Convex Optimization for Deformable Surface 3-D Tracking" Proc. ICCV 2007

Feature trajectories provided by Y. Tian and S. Narasimhan, "A Globally-Optimal Data-Driven Approach for Image Distortion Estimation," Proc. CVPR 2010
scarf.tar.gz Feature trajectories provided by Y. Tian and S. Narasimhan, "A Globally-Optimal Data-Driven Approach for Image Distortion Estimation," Proc. CVPR 2010
two_cloths.tar.gz Feature trajectories provided by Y. Tian and S. Narasimhan, "A Globally-Optimal Data-Driven Approach for Image Distortion Estimation," Proc. CVPR 2010
 
Motion Capture Sequences
 
wind.tar.gz
jacky.tar.gz Sequence from same capture session used by L. Torresani, A. Hertzmann and C. Bregler, "Non-Rigid Structure-From-Motion: Estimating Shape and Motion with Hierarchical Priors" IEEE Trans. on PAMI, May 2008.
bend.tar.gz
rip.tar.gz
two_papers.tar.gz
Acknowledgements
 
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under the RGPIN, Discovery and PGS-D programs.