CADSim: Robust and Scalable in-the-wild
3D Reconstruction for Controllable Sensor Simulation

In Conference on Robot Learning (CoRL), 2022

CADSim recovers shape, appearance and illumination in a robust and scalable way from sensor observations. The reconstructed assets are of high fidelity, part-aware, geometry-aligned and compatible with graphics engine, enabling efficient, realistic and controllable simulation.



Realistic simulation is key to enabling safe and scalable development of self-driving vehicles. A core component is simulating the sensors so that the entire autonomy system can be tested in simulation. Sensor simulation involves modeling traffic participants, such as vehicles, with high-quality appearance and articulated geometry, and rendering them in real-time. The self-driving industry has employed artists to build these assets. However, this is expensive, slow, and may not reflect reality. Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set that provides good real-world coverage. However, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise. To tackle these issues, we present CADSim which combines part-aware object-class priors via a small set of CAD models with differentiable rendering to automatically reconstruct vehicle geometry, including articulated wheels, with high-quality appearance. Our experiments show our approach recovers more accurate shape from sparse data compared to existing approaches. Importantly, it also trains and renders efficiently. We demonstrate our reconstructed vehicles in a wide range of applications, including accurate testing of autonomy perception systems.

Examples (baked texture) CADSim Applications (NVS, Mixed Reality CameraSim)

  author    = {Wang, Jingkang and Manivasagam, Sivabalan and Chen, Yun and Yang, Ze, and and Yang, Anqi and Bârsan Andrei and Ma, Wei-Chiu and Urtasun, Raquel}
  title     = {CADSim: Robust, Scalable and Controllable in-the-wild 3D Reconstruction for Sensor Simulation},
  booktitle = {{CoRL}},
  year      = {2022}