Jun Gao

I am a PhD. student in the Machine Learning Group at the University of Toronto, supervised by Professor Sanja Fidler. I am also a research scientist at NVIDIA Toronto AI Lab.

My research interests focus on the intersection of 3D computer vision and computer graphics, particularly developing machine learning tools to enable 3D content creation at scale and making impact in real-world applications.

I graduated from Peking University in 2018 with a Bachelor degree, where I was fortunate to work with Professor Liwei Wang. I also interned in Stanford, MSRA and NVIDIA.

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

profile photo
Featured Research

My research contributes to the fundamental problems in 3D content creation, from the 3D representation, to learning algorithm, and to interactive control with texts. My research enables generating high quality 3D meshes with textures and arbitrary topologies, from multi-view images, in a generative model, or from text prompts. Many of my work have been successfully depolyed in real-world applications, including GANVerse3D, Neural DriveSim, Toronto Annotation Suite etc.

Representative papers are highlighted, with full publication list in Google Scholar.

3D Generation
Magic3D: High-Resolution Text-to-3D Content Creation
Chen-Hsuan Lin*, Jun Gao*, Luming Tang*, Towaki Takikawa*, Xiaohui Zeng*, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu and Tsung-Yi Lin, (*, denote equal contribution)
CVPR, 2023
Project page / arXiv
We create high-quality 3D meshs with textures from text prompts, utilizing a two-stage pipeline with different diffusion models for fast and high-resolution text-to-3D generation.
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic and Sanja Fidler
NeurIPS, 2022
Project page / Code / PDF / arXiv / Two-Minute Paper Video (330k+ views)
We develop a 3D generative model to generate meshes with textures, bridging the success in the differentiable surface modeling, differentiable rendering and 2D GANs.
Extracting Triangular 3D Models, Materials, and Lighting From Images
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller and Sanja Fidler
CVPR, 2022   (Oral Presentation)
Project page / arXiv / PDF / Code / Two-Minute Paper Video (370k+ views)
Nvdiffrec reconstructs 3D mesh with materials from multi-view images by combining diff surface modeling with diff renderer. The method supports Nvidia Neural Drivesim.
Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
Yuxuan Zhang*, Wenzheng Chen*, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler (* denotes equal contribution)
ICLR, 2021   (Oral Presentation)
Project page / arXiv
We exploit pretrained 2D StyleGAN to generating "pseudo" multi-view images for training single image 3D reconstruction network. We deployed it into Omniverse AI Toybox (I led the effort).

2D/3D Representation
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu and Sanja Fidler
NeurIPS, 2021
Project page / arXiv / Video
DMTet differentiablely extracts the iso-surface from an implicit field via Marching Tetrahedra. We demonstrate the effectiveness in generating surface details from coarse voxels.
Learning Deformable Tetrahedral Meshes for 3D Reconstruction
Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan Mcguire, and Sanja Fidler
NeurIPS, 2020  
Project page / Code / arXiv
We introduce tetrahedral meshe representation into 3D vision to support generating mesh with arbitrary topologies. A differentiable renderer is also devised for DefTet.
Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid
Jun Gao, Zian Wang Jinchen Xuan, and Sanja Fidler
ECCV, 2020
Project page / arXiv / Code / Presentation Video (short, long)
We introduce a deformable grid representation for 2D images to better capture the high-frequency image content. We showcase applications including: replacing standard pooling methods with learnable geometric downsampling, semantic segmentation, object annotation, and unsupervised image partitioning.
Fast Interactive Object Annotation with Curve-GCN
Huan Ling*, Jun Gao*, Amlan Kar, Wenzheng Chen, and Sanja Fidler (* denotes equal contribution)
CVPR, 2019  
arXiv / Code/ Video / Demo
We utilize polygons to represent object boundaries, facilitating interactive object annotation. Our work has been successfully integrated into Toronto Annotation Suite, making real-world annotation faster!
DeepSpline: Data-Driven Reconstruction of Parametric Curves and Surfaces
Jun Gao, Chengcheng Tang, Vignesh Ganapathi-Subramanian, Jiahui Huang, Hao Su and Leonidas J. Guibas
arXiv (Non peer-reviewed, undergrad work)
arXiv / Code
We predict spline curves to represent images (e.g. MNIST digits), and further extend it with the surface of revolution to represent 3D objects (e.g. vase).

Differentiable Rendering
DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany and Sanja Fidler
NeurIPS, 2021
Project page / arXiv / bibtex
DIB-R++ combines rasterization and ray-tracing to differentiably render the image with specular reflections, enabling us to predict geometry, reflectance and lighting from single image.
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Wenzheng Chen Jun Gao* Huan Ling*, Edward J. Smith*, Jaakko Lehtinen, Alec Jacobson and Sanja Fidler (* denotes equal contribution)
NeurIPS, 2019  
Project page] / arXiv / Code/ Two-Minute Paper Video (240k+ views)
DIBR is an interpolation-based 3D mesh differentiable renderer that supports vertex, vertex color, multiple lighting models, texture mapping and could be easily embedded in neural network.
Academic Service
Area Chair: NeurIPS 2023

Conference Reviewer: CVPR, ICCV, ECCV, SIGGRAPH, NeurIPS, ICML, ICLR

Research Experience
Research Scientist at NVIDIA Dec. 2019 - Present

Research Intern at NVIDIA Oct. 2018 - Dec. 2019

Research Intern at Microsoft Research, Asia Sep. 2017 - Jun. 2018

Research Intern at Stanford University Jun. 2017 - Aug. 2017

Research Assistant at Peking University Apr. 2016 - Jun. 2018

Invited Talks
Towards Generative Modeling of 3D Objects Learned from Images. ETH, Oxford, JHU, UofT, PKU, BAAI (2022)

Learning Geometric Representation for Computer Vision. (GAMES-CN) Chinese [Link] (2020)

Learning Geometric Representation from Images. University of Alberta (2020)

Mentored Students/Interns
Tianchang Shen (PhD, CS UofT) Working on differentiable isosurfacing methods.

Weiwei Sun (PhD, CS UBC) Working on 3D generative models.

Zian Wang (PhD, CS UofT) Working on lighting decomposition with differentiable renderers.

Jinchen Xuan (Undergrad, CS PKU) Working on geometric image representation.

Gary Leung (MSc, CS UofT) Working on Transformer for images.

I borrowed the template from , ,,.