Jingkang Wang

PhD candidate
Machine Learning Group
Computer Science
University of Toronto

Email: wangjk (at) cs (dot) toronto (dot) edu

LinkedIn / Twitter / Google Scholar / CV

Bio

I am a Ph.D. candidate at University of Toronto. My advisor is Prof. Raquel Urtasun. I am also part of Vector Institute and a Sensor Researcher at Waabi. From 2019-2021, I was also a research scientist at Uber-ATG Toronto. I received my B.S. degree from Shanghai Jiao Tong University working with Prof. Cewu Lu and Prof. Gongshen Liu. Prior to UofT, I spent a wonderful year working with Prof. Bo Li at UIUC on trustworthy ML as a research intern.

I am interested in 3D vision and robotics, in particular developing simulation systems to capture the long-tail scenarios that rarely happen in the real world for safe self-driving vehicles. The goal is to build realistic, robust and scalable simulation systems for safe autonomy. Then we leverage them for comprehensive testing or robustness enhancement of full autonomy stack for safe deployment in real world.

News

Pro Bono: I commit 1 hour every week to provide guidance, suggestions, and/or mentorships for students from underrepresented groups or whoever is in need. Please fill in this form if you are interested.

^NEW 2024/07: G3R is accepted at ECCV 2024. Checkout our latest generalizable large reconstruction models for large scenes (full simulation capacities supported)!

^NEW 2024/05: Congrats to Ava on receiving the Jessie W.H. Zou Memorial Award!

2023/08: LightSim is accepted at NeurIPS 2023. Congrats to Ava and Gary! Checkout our latest neural lighting simulation system for self-driving.

2023/08: Adv3D is accepted at CoRL 2023. Congrats to Jay!

2023/07: Two papers are accepted at ICCV 2023! NeuRas: neural scene rasterization for large urban scenes and LiDAR-DG: paired evaluation paradigm to study lidar simulation domain gap.

2023/06: Waabi research hub online, check websites for UniSim, UltraLiDAR, NeuSim and CADSim!

2023/03: UniSim and UltraLiDAR are accepted at CVPR 2023. Check our latest works in closed-loop sensor simulation (LiDAR + Camera with full controllability) and LiDAR generation!

2023/01: NeuSim is accepted at ICRA 2023: in-the-wild neural object reconstruction.

2022/10: CADSim is accepted at CoRL 2022: part-aware inverse rendering from CAD templates.

2021/11: 2021 Baidu Fellowship Finalist.

2021/10: One paper is accepted at CoRL 2021: cost-aware active learning for autonomy.

2021/09: Two papers are accepted at NeurIPS 2021: min-max attack, robust loss in RL/IL.

2021/07: One paper is accepted at ICCV 2021: trustworthy multi-agent communication.

2021/06: Give a talk (Safety-critical scenario generation for full autonomy testing) in CVPR21 tutorial.

2021/03: Joined Waabi to work on sensor simulation for safe self driving.

2021/03: AdvSim is accepted at CVPR 2021. Check out our framework for mixed-reality LiDAR simulation and safety-critical scenario generation with full autonomy involved.

Publications (show selected)

My research interest is to scale the data-driven simulation (realistic, robust, scalable, generalizable, efficient) by inferring or grounding the 3D world (graphics, physics) to ensure the safe deployment of robotic systems.

	G3R: Generalizable Gradient-Guided Reconstruction Yun Chen, Jingkang Wang, Ze Yang, Sivabalan Manivasagam, Raquel Urtasun European Conference on Computer Vision (ECCV), 2024 project page / paper / video / bibtex A generalizable large reconstruction model for large dynamic scenes: reconstruct unseen large scenes in 2' and support real-time (FPS>100) full camera simulation.
	LightSim: Neural Lighting Simulation for Urban Scenes Ava Pun, Gary Sun, Jingkang Wang, Yun Chen, Ze Yang, Sivabalan Manivasagam, Wei-Chiu Ma, Raquel Urtasun Advances in Neural Information Processing Systems (NeurIPS), 2023* project page / paper / video / 4K demo / bibtex A data-driven neural lighting simulation system for urban scenes that generates diverse, controllable, and realistic videos.
	Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation Jay Sarva, Jingkang Wang, James Tu, Yuwen Xiong, Sivabalan Manivasagam, Raquel Urtasun Conference on Robot Learning (CoRL), 2023 project page / paper / video / poster / bibtex Closed-loop simulation on real-world scenarios to identify challenging shapes and failure modes.
	Towards Zero Domain Gap: A Comprehensive Study of Realistic LiDAR Simulation for Autonomy Testing Sivabalan Manivasagam, Ioan Andrei Bârsan, Jingkang Wang, Ze Yang, Raquel Urtasun International Conference on Computer Vision (ICCV), 2023 project page / paper / video / poster / bibtex We analyze the impact of LiDAR sensor effects on the full autonomy stack.
	Real-Time Neural Scene Rasterization for Large Scenes Jeffrey Liu, Yun Chen, Ze Yang, Jingkang Wang, Sivabalan Manivasagam, Raquel Urtasun International Conference on Computer Vision (ICCV), 2023 project page / paper / video / poster / bibtex Neural texture features + MLP shader for real-time rendering (>120 FPS for 1080p).
	UniSim: A Neural Closed-Loop Sensor Simulator Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight) project page / paper / video / poster / 4K demo / bibtex A data-driven closed-loop sensor simulator to generate realistic counter-factual scenarios from single driving pass.
	UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 project page / paper / video / poster / bibtex 3D World model for LiDAR: VQVAE tokenizer + autoregressive decoding.
	NeuSim: Reconstructing Objects in-the-wild for Realistic Sensor Simulation Ze Yang, Sivabalan Manivasagam, Yun Chen, Jingkang Wang, Rui Hu, Raquel Urtasun IEEE International Conference on Robotics and Automation (ICRA), 2023 project page / video / poster / bibtex Object reconstruction using physics-guided NeRF from sparse and noisy data.
	CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Simulation Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Anqi Joyce Yang, Wei-Chiu Ma, Raquel Urtasun Conference on Robot Learning (CoRL), 2022 project page / paper / video / bibtex Inverse rendering (articulated geometry, PBR material, lighting) from CAD templates via energy minimization.
	AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2021 project page / paper / poster / video / bibtex Safety-critical scenario generation (w.r.t full autonomy stack) using realistic mixed-reality LiDAR simulation and kinematic bicycle model.
	Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes Sean Segal, Nishanth Kumar, Sergio Casas, Wenyuan Zeng, Mengye Ren, Jingkang Wang, Raquel Urtasun Conference on Robot Learning (CoRL), 2021 paper / poster / bibtex Cost-aware active learning allows for fine-grained selection of examples through partially labeled scenes.
	Adversarial Attacks on Multi-Agent Communication James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, Raquel Urtasun International Conference on Computer Vision (ICCV), 2021 paper / video / bibtex Adversarial robustness in cooperative multi-agent systems.
	Policy Learning Using Weak Supervision Jingkang Wang, Hongyi Guo, Zhaowei Zhu, Yang Liu Advances in Neural Information Processing Systems (NeurIPS), 2021* project page / paper / code / poster / bibtex A framework with correlated aggreement to handle weak supervision in IL/RL.
	Adversarial Attack Generation Empowered by Min-Max Optimization Jingkang Wang, Tianyun Zhang, Sijia Liu, Pin-Yu Chen, Jiacen Xu, Makan Fardad, Bo Li Advances in Neural Information Processing Systems (NeurIPS), 2021 project page / paper / code / poster / bibtex A unified adversarial attack/defense by min-max optimization over multiple domains.
	Learning to Communicate and Correct Pose Errors Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, Raquel Urtasun Conference on Robot Learning (CoRL), 2020 paper / video / bibtex Robust multi-agent message passing under localization noise.
	Cost-Efficient Online Hyperparameter Optimization Jingkang Wang, Mengye Ren, Ilija Bogunovic, Yuwen Xiong, Raquel Urtasun International Conference on Machine Learning (ICML), RealML Workshop, 2020 paper / video / bibtex Cost-efficient time-varying Gaussian Process (GP).
	Zero-Shot Compositional Policy Learning via Language Grounding Tianshi Cao, Jingkang Wang, Annie Zhang, Sivabalan Manivasagam International Conference on Learning Representations (ICLR), BeTR-RL Workshop, 2020 paper / code / media / bibtex Language grounding to improve the generalization of agents across environments with varied dynamics.
	Reinforcement Learning with Perturbed Rewards Jingkang Wang, Yang Liu, Bo Li AAAI Conference on Artificial Intelligence (AAAI), 2020 (Spotlight) paper / code / bibtex An unbiased reward estimator that enables RL agent to learn in the noisy environment when only observing perturbed rewards.
	Multiple Character Embeddings for Chinese Word Segmentation Jingkang Wang, Jianing Zhou, Jie Zhou, Gongshen Liu Annual Meeting of the Association for Computational Linguistics (ACL), Student Research Workshop, 2019 paper / code / bibtex Multiple character embeddings (pinyin and wubi) to enhance CWS.
	LiDAR-Video Driving Dataset: Learning Driving Policies Effectively Yiping Chen, Jingkang Wang, Jonathan Li, Cewu Lu, Zhipeng Luo, Han Xue, Cheng Wang Conference on Computer Vision and Pattern Recognition (CVPR), 2018 project page / paper / code / bibtex End-to-end driving policy learned from LiDAR and camera data.
	G3R: Generalizable Gradient-Guided Reconstruction Yun Chen, Jingkang Wang, Ze Yang, Sivabalan Manivasagam, Raquel Urtasun European Conference on Computer Vision (ECCV), 2024 project page / paper / video / bibtex A generalizable large reconstruction model for large dynamic scenes: reconstruct unseen large scenes in 2' and support real-time (FPS>100) full camera simulation.
	LightSim: Neural Lighting Simulation for Urban Scenes Ava Pun, Gary Sun, Jingkang Wang, Yun Chen, Ze Yang, Sivabalan Manivasagam, Wei-Chiu Ma, Raquel Urtasun Advances in Neural Information Processing Systems (NeurIPS), 2023* project page / paper / video / 4K demo / bibtex A data-driven neural lighting simulation system for urban scenes that generates diverse, controllable, and realistic videos.
	UniSim: A Neural Closed-Loop Sensor Simulator Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight) project page / paper / video / poster / 4K demo / bibtex A data-driven closed-loop sensor simulator to generate realistic counter-factual scenarios from single driving pass.
	CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Simulation Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Anqi Joyce Yang, Wei-Chiu Ma, Raquel Urtasun Conference on Robot Learning (CoRL), 2022 project page / paper / video / bibtex Inverse rendering (articulated geometry, PBR material, lighting) from CAD templates via energy minimization.
	AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2021 project page / paper / poster / video / bibtex Safety-critical scenario generation (w.r.t full autonomy stack) using realistic mixed-reality LiDAR simulation and kinematic bicycle model.

Honors and Awards

Baidu Fellowship Finalist (Top 20)

2021

Outstanding Reviewer, CVPR21

2021

National Scholarships

2016, 2017, 2018

SJTU Scholarships (%1)

2016, 2017, 2018

Excellent Bachelor Thesis (Top %1) of SJTU

2019

Outstanding Undergraduate in Shanghai

2019

First Prize in National College Student Information Security Contest

2018

Meritorious Winner Prize in The Mathematical Contest in Modeling (MCM)

2018

Second Prize in The Chinese Mathematics Competition (CMC, Shanghai)

2017

Second Prize in National College Students Information Security Contest

2017

First Prize in Chinese Mathematical Olympiad (CMO, 10th in Shanxi)

2014

Work Experience Waabi Innovation Inc. Senior Researcher Advisor: Professor Raquel Urtasun Focus: Sensor Simulation, Self-Driving Safety
Uber-ATG Toronto Research Scientist Advisor: Professor Raquel Urtasun Focus: Sensor Simulation, Self-Driving Safety
University of Illinois at Urbana-Champaign Research Intern at Secure Learning Lab (remote work) Advisor: Professor Bo Li Focus: Trustworthy Machine Learning
University of California, Berkeley Research Intern at BAIR Lab (remote work) Advisor: Professors Bo Li and Dawn Song Focus: Trustworthy Machine Learning

Mentorship

Ava Pun (Undergrad, CS UWaterloo) neural lighting simulation and inverse rendering, now PhD at CMU.
Gary Sun (Undergrad, CS UWaterloo) neural lighting simulation, now Researcher at Citadel.
Jay Sarva (Undergrad, CS Brown) adversarial closed loop simulation, now visiting Researcher at Harvard.
Rishi Menon (Undergrad, CS UWaterloo) generalizable asset reconstruction, now Engineer at Waabi.

Service

Conference reviewer: ICLR 2024, NeurIPS 2022-2023, ICML 2024, ICCV 2021-2023, CVPR 2021-2024, ECCV 2022-2024, WACV 2024, CoRL 2022, ICRA 2023, IROS 2023, AutoML 2023, ACL 2021-2022, NAACL 2022, EMNLP 2021, KDD 2020

Journal reviewer: IEEE TIP, IEEE TNNLS, CVIU, T-ITS, IEEE TSP, RA-L

Last update: June, 2024.

Thanks jonbarron for this amazing work.

	G3R: Generalizable Gradient-Guided Reconstruction Yun Chen, Jingkang Wang, Ze Yang, Sivabalan Manivasagam, Raquel Urtasun European Conference on Computer Vision (ECCV), 2024 project page / paper / video / bibtex A generalizable large reconstruction model for large dynamic scenes: reconstruct unseen large scenes in 2' and support real-time (FPS>100) full camera simulation.
	LightSim: Neural Lighting Simulation for Urban Scenes Ava Pun, Gary Sun, Jingkang Wang, Yun Chen, Ze Yang, Sivabalan Manivasagam, Wei-Chiu Ma, Raquel Urtasun Advances in Neural Information Processing Systems (NeurIPS), 2023* project page / paper / video / 4K demo / bibtex A data-driven neural lighting simulation system for urban scenes that generates diverse, controllable, and realistic videos.
	Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulation Jay Sarva, Jingkang Wang, James Tu, Yuwen Xiong, Sivabalan Manivasagam, Raquel Urtasun Conference on Robot Learning (CoRL), 2023 project page / paper / video / poster / bibtex Closed-loop simulation on real-world scenarios to identify challenging shapes and failure modes.
	Towards Zero Domain Gap: A Comprehensive Study of Realistic LiDAR Simulation for Autonomy Testing Sivabalan Manivasagam, Ioan Andrei Bârsan, Jingkang Wang, Ze Yang, Raquel Urtasun International Conference on Computer Vision (ICCV), 2023 project page / paper / video / poster / bibtex We analyze the impact of LiDAR sensor effects on the full autonomy stack.
	Real-Time Neural Scene Rasterization for Large Scenes Jeffrey Liu, Yun Chen, Ze Yang, Jingkang Wang, Sivabalan Manivasagam, Raquel Urtasun International Conference on Computer Vision (ICCV), 2023 project page / paper / video / poster / bibtex Neural texture features + MLP shader for real-time rendering (>120 FPS for 1080p).
	UniSim: A Neural Closed-Loop Sensor Simulator Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight) project page / paper / video / poster / 4K demo / bibtex A data-driven closed-loop sensor simulator to generate realistic counter-factual scenarios from single driving pass.
	UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 project page / paper / video / poster / bibtex 3D World model for LiDAR: VQVAE tokenizer + autoregressive decoding.
	NeuSim: Reconstructing Objects in-the-wild for Realistic Sensor Simulation Ze Yang, Sivabalan Manivasagam, Yun Chen, Jingkang Wang, Rui Hu, Raquel Urtasun IEEE International Conference on Robotics and Automation (ICRA), 2023 project page / video / poster / bibtex Object reconstruction using physics-guided NeRF from sparse and noisy data.
	CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Simulation Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Anqi Joyce Yang, Wei-Chiu Ma, Raquel Urtasun Conference on Robot Learning (CoRL), 2022 project page / paper / video / bibtex Inverse rendering (articulated geometry, PBR material, lighting) from CAD templates via energy minimization.
	AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2021 project page / paper / poster / video / bibtex Safety-critical scenario generation (w.r.t full autonomy stack) using realistic mixed-reality LiDAR simulation and kinematic bicycle model.
	Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes Sean Segal, Nishanth Kumar, Sergio Casas, Wenyuan Zeng, Mengye Ren, Jingkang Wang, Raquel Urtasun Conference on Robot Learning (CoRL), 2021 paper / poster / bibtex Cost-aware active learning allows for fine-grained selection of examples through partially labeled scenes.
	Adversarial Attacks on Multi-Agent Communication James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, Raquel Urtasun International Conference on Computer Vision (ICCV), 2021 paper / video / bibtex Adversarial robustness in cooperative multi-agent systems.
	Policy Learning Using Weak Supervision Jingkang Wang, Hongyi Guo, Zhaowei Zhu, Yang Liu Advances in Neural Information Processing Systems (NeurIPS), 2021* project page / paper / code / poster / bibtex A framework with correlated aggreement to handle weak supervision in IL/RL.
	Adversarial Attack Generation Empowered by Min-Max Optimization Jingkang Wang, Tianyun Zhang, Sijia Liu, Pin-Yu Chen, Jiacen Xu, Makan Fardad, Bo Li Advances in Neural Information Processing Systems (NeurIPS), 2021 project page / paper / code / poster / bibtex A unified adversarial attack/defense by min-max optimization over multiple domains.
	Learning to Communicate and Correct Pose Errors Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, Raquel Urtasun Conference on Robot Learning (CoRL), 2020 paper / video / bibtex Robust multi-agent message passing under localization noise.
	Cost-Efficient Online Hyperparameter Optimization Jingkang Wang, Mengye Ren, Ilija Bogunovic, Yuwen Xiong, Raquel Urtasun International Conference on Machine Learning (ICML), RealML Workshop, 2020 paper / video / bibtex Cost-efficient time-varying Gaussian Process (GP).
	Zero-Shot Compositional Policy Learning via Language Grounding Tianshi Cao, Jingkang Wang, Annie Zhang, Sivabalan Manivasagam International Conference on Learning Representations (ICLR), BeTR-RL Workshop, 2020 paper / code / media / bibtex Language grounding to improve the generalization of agents across environments with varied dynamics.
	Reinforcement Learning with Perturbed Rewards Jingkang Wang, Yang Liu, Bo Li AAAI Conference on Artificial Intelligence (AAAI), 2020 (Spotlight) paper / code / bibtex An unbiased reward estimator that enables RL agent to learn in the noisy environment when only observing perturbed rewards.
	Multiple Character Embeddings for Chinese Word Segmentation Jingkang Wang, Jianing Zhou, Jie Zhou, Gongshen Liu Annual Meeting of the Association for Computational Linguistics (ACL), Student Research Workshop, 2019 paper / code / bibtex Multiple character embeddings (pinyin and wubi) to enhance CWS.
	LiDAR-Video Driving Dataset: Learning Driving Policies Effectively Yiping Chen, Jingkang Wang, Jonathan Li, Cewu Lu, Zhipeng Luo, Han Xue, Cheng Wang Conference on Computer Vision and Pattern Recognition (CVPR), 2018 project page / paper / code / bibtex End-to-end driving policy learned from LiDAR and camera data.
	G3R: Generalizable Gradient-Guided Reconstruction Yun Chen, Jingkang Wang, Ze Yang, Sivabalan Manivasagam, Raquel Urtasun European Conference on Computer Vision (ECCV), 2024 project page / paper / video / bibtex A generalizable large reconstruction model for large dynamic scenes: reconstruct unseen large scenes in 2' and support real-time (FPS>100) full camera simulation.
	LightSim: Neural Lighting Simulation for Urban Scenes Ava Pun, Gary Sun, Jingkang Wang, Yun Chen, Ze Yang, Sivabalan Manivasagam, Wei-Chiu Ma, Raquel Urtasun Advances in Neural Information Processing Systems (NeurIPS), 2023* project page / paper / video / 4K demo / bibtex A data-driven neural lighting simulation system for urban scenes that generates diverse, controllable, and realistic videos.
	UniSim: A Neural Closed-Loop Sensor Simulator Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2023 (Highlight) project page / paper / video / poster / 4K demo / bibtex A data-driven closed-loop sensor simulator to generate realistic counter-factual scenarios from single driving pass.
	CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Simulation Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Anqi Joyce Yang, Wei-Chiu Ma, Raquel Urtasun Conference on Robot Learning (CoRL), 2022 project page / paper / video / bibtex Inverse rendering (articulated geometry, PBR material, lighting) from CAD templates via energy minimization.
	AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun Conference on Computer Vision and Pattern Recognition (CVPR), 2021 project page / paper / poster / video / bibtex Safety-critical scenario generation (w.r.t full autonomy stack) using realistic mixed-reality LiDAR simulation and kinematic bicycle model.