Renjie Liao  

I'm a PhD candidate in Department of Computer Science, University of Toronto.
I also work at Uber Advanced Technology Group and Vector Institute.

My research interests are machine learning, computer vision, and self-driving.
I recently focused on deep probabilistic models of (graph) structured data and its applications. Highlights include:

  • Deep generative models of graphs: Graph Recurrent Attention Networks (GRAN)
  • Multi-scale spectral graph convolutional networks: LanczosNet
  • Implicit differentiation: Recurrent Back-Propagation (RBP)
  • Differentiable bipartite matching: DMM-Net

  • My advisors are Richard Zemel and Raquel Urtasun. My M.Phil. advisor was Jiaya Jia at the Chinese University of Hong Kong, where I worked mostly on computer vision. I got B.Eng. degree from Beihang University. I did internships at Microsoft Research Cambridge and Asia. I had the good fortune of working with some smart "neural networks".

    Curriculum Vitae
    Email: rjliao at cs dot toronto dot edu



    "There is nothing as practical as a good theory." - Kurt Lewin
    "The test of all knowledge is experiment." - Richard Feynman

    Selected Publications

    * below indicates equal contribution
    Link
    Efficient Graph Generation with Graph Recurrent Attention Networks
    Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Charlie Nash, William L. Hamilton, David Duvenaud, Raquel Urtasun, Richard S. Zemel
    Neural Information Processing Systems (NeurIPS), 2019

    [Code]

    Link
    Incremental Few-Shot Learning with Attention Attractor Networks
    Mengye Ren, Renjie Liao, Ethan Fetaya, Richard S. Zemel
    Neural Information Processing Systems (NeurIPS), 2019

    [Code]

    Link
    DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation
    Xiaohui Zeng*, Renjie Liao*, Li Gu, Yuwen Xiong, Sanja Fidler, Raquel Urtasun
    IEEE International Conference on Computer Vision (ICCV), 2019

    [Code] [Video]

    Link
    Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction
    Ajay Jain*, Sergio Casas Romero*, Renjie Liao*, Yuwen Xiong*, Song Feng, Sean Segal, Raquel Urtasun
    Conference on Robot Learning (CoRL), 2019

    [Code Coming Soon!]

    Link
    Lorentzian Distance Learning for Hyperbolic Representations
    Marc T. Law, Renjie Liao, Jake Snell, Richard S. Zemel
    International Conference on Machine Learning (ICML), 2019

    [Code]

    Link
    Inference in Probabilistic Graphical Models by Graph Neural Networks
    KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard S. Zemel, Xaq Pitkow
    ICML Workshop on Tractable Probabilistic Modeling, 2019

    [ICLR 2018 Workshop][Code Coming Soon!] [Best Paper Award]

    Link
    UPSNet: A Unified Panoptic Segmentation Network
    Yuwen Xiong*, Renjie Liao*, Hengshuang Zhao*, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    [Code] [Oral Presentation, 288/5160 (5.6%)]

    Link
    DARNet: Deep Active Ray Network for Building Segmentation
    Dominic Cheng, Renjie Liao, Sanja Fidler, Raquel Urtasun
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    [Code]

    Link
    LanczosNet: Multi-Scale Deep Graph Convolutional Networks
    Renjie Liao, Zhizhen Zhao, Raquel Urtasun, Richard S. Zemel
    International Conference on Learning Representations (ICLR), 2019

    [Code] [NeurIPS 2018 R2L Workshop] [Score Rank: 69/1591 (4.4%)]

    Link
    Neural Guided Constraint Logic Programming for Program Synthesis
    Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, William E. Byrd, Matthew Might, Raquel Urtasun, Richard S. Zemel
    Neural Information Processing Systems (NeurIPS), 2018

    [Code] [ICLR 2018 Workshop]

    Link
    Reviving and Improving Recurrent Back-Propagation
    Renjie Liao*, Yuwen Xiong*, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, Richard S. Zemel
    International Conference on Machine Learning (ICML), 2018

    [Code] [Video] [Full Oral Presentation, 212/2473 (8.6%)]

    Link
    Graph Partition Neural Networks for Semi-Supervised Classification
    Renjie Liao, Marc Brockschmidt, Daniel Tarlow, Alexander Gaunt, Raquel Urtasun, Richard S. Zemel
    International Conference on Learning Representations Workshop (ICLR), 2018

    [Code]

    Link
    NerveNet: Learning Structured Policy with Graph Neural Networks
    Tingwu Wang*, Renjie Liao*, Jimmy Ba, Sanja Fidler
    International Conference on Learning Representations (ICLR), 2018

    [Project] [Code] [Video]

    Link
    Understanding Short-Horizon Bias in Stochastic Meta-Optimization
    Yuhuai Wu, Mengye Ren, Renjie Liao, Roger Grosse
    International Conference on Learning Representations (ICLR), 2018

    [Code]

    Link
    GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation
    Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, Jiaya Jia
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    [Code]

    Link
    Learning Deep Structured Active Contours End-to-End
    Diego Marcos, Devis Tuia, Benjamin Kellenberger, Lisa Zhang, Min Bai, Renjie Liao, Raquel Urtasun
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    [Code] [Spotlight Presentation, 224/3303 (6.8%)]

    Link
    3D Graph Neural Networks for RGBD Semantic Segmentation
    Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, Raquel Urtasun
    IEEE International Conference on Computer Vision (ICCV), 2017

    [Code] [ PyTorch Implementation ] [Oral Presentation, 45/2143 (2.1%)]

    Link
    Situation Recognition with Graph Neural Networks
    Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, Sanja Fidler
    IEEE International Conference on Computer Vision (ICCV), 2017

    [Code]

    Link
    Detail-revealing Deep Video Super-Resolution
    Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia
    IEEE International Conference on Computer Vision (ICCV), 2017

    [Code] [Oral Presentation, 45/2143 (2.1%)]

    Link
    Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
    Mengye Ren*, Renjie Liao*, Raquel Urtasun, Fabian H. Sinz, Richard S. Zemel
    International Conference on Learning Representations (ICLR), 2017

    [Code]

    Link
    Learning to Generate Images with Perceptual Similarity Metrics
    Jake Snell, Karl Ridgeway, Renjie Liao, Brett D. Roads, Michael C. Mozer, Richard S. Zemel
    International Conference on Image Processing (ICIP), 2017

    Link
    Learning Deep Parsimonious Representation
    Renjie Liao, Alexander Schwing, Richard S. Zemel, Raquel Urtasun
    Neural Information Processing Systems (NIPS), 2016

    [Code]

    Link
    Video Super-Resolution via Deep Draft-Ensemble Learning
    Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, Jiaya Jia
    IEEE International Conference on Computer Vision (ICCV), 2015

    [Project & Code]

    Link
    Semantic Segmentation With Object Clique Potential
    Xiaojuan Qi, Jianping Shi, Shu Liu, Renjie Liao, Jiaya Jia
    IEEE International Conference on Computer Vision (ICCV), 2015

    Link
    Handling Motion Blur in Multi-Frame Super-Resolution
    Ziyang Ma, Renjie Liao, Xin Tao, Li Xu, Jiaya Jia, Enhua Wu
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2015

    [Project & Code]

    Link
    Deep Edge-Aware Filters
    Li Xu, Jimmy Ren, Qiong Yan, Renjie Liao, Jiaya Jia
    International Conference on Machine Learning (ICML), 2015

    [Code]

    Link
    Nonparametric Bayesian Upstream Supervised Multi-Modal Topic Models
    Renjie Liao, Jun Zhu, Zengchang Qin
    ACM International Conference on Web Search and Data Mining (WSDM), 2014

    Link
    Learning Important Spatial Pooling Regions for Scene Classification
    Di Lin, Cewu Lu, Renjie Liao, Jiaya Jia
    International Conference on Computer Vision and Pattern Recognition (CVPR), 2014

    Link
    CoDeL: An Efficient Human Co-detection and Labeling Framework
    Jianping Shi*, Renjie Liao*, Jiaya Jia
    IEEE International Conference on Computer Vision (ICCV), 2013

    [Project]

    Link
    Image Super-Resolution Using Local Learnable Kernel Regression
    Renjie Liao, Zengchang Qin
    Asian Conference on Computer Vision (ACCV), 2012

    [Code]

    Preprints

    Link
    Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data
    Sergio Casas, Cole Gulino, Renjie Liao, Raquel Urtasun
    arXiv preprint arXiv:1910.08233 (2019)

    Link
    Deformable Filter Convolution for Point Cloud Reasoning
    Yuwen Xiong, Mengye Ren, Renjie Liao, Kelvin Wong, Raquel Urtasun
    arXiv preprint arXiv:1907.13079 (2019)

    Link
    Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models
    Guangyong Chen, Pengfei Chen, Chang-Yu Hsieh, Chee-Kong Lee, Benben Liao, Renjie Liao, Weiwen Liu, Jiezhong Qiu, Qiming Sun, Jie Tang, Richard S. Zemel, Shengyu Zhang
    arXiv preprint arXiv:1906.09427 (2019)

    [Dataset + Competition]

    Link
    Bounded-Distortion Metric Learning
    Renjie Liao, Jianping Shi, Ziyang Ma, Jun Zhu, Jiaya Jia.
    arXiv preprint arXiv:1505.02377 (2015)

    [Code]

    Talks

    Teaching

    Services

    Honors & Awards

    Best Reviewer, NeurIPS 2019
    RBC Graduate Fellowship, 2019
    Connaught International Scholarship for Doctoral Students (University-wide 20), University of Toronto, 2015.
    Departmental Entrance Scholarship (Top %2 applicant), Department of Computer Science, University of Toronto, 2015.
    3rd prize (1%), China Finals in Software Design Competition of Imagine Cup, Microsoft, 2010.
    1st prize (10 out of candidates of all majors), The 20th Feng Ru Cup, Beihang University, 2010.
    Travel Award: ICLR 2019, ICML 2018, ICLR 2017, ACCV 2012, MLSS 2011.

    Miscellany

    Hobbies: Fingerstyle Guitar, Basketball, Soccer, PC Games.

    My Erdös number is 4 (F. Sinz = 3, G. Rätsch = 2, A. Jagota = 1, P. Erdös = 0).

    Books I have read/am reading/will read.

    Last Updated by Renjie: Dec. 11, 2019