Machine Learning Group
University of Toronto
Vector Institute
Canada

Email: gdzhang dot cs at gmail dot com

Follow Me

"Many things are only trivial once you know them." - Herman Chernoff

About Me

I am a researcher in machine learning and artificial intelligence, focusing on training, tuning, and aligning large language models good.

Curriculum Vitae

My CV can be downloaded from this link: [CV].

Selected Publications (Google Scholar or Semantic Scholar for Full List)

* below indicates equal contribution

Multi-agent Optimization or its Application

• Learning to Give Checkable Answers with Prover-Verifier Games
  Cem Anil, Guodong Zhang, Yuhuai Wu, Roger Grosse
  Preprint.

• Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization
  Guodong Zhang, Yuanhao Wang, Laurent Lessard, Roger Grosse
  International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
  

• A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraints
  Guodong Zhang, Xuchan Bao, Laurent Lessard, Roger Grosse
  Journal of Machine Learning Research (JMLR), 2021.
  

• On the Suboptimality of Negative Momentum for Minimax Optimization
  Guodong Zhang, Yuanhao Wang
  International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
  

• On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach
  Yuanhao Wang*, Guodong Zhang*, Jimmy Ba
  International Conference on Learning Representations (ICLR), 2020.
  (Selected as one of three contributed talks in SGO workshop, NeurIPS, 2019)

Deep Learning Dynamics

• Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers
  Guodong Zhang, Alex Botev, James Martens
  International Conference on Learning Representations (ICLR), 2022.
  

• An Empirical Study of Stochastic Gradient Descent with Structured Covariance Noise
  Yeming Wen*, Kevin Luk*, Maxime Gazeau*, Guodong Zhang, Harris Chan, Jimmy Ba
  International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
  

• Which Algorithmic Choices Matter at Which Batch Sizes?
  Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva,
  George E. Dahl, Christopher J. Shallue, Roger Grosse
  Neural Information Processing Systems (NeurIPS), 2019.
  

• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
  Guodong Zhang, James Martens, Roger Grosse
  Neural Information Processing Systems (NeurIPS), 2019.
  

• Three Mechanisms of Weight Decay Regularization
  Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse
  International Conference on Learning Representations (ICLR), 2019.
  

Bayesian Deep Learning

• Differentiable Annealed Importance Sampling and the Perils of Gradient Noise
  Guodong Zhang, Kyle Hsu, Jianing Li, Chelsea Finn, Roger Grosse
  Neural Information Processing Systems (NeurIPS), 2021.
  (18/9122 in terms of average score, see our review)

• Functional Variational Bayesian Neural Networks
  Shengyang Sun*, Guodong Zhang*, Jiaxin Shi*, Roger Grosse
  International Conference on Learning Representations (ICLR), 2019.
  

• Noisy Natural Gradient as Variational Inference
  Guodong Zhang*, Shengyang Sun*, David Duvenaud, Roger Grosse
  International Conference on Machine Learning (ICML), 2018.
  

• Differentiable Compositional Kernel Learning for Gaussian Processes
  Shengyang Sun, Guodong Zhang, Chaoqi Wang, Wenyuan Zeng, Jiaman Li, Roger Grosse
  International Conference on Machine Learning (ICML), 2018.
  

Avocation

• Picking Winning Tickets Before Training by Preserving Gradient Flow
  Chaoqi Wang, Guodong Zhang, Roger Grosse
  International Conference on Learning Representations (ICLR), 2020.
  

• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
  Chaoqi Wang, Roger Grosse, Sanja Fidler, Guodong Zhang
  International Conference on Machine Learning (ICML), 2019.
  

• Deformable Convolutional Networks
  Jifeng Dai*, Haozhi Qi*, Yuwen Xiong*, Yi Li*, Guodong Zhang*, Han Hu, Yichen Wei
  IEEE International Conference on Computer Vision (ICCV), 2017.
  (45/2143, Oral)

Teaching

• CSC 311: Introduction to Machine Learning (2021 Fall)

• CSC 2541: Neural Network Training Dynamics (2021 Winter)

• CSC 2541: Neural Network Training Dynamics (2021 Winter)
• CSC 311: Introduction to Machine Learning (2020 Fall)
• CSC 2515: Machine Learning (2019 Fall)
• CSC 411: Machine Learning and Data Mining (2018 Fall)
• CSC 321: Introduction to Neural Networks and Machine Learning (2018 Winter)
• CSC 384: Introduction to Artificial Intelligence (2017 Fall, 2018 Summer, 2019 Winter)

Services

• Conference Reviewer: UAI, AISTATS, NeurIPS, ICLR, ICML
• Journal Reviewer: JMLR

Selected Honors & Awards