Shang Wang

Co-founder & CTO | CentML
PhD Student | Department of Computer Science, University of Toronto


Biography

I was born and raised in the beautiful city of Qingdao, Shandong Province, China. I attended the Qingdao No. 2 Middle School (it is actually a high school instead of a middle school; this mistranslation legacy still seems to remain) where I had the honor to grow up with the most talented of my peers.

I graduated with the Bachelor of Applied Science (BASc) degree from the Electrical and Computer Engineering (ECE) program at the University of Toronto in June 2018. I am one of the two Adel S. Sedra Gold Medal winners of my year. Here are a few more links relevant to my undergraduate study: link, link.

During my undergraduate years, I had the honor to work on a summer research project with Prof. Jonathan Rose, and to participate in internships at Google twice and the Intel Programmable Solutions Group (formerly known as Altera) twice.

I graduated with the Master of Science (MSc) degree from the Department of Computer Science (DCS) at the University of Toronto in Jan. 2020. I had the honor to be advised by Prof. Gennady Pekhimenko. I also had affiliation with the Vector Institute as a Graduate Student Researcher. Here is a link to my MSc thesis. My MSc thesis is the CAIAC AI Masters Thesis Award nominee from the University of Toronto.

From March 2020 to June 2022, I worked at NVIDIA as a Deep Learning Software Engineer. I helped ML researchers and developers to optimize their training workloads for better performance on the GPUs. I contributed to NVIDIA Deep Learning Examples that demonstrate how ML/DL training/inference is supposed to be implement to best utilize the GPUs. I participated and contributed to the MLCommons (a.k.a., MLPerf) Benchmark Infra Working Group, representing NVIDIA. Last but not least, I created and later led the development of the LDDL library from scratch. I was promoted to Senior Deep Learning Software Engineer in June 2022.

Since September 2021, I began my PhD study in the Department of Computer Science (DCS) at the University of Toronto. I have the honor to be advised by Prof. Gennady Pekhimenko. I am affiliated with the Vector Institute as a Graduate Student Researcher. I am one of the winners of the 2022-2023 Ontario Graduate Scholarship (OGS) and the 2023-2026 Canada Graduate Scholarship-Doctoral (CGS-D). I am one of the finalists of the 2023 Meta Research PhD Fellowship.

Since July 2022, I have the immense priviledge of leading the research and engineering efforts at CentML where I am one of the co-founders and serve as the CTO.


Research Interests

ML/DL workloads nowadays usually consume an insane amount of compute resources. I like novel ideas. I also like to make things run fast. Hence, I am generally interested in optimization techniques related to ML/DL systems.


Papers


Talks

  • Think Your Models Run Efficiently? Think Again!:
  • Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models:
    • oral at MLSys 2021
  • BPPSA: Scaling Back-propagation by Parallel Scan Algorithm:
  • Recent Trend in Machine Learning Compilers: A Survey:
    • oral at the 17th CASCON Workshop on Compiler Driven Performance

Posters


Teaching Assistant

  • Fall 2018, CSC369: Operating Systems
  • Winter 2019, CSC369: Operating Systems
  • Fall 2019, ECE244: Programming Fundamentals