CSC2125: Safety and Certification of Autonomous Vehicles
Winter 2019
Reading List
General background on self-driving
[1] Liu, S., Li, L., Tang, J., Wu, S., & Gaudiot, J. L. (2017). Creating Autonomous Vehicle Systems. Synthesis Lectures on Computer Science, 6(1), i-186. (Downloadable PDF ref)
[2] J. Tang et al., "Teaching Autonomous Driving Using a Modular and Integrated Approach," 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Tokyo, 2018, pp. 361-366. doi: 10.1109/COMPSAC.2018.00057
[3] Miroslaw Staron, Automotive Software Architectures: An Introduction, Springer, 2017 ISBN 978-3-319-58610-6
General background on ML
[4] Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning. MIT Press. http://www.deeplearningbook.org, 2016.
[5] Lectures 1 and 3 of MIT course on Deep Learning and Self-Driving. https://selfdrivingcars.mit.edu
General background on verification
[6] Lecture notes, csc2108, Fall’07. http://www.cs.toronto.edu/~chechik/courses07/csc2108/index.html
[7] Model Checking, 2nd Edition, MIT Press. Clarke, Grumberg, Peled, Veith
General background on safety, including automotive safety
[7] Miroslaw Staron, Automotive Software Architectures: An Introduction, Springer, 2017 ISBN 978-3-319-58610-6. Chapters 7-8: Functional Safety of Automotive Software
[8] Michael W. Whalen, Darren D. Cofer, Andrew Gacek: Requirements and Architectures for Secure Vehicles. IEEE Software 33(4): 22-25 (2016)
[9] Andrew Gacek, John Backes, Darren D. Cofer, Konrad Slind, Mike Whalen: Resolute: an assurance case language for architecture models. HILT 2014: 19-28
[10] Gillula, Jeremy H., and Claire J. Tomlin. "Reducing conservativeness in safety guarantees by learning disturbances online: iterated guaranteed safe online learning." Robotics: Science and Systems VIII (2013): 81.
[11] Althoff, Matthias, and John M. Dolan. "Online verification of automated road vehicles using reachability analysis." IEEE Transactions on Robotics 30.4 (2014): 903-918.
ML + driving
[12] K. Czarnecki and R. Salay, Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Driving, AG 2018 (Safecomp’18 workshops)
Safety of ML and safety of (self-)driving
[13] SAE International. Surface Vehicle Recommended Practice, issued 2014-01, revised 2018-06
[14] Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua, On a Formal Model of Safe and Scalable Self-driving Cars, Mobileye, 2017 https://arxiv.org/pdf/1708.06374.pdf
[15] R. Salay, K. Czarnecki. Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262. https://arxiv.org/ftp/arxiv/papers/1808/1808.01614.pdf
[16] AI discussion on safety: https://arxiv.org/abs/1606.06565
[17] Responsibility sensitive safety by Mobileye [Shalev-Shwartz et al] - sort of the equivalent of air traffic control rules for autonomous vehicles??
[18] Hazard analysis of ADS - from thesis of Mark Lawford’s students
[19] T. Yamaguchi, T. Kaga, A. Donze, Danjit Seshia. Combining requirement mining, software model checking and simulation-based verification for industrial automotive systems. Technical Report UCB/EECS-2016-124, EECS Department, University of California, Berkeley, June 2016.
[20] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
Testing and
Verification of Machine-Learned Systems
[21] A blog post on adversarial verification, a nice animation of the difference between testing and verification and lots of references to verification literature.
http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html
[22 – repo!] A significant repository of testing and verification papers until 2018. https://sdle2018.github.io/SDLE/V1.1/en/Repository.html. The paper describing the repository is Ma, Lei, Felix Juefei-Xu, Minhui Xue, Qiang Hu, Sen Chen, Bo Li, Yang Liu, Jianjun Zhao, Jianxiong Yin, and Simon See. "Secure Deep Learning Engineering: A Software Quality Assurance Perspective." arXiv preprint arXiv:1810.04538(2018).
[23] Automated Testing of Deep-Neural-Network-driven Autonomous Cars: https://arxiv.org/pdf/1708.08559.pdf
[24] Sanjit Seshia, Dorsa Sadigh, Shankar Sastry. Towards Verified Artificial Intelligence. arXiv:1606.0851v3
[25] Sanjit Seshia, Dorsa Sadigh, Shankar Sastry. Formal methods for semi-autonomous driving. In Proceedings of the Design Automation Conference (DAC), pp. 148:1-148:5, June 2015
[26] G. Katz, C. Barrett, D. Dill, K. Julian, M. Kochenderfer. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks, 2017. https://arxiv.org/abs/1702.01135
[27] Problems of and principles for AI verification: https://arxiv.org/pdf/1606.08514.pdf
[28] Xiang, Weiming, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, and Taylor T. Johnson. "Verification for Machine Learning, Autonomy, and Neural Networks Survey." arXiv preprint arXiv:1810.01989 (2018).
[29] Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana, “DeepXplore: Automated Whitebox Testing of Deep Learning Systems”, SOSP’17. http://www.cs.columbia.edu/~junfeng/papers/deepxplore-sosp17.pdf
[30] Adversarial examples for deep learning in real contexts: https://people.eecs.berkeley.edu/~sseshia/pubdir/sadl-cav18.pdf
[31] Matthew Wicker, Xiaowei Huang, Marta Kwiatkowska: Feature-Guided Black-Box Safety Testing of Deep Neural Networks. TACAS (1) 2018: 408-426
[32] Provable guarantees in verification via building adversarial examples. Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, Daniel Kroening: Concolic testing for deep neural networks. ASE 2018: 109-119
[33] Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska: Reachability Analysis of Deep Neural Networks with Provable Guarantees. IJCAI 2018: 2651-2659
[34] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu: Safety Verification of Deep Neural Networks. CAV (1) 2017: 3-29
[35 – entire project] SafeAI project in ETH Zurich. http://safeai.ethz.ch/
[36] W. Xiang, P. Musau, A. Wild, D. M. Lopez, N. Hamilton, X. Yang, J. Rosenfeld, T. Johnson. Verification for Machine Learning, Autonomy, and Neural Networks Survey, 2018. arXiv:1810.01989v1
Synthesis and design
[37] Andre Platzer. Building safety envelope. CMU. Talk at FLOC ML symposium. Formal model to put boundaries of what reinforcement learning can try. Related to safety envelope of Claire Tomlin (see [10])
[38] Shixiang Gu, Luca Rigazio. “Towards Deep Neural Network Architectures Robust to Adversarial Examples”. https://arxiv.org/abs/1412.5068
[39] Yasser Shoukry, Michelle Chong, Masashi Wakaiki, Pierluigi Nuzzo, Alberto L. Sangiovanni-Vincentelli, Sanjit A. Seshia, Joăo Pedro Hespanha, Paulo Tabuada: SMT-Based Observer Design for Cyber-Physical Systems under Sensor Attacks. TCPS 2(1): 5:1-5:27 (2018)
[40] Tommaso Dreossi, Somesh Jha, and Sanjit A. Seshia. Semantic Adversarial Deep Learning. CAV’18. https://link.springer.com/chapter/10.1007/978-3-319-96145-3_1
Reading list from MIT: https://docs.google.com/spreadsheets/d/1DnfJPZDhrDEB2LTDsqrI4UF-iFtps0LhjjYPBoXJQAc/edit#gid=1966979109
--------------------------------------------------------
A few other ideas from the Aachen seminar – talk to me if you want to present any of these. Some of them are about AI for verification rather than verification/testing of AI-generated systems
Verification of Properties
1. Tomás Brázdil, Krishnendu Chatterjee, Martin Chmelik, Vojtech Forejt, Jan Kretínský, Marta Z. Kwiatkowska, David Parker, Mateusz Ujma: Verification of Markov Decision Processes Using Learning Algorithms. ATVA 2014
2. Luca Bortolussi, Dimitrios Milios, Guido Sanguinetti: Smoothed model checking for Uncertain Continuous-Time Markov Chains. Inf. Comput. 247: 235-253 (2016)
3. Rüdiger Ehlers: Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks. ATVA 2017
4. Gethin Norman, David Parker, Xueyi Zou: Verification and control of partially observable probabilistic systems. Real Time Systems, 2017
5. Tomás Brázdil, Krishnendu Chatterjee, Martin Chmelik, Andreas Fellner, Jan Kretínský: Counterexample Explanation by Learning Small Strategies in Markov Decision Processes. CAV 2015
Learning of Invariants
1. Pranav Garg, Christof Löding, P. Madhusudan, Daniel Neider: ICE: A Robust Framework for Learning Invariants. CAV 2014
2. Pranav Garg, Christof Löding, P. Madhusudan, Daniel Neider: Learning Universally Quantified Invariants of Linear Data Structures. CAV 2013
3. Grigory Fedyukovich and Rastislav Bodik. Accelerating Syntax-Guided Invariant Synthesis. TACAS 2018
4. Synthesizing Inductive Invariants. Yakir Vizel, Arie Gurfinkel, Sharon Shoham, Sharad Malik: IC3 – Flipping the E in ICE. VMCAI 2017
Model Learning
1. Generating Models of Communication Protocols
Synthesis of Programs and Algorithms
1. Ezio Bartocci, Luca Bortolussi, Tomás Brázdil, Dimitrios Milios, Guido Sanguinetti: Policy learning in continuous-time Markov decision processes using Gaussian Processes. Perform. Eval. 116: 84-100 (2017)
2. Sebastian Junges, Nils Jansen, Christian Dehnert, Ufuk Topcu, Joost-Pieter Katoen: Safety-Constrained Reinforcement Learning for MDPs. TACAS 2016
3. Pavol Bielik, Veselin Raychev, Martin T. Vechev: Learning a Static Analyzer from Data. CAV 2017
4. Nader H. Bshouty, Dana Drachsler-Cohen, Martin T. Vechev, Eran Yahav: Learning Disjunctions of Predicates. COLT 2017
5. Richard Evans, Edward Grefenstette: Learning Explanatory Rules from Noisy Data. To appear in Journal of Artificial Intelligence Research
6. Rezaul Chowdhury, Pramod Ganapathi, Stephen L. Tschudi, Jesmin Jahan Tithi, Charles Bachmeier, Charles E. Leiserson, Armando Solar-Lezama, Bradley C. Kuszmaul, Yuan Tang: Autogen: Automatic Discovery of Efficient Recursive Divide-&-Conquer Algorithms for Solving Dynamic Programming Problems. TOPC 4(1): 4:1-4:30 (2017)
7. Kevin Ellis, Armando Solar-Lezama, Joshua B. Tenenbaum: Unsupervised Learning by Program Synthesis. NIPS 2015
8. Bruno Lacerda, David Parker, Nick Hawes: Multi-Objective Policy Generation for Mobile Robots under Probabilistic Time-Bounded Guarantees. ICAPS 2017
Last updated on January 11, 2019