CS2125 Paper Review Form - Winter 2019 Reviewer: Yasaman Rohanifar Paper Title: Autonomous Vehicle Safety - An Interdisciplinary Challenge Author(s): Philip Koopman, Michael Wagner 1) Is the paper technically correct? [*] Yes [ ] Mostly (minor flaws, but mostly solid) [ ] No 2) Originality [ ] Very good (very novel, trailblazing work) [ ] Good [ ] Marginal (very incremental) [*] Poor (little or nothing that is new) 3) Technical Depth [ ] Very good (comparable to best conference papers) [ ] Good (comparable to typical conference papers) [*] Marginal depth [ ] Little or no depth 4) Impact/Significance [ ] Very significant [ ] Significant [*] Marginal significance. [ ] Little or no significance. 5) Presentation [*] Very well written [ ] Generally well written [ ] Readable [ ] Needs considerable work [ ] Unacceptably bad 6) Overall Rating [ ] Strong accept (award quality) [ ] Accept (high quality - would argue for acceptance) [*] Weak Accept (borderline, but lean towards acceptance) [ ] Weak Reject (not sure why this paper was published) 7) Summary of the paper's main contribution and rationale for your recommendation. (1-2 paragraphs) This paper argues about the main disciplines that should be considered in the field of autonomous vehicles. It argues that to deploy fleets of fully autonomous vehicles, we must take a multi-disciplinary approach, across all levels, in various disciplines such as: safety engineering, making ultra-dependable robots, software, computing hardware, testing, security, human-computer interaction, legal, and social acceptance. The 9 aforementioned discplines were discussed in an abstract way throughout the paper. The authors point out the open problems with autonomous vehicles such as legal issues, and call for attention for developing a holistic and different approach to deal with autnomous vehicles development and testing as it is radically different from the ways we deal with traditional vehicles. This paper also calls for safety certification strategy of some sort for fully autonomous vehicles (cross-disciplinary approach), discusses some edge cases in the driving scenarios, subtle trade offs in decision making in this regard, cross-coupling trade offs between areas, and updating practices and validation processes to address safety concerns that can lead to an end-to-end design. It also emphasizes on challenges with validating ML based systems to ultra-dependable levels required for autonomous vehicles fleet and how this issue connects with other discussed areas. I'm leaning towards the weak acceptance of this paper. Though it does not show any technical aspect of the issue at hand, it provides a holistic view of all the disciplines involved with autonomous vehicles and calls for attention in some less discussed areas related to AUVs. Furthermore, as this paper was published as an article in IEEE magazine, I think it suits the context of a magazine and deserved to be published there. 8) List 1-3 strengths of the paper. (1-2 sentences each,identified as S1, S2, S3.) S1: For a scientific article in IEEE magazine, I argue that it's concise and has covered a considerable amount of disciplines within the scope of paper which gives a good fish-eye-view of the important concepts to consider in the autonomous vehicles field. Also, it sheds light on some of the less noticable sides of deploying autonomous vehicles in the scientific community such as legal issues and social acceptance. S2: It is very well-written and easy-to-read in a sense that even people with non-technical backgrounds are able to follow and understand it. Considering that it was published in a magazine and it was aimed at a broader audience than people who are specialized in each of the aforementioned areas, I believe they did a good job. S3: It sheds a good light on the stark differences between probablistic systems and current models (such as V model) for developing and testing vehicles and calls for attention on developing new techniques for the new demand in the market of autonomous vehicles. It repeatedly emphasizes on the fact that due to the nature of real-time learning systems, current testing and engineering models are not sufficient and will not work thus there's a need for new comprehensive techniques. 9) List 1-3 weaknesses of the paper (1-2 sentences each, identified as W1, W2, W3.) W1: Some of the assumptions and suggestions that this paper provides are unrealistic. For example, it suggests that in order to make computer-based automotive systems safe, a solution would be to set the “controllability” aspect of autonomous systems to zero which does not make sense in the context of autonomous vehicles! W2: Although this article does not go deep into any of the disciplines it discusses, some of the areas like security, which are of top priority, could be more studied in the paper. As the paper is written in a broad context and does not go deep into details, it is hard to detect major strengths and weaknesses of it!