CS2125 Paper Review Form - Winter 2019

Reviewer: Zi Yi Chen

Paper Title: Verification for Machine Learning, Autonomy, and Neural Networks Survey 

Author(s): Weiming Xiang, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, Taylor T. Johnson

1) Is the paper technically correct?
 [X] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [X] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [X] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [ ] Significant
 [X] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [X] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [X] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

This paper surveys some of the AI and ML components published by other scientists with the focus on how they can be safely integrated into autonomous cyber-physical systems.  For each section of this paper, the authors first give a light introduction of the topic (eg. what is reinforcement learning), then discuss its limitations in safety, and suggest few literatures that attempt to address the issue.  

Overall, I think this paper is well organized, easy to understand for those with the proper background knowledge.  But I had trouble understanding some of the papers that the authors survey because they only gave a very high-level description of what the paper is about, which is why I gave “Marginal Depth” for technical depth of this paper.  


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1: The paper is well organized and tries to be beginner friendly by briefly introducing each topic before presenting the surveyed papers

S2: The paper explains the limitations of each technology when it comes to safety integration, something that I hadn’t consider when I was learning ML or RL


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1: I wish the authors went a bit more in depth with each paper.  Some were hard to understand without reading the actual paper.