CS2125 Paper Review Form - Winter 2019

Reviewer: Yasaman Rohanifar

Paper Title: Verification for Machine Learning, Autonomy, and Neural Networks Survey

Author(s): Weiming Xiang, Patrick Musau, Ayana A. Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, Taylor T. Johnson

1) Is the paper technically correct?
 [∆] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [ ] Good 
 [ ] Marginal (very incremental)
 [∆] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [∆] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [ ] Significant
 [ ] Marginal significance.
 [∆] Little or no significance.

5) Presentation
 [∆] Very well written
 [ ] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [∆] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

   This paper is a survey that integrates summaries of some related papers in verification techniques for autonomous systems, focusing on formal methods, verification and validation, architectural approaches to analyze and assurance of the safety of cyber-physical systems (CPS) that include the current state-of-the-art for safely integrating AI/ML components, such as neural networks (which is termed "learning enabled components (LECs) by the authors"). 
   It is made up from 6 sections, starting from justifying the motivation behind their work and addressing the prevalence of AI, ML, and DNNs in many applications. Thereafter, it talks about applications in autonomous driving, architecture in safe monitoring and control, intelligence control, specification inference and learning, and verification for AI/ML components and systems(LECs). 


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)
S1: Provides easy to understand, and short description of each paper discussed, followed by concise summaries of the applied methods and results. 
S2: Could be used as a good starting point and a comprehensive directory/reference when researching on various areas in autonomous systems and other applicable, related fields.
S3: Progressively articulates connected ideas, explaining some concepts with examples. 


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)
W1: Some spelling and grammatical mistakes.


MISC:
- Since this paper is just a survey and has not introduced any new technique or novel idea, it is hard to measure its significance, impact, and originality. 
- It is not original in a sense that it is just stating other papers' ideas.
- It is low in technical depth since it is just summarizing the main points of each paper.
- Since it is not novel, it doesn't have the highest impact as a paper presenting original, new ideas.
- Since the stated works are not authors' ideas, not much can be said in terms of strengths and weaknesses. 
- Some sections and papers have been discussed in more details than others which indicates the inclination of this paper (e.g. learning and intelligent control)