CS2125 Paper Review Form - Winter 2019

Reviewer: Eric Langlois

Paper Title: Verification for Machine Learning, Autonomy, and Neural Networks
Survey

Author(s): W. Xiang, P. Musau, A. A. Wild, D. M. Lopez, N. Hamilton, X. Yang, J.
Rosenfeld, T. T. Johnson

1) Is the paper technically correct?
 [ ] Yes
 [X] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [ ] Good 
 [ ] Marginal (very incremental)
 [X] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [X] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [ ] Significant
 [X] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [ ] Generally well written
 [ ] Readable
 [X] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [ ] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [X] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

This paper is a survey of verification techniques for autonomous systems,
particularly those using neural networks. The authors survey a large number of
papers over a range of verification-relevant domains. Each section consists of
a brief introduction followed by a list of paragraphs, one per paper, describing
the cited works. While the report was informative overall, many sections were
difficult to follow. There is little narrative structure describing what the
important research topics problems are, how the papers relate to the subject at
hand, or how the papers relate to each other. The survey would be improved by
abandoning the rigid one-paragraph-per-paper structure in favour of summarizing
related works together.

The paper scores low on several categories by virtue of being a survey but,
for the reasons given above, I also find it to be a low quality survey paper and
would not recommend it for acceptance. With some editing and added structure the
paper could be substantially improved.


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1: 
This survey represents multiple disciplines: control theory, automata theory, and
machine learning. It enables researches in one area to better discover safety
research in other areas.

S2:
The section on safe reinforcement learning is well-written and serves as a good
introduction and survey. The authors provide background information, describe
the core issues, and highlight useful references.

S3:
The catalog of software tools appears to be fairly comprehensive and as far
as I know it is original.

9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1:
Much of the article fails to provide sufficient structure and context for the
papers that it references. The papers are presented as a nearly disconnected
list of descriptions without explanation of why the work is interesting or what
are the core research problems to be addressed.

W2:
Sections 2 and 3 are poorly structured. There are a small number of papers
referenced in each and many of the papers seem to have little to do with each
other or the section topic.

W3:
Many of the cited papers are quite new, not yet published outside of arXiv,
and/or have few citations. It would give me more confidence in the thoroughness
of the survey if more if the papers I checked were more mature.