CS2125 Paper Review Form - Winter 2019

Reviewer: Mohammad Rashidujjaman Rifat

Paper Title: AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation

Author(s):

1) Is the paper technically correct?
 [✔] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [✔] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [✔] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [✔] Significant
 [ ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [✔] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [✔] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

This paper provides a technique for automatically proving safety properties in deep neural networks. To do so, it takes advantage of ongoing research in the area of classic abstract interpretation. Particularly, the paper presents the technique of transforming different layers of a deep neural network architecture to an abstract presentation as a way to characterize the layers in the network. While several scholarships show methods for handling safety and security in some deep neural network, they fall short in analyzing some modern architecture of neural networks, such as convolutional neural networks.

The paper is generally well written. The paper is technically sound and solid. Coming from a background of HCI, I would appreciate a more engaging discussion of a literature review to pinpoint the papers' originality and contributions. However, I also understand the community norm of this research. From what I have learned from this paper and a cursory look from a few other works, the paper made a sound contribution in the area of safety as it has shown techniques for modern neural network architectures. From the presentation point of view, I like that the paper mentioned and described some of the related background concepts. Otherwise, I found the paper lucid and coherent. Considering the evolution of deep neural network architecture and wide use of them in various sensitive applications (such as self-driving cars), this paper contributes to determining safety measures for the applications. At the same time, the paper could open up avenues for further research.


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1: This paper provides a scalable and robust analyzer for neural nets.
S2: The method describe in this paper has a wide application area as most of the neural networks use the types of layers this method involves.


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1: This technique is a binary measure of robustness and not a statistical method.
W2: There might be other architectures different from the simple feed-forward network that this techniques can handle.