CS2125 Paper Review Form - Winter 2019

Reviewer: Yilin Han    

Paper Title: DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems

Author(s): Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid.

1) Is the paper technically correct?
 [X] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [X] Very good (very novel, trailblazing work)
 [ ] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [X] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [X] Significant
 [ ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [X] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [X] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

This paper developed a generative adversarial network(GAN) testing model, called DeepRoad, that can verify different driving scenes (such as snow, rain) as well as sampling techniques that used for input validation. With such a model, they can detect the inconsistent behaviors in autonomous driving systems. For input validation part, their model can validate input online that tells driving they should take over the control of a car while input is invalidated. Their approach is to measures the distance between the features of training and input dataset. 

I will recommend this paper because using GAN-based model is challenging, and this paper showed good results in the experiments. Another strong reason is their model consist of input validation part. I think it is very essential for an autonomous driving car. 

8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1: GAN-based model is novel in the autonomous vehicle testing system. GAN is probably one of the most promising approaches to perform ML testing. 

S2: The model considers online input validation. I think it is a more industrial concerned case that proper input validation can notify the driver of the potential dangers and the effectiveness of a machine learning model. 


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1: Dataset is relatively small. GAN model generation is relatively hard to control the performance, so if a bigger dataset model training and experiments will be much better.