CS2125 Paper Review Form - Winter 2018

Reviewer: Ramy Shahin

Paper Title: Model Checking Lots of Systems: Efficient Verification of Temporal Properties in Software Product Lines

Author(s): Andreas Classen et al.

1) Is the paper technically correct?
 [*] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [*] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [*] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [*] Significant
 [ ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [*] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [*] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)
This paper takes model checking of individual models to the level of product lines of models. First, the Featured Transition System (FTS) notation is introduced. FTSs are transition systems augmented with Feature Diagrams (FDs), and transitions are labeled with features. In addition, a partial order over transitions can define relative priorities that implicitly allow features to remove transitions from base models. The syntax and semantics of FTSs in addition to state reachability are defined formally in the paper.
 
The paper then explains the model checking algorithm for FTSs. Both regular safety and omega-regular property checking are explained. The synchronous product of an FTS model and the negation of the property to be checked is defined. The state reachability algorithm is also explained in detail.

8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1 - The formal treatment of the FTS abstraction, synchronous product and reachability provide a solid foundation for this line of work.

S2 - FTSs together with their model checking algorithm provide an SPL-based model checking system that outperforms model checking inidividual products.


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1 - The mine pump controller model used for evaluation is a relatively small exemplar. Such an exemplar by itself is not adequate for a thorough evaluation.

W2 - The priority-based approach for removing transitions is not robust enough for complex models, where features not directly related in the FD need to remove transitions from each other.

W3 - FTSs as presented in the paper only support singular features as labels for transitions. Typically a boolean expression is used to represent a set of products instead of a single feature.