CS2125 Paper Review Form - Winter 2018

Reviewer: Azadeh Assadi

Paper Title: Model Checking Lots of Systems – Efficient Verification of Temporal Properties in Software Product Lines

Author(s): Andreas Classen, Axel Legay, Jean-Francois Raskin

1) Is the paper technically correct?
 [X] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [X] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [X] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [ ] Significant
 [X] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [X] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [X] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)
The authors in this paper explain their approach at producing a scalable model with efficient verifications of a system’s behaviour within a software product line. The authors first begin by introducing software product lines and explaining that the typical process of abstraction and behavioural modeling leads to a loss of recognition in the various features which leads to a lack of relation between the products and respective behaviours. They also explain that the current mechanisms of checking behaviour are not adequate for the large scale of such systems nor the behavioural checking against temporal properties. The authors then begin to explain their approach using an example from vending machines. The feature transition systems are derived first by the development of a model of all the features within the software product line at question. Then priority list of transitions is developed based on which various transformations lead to the derivation of various feature diagrams. Behaviour of products is achieved through projection of a set of features resulting in the desired Transition System. The model checking occurs through various steps. One method is testing for reachability in FTS stating that the initial states should be reachable from all products. In this process, those models that violate this validation check are added to the set of bad states. To further check bad states, the authors propose performing a double DFS via the IsPersistent function first performing an outer DFS search for a reachable bad state (i.e. an IsReachable function) and then if this relationship is found, a second inner DFS check is done to determine whether the state is on a cycle. The authors evaluate their product with a n example of the analysis of the mine pump controller exemplar and time the various checking before comparing them to the existing model checking techniques.    



8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)
S1 – well written overall with good examples from the vending machine scenario



9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)
W1 – It would have been more beneficial if the related work and survey of existing systems was done early in the paper so as to demonstrate the lay of the land.
W2 – Functions are generally not very well defined and require reading multiple other papers to completely understand their working
W3 – The authors have a pre-defined set of properties (i.e. states, actions, and transitions) that are valid and used to validate and check the models as suggested. The process of generating a valid list of such properties in and of itself requires a fair bit of time which should be factored into the overall efficiency measurements of the proposed product.