CS2125 Paper Review Form - Winter 2018

Reviewer: Ramy Shahin

Paper Title: Assurance Based Development of Critical Systems

Author(s): Graydon et al.

1) Is the paper technically correct?
 [ ] Yes
 [*] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [ ] Good 
 [*] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [*] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [ ] Significant
 [*] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [*] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [ ] Accept (high quality - would argue for acceptance)
 [*] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

This paper presents Assurance Based Development (ABD), an iterative development process of critical systems that interleaves development with assurance. The authors promote the idea of starting with a high level assurance case, and then refine it with each design decision made. This way developers don’t go astray from assurance requirements, and at the same time the assurance case is constructed based on the design decisions made, instead of going off track, diverging from the developed artifacts.

A significant part of the paper is a Runway Safety Monitor (RSM) case study, which is a part of an avionics Runway Incursion Prevention System (RIPS). The assurance case for RSM is step-wise refined as each of the first four design decisions is made.


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1 - The case study is probably the best thing about this paper. It is well explained, and the connection between design decisions and assurance requirements is well presented.

S2 - The paper is well written overall. It can serve as a good introductory reading to assurance/safety cases.


9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1 - ABD as a process seems to be very much inspired by Test Driven Development (TDD). ABD is lifted though to assurance requirements, while TDD is mostly concerned with functional requirements. This connection cannot be found anywhere in the paper.

W2 - The seven criteria for selecting system development choices (design decisions) were listed without giving any rationale behind how they were picked. 

W3 - The case study illustrated the iterative nature of ABD, but the application of ABD to this case study was not evaluated in any way.