In this paper, we report on a between-subjects experiment we conducted with fifteen graduate students familiar with requirements engineering. The experiment investigates the effectiveness and usability of Evolving Intentions, Simulation over Evolving Intentions, and GrowingLeaf.
A. M. Grubb and M. Chechik.
Modeling and Reasoning with Changing Intentions: An Experiment.
2017 IEEE 25th International Requirements Engineering Conference (RE), 2017.
© IEEE 2017.
This page discusses supplemental material. It is recommended that you read the paper prior to continuing here.
Here are the study materials.
Study Protocol:
Models:
Videos and Handouts:
Tool Versions:
R Files:
Subject Recruitment:
We originally created Stochastically Evolving Intentions as a way of testing the functionality of GrowingLeaf's user-interface and the communication protocol with the backend. However in tool testing, we found this functionality so useful in understanding models, we decided to add it to the study as an intermediate between no analysis and time-based analysis. This allowed us to see if it was the many versions of the model or our actual technique, that helped subjects answer questions.
Forward Analysis is a standard technique used in goal modeling and the "state of the art" at the time of our extension. In this study, we wanted to compare the simulations offered in GrowingLeaf with modelers' ability to use standardly available techniques/tool to perform time-based analysis.
As part of RQ2 we asked, “For each alternative decision, which goal has the greatest impact on the decision?”. This question turned out to be poorly worded on our part, because subjects were not sure what was meant by greatest impact. GroupA subjects who used the simulation selected Space in Dump and/or Reduce Operating Costs. Those who didn’t use simulation chose Produce Green Waste. Group B chose (with frequency) Manage City Waste (4), Reduce Operating Cost (3), Produce Green Waste (2), Enjoy City (1), and Space in Dump (1). Group C chose (with frequency) Reduce Operating Cost (2), Manage City Waste (1), Positive City Image (1), and two subjects said their tradeoffs had equal impact. We cannot make inferences from this data but find it interesting that GroupA was uniform in answering, but GroupB and GroupC diverged in their understanding. This shows that the question may have been biased towards GroupA and thus we do not consider it further.
If you have any questions or think we can further clarify some part of this email amgrubb at cs dot utoronto dot ca.
