CS2125 Paper Review Form - Winter 2019 Reviewer: Yasaman Rohanifar Paper Title: Can Autonomous Vehicles be Safe and Trustworthy? Effects of Appearance and Autonomy of Unmanned Driving Systems Author(s): Jae-Gil Lee, Ki Joon Kim, Sangwon Lee & Dong-Hee Shin 1) Is the paper technically correct? [ ] Yes [*] Mostly (minor flaws, but mostly solid) [ ] No 2) Originality [ ] Very good (very novel, trailblazing work) [*] Good [ ] Marginal (very incremental) [ ] Poor (little or nothing that is new) 3) Technical Depth [ ] Very good (comparable to best conference papers) [*] Good (comparable to typical conference papers) [ ] Marginal depth [ ] Little or no depth 4) Impact/Significance [ ] Very significant [*] Significant [ ] Marginal significance. [ ] Little or no significance. 5) Presentation [*] Very well written [ ] Generally well written [ ] Readable [ ] Needs considerable work [ ] Unacceptably bad 6) Overall Rating [ ] Strong accept (award quality) [ ] Accept (high quality - would argue for acceptance) [*] Weak Accept (borderline, but lean towards acceptance) [ ] Weak Reject (not sure why this paper was published) 7) Summary of the paper's main contribution and rationale for your recommendation. (1-2 paragraphs) This paper explores whether applying layers of anthropomorphic cues to artificial driving agent promotes positive evaluations and perceptions of an unmanned driving system. The authors chose one external (human-like appearance) and one internal (autonomy) as anthropomorphic cues and they studied how these cues induce greater social presence, intelligence, safety, and trust in unmanned autonomous vehicles. They conducted a 2 x 2 factorial between-subjects experiment with four conditions representing two types of appearance (NAO robot for human-like vs. smartphone for gadget-like) and two levels of autonomy (asking questions to take actions in low vs. reporting only in high) to examine their hypotheses. They found that both of these qualities have positive effects on the measured variables (social presence, intelligence, safety, cognitive and affective trust) and can be stratigically manipulated to provide more positive and socially meaningful interactions with technology an demonstrates the feasibility of psychologically increasing the safety and trustworthiness of autonomous vehicles. If their method of experiment was more realistic, I would have argued more for the acceptance of this paper. I think the psychological approach for inducing trust was interesting and it would be a publishable and acceptable paper if they conducted their experiments in a more realistic and life-like way. 8) List 1-3 strengths of the paper. (1-2 sentences each, identified as S1, S2, S3.) S1: The authors used a scientific method for conducting their research. Use of Cronbach's alpha, Pearson's correlation analyses, multivariate analysis of variance (MANOVA), etc. shows the statistical knowledge of the authors. S2: Their approach to open a discourse from a psychological perspective to induce trust in the emerging technology of unmanned vehicles was interesting. S3: Good interpretation and validation of the results, addressing some of their limitations, and design suggestions for future work. 9) List 1-3 weaknesses of the paper (1-2 sentences each, identified as W1, W2, W3.) W1: The experimental conditions used to test their hypothesis were not similar to actual driving context at all. For example, it is highly unlikely that small predetermined driving courses or a remote-controlled car for children with a maximum speed of 6 km/hr will induce an actual, real-life perception of an unmanned autonomous driving system in the users' mind. The design of their experiment was very unrealistic. W2: Other than the physcial setup of their experiments, their decision on measuring other factors such as level of autonomy is also questionable. Currently, the semi-autonomous, non-autonomous, or even fully autonomous cars don't ask the driver to stop or start the car (this would be annoying). Furthermore autonomous vehicles don't inform the driver of every action they take! Moreover, in their experiment, their so-called autonomous car was not even autonomous and was remotely controlled by an experimenter. Which is ironic and deceptive given their context of research to assess autonomous cars. W3: There is a huge lack of information in regard with their quesionnaire as well as histograms and charts. They mentioned that they adapted the questionnaire from a validated prior research but there is no mention of that research nor the questions. They did not fully disclose the questionnaire as an appendix or a supplement to the paper. The same goes with the results. They mention that they met the assumptions of normal distribution and the absence of notable outliers confirmed by plots, histograms, skewness, kurtosis z scores, etc. but there is no evidence of such plots or calculations in the paper. - Please notice the unscientific chart (figure 4) which is not the common scientific way of showing such results. - In the study, they don't ask about the participants' driving experiences and status of car their ownership which has a huge impact on the results obtained from them, but in the discussion section they only give statistics about the number of people younger than 30 who own a car, and they guess this may negatively impact their results. - Some data in their reported data and analysis is missing. For example, in Table 3, the row representing the B value, as well as upper and lower limit of Confidence Interval of Appearance for Affective trust was missing from the table. - It seems like they knew the results beforehand and tried to design an experiment to validate them.