CSC2130: Empirical Research Methods for Computer Scientists

(Fall session, 2014)


Instructors: Steve Easterbrook and Barbara Barbosa Neves
Seminars: Thursdays, 2-5pm. BA B025 (basement of the Bahen building)
First class: Sept 11, 2014.
Breadth credit for CS grad program: RA14 & M4

Also, we'll be using Piazza for online discussions. Course link is

About the Course

This course will explore the role of empiricism in computer science research, and will prepare students for advanced research by examining how to plan, conduct and report on empirical investigations. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing. The course will cover the principal research methods used to study human interaction with computer technology: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies. We will relate these methods to relevant metatheories in the philosophy and sociology of science, and critically review published examples of work that used each of the principle methods in computer science.

The course is aimed at students who plan to conduct research that requires some form of empirical validation involving human subjects, or who wish to establish an empirical basis for a research programme. In most cases, such empirical work focuses on people - examining the ways in which people interact with computers, and how new tools and techniques affect them. For this reason we draw heavily on the behavioural sciences for suitable empirical methods, and we focus on both qualitative and quantitative data analysis. The course will be especially relevant to students in HCI, Software Engineering, Cognitive Science, and interdisciplinary computing. Note that we will *not* cover the kinds of performance evaluation techniques used in systems areas.

Pages from previous years' iterations of this course are still available:

Course Outline

Note: This is a draft - topics and readings may change as we get into the course!

  Seminar Topic & Notes Background Readings

Week 1
(Sept 11, 2014)

Introduction & Orientation (slides)

  • Course objectives & Assessments
  • "Research Idioms"
  • Intro to philosophy of science
  • Role of theory building

Read these papers sometime this week:

  1. Easterbrook et al: Selecting Empirical Methods for Software Engineering Research
  2. McComas: The Principal Elements of the Nature of Science: Dispelling the Myths
  3. Optional - explore this website: How science really works
Week 2
(Sept 18, 2014)

Research Design and Ethics (slides)

  • Evidence-based practice
  • Research Design
  • Research Ethics

Read these two papers prior to the seminar:

  1. Creswell: Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (Chapter 1)
  2. Singer & Vinson: Ethical Issues in Empirical Studies of Software Engineering
More background Reading:
Week 3
(Sept 25, 2014)

Basics of Doing Research (slides)

  • Finding good research questions
  • Theory building
  • Evidence and Measurement
  • Replication
  • Peer Review

Read these three papers prior to the seminar:

  1. Tedre & Sutinen: Three traditions of computing: what educators should know
  2. Ioannidis: Why Most Published Research Findings Are False.
  3. Sjoberg et al: Building Theories in Software Engineering

Week 4
(Oct 2, 2014)

Laboratory Experiments (slides)

  • Controlled Experiments
  • Quasi-experiments
  • Sampling
  • Confounding Variables

Read these papers prior to the seminar:

Week 5
(Oct 9, 2014)

Quantitative Analysis (slides)

  • Basic Stats
  • Choosing a statistical model
  • Understanding Statistical Significance
  • Statistical Power Analysis
  • Meta-Analysis

Read these two papers prior to the seminar:

More Background reading:

Week 6
(Oct 16, 2014)

Interviews and Observation (slides)

  • Conducting interviews
  • Focus Groups
  • Participant Observation
  • Ethnographies
  • Collecting Field Notes

Read these papers prior to the seminar:

Week 7
(Oct 23, 2014)

Qualitative Analysis (slides)

  • Coding Strategies
  • Grounded Theory
  • Phenomenography

Read these prior to the seminar:

Week 8
(Oct 30, 2014)

Case Studies (slides)

  • Single and Multi-case
  • Longitudinal Case Studies
  • Approaches to Data Collection

Read these prior to the seminar:

Week 9
(Nov 6, 2014)

Survey Research (slides)

  • Designing Questionnaires
  • Sample Size

Read these papers prior to the seminar:

Week 10
(Nov 13, 2014)

Intervention Methods

  • Action Research
  • Pilot Studies
  • Benchmarking

Read these prior to the seminar:

Week 11
(Nov 20, 2014)

Replication and Beyond

  • How important is replication?
  • Bias and Influences
  • Threats to Validity (and how to reduce them)
  • When to use empirical methods
  • When NOT to use empirical methods


Week 12
(Nov 27, 2014)

Publishing and Reviewing

  • What makes a good paper?
  • Where to publish
  • The peer review process

Class will be a "mock" program committee meeting. Before class, read these two papers and write a short review of each. Your review should start with a 1 paragraph summary of the paper, followed by comments on its strengths and weaknesses, and finish with a recommendation (accept, accept with revisions, or reject):

Background reading on peer review:

Books & other Resources

See also the resource list at ESERNET - the Experimental Software Engineering Network

1. Books

Blum. B. I. Beyond Programming: To A New Era of Design. Oxford University Press. 1996.

Chalmers, A. Science and Its Fabrication. University of Minnesota Press. 1990.

Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Second Edition. Sage. 2002.

Crotty, M. Foundations of Social Research: Meaning and Perspective in the Research Process. Sage. 1998.

Endres, A.; Rombach, D.; A Handbook of Software and Systems Engineering: Empirical Observations, Laws, and Theories. Addison Wesley, 2003.

Lazar, J., Feng, J. H., & Hochheiser, H. Research methods in human-computer interaction. John Wiley & Sons. 2010

Meltzoff, J. Critical Thinking About Research: Psychology and Related Fields. American Psychological Association, 1998.

Shull, F.; Singer J.; Sjoberg, D.I.K. (eds); Guide to Advanced Empirical Software Engineering. Springer, 2007.

Strauss, A; Corbin, J; Basics of Qualitative Research : Techniques and Procedures for Developing Grounded Theory. Sage 1998.

2. Papers on Software Engineering Research Methodology

NOTE: I've arranged these in chronological order - there has been considerable maturation of empirical software engineering over the past 15 years or so - the later papers offer increasingly sophisticated ideas about how to do empirical research in SE.

Curtis, B.; Measurement and experimentation in software engineering. Proceedings of the IEEE 68(9) 1144-1157. Sept. 1980.

Galliers R. D.; Land F. F.; Viewpoint: choosing appropriate information systems research methodologies. Communications of the ACM 30(11) 901 - 902. November 1987. (see also followup correspondence in CACM 31 (12) 1502-1506.)

Shaw,M.; Prospects for an Engineering Discipline of Software. IEEE Software 7 (6) 15-24. November 1990.

Fenton, N.; Pfleeger, S.L.; Glass, R.L.; Science and substance: a challenge to software engineers. IEEE Software 11 (4) 86-95. Jul 1994.

Pfleeger, S.L.; Experimental design and analysis in software engineering. Annals of Software Engineering 1, 219–253. 1995.

Kitchenham, B.; Pickard, L.; Pfleeger, S.L. Case studies for method and tool evaluation. IEEE Software 12 (4) 52-62.  July 1995.

Briand, L. C.; Emam, K. E.; and Morasca, S.; On the Application of Measurement Theory in Software Engineering. Empirical Software Engineering; 1(1) 61-88,1996

Basili, V.R.; The role of experimentation in software engineering: past, current, and future. Proceedings of the 18th International Conference on Software Engineering (ICSE 1996) p442-449. 25-30 March 1996.

Marvin V. Zelkowitz and Dolores Wallace. Experimental validation in software engineering. Information and Software Technology 39 (11) 735-743. 1997.

Potts, C.; Newstetter, W.C.; Naturalistic inquiry and requirements engineering: reconciling their theoretical foundations. Proceedings of the Third IEEE International Symposium on Requirements Engineering, (RE'97), p118-127. 6-10 Jan.1997

Tichy, W.F.; Should computer scientists experiment more? IEEE Computer 31 (5) 32-40. May 1998.

Robillard, P. N.; d'Astous, P.; Detienne, F.; Visser W.; Measuring Cognitive Activities in Software Engineering. Proceedings of the 20th International Conference on Software Engineering (ICSE'98), 19-25 April 1998.

Tomayko, J. E.; Forging a discipline: An outline history of software engineering education. Annals of Software Engineering 6 (1) 3-18. March 1998.

Lau, F; Towards a framework for action research in information systems studies. Information Technology and People 12 (2) 148-175. 1999.

Klein, H. K.; Myers, M. D.; A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Quarterly 23(1) 67-93. March 1999.

Kellner, M. I.; Madachy, R. J.; Raffo, D. M.; Software Process Simulation Modeling: Why? What? How? Journal of Systems and Software 46 (2-3) 91-105, April 1999.

Atkins, D., Ball, T., Graves, T., Mockus, A.; Using version control data to evaluate the impact of software tools. In Proceedings of the 21st IEEE International Conference on Software Engineering (ICSE'99), Los Angeles, May 16 - 22, 1999.

Seaman, C. B. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 25 (4) 557-572. July 1999.

Bryant, A.; Metaphor, myth and mimicry: The bases of software engineering. Annals of Software Engineering 10 (1-4) 273 - 292. March 2000.

Lewerentz, C.; Rust, H.; Are software engineers true engineers? Annals of Software Engineering 10 (1-4) 311-328. March 2000.

Perry, D. E.; Porter, A. A.; Votta, L. G.; Empirical studies of software engineering: a roadmap. The Future of Software Engineering p345-355. ACM Press, May 2000.

Maibaum. T; Mathematical foundations of software engineering: a roadmap. The Future of Software Engineering p161-172. ACM Press, May 2000.

Tichy, W. F.; Hints for Reviewing Empirical Work in Software Engineering. Empirical Software Engineering 5 (4) 309-312. December 2000.

Pfleeger, S. L; Kitchenham, B. A; Principles of Survey Research Part 1. Part 2. Part 3. Part 4. Part 5. Part 6. Software Engineering Notes, Nov 2001 - Mar 2003

Singer J.; Vinson, N.; Why and How Research Ethics Matters to You. Yes, You! Empirical Software Engineering 6 (4) 287-290. December 2001. (This is an intro to a special issue on research ethics. The rest of the issue is also relevant)

Bratthall, L.; Jørgensen, M.; Can you Trust a Single Data Source Exploratory Software Engineering Case Study? Empirical Software Engineering 7 (1) 9 - 26. March 2002.

Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.; El Emam, K.; Rosenberg, J.; Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28 (8), 721-734. Aug. 2002.

Shull, F.; Basili, V.; Carver, J.; Maldonado, J.C.; Travassos, G.H.; Mendonca, M.; Fabbri, S.; Replicating software engineering experiments: addressing the tacit knowledge problem. Proceedings, International Symposium on Empirical Software Engineering (ISES'02). p7-16. 3-4 Oct. 2002.

Singer, J; Vinson, N. G; Ethical Issues in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 28 (12) 1171-1180. December, 2002.

Shaw, M.; Writing good software engineering research papers. Proceedings. 25th International Conference on Software Engineering (ICSE 2003). p726-736. 3-10 May 2003. (also: Mary Shaw's slides from this minitutorial)

Sim, S.; Easterbrook, S. M.; Holt, R. C.; Using Benchmarking to Advance Research: A Challenge to Software Engineering. Proceedings, ICSE-2003

Kitchenham, B.; Fry, J.; Linkman, S.; The case against cross-over designs in software engineering. Eleventh Annual International Workshop on Software Technology and Engineering Practice p65 - 67. 19-21 Sept. 2003.

Davison, R. M.; Martinsons, M. G.; Kock, N.; Principles of Canonical Action Research. Information Systems Journal 14, 65-86, January 2004.

Kitchenham, B.A.; Dyba, T.; Jorgensen, M.; Evidence-based software engineering. Proceedings. 26th International Conference on Software Engineering (ICSE 2004), p273-281. 23-28 May 2004.

Jorgensen, M.; Sjoberg, D. I.; Generalization and Theory-Building in Software Engineering Research. Proceedings, Workshop on Empirical Assessment in Software Engineering (EASE'04), at ICSE'04 p29-36. IEE Proceedings.

Carver, J.; VanVoorhis, J.; Basili, V.; Understanding the impact of assumptions on experimental validity. Proceedings, International Symposium on Empirical Software Engineering, (ISESE '04), p251-260. 19-20 Aug 2004.

Lethbridge, T. C.; Sim, S. E.; Singer, J; Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering 10 (3) 311-341. July 2005

Ioannidis, J. P. A.; Why Most Published Research Findings Are False. PLoS Med. 2005 August; 2(8): e124.

Sjoeberg, D.I.K.; Hannay, J.E.; Hansen, O.; Kampenes, V.B.; Karahasanovic, A.; Liborg, N.-K.; Rekdal, A.C.; A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31 (9) 733-753. Sept. 2005.

Flyvbjerg, B.; Five Misunderstandings about Case Study Research. Qualitative Inquiry 12 (2) 219-245, April 2006.

Sjoberg, D. I., Dyba, T., and Jorgensen, M. 2007. The Future of Empirical Methods in Software Engineering Research. In 2007 Future of Software Engineering (May 23 - 25, 2007). International Conference on Software Engineering. IEEE Computer Society, Washington, DC, 358-378.

Lutters W. G., and C. B. Seaman, Revealing actual documentation usage in software maintenance through war stories. Information and Software Technology, Volume 49 , Issue 6 (June 2007) Pages 576-587

Parnas, D. Stop the Numbers Game. Communications of the ACM, Volume 50 , Issue 11, Pages: 19 - 21, November 2007.

Easterbrook, S. M., Singer, J., Storey, M, and Damian, D. Selecting Empirical Methods for Software Engineering Research. In F. Shull and J. Singer (eds) "Guide to Advanced Empirical Software Engineering", Springer, 2007.

Shull, F. J., Carver, J. C., Vegas, S. and Juristo, N., The role of replications in Empirical Software Engineering. Empirical Software Engineering 13 (2), p211-218. April 2008

Kitchenham, B. The Role of Replications in Software Engineering - A word of Warning. Empirical Software Engineering 13 (2), p219-221. April 2008.

Lung, J., Aranda, J., Easterbrook, S. M. and Wilson, G. V., On the Difficulty of Replicating Human Subjects Studies in Software Engineering. 30th ACM/IEEE International Conference on Software Engineering (ICSE'2008), Leipzig, Germany, May 10-18, 2008.

Zelkowitz, M. An update to experimental models for validating computer technology. Journal of Systems and Software, 82 (3) p373-376, March 2009.

Kaptein M, Robertson J. Rethinking statistical analysis methods for CHI. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI'2012.

Stanley, D. J., & Spence, J. R. (2014). Expectations for Replications: Are Yours Realistic? Perspectives on Psychological Science, 9(3), 305–318. doi:10.1177/1745691614528518