CSC2130S: Empirical Research Methods in Software Engineering

(Winter session, 2012)


Course Seminars are scheduled for Wednesday afternoons, 2-5pm in HA 410 1-4pm in BA5256. The first seminar will be on Wednesday Jan 11, 2012.

newNote: Final term papers are due May 4, 2012. We're using the IEEE format templates.

About the Course

This course will explore the role of empiricism in software engineering research, and will prepare students for advanced research in SE by examining how to plan, conduct and report on empirical investigations. The course will cover all of the principal methods applicable to SE: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies, and will relate these methods to relevant metatheories in the philosophy and sociology of science. The course will critically review published examples of work that used each of the principle methods, both from within SE and from other disciplines. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing.

Please note that this is an advanced software engineering course - students will be expected to already have a strong grasp of the key principles of software engineering and the common methods used in software practice.

The course is aimed at students who plan to conduct software engineering research that demands some form of empirical validation, or who wish to establish an empirical basis for a research programme. In most cases, such empirical work focuses on people - examining the ways in which software developers work, and how new tools and techniques affect their ability to construct high quality software efficiently. For this reason we draw heavily on the behavioural sciences for suitable empirical methods, and we focus on both qualitative and quantitative data analysis. Beyond software engineering, the course may be of interest to students in HCI, and perhaps those interested in studies of human cognition. Note that we will *not* cover the kinds of experimental techniques used in systems areas.

Pages from previous years' iterations of this course are still available:

Some similar courses at other Universities exist, and may have useful material relevant to this course:

Course Outline

CSC2130 Course Materials
  Seminar Topic & Notes Background Readings

Week 1
(Jan 11, 2012)

Introduction & Orientation (slides)
  • Course objectives
  • Assessments
  • Intro to Research Idioms

Read this sometime this week:

  1. Easterbrook et al: Selecting Empirical Methods for Software Engineering Research
Week 2
(Jan 18, 2012)

What is Science? (slides)

  • Philosophy of Science
  • Sociology of Science
  • Metatheories

Read these two papers prior to the seminar:

  1. McComas, "The Principal Elements of the Nature of Science: Dispelling the Myths
  2. Creswell - "Research Design: Qualitative, Quantitative, and Mixed Methods Approaches" (Chapter 1)
  3. Bonus: How science really works

Alex and Natasha's Discussion Slides

Week 3
(Jan 25, 2012)

Research Design and Ethics

  • Evidence-based software engineering
  • What makes a good research paper?
  • Research Design
  • Research Ethics

Read these two papers prior to the seminar:

  1. Kitchenham et al: Preliminary guidelines for empirical research in software engineering
  2. Singer & Vinson: Ethical Issues in Empirical Studies of Software Engineering
Bonus: Shaw: "Writing good software engineering research papers"

Tah, Lev & Elizabeth's discussion slides

Week 4
(Feb 1, 2012)

Basics of Doing Research (slides)

  • Finding good research questions
  • Theory building
  • Evidence and Measurement
  • Peer Review Process

Read these papers prior to the seminar:

  1. Tedre & Sutinen: Three traditions of computing: what educators should know
  2. Ioannidis: Why Most Published Research Findings Are False.
  3. Jorgensen & Sjoberg's Generalization and Theory-Building in Software Engineering Research.

Uzma's discussion slides

Bonus paper: Sjoberg et al, Building Theories in Software Engineering

Week 5
(Feb 8, 2012)

Experiments (slides)

  • Controlled Experiments
  • Quasi-experiments
  • Sampling
  • Replication

Read these papers prior to the seminar:

Frank & Mike's Discussion Slides

Week 6
(Feb 15, 2012)

Case Studies (slides)

  • Single and Multi-case
  • Longitudinal Case Studies
  • Approaches to Data Collection

Read these prior to the seminar:

Janis & Cloud's Discussion slides

Week 7
(Feb 22, 2012)

Reading Week - No seminar


Week 8
(Feb 29, 2012)

Survey and Observation

  • Surveys
  • Focus Groups
  • Ethnographies

Read these prior to the seminar:

Week 9
(Mar 7, 2012)


  • Action Research
  • Pilot Studies
  • Benchmarking

Read these prior to the seminar:

Jan & Josh's discussion slides

Week 10
(Mar 14, 2012)

Qualitative Analysis

  • Grounded Theory
  • Phenomenography
  • Mixed Methods Research

Read these prior to the seminar:

Nick's discussion slides

Bonus papers: Guest, How Many Interviews are Enough?

Week 11
(Mar 21, 2012)

Quantitative Analysis

  • Basic Stats
  • Significant figures!
  • Choosing a statistical model
  • Statistical Power Analysis
  • Meta-Analysis

Read these prior to the seminar:

Yi's discussion slides and Yanshuai's discussion slides

Background papers: The introductory sections of Statsoft's electronic textbook; and Cohen: A Power Primer; see also The dance of the P values.

Week 12
(Mar 28, 2012)

Publishing and Reviewing

  • Where to publish
  • The peer review process

Class will be a "mock" PC meeting. Background reading on peer review:

Week 13
(Apr 4, 2012)

Replication and Beyond

  • How important is replication?
  • Bias and Influences
  • Threats to Validity (and how to reduce them)
  • When to use empirical methods
  • When NOT to use empirical methods
Readings TBD

Books & other Resources

See also the resource list at ESERNET - the Experimental Software Engineering Network

1. Books

Blum. B. I. Beyond Programming: To A New Era of Design. Oxford University Press. 1996.

Chalmers, A. Science and Its Fabrication. University of Minnesota Press. 1990.

Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Second Edition. Sage. 2002.

Crotty, M. Foundations of Social Research: Meaning and Perspective in the Research Process. Sage. 1998.

Endres, A.; Rombach, D.; A Handbook of Software and Systems Engineering: Empirical Observations, Laws, and Theories. Addison Wesley, 2003.

Meltzoff, J. Critical Thinking About Research: Psychology and Related Fields. American Psychological Association, 1998.

Shull, F.; Singer J.; Sjoberg, D.I.K. (eds); Guide to Advanced Empirical Software Engineering. Springer, 2007.

Strauss, A; Corbin, J; Basics of Qualitative Research : Techniques and Procedures for Developing Grounded Theory. Sage 1998.

2. Papers on Software Engineering Research Methodology

NOTE: I've arranged these in chronological order - there has been considerable maturation of empirical software engineering over the past 15 years or so - the later papers offer increasingly sophisticated ideas about how to do empirical research in SE.

Curtis, B.; Measurement and experimentation in software engineering. Proceedings of the IEEE 68(9) 1144-1157. Sept. 1980.

Galliers R. D.; Land F. F.; Viewpoint: choosing appropriate information systems research methodologies. Communications of the ACM 30(11) 901 - 902. November 1987. (see also followup correspondence in CACM 31 (12) 1502-1506.)

Shaw,M.; Prospects for an Engineering Discipline of Software. IEEE Software 7 (6) 15-24. November 1990.

Fenton, N.; Pfleeger, S.L.; Glass, R.L.; Science and substance: a challenge to software engineers. IEEE Software 11 (4) 86-95. Jul 1994.

Pfleeger, S.L.; Experimental design and analysis in software engineering. Annals of Software Engineering 1, 219–253. 1995.

Kitchenham, B.; Pickard, L.; Pfleeger, S.L. Case studies for method and tool evaluation. IEEE Software 12 (4) 52-62.  July 1995.

Briand, L. C.; Emam, K. E.; and Morasca, S.; On the Application of Measurement Theory in Software Engineering. Empirical Software Engineering; 1(1) 61-88,1996

Basili, V.R.; The role of experimentation in software engineering: past, current, and future. Proceedings of the 18th International Conference on Software Engineering (ICSE 1996) p442-449. 25-30 March 1996.

Marvin V. Zelkowitz and Dolores Wallace. Experimental validation in software engineering. Information and Software Technology 39 (11) 735-743. 1997.

Potts, C.; Newstetter, W.C.; Naturalistic inquiry and requirements engineering: reconciling their theoretical foundations. Proceedings of the Third IEEE International Symposium on Requirements Engineering, (RE'97), p118-127. 6-10 Jan.1997

Tichy, W.F.; Should computer scientists experiment more? IEEE Computer 31 (5) 32-40. May 1998.

Robillard, P. N.; d'Astous, P.; Detienne, F.; Visser W.; Measuring Cognitive Activities in Software Engineering. Proceedings of the 20th International Conference on Software Engineering (ICSE'98), 19-25 April 1998.

Tomayko, J. E.; Forging a discipline: An outline history of software engineering education. Annals of Software Engineering 6 (1) 3-18. March 1998.

Lau, F; Towards a framework for action research in information systems studies. Information Technology and People 12 (2) 148-175. 1999.

Klein, H. K.; Myers, M. D.; A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Quarterly 23(1) 67-93. March 1999.

Kellner, M. I.; Madachy, R. J.; Raffo, D. M.; Software Process Simulation Modeling: Why? What? How? Journal of Systems and Software 46 (2-3) 91-105, April 1999.

Atkins, D., Ball, T., Graves, T., Mockus, A.; Using version control data to evaluate the impact of software tools. In Proceedings of the 21st IEEE International Conference on Software Engineering (ICSE'99), Los Angeles, May 16 - 22, 1999.

Seaman, C. B. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 25 (4) 557-572. July 1999.

Bryant, A.; Metaphor, myth and mimicry: The bases of software engineering. Annals of Software Engineering 10 (1-4) 273 - 292. March 2000.

Lewerentz, C.; Rust, H.; Are software engineers true engineers? Annals of Software Engineering 10 (1-4) 311-328. March 2000.

Perry, D. E.; Porter, A. A.; Votta, L. G.; Empirical studies of software engineering: a roadmap. The Future of Software Engineering p345-355. ACM Press, May 2000.

Maibaum. T; Mathematical foundations of software engineering: a roadmap. The Future of Software Engineering p161-172. ACM Press, May 2000.

Tichy, W. F.; Hints for Reviewing Empirical Work in Software Engineering. Empirical Software Engineering 5 (4) 309-312. December 2000.

Pfleeger, S. L; Kitchenham, B. A; Principles of Survey Research Part 1. Part 2. Part 3. Part 4. Part 5. Part 6. Software Engineering Notes, Nov 2001 - Mar 2003

Singer J.; Vinson, N.; Why and How Research Ethics Matters to You. Yes, You! Empirical Software Engineering 6 (4) 287-290. December 2001. (This is an intro to a special issue on research ethics. The rest of the issue is also relevant)

Bratthall, L.; Jørgensen, M.; Can you Trust a Single Data Source Exploratory Software Engineering Case Study? Empirical Software Engineering 7 (1) 9 - 26. March 2002.

Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.; El Emam, K.; Rosenberg, J.; Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28 (8), 721-734. Aug. 2002.

Shull, F.; Basili, V.; Carver, J.; Maldonado, J.C.; Travassos, G.H.; Mendonca, M.; Fabbri, S.; Replicating software engineering experiments: addressing the tacit knowledge problem. Proceedings, International Symposium on Empirical Software Engineering (ISES'02). p7-16. 3-4 Oct. 2002.

Singer, J; Vinson, N. G; Ethical Issues in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 28 (12) 1171-1180. December, 2002.

Shaw, M.; Writing good software engineering research papers. Proceedings. 25th International Conference on Software Engineering (ICSE 2003). p726-736. 3-10 May 2003. (also: Mary Shaw's slides from this minitutorial)

Sim, S.; Easterbrook, S. M.; Holt, R. C.; Using Benchmarking to Advance Research: A Challenge to Software Engineering. Proceedings, ICSE-2003

Kitchenham, B.; Fry, J.; Linkman, S.; The case against cross-over designs in software engineering. Eleventh Annual International Workshop on Software Technology and Engineering Practice p65 - 67. 19-21 Sept. 2003.

Davison, R. M.; Martinsons, M. G.; Kock, N.; Principles of Canonical Action Research. Information Systems Journal 14, 65-86, January 2004.

Kitchenham, B.A.; Dyba, T.; Jorgensen, M.; Evidence-based software engineering. Proceedings. 26th International Conference on Software Engineering (ICSE 2004), p273-281. 23-28 May 2004.

Jorgensen, M.; Sjoberg, D. I.; Generalization and Theory-Building in Software Engineering Research. Proceedings, Workshop on Empirical Assessment in Software Engineering (EASE'04), at ICSE'04 p29-36. IEE Proceedings.

Carver, J.; VanVoorhis, J.; Basili, V.; Understanding the impact of assumptions on experimental validity. Proceedings, International Symposium on Empirical Software Engineering, (ISESE '04), p251-260. 19-20 Aug 2004.

Lethbridge, T. C.; Sim, S. E.; Singer, J; Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering 10 (3) 311-341. July 2005

Ioannidis, J. P. A.; Why Most Published Research Findings Are False. PLoS Med. 2005 August; 2(8): e124.

Sjoeberg, D.I.K.; Hannay, J.E.; Hansen, O.; Kampenes, V.B.; Karahasanovic, A.; Liborg, N.-K.; Rekdal, A.C.; A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31 (9) 733-753. Sept. 2005.

Flyvbjerg, B.; Five Misunderstandings about Case Study Research. Qualitative Inquiry 12 (2) 219-245, April 2006.

Sjoberg, D. I., Dyba, T., and Jorgensen, M. 2007. The Future of Empirical Methods in Software Engineering Research. In 2007 Future of Software Engineering (May 23 - 25, 2007). International Conference on Software Engineering. IEEE Computer Society, Washington, DC, 358-378.

Lutters W. G., and C. B. Seaman, Revealing actual documentation usage in software maintenance through war stories. Information and Software Technology, Volume 49 , Issue 6 (June 2007) Pages 576-587

Parnas, D. Stop the Numbers Game. Communications of the ACM, Volume 50 , Issue 11, Pages: 19 - 21, November 2007.

Easterbrook, S. M., Singer, J., Storey, M, and Damian, D. Selecting Empirical Methods for Software Engineering Research. In F. Shull and J. Singer (eds) "Guide to Advanced Empirical Software Engineering", Springer, 2007.

Shull, F. J., Carver, J. C., Vegas, S. and Juristo, N., The role of replications in Empirical Software Engineering. Empirical Software Engineering 13 (2), p211-218. April 2008

Kitchenham, B. The Role of Replications in Software Engineering - A word of Warning. Empirical Software Engineering 13 (2), p219-221. April 2008.

Lung, J., Aranda, J., Easterbrook, S. M. and Wilson, G. V., On the Difficulty of Replicating Human Subjects Studies in Software Engineering. 30th ACM/IEEE International Conference on Software Engineering (ICSE'2008), Leipzig, Germany, May 10-18, 2008.

Zelkowitz, M. An update to experimental models for validating computer technology. Journal of Systems and Software, 82 (3) p373-376, March 2009.

Kaptein M, Robertson J. Rethinking statistical analysis methods for CHI. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI'2012.