CSC2130S: Empirical Research Methods in Software Engineering

(Winter session, 2009)

NOTE: This is an old version of the course. The current course is here.


Note:

Course Seminars are scheduled for Wednesday afternoons, 2-5pm in BA5256. There is no seminar in the first week of term; the first seminar will be on Wednesday Jan 14, 2009.

About the Course

This course will explore the role of empiricism in software engineering research, and will prepare students for advanced research in SE by examining how to plan, conduct and report on empirical investigations. The course will cover all of the principal methods applicable to SE: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies, and will relate these methods to relevant metatheories in the philosophy and sociology of science. The course will critically review published examples of work that used each of the principle methods, both from within SE and from other disciplines. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing.

Please note that this is an advanced software engineering course - students will be expected to already have a strong grasp of the key principles of software engineering and the common methods used in software practice.

The course is aimed at students who plan to conduct software engineering research that demands some form of empirical validation, or who wish to establish an empirical basis for a research programme. In most cases, such empirical work focuses on people - examining the ways in which software developers work, and how new tools and techniques affect their ability to construct high quality software efficiently. For this reason we draw heavily on the behavioural sciences for suitable empirical methods, and we focus on both qualitative and quantitative data analysis. Beyond Software Engineering, the course may be of interest to students in HCI, and perhaps those interested in studies of human cognition. Note that we will *not* cover the kinds of experimental techniques used in systems areas.

Pages from previous years' iterations of this course are still available:

Some similar courses at other Universities exist, and may have useful material relevant to this course:

Project portal for this course: DrProject

Course Outline

CSC2130 Course Materials
  Seminar Topic & Notes Background Readings

Week 2
(Jan 14, 2009)

Introduction & Orientation
  • Course objectives
  • Assessments
  • Intro to Research Idioms

Slides used this week

Read this sometime this week:

Week 3
(Jan 21, 2009)

What is Science?

  • Philosophy of Science
  • Sociology of Science
  • Metatheories

Read these papers prior to the seminar:

  1. Creswell - "Research Design: Qualitative, Quantitative, and Mixed Methods Approaches" (Chapter 1)
  2. McComas, "The Principal Elements of the Nature of Science: Dispelling the Myths

Bonus: How science really works

Week 4
(Jan 28, 2009)

What is Software Engineering?

  • Engineering & Design
  • Disciplinary Analogies for SE
  • Evidence-based software engineering
  • What makes a good SE research paper?
  • slides used this week

Read these papers prior to the seminar:

  1. Shaw: "Writing good software engineering research papers"
  2. Kitchenham et al: Preliminary guidelines for empirical research in software engineering
Week 5
(Feb 4, 2009)

Basics of Doing Research

  • Finding good research questions
  • Theory building
  • Research Design
  • Ethics
  • Evidence and Measurement
  • Sampling
  • Peer Review Process
  • Intro to basic stats

Read these papers prior to the seminar:

  1. Singer & Vinson: Ethical Issues in Empirical Studies of Software Engineering
  2. Two short papers
    1. Ioannidis: Why Most Published Research Findings Are False.
    2. Jorgensen & Sjoberg's Generalization and Theory-Building in Software Engineering Research.
Week 6
(Feb 11, 2009)

Experiments

Read these papers prior to the seminar:

Week 7
(Feb 18, 2009)
Reading Week - No seminar
Week 8
(Feb 25, 2009)

Case Studies

Read these prior to the seminar:

Week 9
(Mar 4, 2009)

Survey and Observation

  • Surveys
  • Focus Groups
  • Ethnographies

Read these prior to the seminar:

Week 10
(Mar 11, 2009)

Histories and Simulations

  • Artifact Analysis
  • Archival Analysis and Post-mortems
  • Simulation Techniques

Read these prior to the seminar:

Week 11
(Mar 18, 2009)

Interventions

  • Action Research
  • Pilot Studies
  • Benchmarking

Read these prior to the seminar:

Week 12
(Mar 25, 2009)

Analysis Methods

  • Qualitative, Quantitative and Mixed approaches
  • Statistical Analysis
  • Grounded Theory

Slides used this week

Read these prior to the seminar:

Week 13
(Apr 1, 2009)

More Stats

  • Statistical Power Analysis
  • Meta-Analysis

Read these prior to the seminar:

Week 14
(Apr 8, 2009)

Reporting and Publishing

  • Displaying data
  • Writing up results
  • Where to publish
No Readings - class will be a "mock" PC meeting.

Extra Seminar

TBD

Course Summary

  • When to use empirical methods
  • When NOT to use empirical methods
  • How to select a research method
TBD

Books & other Resources

See also the resource list at ESERNET - the Experimental Software Engineering Network

1. Books

Blum. B. I. Beyond Programming: To A New Era of Design. Oxford University Press. 1996.

Chalmers, A. Science and Its Fabrication. University of Minnesota Press. 1990.

Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Second Edition. Sage. 2002.

Crotty, M. Foundations of Social Research: Meaning and Perspective in the Research Process. Sage. 1998.

Endres, A.; Rombach, D.; A Handbook of Software and Systems Engineering: Empirical Observations, Laws, and Theories. Addison Wesley, 2003.

Meltzoff, J. Critical Thinking About Research: Psychology and Related Fields. American Psychological Association, 1998.

Shull, F.; Singer J.; Sjoberg, D.I.K. (eds); Guide to Advanced Empirical Software Engineering. Springer, 2007.

Strauss, A; Corbin, J; Basics of Qualitative Research : Techniques and Procedures for Developing Grounded Theory. Sage 1998.

2. Papers on Software Engineering Research Methodology

NOTE: I've arranged these in chronological order - there has been considerable maturation of empirical software engineering over the past 15 years or so - the later papers offer increasingly sophisticated ideas about how to do empirical research in SE.

Curtis, B.; Measurement and experimentation in software engineering. Proceedings of the IEEE 68(9) 1144-1157. Sept. 1980.

Galliers R. D.; Land F. F.; Viewpoint: choosing appropriate information systems research methodologies. Communications of the ACM 30(11) 901 - 902. November 1987. (see also followup correspondence in CACM 31 (12) 1502-1506.)

Shaw,M.; Prospects for an Engineering Discipline of Software. IEEE Software 7 (6) 15-24. November 1990.

Fenton, N.; Pfleeger, S.L.; Glass, R.L.; Science and substance: a challenge to software engineers. IEEE Software 11 (4) 86-95. Jul 1994.

Pfleeger, S.L.; Experimental design and analysis in software engineering. Annals of Software Engineering 1, 219–253. 1995.

Kitchenham, B.; Pickard, L.; Pfleeger, S.L. Case studies for method and tool evaluation. IEEE Software 12 (4) 52-62.  July 1995.

Briand, L. C.; Emam, K. E.; and Morasca, S.; On the Application of Measurement Theory in Software Engineering. Empirical Software Engineering; 1(1) 61-88,1996

Basili, V.R.; The role of experimentation in software engineering: past, current, and future. Proceedings of the 18th International Conference on Software Engineering (ICSE 1996) p442-449. 25-30 March 1996.

Marvin V. Zelkowitz and Dolores Wallace. Experimental validation in software engineering. Information and Software Technology 39 (11) 735-743. 1997.

Potts, C.; Newstetter, W.C.; Naturalistic inquiry and requirements engineering: reconciling their theoretical foundations. Proceedings of the Third IEEE International Symposium on Requirements Engineering, (RE'97), p118-127. 6-10 Jan.1997

Tichy, W.F.; Should computer scientists experiment more? IEEE Computer 31 (5) 32-40. May 1998.

Robillard, P. N.; d'Astous, P.; Detienne, F.; Visser W.; Measuring Cognitive Activities in Software Engineering. Proceedings of the 20th International Conference on Software Engineering (ICSE'98), 19-25 April 1998.

Tomayko, J. E.; Forging a discipline: An outline history of software engineering education. Annals of Software Engineering 6 (1) 3-18. March 1998.

Lau, F; Towards a framework for action research in information systems studies. Information Technology and People 12 (2) 148-175. 1999.

Klein, H. K.; Myers, M. D.; A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Quarterly 23(1) 67-93. March 1999.

Kellner, M. I.; Madachy, R. J.; Raffo, D. M.; Software Process Simulation Modeling: Why? What? How? Journal of Systems and Software 46 (2-3) 91-105, April 1999.

Atkins, D., Ball, T., Graves, T., Mockus, A.; Using version control data to evaluate the impact of software tools. In Proceedings of the 21st IEEE International Conference on Software Engineering (ICSE'99), Los Angeles, May 16 - 22, 1999.

Seaman, C. B. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 25 (4) 557-572. July 1999.

Bryant, A.; Metaphor, myth and mimicry: The bases of software engineering. Annals of Software Engineering 10 (1-4) 273 - 292. March 2000.

Lewerentz, C.; Rust, H.; Are software engineers true engineers? Annals of Software Engineering 10 (1-4) 311-328. March 2000.

Perry, D. E.; Porter, A. A.; Votta, L. G.; Empirical studies of software engineering: a roadmap. The Future of Software Engineering p345-355. ACM Press, May 2000.

Maibaum. T; Mathematical foundations of software engineering: a roadmap. The Future of Software Engineering p161-172. ACM Press, May 2000.

Tichy, W. F.; Hints for Reviewing Empirical Work in Software Engineering. Empirical Software Engineering 5 (4) 309-312. December 2000.

Pfleeger, S. L; Kitchenham, B. A; Principles of Survey Research Part 1. Part 2. Part 3. Part 4. Part 5. Part 6. Software Engineering Notes, Nov 2001 - Mar 2003

Singer J.; Vinson, N.; Why and How Research Ethics Matters to You. Yes, You! Empirical Software Engineering 6 (4) 287-290. December 2001. (This is an intro to a special issue on research ethics. The rest of the issue is also relevant)

Bratthall, L.; Jørgensen, M.; Can you Trust a Single Data Source Exploratory Software Engineering Case Study? Empirical Software Engineering 7 (1) 9 - 26. March 2002.

Kitchenham, B.A.; Pfleeger, S.L.; Pickard, L.M.; Jones, P.W.; Hoaglin, D.C.; El Emam, K.; Rosenberg, J.; Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28 (8), 721-734. Aug. 2002.

Shull, F.; Basili, V.; Carver, J.; Maldonado, J.C.; Travassos, G.H.; Mendonca, M.; Fabbri, S.; Replicating software engineering experiments: addressing the tacit knowledge problem. Proceedings, International Symposium on Empirical Software Engineering (ISES'02). p7-16. 3-4 Oct. 2002.

Singer, J; Vinson, N. G; Ethical Issues in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering 28 (12) 1171-1180. December, 2002.

Shaw, M.; Writing good software engineering research papers. Proceedings. 25th International Conference on Software Engineering (ICSE 2003). p726-736. 3-10 May 2003. (also: Mary Shaw's slides from this minitutorial)

Sim, S.; Easterbrook, S. M.; Holt, R. C.; Using Benchmarking to Advance Research: A Challenge to Software Engineering. Proceedings, ICSE-2003

Kitchenham, B.; Fry, J.; Linkman, S.; The case against cross-over designs in software engineering. Eleventh Annual International Workshop on Software Technology and Engineering Practice p65 - 67. 19-21 Sept. 2003.

Davison, R. M.; Martinsons, M. G.; Kock, N.; Principles of Canonical Action Research. Information Systems Journal 14, 65-86, January 2004.

Kitchenham, B.A.; Dyba, T.; Jorgensen, M.; Evidence-based software engineering. Proceedings. 26th International Conference on Software Engineering (ICSE 2004), p273-281. 23-28 May 2004.

Jorgensen, M.; Sjoberg, D. I.; Generalization and Theory-Building in Software Engineering Research. Proceedings, Workshop on Empirical Assessment in Software Engineering (EASE'04), at ICSE'04 p29-36. IEE Proceedings.

Carver, J.; VanVoorhis, J.; Basili, V.; Understanding the impact of assumptions on experimental validity. Proceedings, International Symposium on Empirical Software Engineering, (ISESE '04), p251-260. 19-20 Aug 2004.

Lethbridge, T. C.; Sim, S. E.; Singer, J; Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering 10 (3) 311-341. July 2005

Ioannidis, J. P. A.; Why Most Published Research Findings Are False. PLoS Med. 2005 August; 2(8): e124.

Sjoeberg, D.I.K.; Hannay, J.E.; Hansen, O.; Kampenes, V.B.; Karahasanovic, A.; Liborg, N.-K.; Rekdal, A.C.; A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31 (9) 733-753. Sept. 2005.

Flyvbjerg, B.; Five Misunderstandings about Case Study Research. Qualitative Inquiry 12 (2) 219-245, April 2006.

Sjoberg, D. I., Dyba, T., and Jorgensen, M. 2007. The Future of Empirical Methods in Software Engineering Research. In 2007 Future of Software Engineering (May 23 - 25, 2007). International Conference on Software Engineering. IEEE Computer Society, Washington, DC, 358-378.

newLutters W. G., and C. B. Seaman, Revealing actual documentation usage in software maintenance through war stories. Information and Software Technology, Volume 49 , Issue 6 (June 2007) Pages 576-587

newParnas, D. Stop the Numbers Game. Communications of the ACM, Volume 50 , Issue 11, Pages: 19 - 21, November 2007.

Easterbrook, S. M., Singer, J., Storey, M, and Damian, D. Selecting Empirical Methods for Software Engineering Research. In F. Shull and J. Singer (eds) "Guide to Advanced Empirical Software Engineering", Springer, 2007.

Shull, F. J., Carver, J. C., Vegas, S. and Juristo, N., The role of replications in Empirical Software Engineering. Empirical Software Engineering 13 (2), p211-218. April 2008

Kitchenham, B. The Role of Replications in Software Engineering - A word of Warning. Empirical Software Engineering 13 (2), p219-221. April 2008.

Lung, J., Aranda, J., Easterbrook, S. M. and Wilson, G. V., On the Difficulty of Replicating Human Subjects Studies in Software Engineering. 30th ACM/IEEE International Conference on Software Engineering (ICSE'2008), Leipzig, Germany, May 10-18, 2008.

newZelkowitz, M. An update to experimental models for validating computer technology. Journal of Systems and Software, 82 (3) p373-376, March 2009.