Some Thoughts on Guidelines for Empirical Research

1. Some Thoughts on Guidelines for Empirical Research Mike Barker Visiting Scholar, NAIST

2. A Guide to theEmpirical ResearchBody of Knowledge2005 VersionERBOK

3. Problems(aka Why do we need guidelines?) • Too little testable theory and evidence • Too little technology transfer (lack of context, evidence, applicability) • Too little comparison, contrast, combination, and replication • Too little life-cycle work: examination of phenomenon, understanding, theory building, theory testing, replication, theory revision, re-evaluation of phenomenon • Empirical research should start with the first introduction to software engineering course – how do we provide a framework and guidance to make those experiments a success? • Empirical research should be useful to other researchers and practitioners – how do we provide a framework and guidance to make the results easily compatible with other research and useful?

4. Questions • What is the purpose of guidelines for empirical research? Who is the audience and what is the message? • Support conduct of research by non-researchers? (e.g. introductory programmers, practitioners, etc.?): focus on standards for doing research • Ensure that results of research are replicable, comparable, etc.?: focus on standards for doing research • Ensure that results of research are easy to apply to practical use?: focus on standards for reporting research

5. Guidelines for Empirical Studies • What would make it easy to use? • Where in the lifecycle is it (e.g. exploratory study, theory testing, replication)? • Fill in the pieces templates, checklists: • Experimental Context • Experimental design • Experimental conduct and data collection • Analysis • Presentation of results • Interpretation of results • Provide guidelines for statistics and other parts of an empirical study that are likely to have problems • Provide standard tools (survey forms, data collection and analysis software, etc.) to make doing empirical studies easy.

6. Guidelines for Presentation/Publication

7. Benefits • Make comparison, contrast, replication significantly easier • Make practical use of results easier • Make original research easier – by giving a background of well-known standard research for comparison. • Build common terminology • Reduce effort and cost for standard research • Make unifying results and knowledge easier • Help to identify empirical research and empirical researchers

8. Deployment • Building on Current Standards: • Guide to the Empirical Research Body of Knowledge (ERBOK)? OR • addition to IEEE SWEBOK as an Appendix or Practice Standard? • Website • Articles • Let's not forget, guidelines need testing too! • Review • Actual use • Revision process • Need to be sure we include a plan for modifications, additions, removals, and extensions.

9. Other • Reasons we SHOULD NOT have guidelines for empirical research: • Loss of originality and exploratory: people will stop pushing the boundaries and just follow the guidelines • Insufficient material: there is a wealth of material on empirical research and research methods. The SE field isn't unique, just use what's already available. • Confuses methodology with relevance and utility: guidelines that show how to should NOT substitute for careful consideration of research goals and application

10. References SWEBOK 2004. Available from <http://www.swebok.org/> PMBOK <http://www.pmi.org/> R. Jeffery and L. Scott, "Has Twenty-five Years of Empirical Software Engineering Made a Difference?", in Proceedings of the Ninth Asia-Pacific Software Engineering Conference (APSEC'02). B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg, "Preliminary Guidelines for Empirical Research in Software Engineering," IEEE Transactions on Software Engineering, vol. 28, pp. 721-734, 2002 Shull, F., Mendonca, M. G., Basili, V., Carver, J., Maldonado, J. C., Fabbri, S. A., Travassos, G. H., & Ferreira, M. C. (2004). Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering, 9(1/2), 111-137. Retrieved July 7, 2004, from the kluweronline database. D. Port & D. Klappholtz, "Performing Empirical Software Engineering Research in the Classroom", Proceedings of the 17th Conference on Software Engineering Education and Training (CSEET’04) D. E. Perry, A. A. Porter, & L. G. Votta, "Empirical Studies of Software Engineering: A Roadmap", in "The Future of Software Engineering", 2000. Available at http://www.cs.ucl.ac.uk/staff/A.Finkelstein/fose/finalporter.pdf