1 / 32

Readers Project: Quality aspects in Empirical Studies

Readers Project: Quality aspects in Empirical Studies. 13th International Software Engineering Research Network ISERN 2005. José Carlos Maldonado jcmaldon@icmc.usp.br ICMC/USP Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo – São Carlos.

martha
Download Presentation

Readers Project: Quality aspects in Empirical Studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Readers Project:Quality aspects in Empirical Studies 13th International Software Engineering Research Network ISERN 2005 José Carlos Maldonado jcmaldon@icmc.usp.br ICMC/USP Instituto de Ciências Matemáticas e de ComputaçãoUniversidade de São Paulo – São Carlos Noosa Heads, Austrália - November, 2005

  2. Readers’ Team ISERN-2005 Universidade de São Paulo READERS: PBR studies University of Maryland José Carlos Maldonado (Brazil Coordinator) Victor Basili (USA Coordinator) Forrest Shull Jeffrey Carver PBR Secondary Study Universidade Federal do Rio de Janeiro Universidade Federal de São Carlos Competence Centers Portal Sandra Fabbri Guilherme Horta Travassos Manoel Mendonça Universidade Salvador

  3. Readers Project ISERN-2005 Readers Project • NSF-CNPQ Readers Project • - A Collaborative Research to Develop, Validate, and Package Reading Techniques for Software Defect Detection • Investigating replications and transfer of experimental knowledge issues • Basili-Selby (Extended) Experiment • Implementation Level • Inspection technique X Testing techniques • PBR Experiment • Requirement Specification Level • Inspection techniques

  4. Readers: PBR studies ISERN-2005 Readers Project PBR teams X Non-PBR teams

  5. PBR Studies: questions ISERN-2005 Readers Project 1) Do teams applying PBR detect more defects than teams applying the comparison technique? 2) Do individual reviewers using PBR find more defects than those using the comparison technique? 3) Does a reviewer’s experience affect her effectiveness? 4) Does a reviewer, individually, find out different defects applying PBR and non-PBR techniques? 5) Do the PBR perspectives have the same effectiveness and efficiency? 6) Do PBR perspectives find different defects?

  6. Replications: difficulties ISERN-2005 Readers Project • Difficulty to assemble a complete and consistent lab package for the replication; • Version control and configuration problems; • Replications had different cultural settings; • Language, application domain, … • Training; • Meta-analysis: data collection, … • Pilot study • To understand lab package and how the experimental process should be conducted.

  7. Replications: Lab Package Evolution ISERN-2005 Readers Project • Criteria for artifact (lab package) evolution??? • Lists of • defects • false positives • non-defects (should be elaborated?) • Updated Forms • Defect Collection (additional fields) • Annalist Survey • Feedback • Training material • Steps • High x low level • PBR underlying techniques • Error guessing • Equivalence partitioning • Smaller Requirement Document

  8. Experimental Knowledge Sharing Model ISERN-2005 Readers Project SHULL, F.; MENDONÇA, M.; BASILI, V.; CARVER, J.; MALDONADO, J.; FABBRI, S.; TRAVASSOS, G.; OLIVEIRA, M. Knowledge-Sharing Issues in Experimental Software Engineering. Empirical Software Engineering An International Journal. Kluwer Academic Publishers, v.9, n.1-2, p.111 - 137, 2004.

  9. Replications: collaboration mechanisms ISERN-2005 Readers Project • Knowledge Transfer Process • Series of team’s workshops (6 months interval) • e-mail and Web Portals • Knowledge Repository • e-Workshops and Local Workshops >>>>supporting the steps of EKSM

  10. Experimental Improvement Paradigm ISERN-2005 Readers Project MENDONÇA, M., MALDONADO, J. C., OLIVEIRA, M. C. F., FABBRI, S., SHULL, F., CARVER, J.; BASILI, V. and TRAVASSOS, G. H. The Experimentation Improvement Paradigm for Software Engineering. (to be submitted).

  11. Lab Package Quality Goals ISERN-2005 Readers Project • Access to experimental artifacts • Adequate and complete training materials • Accurate estimates for the time required for subjects to execute the experiment • Presence of oracles and golden artifacts • Ease of package evolution and experimental feedback • Clear description of communications channels by which the original experimenters and, If possible, related experimental community can be contacted

  12. Lab Package Quality Goals ISERN-2005 Readers Project • Complete descriptions of the • analysis and goal of the experiment • experimental design • Including threats to validity • context(s) in which the experiment was run • process to run the experiment • Process conformance • Tacit knowledge • Collaboration structure

  13. Knowledge Sharing in a Technology Transfer to Industry using a Lab Package ISERN-2005 • Lab Package adaptation • Application domain and techniques training • Selection/elaboration of specific domain artifacts • Running and evaluating pilot study; • Evaluation and adjustment of the lab package in the specific context • Running another pilot study to valuate the adapted lab package. Readers Project • Replicator training • Study lab package, experimentation process • Pilot study: observation, running and evaluation • Disseminate into the organization EKSM in a Technology Transfer to Industry

  14. Replications: Main Agents ISERN-2005 Readers Project Replicator: • What are the roles of the Replicator? • What would be the “conditions” (requirements) to be satisfied? • Should the Replicator be trained? • Has s/he been a subject? • Has s/he been a observer of a replication? • Should s/he be evaluated against the concepts of experimental studies and experimental projects? • ??? Replications: Main Agents

  15. Replications: Main Agents ISERN-2005 Readers Project Trainer: • What would be the “requirements” to be a trainer? • Should the trainer be trained? • Has s/he been a subject? • Has s/he been a replication observer? • Should s/he be evaluated against the techniques involved (independent variables)? • ??? Others.. Replications: Main Agents

  16. Replications: Main Agents ISERN-2005 Readers Project Data Analyzer: • What would be the requirements to be a Data Analyzer? • Background on statistical techniques? X • Background on experimentation and the techniques under study? Replications: Main Agents

  17. Secondary Study: observations on PBR studies ISERN-2005 Systematic Review • 19 experimental studies about PBR found. • 14 studies about PBR effectiveness against (Checklist or ad-hoc) effectiveness • 57% - PBR is more effective. • 29% - not succeed to indicate PBR effectiveness superiority. • 14% - not obtain expressive difference. Secondary studies Systematic Review MAFRA, S. N.; TRAVASSOS, G. H. Técnicas de Leitura de Software: Uma Revisão Sistemática. In Proceedings of 19th Brazilian Symposium on Software Engineering. P. 72-87. October, 2005.

  18. Secondary Study: observations on PBR studies ISERN-2005 Systematic Review • Controversial Points: • PBR process conformance • Critical defects • PBR perspectives overlaying • Reviewer’s expertise X effectiveness Secondary studies Systematic Review MAFRA, S. N.; TRAVASSOS, G. H. Técnicas de Leitura de Software: Uma Revisão Sistemática. In Proceedings of 19th Brazilian Symposium on Software Engineering. P. 72-87. October, 2005.

  19. Competence Center Portals ISERN-2005 • Highly connected clusters (“communities”) [close-knit circles of “competences” (friends)] linked by a few weak ties between acquaintances belonging to different “circles” of “competences” • Experimentation Process • VV&T Competence • Formal Methods • Software Process • Open Source Software • …. Competence Center Portals

  20. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Competence Centers Portal

  21. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Source Software Competence Center Competence Centers Portal

  22. Experiment Types ISERN-2005 Competence Centers Portal Experiment Types TRAVASSOS, G.H.; BARROS, M.O. Contributions of In virtuo and In Silico Experiments for the future of Empirical Studies in Software Engineering. Proceedings of the ESEIW 2003 Workshop on Empirical Studies in Software Engineering - WSESE 2003 Workshop Series on Empirical Studies in Software Engineering (discussed at the ISERN 2003)

  23. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Source Software Competence Center Competence Centers Portal

  24. Visual Data Mining ISERN-2005 Competence Centers Portal Visual Data Mining based process

  25. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Source Software Competence Center Competence Centers Portal

  26. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Source Software Competence Center Competence Centers Portal

  27. ESE maturity process ISO 12207, generic experimentation process Packaging process Types of Experiments, experimentation cycle, instruments, techniques, tools Experiment’s features Definition Standard Experimentation Process (SEP) Specialization by type of experiment (In Virtuo, In Vivo, In Vitro, In Silico) SEP Specialized 1 ... SEP Specialized N SEP Specialized 2 Instantiation by experiment Process of the Experiment 1 Process of the Experiment 2 ... Process of the Experiment N Defining experimentation processes for eSEE using TABA Workstation infra-structure (see ISESE 2005 late breaking paper session) ISERN-2005 Competence Centers Portal ESE maturity process www.cos.ufrj.br/~ese

  28. Competence Centers Portal ISERN-2005 • VV&T Competence Center Portal • Lessons Learned • Experiments • Lab packages • Experiment Types • Visual Data Mining • Distributed Experiments • Open Source Tools • Open Source Training Material • ESE maturity process • Open Source Software Competence Center Competence Centers Portal

  29. Qualipso Proposal ISERN-2005 Competence Centers Portal

  30. Industry and Academy cooperation Industry Academy Experimental Studies Real Projects Experimental Studies Knowledge Base (ISERN, CeBASE, …) ISERN-2005 Readers Project Cooperation and integration between industry and academy in the technology development and evaluation are fundamental. Industry and Academy Cooperation HÖHN, E. N.; MALDONADO, J. C.; MENDONÇA, M.; FABBRI, S. VILLAS BOAS, A. L.; TAMBASCIA, C.; FREITAS, M. E.; PAGLIUSO, P. PBR: Transferência de Tecnologia Baseada em Pacotes de Experimentação. In Proceedings of SBQS. 2004.

  31. ISERN-2005 Quality depends on the ESE Process Maturity Level • ( ) 1 • ( ) 2 • ( ) 3 • ( ) 4 • ( ) 5

  32. ISERN-2005 ESE Process Maturity Level?? • ( ) 1 • ( ) 2 • ( ) 3 • ( ) 4 • ( ) 5  FIM

More Related