Obtaining Well-Founded Practices about Elicitation Techniques by Means of an Update of a Previous Systematic Review

1. Obtaining Well-Founded Practices about Elicitation Techniques by Means ofan Update of a Previous Systematic Review Oscar Dieste, Fraunhofer IESE Marta López, Complutensian University of Madrid Felicidad Ramos, Indra Systems

2. Systematic reviews (SR) • Goal: • Pool together the results obtained in different empirical studies and propose recommendations based on the best available evidence • Critical factors: • Amount of evidence available at the time SR is done • Need for updating: • If new empirical studies are discovered (because they were not identified before) or carried out, the conclusions of earlier SRs should be updated, either confirming or refuting the previous findings

3. Our SR instantiation • SR on Requirements Engineering field • Elicitation techniques selection • Relevance: • Techniques applied to extract knowledge from the requirements stakeholders • Critical for improving the communication process among software engineers and stakeholders

4. SRs performed • Previous SR: 53 publications concerningindividual elicitation techniques 26 selected publications (Previous SR review) 27 unavailable publications(initial search)

5. SRs performed • Updating SR: 53 publications concerningindividual elicitation techniques 26 selected publications (Previous SR review) 27 unavailable publications(initial search) 14 unavailable public.(grey literature) 4 useless public. (no empirical studies) 9 useful public. (Update SR review)

6. SR Methodology 0. Training on SR 1. Selection of primary studies 2. Primary studies reading and data extraction 3. Setting up supporting tables 6.Development of results table 4. Treatment analysis 5. Response variables analysis 7. Generalization Pending treatments/response variables? 8. New evidence extraction Yes 9. Aggregation 10. Final results decodification No

7. SR Main Findings • After performing the SR, more than 60 new empirical results were obtained • Combining those empirical results among themselves, as well as with the results of previous SR, evidences which can be later used to identify in which situations a given elicitation technique is useful can obtained • Aggregation process, see • Dieste, O. and Juristo, N., "Systematic Review and Aggregation of Empirical Studies on Elicitation Techniques," IEEE Transactions on Software Engineering, vol. 2008 • Dieste, O."TR5. Agregación de las evidencias obtenidas de los estudios empíricos relevantes," http://grise.upm.es/research_documents.php, 2006

8. SR Main Findings • After performing the SR, more than 60 new empirical results were obtained

9. SR Main Findings • Types of findings: • Refutes • The newly gathered results refutes a previous evidence • Reduces • The new results cannot refute a previous evidence, but reduces our confidence in it • Reinforces • A previous evidence is supported with new compatible results • New • A new evidence, not present in the existing set, has been identified

10. Discussion • Focusing on interviews, • Effectiveness and efficiency: • In the previous SR, it was found that interviews were overall the most effective elicitation technique, although some contrived techniques like laddering or card sorting were equally effective in some cases. The current work strengthens this conclusion. No empirical result contesting such effectiveness has been identified • Time-consuming: • Interviews may be more time-consuming than other more-focused techniques like sorting or protocol analysis. Evidence 16 supports that fact (study suggest that interviews take more time than protocol analysis)

11. Conclusions • Updating Systematic Review • Identifying well-founded practices when selecting an elicitation requirements technique • Results: • Interviews are in average the most effective elicitation technique • Future work: • Combine findings from empirical and theoretical works and expect opinion to develop a comprehensive theory concerning the application of elicitation techniques

12. SR References • References used in the updating work [1] Burton, A. M., Shadbolt, N. R., Rugg, G., and Hedgecock, A. P., "Knowledge elicitation techniques in classification domains," Proceedings of the 8th Conference in Artificial Intelligence ECAI-88, 1988. [2] Chao, C.-J. and Salvendy, G., "Impact of cognitive abilities of experts on the effectiveness of elicited knowledge," Behaviour & Information Technology , vol. 14, pp. 174-182, 1995. [3] Holsapple, C. W. and Raj, V. S., "An exploratory study of two KA methods," Expert Systems, vol. 11, pp. 77-87, 1994. [4] Wood, L. E., Davis, T. C., Clay, S. L., Ford, J. M., and Lammersen, S., "Evaluation of interviewing methods and mediating representations for knowledge acquisition," International Journal of Expert Systems, vol. 8, pp. 1-23, 1995. [5] Brandt, J. P. and Shook, S. R., "Attribute elicitation: Implications in the research context," Wood and Fiber Science, vol. 37, pp. 127-146, 2005. [6] Bradburn, B. , "A comparison of knowledge elicitation methods," International Conference on Engineering Design (ICED'91), pp. 298-305, 1991. [7] Maiden, N. A. M. and Rugg, G., "Knowledge acquisition techniques for requirements engineering," Proceedings of the Workshop on Requirements Elicitation for System Specification, Keele, UK, 1994. [8] Grabowski, M., "Knowledge acquisition methodologies: Survey and empirical assessment," Proceedings of the Ninth International Conference on Information Systems, Minneapolis, MN, USA, pp. 47-54, 1988. [9] Steenkamp, J.-B. E. M. and Van Trijp, H. C. M., "Attribute elicitation in marketing research: A comparison of three procedures," Marketing Letters, vol. 8, pp. 153-165, 1997.