Software testing research

1. Software testing research • Ossi Taipale • November 2011 • Lappeenranta University of Tech.

2. Contents • Background • Research process • Research methods • Research projects, research objectives and results: • Basic research of software testing, ANTI-project • Reference model of software testing, MASTO-project • Software testing in the cloud, STaaS-project • Intended quality, STX-project • Research co-operation

3. Background, why testing • During the last decades productivity of Finland has been higher than in other EU countries, but now the productivity in the product and service sectors is slowing down meaning lower growth rate of GDP. • The key to increase productivity is in the hands of industry and service sectors that produce or apply ICT. • Productivity can be increased by creating tangible products or service products. • Testing enables product creation.

4. Background, software testing research Top-down approach: • 1. Years 2004–2007 ANTI-project: Basic research of software testing, affecting factors. • 2. Years 2008–2011 MASTO-project: Software testing strategy and management. Theses: Software Testing Strategy and Management and Software Testing in the Cloud. The reference model of software testing ISO/IEC 29119 Software Testing Standard. 1. Vocabulary, 2. Testing Processes, 3. Reporting. Part 4. Methods is under development. Ossi Taipale is a member of the WG26. • 3. Years 2011-2014, new STX-project: Testing level and ”Testing Spice”.

5. Software testing research projects • Top down approach in the research projects: ANTI-project: Basic research of SW testing, 2004 - 2007 MASTO-project: Reference model of SW testing, 2008 - 2011 STX-project: Intended quality, 2011 - 2014 ISO/IEC 29119: SW testing standard SW testing in the cloud ISO/IEC and IEEE : 29119 and Testing Spice, 15504, IEEE1012, 25010

6. Background, software testing definition • We use the definition: Software testing consists of verification and validation. • By testing, we try to answer two questions: are we building the product right and are we building the right product? • Verification answers the question: are we building the product right? Basic verification methods include inspections, walkthroughs, and technical reviews. Checklists are used as the tools of verification. • Validation answers the question: are we building the right product? Validation activities can be divided into unit testing, integration testing, usability testing, function testing, system testing, and acceptance testing.

7. Selection of standards and constructs: e.g. ISO/IEC 12207, 15504 and 29119 Ontology Beliefs of the essence of phenomena under investigation Research philosophy Empirical study Background assumptions Concrete selections of the study: Delphi, survey, and grounded theory methods Beliefs of the suitable method Epistemology Research process

8. Research methods • Mixed methods approach consisting of quantitative and qualitative analyses (Hirschheim 1985). • The results of the quantitative and qualitative analyses are triangulated to increase the validity of the study. The principle of triangulation means that more than one method, observer or data set is used in a study to complement each other and to verify the findings (Denzin 1978).

9. Research methods The objective of the Delphi method is to achieve the most reliable consensus of opinion of a group of experts. Delphi method can be used in finding good arguments about an ongoing process (Directing the study). The survey method is used to gather information about feelings, motivations, plans, and beliefs. Tools for gathering such information are usually questionnaires or interviews (Fink & Kosecoff 1985). Grounded theory (Strauss & Corbin 1990). The objective of a qualitative study is rather to find or reveal facts than to prove existing theorems. Strauss and Corbin (1990) define qualitative research as any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification.

10. Some results of the ANTI-project (2004-2007) • Hypotheses for improvement, process view: • Testing ought to be adjusted according to the business orientation of the OU. Product oriented organizations should adopt a formal planned testing process. Service oriented organizations should adopt a flexible testing process that allows appropriate coordination with their clients. • Enhanced testability of software components. Consider testability as an important factor in the selection of components. Review the testing process of your suppliers. • Efficient communication and interaction between development and testing seems to reduce costs and improve quality. Product oriented OUs could develop straightforward processes because knowledge transfer is predictable. Service oriented OUs could develop innovative solutions because knowledge transfer varies according to the customer. • The early involvement of testing seems to reduce costs and improve quality. The planning of testing is more straightforward in product oriented than in service oriented OUs. • The risk-based testing strategy helps in avoiding ad hoc decisions on testing, because the decision on what to test is based on a predefined testing strategy.

11. Some results of the ANTI-project (2004-2007) • Hypotheses for improvement, knowledge management view: • Business orientation affects the testing organization. In product oriented OUs, the organization model ought to support repetitive testing tasks that enable development of a deeper expertise. In service oriented OUs, the organization model ought to support adaptation to the processes of the customers that demand broader expertise. • Business orientation affects the knowledge management strategy. OUs ought to adjust their knowledge management strategy according to the business orientation. • Business orientation and the knowledge management strategy affect outsourcing. Codification of knowledge enables testing outsourcing. • Identifying and avoiding barriers and using enablers improve knowledge transfer.

12. Some results of the ANTI-project (2004-2007) • Associations between testing outsourcing and knowledge management: • Outsourcing seems to be more effective when independent testing agencies have enough domain knowledge. • Outsourcing verification tasks is more difficult than outsourcing validation tasks.

13. Publications of the ANTI-project (2004-2007) 1. Finding and Ranking Research Directions for Software Testing, EuroSPI, Budapest, Taipale, O., K. Smolander, H. Kälviäinen. 2. Cost Reduction and Quality Improvement in Software Testing, SQM, Southampton, UK, Taipale, O., K. Smolander, H. Kälviäinen. 3. A Survey on Software Testing, SPICE Conference , Luxembourg, SPICE ,Taipale, O., K. Smolander, H. Kälviäinen (2006). 4. Factors Affecting Software Testing Time Schedule, ASWEC, Sydney, Australia, Taipale, O., K. Smolander, H. Kälviäinen (2006). 5. Improving Software Testing by Observing Practice, ISESE, Rio de Janeiro, Brazil, Taipale, O., K. Smolander. 6. Observing Software Testing Practice from the Viewpoint of Organizations and Knowledge Management, ESEM, Madrid, Spain, Taipale, O., K. Karhu, K. Smolander. 7. Triangulating Testing Schedule Over-runs from Knowledge Transfer Viewpoint, Lappeenranta University of Technology, Research Report 104, Taipale, O., K. Karhu, K. Smolander . 8. Outsourcing and Knowledge Management in Software Testing, EASE, Staffordshire, UK, K. Karhu, O. Taipale, K. Smolander.

14. Overview on MASTO-project (2008-2011) Jussi Kasurinen

15. MASTO • The project was a 3-year academic joint project conducted by LUT and Aalto University, funded by Tekes (Technology and Innovation Agency of Finland) and a group of companies and organizations. • The focus on the project was on software testing in different organizational levels. • MASTO at LUT focused on test process and organizational aspects. • Aalto focused on the testing work and project-level aspects. • Researcher exchange, Jussi Kasurinen – Lund, Sweden, Leah Riungu-Kalliosaari – Limerick, Ireland. Software-producing organization Organizational level, one company has one set of these Project-level work, one company may have several of these Pof these Projecese

16. Collection of qualitative and quantitative data, 65 interviews

17. ISO/IEC 29119 Test process

18. Testing standard and publications Journal article “Software Test Automation in Practice: Empirical Observations” Conference paper “How Test Organizations Adopt New Testing Practices and Methods?” Conference paper: “A Study on Agility and Testing Processes in Software Organizations” Covered in article “Software Test Automation in Practice: Empirical Observations” Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?” Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard” Conference paper: “Exploring Quality Concepts in Software Organizations” Conference paper “Analysis of Problems in Testing Practices” Test strategy constructs (and layers)

19. Quality and quality objectives • According to ISO/IEC 25010 (Software product quality), the quality in the software product is a composition of several quality attributes • These quality attributes define the quality objectives for software

20. Quality characteristics • All at least somewhat important • Only in 9 out of 248 (3,6%) assessments “not important” • Perceived quality, not absolute quality!

21. Important things that affect perceived quality (Survey 2009) • Trust between the customer and developer. • Specifications are distributed freely, things are discussed and more information is available if needed. • Existing conformance with the ISO/IEC 29119 model concepts. • Workflow has a feedback system which is used, plans and made decisions are assessed and changes can be made. • Elaboration of quality • Everyone in the organization knows what kind of quality is preferred and tries to achieve that.

22. Things that do not affect the perceived quality that much even if they seem to have a big influence • End-product criticality • Obviously a rocket control system is tested more thoroughly than a mobile game, but preferred quality comes from the domain, not criticality. • Development method • An agile product can be as high quality as a “traditionally” made product. • Open source is not necessarily better (or worse) than closed software. • Outsourcing • In large organizations, outsourcing does not seem to affect perceived quality. • In smaller organizations it may, but not always.

23. How do the organizations develop their test process • Organizations do not tend to try out new ideas. • Sporadic development is done when the inconveniences overcome acceptable losses. • Even if the test process feedback is collected, it is often neglected if the process is “good enough”.

24. Self assessment framework • Combination of Test Improvement model (TIM) maturity levels and ISO/IEC 29119 processes. • Two results: • General maturity and conformance with the standard model. • Process improvement objectives to develop test process. Maturity levels from TIM Processes from ISO/IEC 29119 Individual assessment of each process area, development ideas General maturity/conformance estimation

25. Publications from the MASTO project (2008 -2011) 1. Test Case Selection and Prioritization: Risk-Based or Design-Based? Jussi Kasurinen, Ossi Taipale and Kari Smolander, ESEM 2. A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard, Jussi Kasurinen, Per Runeson, Leah Riungu and Kari Smolander, EuroSPI 3. Software Test Automation in Practice: Empirical Observations, Jussi Kasurinen, Ossi Taipale, Kari Smolander, AiSE 4. Exploring Perceived Quality in Software Organizations, Jussi Kasurinen, Ossi Taipale, JariVanhanen and Kari Smolander, IEEE 5. Analysis of Problems in Testing Practices, Jussi Kasurinen, Ossi Taipale and Kari Smolander, APSEC 6. A Study on Agility and Testing Processes in Software Organizations, VesaKettunen, Jussi Kasurinen, Ossi Taipale, and Kari Smolander, ISSTA 7. How Test Organizations Adopt New Testing Practices and Methods? Jussi Kasurinen, Ossi Taipale and Kari Smolander, TAICPART 8. Exploring the Perceived End-Product Quality in Software-Developing Organizations, accepted for publication in International Journal of Information System Modeling and Design, IGI Global,Jussi Kasurinen, Ossi Taipale, JariVanhanen and Kari Smolander.

26. Software Testing in the Cloud (2010-2013) Leah Riungu-Kalliosaari

27. Objectives and background • Empirical study aimed at understanding how organizations can successfully use the cloud for testing • Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition). • STaaS: A model for software testing whereby testing of an application is provided as a service across the internet. • Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.

28. Initial observations The trend towards cloud services is growing Traditional desktop applications  online software services e.g. SaaS, StaaS According to a 2009 Gartner report, cloud computing changes the way software is delivered and used. This change means that traditional licensing of software seems to loose ground where as hiring of software services seems to be on the rise. Mixed reactions  Generally positive Positive – due to e.g. cost reduction, flexibility, performance, access to global markets Negative – due to concerns regarding e.g. security, domain knowledge, test data management Neutral – mainly explorative approach

29. Implications on Testing With the cloud becoming more common, the quality expectations are growing. New issues and complexities surrounding these services will need to be addressed. Testing will become more important Need for testing increases Testing is seen as arena of cloud computing where the barriers to entry are relatively low. Testing in the cloud simply entails the use of computing resources and models in the cloud to perform testing. Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.

30. 3. Testing the cloud 2. Testing environments in the cloud 1b. Non-SaaS software 1a. SaaS software Testing in the Cloud • The system or application under test is available online • Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas)) • Testing of the cloud itself Facets of testing in the cloud

31. A roadmap towards testing in the cloud Develop understanding of cloud computing Cloud computing is increasingly becoming a feasible choice for testing. Carry out pilot projects We recommend that organizations carry out pilot projects that enable them to fully explore the potential benefits. Come up with elaborate strategies Cloud testing vendors as well as testing and quality assurance consulting firms will be called upon to offer advice and direction. Enhance team interaction and prepare for complexities Organizations need to be prepared for additional testing brought about by the complexities and new requirements for cloud-based applications and systems. Enhance co-operation between research and industry These include, among others, application issues e.g. the types of applications best suited for testing in the cloud; management issues like how to organize the human resources for cloud-based testing (e.g. crowdsourcing); and legal and financial issues such as the management of test data across different global jurisdictions and how to device appropriate pricing models.

32. Current research problem To better understand how cloud based services can be achieved with intended quality Identify the most important quality requirements for cloud-based services Provide recommendations for achieving overall intended quality of cloud-based services Difference between intended and achieved quality What are the tradeoffs, and what is the optimal balance.

33. Publications from the Testing in Cloud – project (2010-2013) 1. Research Issues for Software Testing in the Cloud, Leah MuthoniRiungu, Ossi Taipale, Kari Smolander, 2nd IEEE International Conference on Cloud Computing Technology and Science. 2. Software Testing as an Online Service: Observations from Practice, Leah MuthoniRiungu, Ossi Taipale, Kari Smolander, Third International Conference on Software Testing, Verification, and Validation Workshops. 3. Testing in the Cloud: Exploring the Practice, IEEE Software, Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander.

34. Software testing and development for intended quality, STX (2011-2014) Tero Pesonen

35. Objective To show how software development, software testing and intended quality depend on one another. Traditional software development and service models Emerging XaaS (Everything as a Service) architectures, technologies and service models. The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole. Testing techniques Testing as a service

36. Research Problem Intended Software Quality Software Development • Software Products • ISO/IEC 12207 • ISO/IEC 15504 • New Services • SaaS ISO/IEC 25010, Software Product Quality Software Testing ISO/IEC 29119, Testing Spice, IEEE Std 1012 StaaS OU’s are evaluated through an assessment framework

37. Research Problem The framework used as a ”grid” through which each OU is assessed. Intended quality vs. achieved quality. Top-down and bottom-up analysis (testing work  project management  organizational level). The assessment framework is defined through international software engineering standards. We need to anchor the assessment to a specific, concrete view on what quality, software testing and software development are. Based on standards, the model remains coherent throughout the assessment. Project results and usability; contribution to standardization work. The results include improvement proposals for the OU’s. Data collection methods include interviews and surveys.

38. Work division, entire project 3 year project 12 months / work package; WP 1 starts 8/2011 WP1: Requirements for intended quality Identifies the quality requirements and attributes that the OU's perceive as key for achieving the intended quality. Defining intended quality, the importance of different quality characteristics in achieving it, traditional vs. SaaS WP2: Building the intended quality Achieved vs. Intended quality vs. the software development process Action research WP3: Assessment How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?

39. Contents of work package (WP) 1 Study units are OU's. What are the quality requirements, and hence the quality attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)? Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here? Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development? Data collection for the first qualitative study September – November / 2011. Additional data from other projects also incorporated. Data analysis may produce first leads. Second data collection round will constitute quantitative studies based on round 1 observations.

40. Publications from the STX project (2011-2014) 1. Observations on eBusiness implementation capabilities in heterogeneous business networks", 11th IFIP International Conference on e-Business, 2011, Kaunas, Lithuania Pesonen T., Smolander K.