International Conference on IT to Celebrate S.Charmonman s 72nd Birthday Charm09

1. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 International Conference on IT to Celebrate S.Charmonman�s 72nd Birthday (Charm09) Distributed t-way Test Suite Generation Algorithm for Combinatorial Interaction Testing

2. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 T-way Combinatorial Testing Systematically reduce the number of test cases. Given any t parameters, every interaction element for t parameters should be covered by at least one test case. Allows faults triggered by interaction among t parameters to be detected. Advantages: Higher test coverage better quality assurance Requires no access to internal source code SUT t-way is is one of the black box testing, which is the selection of test case is done without reference of the internal structure of the software under test or in order word without reference to the source code. This test technique can be used to systematically reduce the number of test case based on interaction coverage of the input parameter. One example of t-way Testing is pairwise testing. In this type of testing, the interaction element which is the combination of parameter value between t ; parameter must be cover by at least one test case. T can be any number start from 2 or above. This technique is quite effective in ensuring error detection of faulty software interaction error among t parameter. Among the advantages using tway is that the test technique provide higher test interaction coverage for higher t. And required no internal access to the source code. There a few related work for tway interaction testing Next slide t-way is is one of the black box testing, which is the selection of test case is done without reference of the internal structure of the software under test or in order word without reference to the source code. This test technique can be used to systematically reduce the number of test case based on interaction coverage of the input parameter. One example of t-way Testing is pairwise testing. In this type of testing, the interaction element which is the combination of parameter value between t ; parameter must be cover by at least one test case. T can be any number start from 2 or above. This technique is quite effective in ensuring error detection of faulty software interaction error among t parameter. Among the advantages using tway is that the test technique provide higher test interaction coverage for higher t. And required no internal access to the source code. There a few related work for tway interaction testing Next slide

3. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Related Work Existing work has mainly focused on pairwise testing such as AETG, IPO, IRPS, DDA, G2Way and Tconfig. Many failures are caused by the interaction involving more than two parameters A recent NIST study by R. Kuhn indicates that failures can be triggered by interactions up to 6 parameters For certain software, pairwise testing discovers a relatively low percentage of faults D. Cohen, et al. provided the AETG system for pair-wise, triple, or t-way combinations of a system�s parameters. Y. Lei, et al. offered IPO strategy for pairwise testing and IPOG for t-way testing. Existing work has mainly focus on pairwise testing and there are many test case generator tool for pairwise testing such as AETG, IPO, IRPS, DDA and Tconfig. Pairwise testing is not enough to detect the failure caused by interaction of more than two Parameter. A study by Rick Kuhn of NIST, revealed that the interaction failure can caused By interaction more than 2 and more up to 6. and for certain software, pairwise only cover low Percentages of faults e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. David Cohen provide AETG for pairwise and tway testing and Yu Lei offerred IPO strategy for pairwise testing and IPOG for t-way testing. All the existing tool is cannot scale up to a very large input parameter e.g more than 1000 Parameter. And 100 or more value. This combination of input parameter can take day to Complete. This problem give a to our research problem. Next slide. Existing work has mainly focus on pairwise testing and there are many test case generator tool for pairwise testing such as AETG, IPO, IRPS, DDA and Tconfig. Pairwise testing is not enough to detect the failure caused by interaction of more than two Parameter. A study by Rick Kuhn of NIST, revealed that the interaction failure can caused By interaction more than 2 and more up to 6. and for certain software, pairwise only cover low Percentages of faults e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. David Cohen provide AETG for pairwise and tway testing and Yu Lei offerred IPO strategy for pairwise testing and IPOG for t-way testing. All the existing tool is cannot scale up to a very large input parameter e.g more than 1000 Parameter. And 100 or more value. This combination of input parameter can take day to Complete. This problem give a to our research problem. Next slide.

4. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Research Problem In real and complex software product development, the number of input parameter, Pi and input parameter values, v is quite large. For high t-way testing with large parameter and input parameter values, the number of interaction element, IE grows rapidly as the value of t increases and leads toward the combinatorial explosion and the computational complexity problem. The test suite generation time will take longer time or can take day to complete or even stall. In real andcomplex Combinatorial Explosion can lead to out of memory error, thus tool generation process is became stall.In real andcomplex Combinatorial Explosion can lead to out of memory error, thus tool generation process is became stall.

5. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Proposed Distributed Strategy for TS-CIT The proposed generation distributed strategy are based on parallel processing using master-worker pattern using tuple spaces technology. Tuple spaces is a middleware that enables communication between different process by means of exchanging tuples through a shared data-space The master and all workers are loosely connected through TSpaces server. Distributes the computational power and memory into different process/worker running on heterogeneous workstation.

6. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Modelling Master-Worker

7. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Strategy TS-CIT Master Roles of master process : The master sends ParmSet, interaction strength t, and the IEG to shared data space. The master generates the exhaustive test set and send by batch to respective worker local memory. The master starts with an empty test suite (TS), and waits for all workers to send a notification on new test case t and maximum interaction element coverage value maxIEC to shared data space. The master reads the maxIEC value from each worker from shared data space. Then, the master chooses the test case t corresponding to the highest maxIEC value to be added to latest test suite, TS. The master issues command to the selected worker via shared data space to delete the selected test case t from their own local memory and to delete interaction elements covered by in IESet and notify for recalculate new highest maxIEC. The master checks if the IESet is empty or not. If empty, the master will consider the latest test suite, TS as the final generated test suite

8. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Strategy TS-CIT Worker Roles of worker process : The worker first connects to shared data space and reads the ParmSet, IEG and t in shared data space. Each worker reads the assigned IEG from shared data space. The worker generates all possible interaction elements for their assigned IEG and send back to shared data space as IESet. Each worker reads all test case t in their local memory and the worker determines the maxIEC of all t and sends the highest maxIEC with t to shared data space. The worker reads the command from shared data space, if it contains delete command the worker deletes t in their local memory and wait for notify recalculation of new maxIEC notification.

9. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Master Generate Test Case

10. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 Conclusion We also describe the ways to parallelize the TS-CIT using the tuple space implementation. Initial prove of correctness (accuracy) is encouraging. For 4 parameter with 3,2,2,2. All possible IE is 44. The exhaustive test case is 24. After 13 iteration all 44 IE is covered by 12 selected test case. In future we hope will be able to implement the algorithms and make experiment on their effectiveness with regard to other tools.

11. Software Engineering Research Group, School of Electrical and Electronic Engineering, USM Charm09 References D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, �The AETG System: An Approach to Testing Based on Combinatorial Design,� IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, �Constructing test suites for interaction testing,� In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland. R. Kuhn, D. Wallace, A. Gallo, �Software Fault Interactions and Implications for Software Testing,� IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6. Alan Hartman, Leonid Raskin, �Problems and algorithms for covering arrays,� Discrete Mathematics 284(1-3): 149-156 (2004) Y. Lei and K. C. Tai , �In-parameter-order: a test generation strategy for pairwise testing,� Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261. K. C. Tai and Y. Lei, �A Test Generation Strategy for Pairwise Testing,� IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1. Changhai Nie, Baowen Xu, Liang Shi, Guowei Dong. Automatic test generation for n-way combinatorial testing. In: Proceedings of Second International Workshop on Software Quality (SOQUA2005), Fair and Convention Center, Erfurt, Germany, 2005. Lecture Notes in Computer Science 3712, 2005: 203-211. Yu Lei et al, IPOG: A General Strategy for T-Way Software Testing, 2006