360 likes | 474 Views
Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006. Integration Testing of Components Guided by Incremental State Machine Learning. Content. Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion.
E N D
Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006 Integration Testing of Components Guided by Incremental StateMachine Learning
Content Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion
Motivation Software development is more and more based on component integration Unknown Internal structures and insufficient documents Only partial and informal information is available Provide a required system assembling COTS of which he has a limited knowledge Unit testing and integration testing Have to rely on intuitions and little support is available Integrator: Component:
Our Approach Infer formal models of components from behaviours observed during tests Perform integration testing based on the formal models Iteratively improve the formal models
Background - Angluin’s Algorithm [1987] Infer formal model (conjecture) from observed behavior (observation table) For deterministic finite-state acceptor Teacher answers Membership query Equivalence query
Process Add strings to Observation Table No No Closed & Cons. Empty Observation Table Membership Query Yes Equivalence Query Yes Yes? End String Accepted Or Not Yes Or Counterexample Conjecture Teacher
Assumptions Black boxes with known interfaces Only external interfaces are controllable Slow environment Test scenarios
Framework Scenario Testing Unit Testing Integration Testing Consult Domain Expert Initial Observation Tables Models of Components
Models C.E. C.E. I. T. I. T. I. T. U. T.: Unit Testing I. T.: Integration Testing C.E.: Counter- example M1(1) M1(2) M1(3) U. T. U. T. U. T. M2(1) M2(2) M2(3) U. T. U. T. U. T. … … … Mn(1) Mn(2) Mn(3) U. T. U. T. U. T.
Scenario Testing A test scenario is an input/output sequence containing external inputs and outputs A test case is constructed for each test scenario Replace problematic component Record observed behaviour of each component in its observation table
Unit Testing Each Component is tested individually For each component Complete the initial observation table Using extended learning algorithm to construct closed and consistent observation table Make conjectured model
Integration Testing Unit Testing Yes Coverage Criterion Satisfied? Discre- pency? Test Generation Test Execution No Yes No
Test Generation based on Composition of FSMs M N [Petrenko 98] An algorithm to compute M N [Lima 97] Assign colour to “global” transitions An algorithm to generate tests covering colours [Kim 02]
References [Petrenko 98] A. Petrenko, N. Yevtushenko, “Solving Asynchronous Equations,” Proceedings of FORTE’98, 1998 [Lima 97] L. Lima Jr., A. Cavalli, “A Pragmatic Approach to Generating Test Sequences for Embedded Systems,” Proceedings of IWTCS’97, 1997 [Kim 02] M. Kim, C. Besse, A. Cavalli, and F. Zaïdi, “Two Methods for Interoperability Tests Generation: An Application to the TCP/IP Protocol,” Proceedings of TestCom 2002, 2002.
Dealing with Discrepancy Observe the behaviour of each component, and compare with its model Discrepancy Unexpected output sequence New input symbol
Refine (relearn) model Counterexample or new input symbol has been found Add corresponding strings into observation table Perform learning algorithm to make the observation table closed and consistent again Make conjectured model
Consulting Domain Expert Models of components and test results Domain expert Identify unexpected behaviours Propose additional test scenarios Iteration
Results of Integration Testing For each component, we have a model The joint behaviour of these components has been systematically tested Faults might be revealed during testing
Conclusion Combines machine learning and model-based testing for the integration of components Reuses Angluin’s algorithm Iteratively refines models
The Models Just a “by-product” Our goal is not to build a complete model of the system Just used as basis for the testing process to: Generate integration tests of components Assess the coverage, hence the quality of the integration testing process
The Next Steps Move to Parameterized FSM presentation by Muzammil Shahbaz Identify the type of approximation needed Experiment and case study Revisit model-based test generation strategies
Observation Table ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E S S.A
Closed ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E tSA, sS, s.t., row(t)=row(s) S SA
Consistent ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E s1S, s2S, row(s1)=row(s2), Then, aA, row(s1a)=row(s2a) S SA
Conjecture ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E q0 S a b a b b q2 q1 SA a
Observation Table a b ε x x a x y b y x aa x y ab y x ba x y bb y x row(ε) a/x b/x b/y row(a) row(b) a/x b/x a/y
Example – Component Hotel Reservation OK/CS1 OK/CD1 No/F q3 q5 No/TL T/PD1 Dbl/F q2 Sgl/F T/PS1 No/HL q4 q6 No/TL H1/TL No/F N/HL q0 q1 H2/TL No/TL No/F q8 q10 T/PD2 Dbl/F q7 Sgl/F T/PS2 q9 q11 No/TL OK/CD2 No/F OK/CS2
Initial Observation Table N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω H2 HL Ω Ω Ω Ω Ω Ω Ω Dbl HL Ω Ω Ω Ω Ω Ω Ω Sgl HL Ω Ω Ω Ω Ω Ω Ω T HL Ω Ω Ω Ω Ω Ω Ω OK HL Ω Ω Ω Ω Ω Ω Ω No HL Ω Ω Ω Ω Ω Ω Ω
Add N to S N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω … … … … … … … … … No HL Ω Ω Ω Ω Ω Ω Ω N/N Ω TL TL Ω Ω Ω Ω Ω N/H1 Ω Ω Ω F F Ω Ω HL N/H2 Ω Ω Ω F F Ω Ω HL N/Dbl Ω TL TL Ω Ω Ω Ω Ω … … … … … … … … … N/No Ω TL TL Ω Ω Ω Ω Ω
Conjecture OK/CD1 No/F row(N-H1-D) row(N-H1-D-T) No/TL T/PD1 No/HL D/F row(N-H1) N/HL H1/TL row(ε) row(N) S/F row(N-H1-S-T) T/PS1 row(N-H1-S) H2/TL No/TL No/F OK/CS1
Example – Component User Interface UI_Dbl/Dbl UI_Sgl/Sgl UI_No/No UI_T/T UI_No/No PD1/UO_PD1 PS1/UO_PS1 PD2/UO_PD2 PS2/UO_PS2 UI_H1/H1 UI_H2/H2 UI_OK/OK UI_No/No UI_Name/N HL/UO_HL F/UO_F TL/UO_TL q0 q1 q2 q3 q4 TL/UO_TL F/UO_F HL/UO_HL CD1/UO_CD1 CS1/UO_CS1 CD2/UO_CD2 CS2/UO_CS2
Integration Testing Environment oMN cM User Interface (M) Hotel Reservation (N) oM oNM
Counterexample UI_Name/UI_H2/UI_Dbl/UI_T The real input sequence of Hotel Reservation is: N/H2/Dbl/T The corresponding output sequence is: HL/TL/F/PD2 According to the model, the output should be: HL/TL/F/PD1
Comparing with Angluin’s Algorithm Add strings to Observation Table Yes No Closed & Cons. No Counter- example? Initial Observation Table Unit Testing Yes Integration Testing End Test Output Output Test Component Component A Component B Scenario Testing