1 / 24

Interoperability Testing Based on a Fault Model for a System of Communicating FSMs

Interoperability Testing Based on a Fault Model for a System of Communicating FSMs. TestCom 2003 May 27, 2003 Vadim Trenkaev (vad@elefot.tsu.ru) Myungchul Kim (mckim@icu.ac.kr) *Soonuk Seol (suseol@icu.ac.kr) Information and Communications University, Korea. Contents.

alaric
Download Presentation

Interoperability Testing Based on a Fault Model for a System of Communicating FSMs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interoperability Testing Based on a Fault Model for a System of Communicating FSMs TestCom 2003 May 27, 2003 Vadim Trenkaev (vad@elefot.tsu.ru) Myungchul Kim (mckim@icu.ac.kr) *Soonuk Seol (suseol@icu.ac.kr) Information and Communications University, Korea

  2. Contents • Introduction and Motivation • Preliminaries • Fault Model • Test Architecture • Interoperability Test Suite Derivation • Application to TCP • Conclusion • References

  3. Introduction • Conformance Testing • It is used when a single protocol entity is tested. The objective is to determine whether an implementation conforms to the specification. • Interoperability Testing • It is used when two or more protocol entities are tested. The objective is to determine whether two or more implementations can interact and if so whether they behave together as expected by the relevant specification.

  4. Introduction and Motivation • Test Generation and Evaluation • The basic technique of interoperability test suite (ITS) derivation is a reachability analysis. • Methods of the evaluation of the ITS quality • calculating the number of executed interactions • calculating the number of detected faults • Motivation • What is interoperability faults? • What is the fault coverage of an ITS? • Why implementations can not communicate or behave together as unexpected ?

  5. Preliminaries (1) • Finite State Machine • FSM A=(S, X, Y,,, s0), whereS - a finite set of states with s0 as the initial state, X - a finite set of inputs, Y - a finite set of outputs, : SXS - thetransition function, : SXY - theoutput function • FSM A=(S, X, Y,1,2, s0), FSM B=(Q, X, Y,2,2, q0)A is equivalent B (AB)if X* [1(s0,) = 2(q0,)] a sequence is said to distinguishA and Bif 1(s0,) 2(q0,)]

  6. channel C12 Y2 X1 E1 E1 FSMA2 FSM A1 E2 E2 X2 Y1 channel C21 Preliminaries (2) • System of Communicating Finite State Machines • slow environment • no live-locks and no input queue • channel is a perfect bounded FIFO queue • component FSM can produces a pair of outputs FSM A1=(Q,X1E2,Y1E1,1,1,q0) and FSM A2=(T,X2E1,Y2E2,2,2,t0)

  7. stable state (q,t,,) x a Case 1 (y,a)=1(q,x) q'=1(q,x) A2 A1 x/(y,a) y (q',t,a,) transient state  is so-called empty symbol transientstate Case 2 (c,b)=2(t,a) t'=2(t,a) (q,t,a,) c a A2 A1 a/(c,b) b (q,t',,b) transient state Preliminaries (3) • Reachability graph • avertex - 4-tuple (q, t, c1, c2),where q - a state of FSM A1, t - a state of FSM A2,c1 (c2) - content of channel C12 (C21)

  8. Fault Model (1) • Interoperability • the capability of protocol entities for interacting as expected. • Why do we need interoperability testing? Interoperability testing is necessary as a result of • incompleteness of specification validation • incompleteness of conformance testing • incompleteness of media/service testing • ambiguities in the specification

  9. Fault Model (2) • Assumptions 1) the commutation media can be faulty 2) conformance testing can be incomplete Primitive to applicationprocess Primtive from application process wrong message message incompleteness of media testing entity A entity B media 1. Communication media is faulty Primitive to application process Primtive from application process incompleteness of conformance testing wrong message entity A entity B 2. Implementation A does not conform to its specification

  10. x e e FSM A2 m: EE FSM A1 m(e)=e, for eE - function of correct media x e' e FSM A2 f: EE FSM A1 f(e)=e', where e'e for some eE or e'= - function of faulty media x e' e x FSM A1 ' FSM A1 (q, x, e, q')- correct transition (q, x, e', q')- transitionwith an output fault Fault Model (3) • Data transmission faults: • corruption of a message, loss of a message, ect. • FSM faults: output faults, transfer faults

  11. Fault Model (4) • Fault Model • single internal output faults • at most one component machine can be faultyA1=(Q,X1E2,Y1E1,1,1,q0) - reference FSM B1=(Q,X1E2,Y1E1,1',1',q0) - implemented FSMFSM B1 has a single internal output faultif for each pair (q,x), qQ, xX1E2, 1'(q,x)=1(q,x) and there exists exactly one pair (q,x) such that 1'(q,x)=(y,e') 1(q,x)=(y,e), e,e'E1, yY1.

  12. Test coordination procedure Tester A Tester B entity B entity A System Under Test Underlying communication service no access is granted to internal outputs Test Architecture

  13. Step 1. Interoperability test suite derivation by use of the existing method. Step 2. Fault coverage analysis of the test suite; if the fault coverage is not reasonable then step 3 otherwise stop. Step 3. Extension of the test suite. Interoperability Test Suite Derivation (1) • Sketch of a proposed method

  14. implemented component with a single internal output fault X1 E1 A1' Y1 E2 Y2 X1 E1 FSMA2 A1 E2 X2 Y1 reference component Replacement Interoperability Test Suite Derivation (2) • Fault Coverage Analysis

  15. Interoperability Test Suite Derivation (3) • Fault Coverage Analysis • FSM N - the composed machine of the specif. system  - the set of composed machines, which is derived by explicit simulation of all single internal output faults of each component FSM. • The definition task For each FSM M that is not equivalentto FSM Nto check if an interoperability test suitehas a sequence thatdistinguishes the FSMs N and M. • a state of a FSM is transparentiffor each internal input the external parts of all outputs produced at the state are not equal to the empty word and are pairwise different. • Proposition. If each state of FSMs A1 and A2 is transparent then a test suite traversing each component transition with a non-empty internal output also detects any internal output fault.

  16. q0t0  xIT, where IT -interoperability test suite k‘ =(q,x,(y,e'),q), where e'e - a faulty transition qt x/ (y,e) k=(q,x,(y,e),q) - a reference transition x/(y,e'') x/(y,e')  q'te q'te' q'te''  Interoperability Test Suite Derivation (4) • Fault Coverage Analysis - technique

  17. Interoperability Test Suite Derivation (5) • Fault Coverage Analysis - algorithm k=(q,x,(y,e),q), where (y,e) =1(q,x), q'=1(q,x) - a reference transition of FSM A1 F(k)={(q,x,(y,e'),q): e'e} ; a transition k‘ from F(k) - a faulty transition G(k‘) - reachability graph G’ for faulty transition k‘ is a reachability graph where a reference transition k of an appropriate component machineis replaced by a faulty transition k'.Input: interoperability test suite ITOutput: verdict about test suite completenessFor each sequence  of the set IT, do 1. Construct the path P() generated by the sequence  in the reference reachability graph. 2. Obtain the set RT() of reference transitions covered by the path P(). 3. For each reference transition kRT(), derive the set F(k) of faulty transitions. 4. Find a set W() as union of sets F(k) over kRT(). 5. For each faulty transition k'W() do- construct a deterministic path P'() generated by the sequence  in the G(k‘); - compare the external projection of the path P() with that of the path P'(); if they are different then add faulty transition k' to the set D(IT); If the set D(IT) has all faulty transitions of each component FSM then the interoperability test suite is complete; otherwise, it is incomplete.

  18. Interoperability Test Suite Derivation (6) • Extension of the Test Suite Extension algorithm Input: interoperability test suite IT, length max of maximum possible extensionOutput: extended interoperability test suite ETFor each sequence  in IT do1. Construct the path P() generated by in the reference reachability graph. 2. Obtain the set S() of faulty transitions which are undetected by . 3. For each faulty transition k'S() assign =and dountil length of the sequence  is less than max do- select randomly symbol x from (X1X2); - concatenate the sequence  with symbol x;- construct paths P() and P'(), where P'() is the path generated by the sequence  in the graph G(k' );- if the Y-projection of the path P() is not equal to that of the path P'() then claim the faulty transition k' to be detected and replace the sequence with the sequence in the test suite IT.4. If the set of undetected transitions is empty then the extended test suite is complete.

  19. Interoperability Test Suite Derivation (7) • Extension of the Test Suite Modified version: to concatenate each test sequence with each possible input sequence of length up to two and to check if the extended test sequence can detect any fault.Let an interoperability test suite be IT, n1and n2 are numbers of states of component FSMs. Proposition. Given a test case IT, if a single internal output fault of any component FSM is undetected with each sequence , where || = 2n1n2 – 1 – ||, then the fault is undetected with any sequence with the prefix .

  20. Application to TCP • A Finite State Machine

  21. call_1 SYN TCP1 TCP2 SYN_ACK call_ask_1 Application to TCP • An Example of Test Case The interoperability test case for the TCP is {listen _1call_2close_1close_2, listen_1listen_2send_data_1, listen_2call_1close_2close_1, listen_2listen_1send_data_2}.The same as that of in “Interoperability Test Suite Derivation for the TCP protocol”, Proceedings of the IFIP Joint Inter. Conf. FORTE ‘1999.

  22. Conclusion In this paper we proposed • a fault model for interoperability testing: interoperability faults are modeled with a single internal output faults of one of component FSM • a test suite derivation method based on the fault model and consisted of three steps: initial test suite, fault coverage analysis, augmentation

  23. References (1) [1] Bochmann, G., Petrenko, A.: Protocol Testing: Review of Methods and Relevance for Software Testing. Sigsoft Software Engineering Notes, spec. issue., USA (1994) 109-124 [2] Luo, G., Bochmann G., Petrenko, A.: Test Selection Based on communicating Nondeterministic Finite-State Machines Using a Generalized Wp-Method. IEEE Transactions on S.E., Vol 20. N 2. (1994) 149-162 [3] Petrenko, A., Yevtushenko, N., Bochmann, G., Dssouli, R.: Testing in context: framework and test derivation. Computer communications, Vol. 19 (1996) 1236-1249 [4] Lee, D., Sabnani, K., Kristol, D., Paul, S.: Conformance testing of protocols specified as communicating finite state machines - a guided random walk based approach. In IEEE Transactions on Communications, vol. 44, N 5 (1996) 631-640 [5] Cavalli, A., Prokopenko, S., Yevtushenko, N.: Fault detection power of a widely used test suite for a system of communicating FSMs. Proceedings of the IFIP TC6/WG6.1 13th Inter. Conf. TestCom2000, (2000) 35-59 [6] Petrenko, A., Yevtushenko, N., Bochmann, G.: Fault models for testing in context. Proceedings of the IFIP 1st Joint Inter. Conf. FORTE/PSTV96, (1996) 163‑178 [7] Rafiq, O., Castanet, R.: From Conformance Testing to Interoperability testing. Proceedings of the 3rd International Workshop on Protocol Test Systems, USA (1990) 371-385 [8] Seol, S., Kim, M., Kang, S., Park, Y.: Interoperability Test Suite Derivation for the TCP protocol. Proceedings of the IFIP Joint Inter. Conf. FORTE XII/PSTV XIX, (1999) 357-376

  24. References (2) [9] Kang, S., Shin, J., Kim, M.: Interoperability Test Suite Derivation for Communication Protocols. Computer Networks, 32 (2000) 347-364 [10] Viho, C., Barbin, S., Tanguy, L.: Towards a formal framework for interoperability testing. Proceedings of the 21st Inter. Conf. FORTE2001, (2001) 51-68 [11] Seol, S., Kim, M., Chanson, S.T.: Interoperability Test Generation for Communication Protocols based on Multiple Stimuli Principle. Proceedings of the IFIP 14th Inter. Conf. TestCom2002, (2002) 151-169 [12] Griffeth, N., Hao, R., Lee, D., Sinha R.K.: Integrated system interoperability testing with application to VOIP. Proceedings of IFIP TC6 WG6.1 Joint Inter. Conf. FORTE XIII / PSTV XX, (2000) 69-84 [13] Revised Working Draft on “Framework: Formal Methods in Conformance Testing”, JTC1/SC21/WG1/Project 54.1 // ISO Interim Meeting / ITU-T on, Paris, (1995) [14] Gill, A.: Introduction to the theory of finite state machine. McGraw-Hill, New York (1962) 207

More Related