Bio-inspired Robustness Testing Approaches

1. Seminário IC- Set/ 2010 Bio-inspired Robustness Testing Approaches Eliane Martins eliane@ic.unicamp.br

2. Overview of the talk The Test and Fault Injection Group: who are we? Robustness: what is it? Why is it important? Robustness testing activities of the group Part I: An evolutionary approach for robustness tests generation Part II: Use of sequence alignment algorithms as test oracles

3. The Test and Fault Injection Group http://www.sed.ic.unicamp.br/

4. Motivation • Testing software to ensure that the required functionality was met … • Example: • Requirement: • When heading in a direction (up / down) an elevator would service all received stop requests in that direction then it would reverse direction and verify if other stops in that direction had to be serviced • Test scenarios: • Does the elevator goes up when a request up is given? • Does the elevator goes down when a request down is given? … is necessary but not enough

5. Why robustness testing? Robustness testing • Requirement: • When heading in a direction (up / down) an elevator would service all received stop requests in that direction then it would reverse direction and verify if other stops in that direction had to be serviced • Test scenarios: • What if a request down is given and the elevator is on the first floor? • What if a request up is given and the elevator is on the last floor? • How does the system behave in presence of • erroneous or unexpected user inputs? • internal or external failures? • stressful conditions? • attacks?

6. Robustness • IEEE Std 610.12-1990 - Glossary of Software Engineering Terminology • “Robustness is the degree to which a software system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.” • [Anderson85]: • “[…] A computing system can be said to berobust if it retains its ability to deliver service inconditions which are beyond its normal domain ofoperation”

7. When Robustness testing is useful? • In independent technical assessment • To perform top-level investigation into the quality of the software being developed and to see how the developing organization reacts when problems are encountered. • During component selection: • To help in the decision of what component better suit the system being developed • During software development • To assess error handling mechanisms Source: SEI-CMU Report [CPK05]

8. Robustness testing characterization Specified Inputs Possible Behavior Outputs Success Should work Valid Component under test Undefined Should return error Failure Invalid Inspired on (Kropp, Koopman, Siewiorek 1998)

9. TIF – Test & Fault Injection Group Exhaustive search Evolutionary search Activity Diagram State Diagram Property-oriented Bioinformatics algorithms External faults: Accidental faults Malicious faults Interface faults Communication faults

10. TIF – Research Projects and Tools

11. TIF – Collaborations with other research groups • Foreign groups: • University of Coimbra • Institut Telecom Sud-Paris • Université Paris-Orsay • Laboratoired’Automatique et d’Analyse des Systèmes (LAAS), Toulouse • Universidad Politécnica de Valencia • Brazilian groups: • INPE • ITA • UFRGS • UFES • UFAL • USP-S.Carlos

12. Part I – Robustness tests generation Exhaustive search Evolutionary search Activity Diagram State Diagram Property-oriented Bioinformatics algorithms External faults: Accidental faults Malicious faults Interface faults Communication faults

13. Robustness testing characterization Specified Inputs Possible Behavior Outputs Success Should work Valid Component under test Undefined Should return error Failure Invalid Inspired on (Kropp, Koopman, Siewiorek 1998)

14. Model-Based Testing (MBT) • Used to test software system based on a model of the system behavior • Test cases are generated from the system behavior model • State-based models: • Commonly used to represent behavior of reactive systems • Techniques proposed since early sixties

15. Why MBT • Useful when source code is not available • Applicable in all test phases: from Unit to System testing • Tests can be prepared early in development cycle • Test preparation can help in uncover requirements faults • Tests can be automatically generated • For robustness testing: • Concerns with abnormal situations early in the development cycle

16. MBT – State-based models Homepage HomeTab HomeTab NewsTab ImageTab NewsTab ImageTab Imagepage Newspage Fonte: H. Robinson, StarWest 2006 • Useful to represent behavior of reactive systems: • Control objects and systems • Communication protocols • User interactions • …

17. MBT – State-based models • Useful to represent behavior of reactive systems: • Control objects and systems • Communication protocols • User interactions • … • There are various models: • FSM (Finite State Machines) • EFSM (Extended-Finite State Machines) • UML State Machines • And others: • Statecharts • State transition systems

18. EFSM Event(P) [Guard] / Actions State2 State1 • EFSM consists of: • States • Transitions • The following elements are associated with each transition: • Events • Guards (enabling conditions) • Actions

19. EFSM Testing Strategies Plavis platform Condado tool ??? tool

20. Exhaustive enumeration approaches • Automata-theoretic approaches • Equivalence between specification model and implementation • Graph-based approaches: • Coverage of model elements • Exhaustive search of the state-space • Algorithms commonly used: • Traveling salesman • Depth-first search

21. Model-based robustness test cases generation Specification of robust behavior Unexpected inputs Specification (nominal) Inopportune inputs Unspecified inputs Inputs that should return errors Test case generation Robustness test cases

22. Difficulties with exhaustive search • Model can be very big, in special, when representing data aspects and abnormal situations • Risk of state explosion • Algorithms like TSM or DFS are no longer adequate • Huge solution space • Generationoflargetestsuites • Big test cases • Highcosts to execute andmaintain

23. Search-based testing • How to deal with the huge solution space? • Use of meta-heuristics to search the solution space for automatic test case generation • Various approaches: • Random testing • Evolutionary algorithms (EA) • Genetic Algorithms (GA) • Generalized Extreme Optimization Algorithms (GEO) • Have been extensively used to search input data for single functions

24. Evolutionary testing schema Initial solution Solution: input data Test execution and monitoring Traversed path Output No solution

25. Code-based x EFSM-based evolutionary testing • Code-based: • Generation: • Input data • Controllability: • Control of code instructions • No control of states and context variables • Path feasibility: • execute inputs to decide whether the required element (instruction, decision, ...) has been traversed • EFSM-based: • Generation: • Sequence of inputs • Controllability: • States, event parameters and context variables • No control of code instructions • Path feasibility: • Static approaches  path feasibility is an undecidable problem

26. Some existing approaches

27. MOST-EFSM: our evolutionary model-based testing approach (Thaise Yano, PhD) • Generation: • A solution = test sequence • Test sequence = sequence of events starting at the initial state + parameters • Test sequence  transition path • Objectives: find a path that covers the test purpose + minimize path size • Controllability: • How to guide the search for an adequate path in the EFSM? • Use of Dependency Analysis • Path feasibility • Use of executable EFSM test paths dynamically derived

28. Steps of the approach Test criterium Érika Almeida - MSc

29. Example – ATM system Test purpose ATM – Automated Teller Machine [Korel et al. 2003]

30. Dependency analysis – ATM example Taffecting(t18) = {t1,t4,t8,t17} Shows control and data dependencies between transitions Backward reachability analysis to obtain Taffecting(tp) Based on algorithm from K. Androutsopoulos et al.2009

31. Example - nominal inputs seq={card(236,234,138), pin(236), spanish(), savings(), deposit(867)} path = { t1 t4 t6 t8 t18 }

32. Example - Inopportune inputs seq={card(236,234,138), receipt(), pin(236), spanish(), pin(236), savings(), deposit(867)} path = { t1 t4 t6 t8 t18 }

33. Example - Inopportune inputs seq={card(236,234,138), receipt(), pin(236), spanish(), pin(236), savings(), deposit(867)} path = { t1 t4 t6 t8 t18 } Assumption: Unexpected input  the machine remains in current state and generates null output

34. Some results t3 t18 Sequence size x Number of Fitness Function evaluations

35. Work in progress • Definition of other test criteria: • For the moment only test purpose coverage • Transition coverage based on dependency analysis • Evaluation of the approach • Cost – effectiveness  use of benchmark models • Scalability  Use in real world applications • Web-services from an industry • Teleduc

36. Open points and opportunities (1) • Development of a framework to support MOST • How to guide users in adjusting the EA for his/her problem? • Use of model transformation to easy dependency analysis • One of the challenges in EFSM is the problem of how to determine control dependency, as state machines may not have a final state • Transform the model to improve its “analysability” • Transform to a control flow graph [J.Li, E.Wong 2002]  use of MDA approaches

37. Open points and opportunities (2) • Comparison with the use of other meta-heuristics for EFSM-based testing: • Given that each heuristic has its own strengths and limitations, some might be more useful for some problems than others • Random testing (e.g, hit-or-jump) • Multi-objective Genetic Algorithms • Multi-objective Ant Colony • … • More long term … • Use of more than two objectives • Use of other approaches: • Memetic algorithms (MA): form of hybrid GA + individual learning procedures to perform local refinements • Hyper-heuristics: use of machine-learning techniques to automate the process of selecting, combining, adapting several heuristics to efficiently solve a problem

39. Part II – Result Analysis Exhaustive search Heuristic-based search Activity Diagram State Diagram Property-oriented Bioinformatics algorithms External faults: Accidental faults Malicious faults Interface faults Communication faults

40. Robustness testing characterization Specified Inputs Possible Behavior Outputs Success Should work Valid Component under test Undefined Should return error Failure Invalid Inspired on (Kropp, Koopman, Siewiorek 1998)

41. How to identify robustness failures? (1) Invalid inputs Valid inputs What if the “golden run” is faulty? Component under test Component under test Outputs (golden run) Outputs No Yes = Success Failure Comparison with a “golden run”

42. How to identify robustness failures? (2) • CRASH scale of the Ballista approach: • Catastrophic: system crashes or reboots • Restart: task execution hangs • Abort: abnormal termination of task execution • Silent: no exception reported when it should be • Hindering: incorrect error code or delayed response

43. However … • Use of the CRASH scale: • A system can fail without aborting or delaying responses or sending error messages • For example: • in an Elevator service, if there is a requestUP(floorID), then the Elevator should moveUp( ), stopAt(floorID) and opendoor( ) or else it stays in the same floor • In case of a request to a valid floor, if the Elevator opendoor( ) before stopAt(floorID), failure

44. A passive approach for robustness testing Points of Observation (PO) Observation Inputs Inputs Client Environment System under Test Outputs Outputs Execution Traces Analysis Pass Fail Inconclusive Trace Analyser Verdict Specification (model or properties)

45. Passive testing • A monitoring approach • The steps: • Add observation mechanisms (PO) to the System under Test (SUT) • Execute the SUT and collect execution traces • Filter the execution trace to keep only the events of interest • Analyse the filtered trace

46. Passive Testing Approaches

47. I1 = RcvInvoke(TID = N)/?, *, TR-Invoke.res/{Ack (TID = N)} I2= RcvInvoke(TID = N) / TR.Invoke.ind, *, TR-Invoke.res / {Ack (TID = N) } A previous invariant analysis approach [Bayse, Cavalli, Nuñez, Zahidi 2005] Behavior model Textual specification verification Invariants in the form of regular expressions

48. Our proposal (Gizelle S. Lemos) • An approach for analyzing results of robustness testing: • Monitoring system execution in the presence of faults, collecting execution trace • Definition of the properties the system should verify even in the presence of faults • Verify whether the system model satisfies the required properties, if the model exists • Checking whether the execution trace satisfies the required properties

49. Aspects to consider How to obtain the properties? What kind of properties to analyze? How to express the properties? How to analyze the trace?

50. How to obtain the properties? • Specification stating what a system should or should not do • Robustness specification should take into account • Invalid and inopportune inputs: • exceptions, • erroneous data, • valid data at the wrong moment, • missing events, • late events • Outputs corresponding to the abnormal inputs: • exception handlers, • error messages, • restart mechanism, • …