Software Architecture based Performance and Reliability Evaluation

1. Software Architecture based Performance and Reliability Evaluation Vibhu S. Sharma Ph.D. Scholar CSE, IITk

2. Client 1 Client 2 Web-server Application Server DB Client nc A web-based software system Software Architecture • Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution. - IEEE 1471-2000. • Software architecture styles are like templates which are followed by commonly used systems: • Layered style : Web-based systems • Pipe and Filter style : Compilers • . . . IIT Kanpur

3. 1-p12 1 p12 2 p23 3 1 C 1-p23 ESA Antenna array configuration tool architecture S0 p12 p23 p(nl-2)(nl-1) p(nl-1)nl 1 S1 S2 . . . . . . . . . . Snl-1 Snl 1-p12 1-p23 1 1- p(nl-1)nl Snl+1 Snl+2 . . . . . . . . . . S2nl-1 S2nl 1 1 1 1 1 S2nl+1 1 Modeling the workflow of the Software Architecture • We use finite absorbing Discrete Time Markov Chains (DTMC) for representing the software architecture workflow. • Examples: • These DTMCs can be used to find the average visit counts to various components for a typical request as well as the limiting probabilities. • Very useful for reliability as well as performance prediction. A typical layered architecture IIT Kanpur

4. 1 pf1 (1-pf1)p12 3 pf3 2 pf2 F 4 pf4 5 pf5 S Architecture based Reliability prediction • Issues/Questions: • What is the reliability of a software system composed of unreliable components? • How does increasing the reliability of a particular component affects overall reliability ? • How does changing the architecture affect reliability ? • Reliability modeling using DTMCs • Model the component failure as a state in the DTMC and associate probabilities of transition (unreliabilities). • Overall reliability is the probability of ending in the correct state. • Another option is using a reward based method IIT Kanpur

5. S0 p12 p23 p(nl-2)(nl-1) p(nl-1)nl 1 S1 S2 . . . . . . . . . . Snl-1 Snl 1-p12 1-p23 1 1- p(nl-1)nl Snl+1 Snl+2 . . . . . . . . . . S2nl-1 S2nl 1 1 1 1 1 S2nl+1 1 Architecture based performance prediction • Issues/Questions: • What is the maximum number of clients the system can able to handle before it saturates ? • What effect does varying the number of clients have on the throughput and the average response time ? • How should software components be allocated to the hardware nodes ? • What changes should be made in the system to improve performance ? • We take an example of layered software architecture. • A Discrete Time Markov Chain (DTMC) is used to characterize the control flow in the layered software architecture. • The DTMC is used to calculate the visit counts to various layers and then a closed queueing network (QN) model is constructed and solved. IIT Kanpur

6. Some Analysis Results • Average response time and throughput and bottleneck analysis • Effect of change in individual reliabilities & workload on overall reliability IIT Kanpur

7. Reliability and Performance Tradeoff • Given a set of components and a set of machines to deploy these onto • What are the performance & reliability characteristics of the setup under a given load ? • Which architecture should be chosen for optimal performance and reliability ? • What is the tradeoff ? • Software failures of different types cause different effects on the performance of the system. • Hardware availability also needs to be taken into account. • Some configurations will result in high throughput with decreased reliability and vice-versa. IIT Kanpur

8. Combining Static Analysis and Testing Frameworks to find Software Defects Vipindeep V

9. Write Code Testing Compile/Build Customer Evaluate Defect Prevention during Software Development Static analysis Test Suite Program Specification IIT Kanpur

10. Testing + Static Analysis – How? • Static Analysis Vs Testing • Different areas altogether, but motive – identify defects • Combining the approaches… improved program checking • Motivations for the Proposed Approach • Static analysis concerns • Testing is done.. Bugs are revealed • Ideal test suite => identify all bugs • Non-ideal scenario of Testing • Identify defects, partially • Some paths/blocks are “not (properly) tested” .. Why?? • Static analysis effort on unexplored parts of the code • Our proposed approach • Focused Static Analysis with Path Pruning using Coverage Data IIT Kanpur

11. Overview of the approach IIT Kanpur

12. Pros and Cons • Advantages: • New real errors can be identified which escaped in _all_paths analysis • Rigorous testing effort counted for better static analysis • Reduce execution time • Reduced warnings => Reduced Noise • Disadvantages: • Some errors escape in the pruned path analysis • Depends on testing framework IIT Kanpur

13. Preliminary Results • Applied the approach with FindBugs, Jlint for JAVA • Results from testing were encouraging • Tested the applicability using Random Path Selection IIT Kanpur

14. Conclusions • Works well in practice • Finds enough real defects to be useful • Noise is low enough that people can use it • Scales well, so works on large very large code bases • More issues to explore.. • Better algorithm to use testing data • Path profiles, Crash dumps – focused static analysis • Use static analysis results for focusing the testing effort • Combining static and dynamic analysis • Usability related issues IIT Kanpur

15. State Based Testing Automation Atul Gupta Ph.D. Scholar

16. State Based Testing • Program behavior can be modeled as a state graph • A formal testing approach at all levels-unit, component or system level. • Effective testing strategy to O-O software • Test Cases and Test Oracles can be automatically generated and evaluated Atul Gupta PhD Scholar IIT Kanpur

17. vend( ) allowVend = 0 curQtrs = 1 allowVend = 0 curQtrs = 0 addQtr( ) addQtr( ) addQtr( ) WAIT Q1 Q2 retQtrs( ) allowVend = 1 curQtrs >= 2 retQtrs( ) An Example : A CoinBox Class • Class CoinBox { • unsigned totalQtrs; • unsigned curQtrs; • unsigned allowVend; • public: • CoinBox( ) { • totalQtrs = 0; • curQtrs = 0; • allowVend = 0; • } • void retQtrs( ) { • curQtrs = 0; • } • void addQtr( ) { • curQtrs = curQtrs +1; • if (curQtrs > 1) • allowVend = 1; • } • void vend( ) { • if (allowVend) { • totalQtrs = totalQtrs + curQtrs; • curQtrs = 0; • allowVend = 0; • } • } • }; IIT Kanpur

18. All-Transition Coverage (AT) All-Transition-pair Coverage (ATP) Full Predicate coverage (FP) Transition-Tree Coverage (TT) All-Round-Trip path coverage (ART) Complete Sequence (CS) vend( ) allowVend = 0 curQtrs = 1 allowVend = 0 curQtrs = 0 addQtr( ) addQtr( ) addQtr( ) WAIT Q1 Q2 retQtrs( ) allowVend = 1 curQtrs >= 2 retQtrs( ) State Based Testing Criteria IIT Kanpur

19. vend( ) allowVend = 0 curQtrs = 1 allowVend = 0 curQtrs = 0 addQtr( ) addQtr( ) addQtr( ) WAIT WAIT WAIT WAIT WAIT Q1 Q1 Q2 Q2 Q2 addQtr( ) retQtrs( ) allowVend = 1 curQtrs >= 2 retQtrs( ) addQtr( ) retQtrs( ) addQtr( ) retQtrs( ) vend( ) Generating Test Sequences IIT Kanpur

20. Research Issues • How to generate test cases automatically from a state model • How to execute them in an automated manner • How to evaluate the results automatically IIT Kanpur

21. Our Approach • Identify State Model of a Class • Generate test sequences based on some Coverage Criteria • Convert these sequences into JUnit format for Automatic Execution and Result evaluation AIM : Complete end-to-end Automation IIT Kanpur