Safe Runtime Validation of Behavioral Adaptations in Autonomic Software

1. Safe Runtime Validation of Behavioral Adaptations in Autonomic Software Presenter: Tariq M. King ATC 2011 September 2-4, Banff, Alberta, Canada

2. Outline • Motivation • Introduction • Testing Approach • Component Design • Prototype • Related Work • Conclusion

3. Motivation A Neglected Self-* Characteristic • Self-* features in autonomic computing systems may incorporate dynamic software adaptation • System adds, removes, replaces its own components at runtime • Raises reliability concerns asnew faults may be introduced due to dynamic adaptation • Most AC research neglects that self-test should be an integralpart of these types of systems Self- Configure Self- Heal Self-Test Self- Optimize Self- Protect FAULTS? FAILURES?

4. Introduction Autonomic self-testing • To address this issue, we introduced the notion of Autonomic Self-Testing (King et. al 2007) • Idea: Investigate howAutonomic Managers(AMs) can be tailored for purpose of testing the system itself • Test Managers (TMs)Monitor, intercept and validate the adaptive changes requests of AMs Test Manager Autonomic Manager Analyze Analyze Plan Plan Monitor Monitor Execute Execute Knowledge Knowledge Autonomic Manager Sensor Effector Managed Resource

5. Introduction Autonomic Self-Testing (cont.) Initial work on AST concentrated on: • Defining validation strategies for TMs • Replication with Validation (RV) – tests changes using separate copies of managed resources • Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources • Developing prototypes to investigate the feasibility of the RV strategy • Lightweight, focused on localized validation

6. Introduction CONTRIBUTIONs In this research, we extend previous work on AST by overcoming two of its limitations. Major Contributions: • Addresses the need for system-wide validation in autonomic software through the description of an automated runtime integration testing approach • Investigates AST of a real-world application in which it is too expensive to maintain copies of managed resources for testing – Validates SAV

7. Overview of Testing Approach System-Wide Validation of Behavioral Adaptations Using Self-Testable Autonomic Components (STACS)

8. State-Based Safety Model 1. Extends Zhang (2004) “Enabling Safe Component-Based Software Adaptation” in WADS ‘04

9. Component Design STAC Definition Self-Testable Autonomic Component (T, A, R, I, K)

10. Managers (T, A) • Abstract MAPE • Generics • Multi-Threading • Safety Mechanisms

11. Resources (R) Full Access to Source Code • Derive suspend, resume, saveand restore state operations from AbstractResource class • Loose coupling of concreteobjects to facilitate stubinjection at runtime Limited Access to Source Code • Use wrapper class (carefully) • Research on improving COTS testability

12. Test Interface (IT) • Test Case Execution • Code Coverage Analysis • Performance

13. Prototype Communication Virtual Machine • Communication Virtual Machine (CVM) is a model-driven platform for realizing user-centric communication services • Collaboration: FIU & Miami Children’s Hospital • CVM separates the concerns of modeling, synthesis, coordination, and delivery of communication services into self-contained layers • Layer in which communication in communication takes place is called the Network Communication Broker (NCB).

14. Prototype Communication Virtual Machine • Allen (2009) leverage AC and open-platform APIs for integrating communication services into NCB NCB LAYER OF CVM

15. Experiment Overview • Conducted two sets of experimental runs as part of the prototype evaluation: • Test Quality Experiments – compared the effectiveness of test sets in revealing faults and exercising the NCB program code • Performance Experiments – investigates ways to reduce the performance overhead of having a runtime testing process interleaved with the NCB • In both cases, a Mutation Analysis technique was used to simulate faulty change requests, induce self-testing, and produce the results

16. Setup Environment • 39 mutants were generated for the prototype by planting artificial faults into the interfaces and/or implementations of Skype and Smack • 15 created manually, 24 automatically generated • Test Support Tools: JUnit, Cobertura, MuJava, Eclipse TPTP • Experiments were performed on a windows-based, webcam-enabled Intel® Core 2 Duo 2.4GHz PCs with 2 GB RAM or greater

17. Results Test Quality • Fault Detection • 35 out of the 39 mutants were detected, producing a mutation score of 89.7% (Threat, CVM Dev/Test) • Code Coverage • Recorded 63% statement coverage and 57% branch coverage of the NCB implementation • 100% method coverage of the communication service interfaces (SkypeAdapter, SmackAdapter)

18. Results (cont.) PERFORMANCE SKYPE TWO-WAY TO SMACK THREE-WAY

19. Lessons Learned • System-wide SAV is feasible and beneficial for preventing runtime faults in the context of a real-world application • Testing adaptive changes in-place using TMs was challenging: high complexity, synchronization • Significant performance overhead, further research • Safety mechanisms (locks) were instrumental during development and runtime testing • Selecting and tailoring open-source testing tools presented unnecessary difficulties

20. Related Work • Da Costa et al. (2010) • JAAF+T: Java Self-Adaptive Framework + Test • Modify MAPE structure to include a self-test activity • Stevens et al. (2007) and Ramirez et al. (2008) • Prototypes of Replication with Validation • Autonomic Container & Autonomic Job Scheduler • Zhang et al. (2009) and Zhao et al. (2005) • Runtime verification of self-* systems

21. Conclusion • Presented a system-wide approach to runtime integration testing in autonomic software • Still many open research issues: • Dynamic test generation • Dynamic regression test scheduling • Runtime test case maintenance • Reducing overhead of runtime testing • Latest Direction: Cloud(CASCON 2011)

22. Acknowledgements • Dr. Masoud Milani, FIU • Dr. S. Masoud Sadjadi, FIU • Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU • Graduate students and research collaborators on CVM project This work has been supported in part by the NSF under grant IIS-0552555.

23. Thank You! Questions?¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις