1 / 24

Engineering Self-Testable Autonomic Software

Engineering Self-Testable Autonomic Software. A COMPARATIVE CASE STUDY. Presenter: T ariq M . King. EASe 2011. April 27-29, Las Vegas, NV, USA. Outline. Motivation Background Applications Experiments Lessons Learned Related Work Conclusion. Motivation.

roscoe
Download Presentation

Engineering Self-Testable Autonomic Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Engineering Self-Testable Autonomic Software A COMPARATIVE CASE STUDY Presenter: Tariq M. King EASe 2011 April 27-29, Las Vegas, NV, USA

  2. Outline • Motivation • Background • Applications • Experiments • Lessons Learned • Related Work • Conclusion

  3. Motivation A Neglected Self-* Characteristic • Self-* features in autonomic computing systems may incorporate dynamic software adaptation • System adds, removes, replaces its own components @ runtime • Raises reliability concerns asnew faults may be introduced due to dynamic adaptation • Most research neglects that self-test should be an integral part of these types of systems Self- Configure Self- Heal Self-Test Self- Optimize Self- Protect FAULTS? FAILURES?

  4. Background Autonomic self-testing (AST) • King (2009) describes an approach that seeks to seamlessly integrate runtime testing into the workflow of Autonomic Managers (AMs) • Idea: Instantiate newAMs that are tailoredfor testing activities • Test Managers (TMs)monitor, intercept & validate the change requests generated byAMs to realize AST Test Manager Autonomic Manager Analyze Analyze Plan Plan Monitor Monitor Execute Execute Knowledge Knowledge Autonomic Manager Sensor Effector Managed Resource

  5. Background AST Strategies & TOOL Support • Under AST, test managers can operate according to two different validation strategies: • Replication with Validation (RV) – tests changes using separate copies of managed resources • Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources • Supported by an OO framework for implementing prototypes of self-testable autonomic software • Java, Reflection, Threads, XML-Based Policies • Interfaces with Automated Software Testing Tools

  6. Applications PRELUDE • During preliminary investigations, we sought to locate freely available autonomic software that could be used to demonstrate our ideas on AST • When no such applications could be found, focus shifted to development through our own avenues: • REU on Autonomic Computinghttp://www.cis.fiu.edu/reu/ • CVM Research Projecthttp://cvm.cs.fiu.edu/

  7. Applications Autonomic Container • Stevens (2007) developed a self-configuring data structureto demonstrate ideas on AST ACT Lightweight StructuralAdaptations RV Strategy

  8. Applications Autonomic Job scheduler • Ramirez (2008) extended the ACT prototype by applying it in the context of short-term scheduling • A pool of agents service jobs according to a high-level scheduling algorithm, and self-configure at runtime, e.g., InteractiveFCFS, Batch SJN • Job requests stored in an autonomic container • Self-Optimization is achieved by adding or removing agents in proportion to the workload • A subsequent point release introduced Self-Protection by monitoring exception throwing

  9. Applications Autonomic Job Scheduler AJS Lightweight(More Complexthan ACT) Structural &Behavioral Adaptations RV Strategy

  10. Applications Communication Virtual Machine • CVM is a model-driven platform for realizing communication services • Collaboration: FIU & Miami Children’s Hospital • Separates concerns of modeling, synthesis, coordination, and delivery of user communication • Users specify their communication requirements, which are represented as schemas • Schemas are then transformed into executable control scripts, which are used to configure the underlying network devices and protocols

  11. Applications Communication Virtual Machine • Allen (2009) leverage AC and open-platform APIs for integrating communication services into CVM CVM Real-World BehavioralAdaptations SAV Strategy NCB LAYER

  12. Experiments overview • Conducted three sets of experiments as part of the case study: • Development Experiments – compared the relative effort required to build the autonomic and self-test infrastructures, and various prototypes • Performance Experiments – compared the runtime performance of different versions and variants of the prototypes • Test Quality Experiments – compared the effectiveness of test sets in revealing faults and exercising the program code of the prototypes

  13. Experiments Setup Environment • Experiments were performed on windows-based Intel® Core 2 Duo 2.4GHz PCs with 2–4 GB RAM • Several tools were used to support automatically collecting results: • Eclipse Metrics – source code size and other complexity metrics for development experiments • Eclipse TPTP – thread and memory analysis for performance experiments • JUnit, Cobertura, MuJava– test failures, code coverage, and mutation analysis for test experiments

  14. Experiments Development COMPARISON OF AUTONOMIC & TEST INFRASTRUCTURES

  15. Experiments Development (Cont.) COMPARISON OF PROTOTYPE DEVELOPMENT

  16. Experiments PERFORMANCE • Non-Self-Testvs. High Transparency Self-Test • Used Java RMI to develop a self-test version of AJS with a distributed RV self-testing process • Performed identical sets of user actions on both systems and compared memory and thread usage • Results: Less than 1% difference • Naivevs. “Enhanced” Interleaving Self-Test • Introduced timeout into TM threads of CVM • Purpose: Seek to achieve better interleaving in favor of core system over self-test during SAV self-testing

  17. Experiments Performance (Cont.) SKYPE TWO-WAY TO SMACK THREE-WAY COMPARISON OF SELF-TEST VARIANTS CVM

  18. Experiments Test quality MUTATIONSETUP

  19. Experiments Test quality (Cont.) MUTATION RESULTS CODE COVERAGE RESULTS

  20. Lessons Learned • RV can be used to provide a transparent self-testing process, while SAV requires optimizations. • Using TMs as a test harness was convenient but problematic: Complexity, Synchronization • Safety mechanisms (locks) were instrumental • Selecting and tailoring open-source testing tools presented unnecessary difficulties • Benefits: Prevention of Runtime Faults Testing Aligned with Development

  21. Related Work • “Testing and assurance are probably the least focused phases in engineering self-adaptive software…” – Salehie (2009), ACM Trans… • King et al. (2007), and Zhang et al. (2007) • Da Costa et al. (2010) • JAAF+T: Java Self-Adaptive Framework + Test • Modify MAPE structure to include a self-test activity • Munoz & Baudry (2009) • Artificial Table Testing of Adaptive Software

  22. Conclusion • Presented a case study on the engineering of autonomic software with self-testing features • Reached a level of “enablement”, still many open research issues: • Dynamic regression test case selection and scheduling based on runtime testing goals • Runtime test case maintenance • Reducing overhead of runtime testing • Current Directions: Cloud, Virtualized RV.

  23. Acknowledgements • Dr. Masoud Milani, FIU • Dr. S. M. Sadjadi, FIU • Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU • Students and collaborators on CVM project This work has been supported in part by the NSF under grants IIS-0552555, and HRD-0833093

  24. Thank You! Questions?¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις

More Related