Tariq M. King

1. Towards a Self-Testing Cloud Computing Infrastructure Tariq M. King STITC 2011 Markham, Ontario, Canada Co-located with

2. Outline • Introduction • Research Questions • Cloud Testability • Test Support as-a-Service • Vision: Self-Testing in the Cloud • Summary • Open Discussion

3. Introduction CLOUD • Cloud provides ubiquitous, on-demand access to computing resources “as services” via the Internet • Operational Perspective • Availability, Scalability, Efficiency, … • Development Perspective • ReusableComponents, Info. Hiding • Highly Complex Local Infrastructure • Testing Perspective?

4. Introduction STITC • Cloud brings new testing challenges and benefits • Software Testing In the Cloud (STITC) • STITC 2010 Workshop focused on three areas: • Testing IN the Cloud – leveraging the cloud for test execution and testing in a virtualized environment • Testing OF the Cloud – validating applications that are hosted and deployed cloud • Testing TO the Cloud – moving the testing process and other assets to the cloud

5. Introduction Cloud Services • Services in cloud computing typically fall intothree categories: Software-as-a-Service| Google Docs, Office Web Apps Platform-as-a-Service| Google App Engine, Windows Azure Infrastructure-as-a-Service | Amazon EC2, Rackspace

6. Introduction Virtualization • Many cloud platforms offer Virtualization support • Creating abstract computing resources from more powerful physical resources Hosted Architecture • Hypervisor Architecture

7. Introduction AUTONOMIC COMPUTING • Cloud Computing shares characteristics with Autonomic Computing (AC), Grid, Utility, … • Central theme of AC is self-management to reduce burden of human administration and integration • King et. al 2008 introduce animplicit self-test characteristic • Regardless of self-* featuresimplemented, system shouldvalidate adaptations at runtime Self- Configure Self- Heal Self-Test Self- Optimize Self- Protect

8. Introduction About this talk • “Migrating Autonomic Self-Testing to the Cloud” • Tariq M. King and Annaji S. Ganti | STITC 2010 • High-level position on how a self-testing approach for autonomic software can be applied to the cloud • In today’s talk, we highlight our recent progress and developments from the above perspective • Extended version of King et al. | CASCON 2011 • Summarizes contents of book chapter submitted to STITC: Perspectives on an Emerging Discipline

9. Research Question #1 • How does cloud computing affect our ability to test applications that are either built or deployed in a cloud environment, or depend on cloud services? Testability Investigates impact of cloud development model on software testability (Positive, Negative)

10. Research Question #2 • How can the test artifacts and tools used to validate a cloud application service during development, be reused to help test derivative applications? Test Reuse Investigates ways to reuse outputs of pre-deployment testing effort (Artifacts, Tooling) TEST

11. Research Question #3 • How can built-in cloud computing technologies such as virtualization be used to improve the process of testing cloud applications? Cloud Leverage Investigates ways in whichthe cloud infrastructure itself can be harnessed for Testing OF the Cloud (Power, Virtualization)

12. Cloud Testability: The Negative (Paradigm Infancy) Autonomy Adaptiveness Information Hiding Statefulness High Dependability Criteria Remoteness Complexity High Performance Criteria

13. Cloud Testability: The Positive Computational Power Storage Virtualization Testing requires lots of resources and the cloud is certainly powerful enough to handle it

14. Testing of the Cloud, In the Cloud • As a step towards Research Questions 2 and 3, we investigate a novel approach to software testing of the cloud, in the cloud | King et al., CASCON ‘11 • Scenario: One host (Provider B) is developing an application service that will extend functionality of a service offered by another host (Provider A)

15. Integration Testing Challenges • Even with its limited scope, our scenario reveals many cloud integration testing challenges: • Difficult for Provider B to Setup Some Tests for his application since he does not have full control and observation of Service A. • Developing Accurate Stubs is Hard since Provider B has no knowledge of Service A’s implementation • Provider B Limited to Black Box Testing of those parts of his application implemented by Service A • Service B can Change w/oProvider B’s Knowledge i.e., Update to Service A Need to Retest Service B

16. Proposed Solution • Our proposed solution introduces the notion of Test Support as-a-Service (TSaaS) in the cloud • Prior to the deployment of Service A, Provider A would have tested it, and created: • Test Artifacts – Test Drivers, Test Stubs, Test Data • Local Test Bed – Hardware/Software Environment • TSaaS seeks to reuse existing test artifacts, tools and configurations of the local test bed, to aid the rapid development and delivery of test support services for collaborating partners

17. Test Support as-a-Service (TSaaS) • Provider Bhas full access to a test copy of Service A • Test hooks, instrumentation, and scaffolding reuse • Test data (non-confidential) can be made available • Testing does not interrupt service in production • Concerns: Maintenance, Scalability, Security Provider A exposes TSaaS to Provider B

18. TSaaS Architecture Hypervisor-Based Virtualization and Secure Access

19. TSaaS Control Flow Control Flow When Realizing Test Support Services

20. TSaaS Development Method Using Existing Test Automation to Guide Development

21. TSaaS Prototype DETailed design • Designed with goal of making its components easily deployable to Windows Azure® • Web & Worker Roles (ASP.NET, IIS7, WCF Services) • VM Role (Windows 2008 R2 Server Image) • Application Domain: Credit Reporting/Banking

22. TSaaS Prototype Programming Interface • We were able to develop and expose a cross section of test support operations via the TSaaS API • Test Specification – user defines fixture operations • Test Execution – runs specified fixture operations • Code Coverage – analyzes branch and lines covered • Server-Side Diagnostics – facilitates configuring web server, including performing a reset after failures • Runtime Virtualization – create, start, stop, delete, take snapshot, and update VHD configuration

23. TSaaS Prototype Graphical User Interface

24. Vision: Self-Testing In The Cloud Provider B monitors Service A for changes Provider B Autonomic Test Manager Test Harness Service B (Test Environment) Analyze Plan Provider A builds a sensor for Service A If a change occurs, Provider B generates a plan for regression testing of Service B Monitor Execute Knowledge Provider B reruns tests on latest build of Service B, which is set to use new version of Service A via TSaaS Provider A Sensor Service A (Production Env.) TSaaS

25. Foreseen Benefits Test Support as-a-Service • Improved Testability and Confidence (Partner) • Control/Observation, White Box Testing, Metrics • Improved Product (Provider) • Partners test your component, bug reporting/fixes • Virtualization Benefits • Snapshots: Speed Up, Bug Replication/Debugging • Scalability, Ease of Maintenance, Hardware Savings, and Environment Testing (OS, Applications)

26. Foreseen Benefits (cont.) Self-Testing in the Cloud via TSaaS • Automatic Regression Testing of changes due to: • Software maintenance and evolution • Dynamic adaptation and updating • Improved Autonomic Decision-Making • Use of validation to determine which service may be the best candidate to select during adaptation • As a proactive form of self-healing (fault detection through formalized testing techniques)

27. Key Challenges • Standardization • Providing standardized interfaces and guidelines for performing testing in the cloud, and developing test support infrastructures • Evaluation • Providing convincing evidence that these approaches are not only feasible to implement but can have real value to practitioners • Validation • Showing virtualization is consistent with production

28. Summary • Described our progress towards our grand vision of migrating autonomic self-testing in the cloud • Proposed an approach to Test Support as-a-Service,which uses the cloud to tackle testing of the cloud • Described the design and implementation of a aprototype of TSaaS • Discussed benefits and key challenges associated with TSaaS and use for self-testing OF/IN the cloud • In terms of our progress, we have barely scratched the surface of the research problem under investigation

29. Acknowledgements • Annaji Ganti | PhD Student, NDSU/Microsoft • Neha Kale | MS Student, NDSU • David Froslie | Test Architect, Microsoft • AlissonSol | Knowledge Engineer, Microsoft • Dr. Scott Tilley | Professor, Florida Tech • Participants and Reviewers • STITC 2010 • CASCON 2011

30. Thank You! Questions?¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις