Software Testing in a Virtualized World

1. Software Testing in a Virtualized World Anuradha Biswas Prakat Solutions

2. Living upto the CIO’s Priorities • Performance • Security • Reliability • Cost

3. Viewpoints on Cloud Computing “I don’t understand what we would do differently in the light of Cloud Computing other than change the wordings of some of our ads” Larry Ellision, Oracle’s CEO “I have not heard two people say the same thing about it [cloud]. There are multiple definitions out there of “the cloud” Andy Isherwood, HP’s Vice President of European Software Sales “It’s stupidity. It’s worse than stupidity: it’s a marketing hype campaign.” Richard Stallman, Free Software Foundation founder

4. IDC’s Viewpoint • Spending on IT cloud services is expected to grow nearly threefold, to $42 billion by 2012 • Cloud computing will also account for 25% of IT spending growth in 2012 and nearly a third of the IT spending growth in 2013

5. News … Microsoft unveiled on Tuesday (16 Nov 2010) the results of its behind-the-scenes work to bring a popular suite of supercomputing software tools to its Azure cloud platform. The company collaborated with the Seattle Children's Hospital on a test job that it says would have cost an estimated $3m (£1.9m) if it had used traditional on-premises hardware, but cost little more than $18,000 using a hybrid approach. Source: http://www.zdnet.co.uk/news/cloud/2010/11/17/microsoft-shows-off-azure-power-with-software-test-40090884/

6. Defining Cloud Computing

7. What is the Fuss all about? Source: Madgeek’s Gallery on Picasa

8. Cost Savings Potential … Source: http://www.testingthefuture.net/

9. Non-Cloud Testing Steps • Requirement Study • Test Planning • Test scenario identification • Test Case / Script Writing • Coverage Review • Test Execution • Environment Preparation • Test Case / Automated suite execution • Defect Reporting and Tracking • Record defects • Report Defect • Non Functional Testing • Performance • Security/ Vulnerability • Accessibility Testing

10. Cloud Testing Steps • Requirement Study • Test Planning • Test scenario identification • Test Case / Script Writing • Coverage Review • Test Execution • Environment Preparation • Test Case / Automated suite execution • Defect Reporting and Tracking • Record defects • Report Defect • Non Functional Testing • Performance • Security/ Vulnerability • Accessibility Testing Exactly Same

11. How Cloud helps traditional Testing challenges • Securing multiple environments (OS/Service Pack/Browser etc) cost and time prohibitive Cloud provides a way to quickly ramp up different environments on the fly without investing in physical environments. • On premise qualified resource to execute testing activities Cloud breaks the physical barrier and allows executing tasks from any place with secured connectivity. • Standardization of toolsets and expectations Testing on cloud requires usage of specific tools and procedures. This in turn enforces certain discipline and data mining and reporting becomes symmetric.

12. Critical Factor for Effective Cloud Testing • A challenge in testing cloud-based applications is the difficulty of replicating the number, diversity, and geographic distribution of the clients. • Testing the behavior and functionality is not sufficient. • It’s difficult to quickly and efficiently generate enough client. A cloud of virtual test clients along with software to orchestrate, record, and analyze the testing. • Performance • Security

13. Performance in the Cloud Testing all the layers — from your application to the cloud service provider Source: Madgeek’s Gallery on Picasa

14. Measuring Performance • Cloud Infrastructure Test performance of the cloud provider(s) to ensure they can sustain the needed load. • Data Test the system with real life data handling scenarios • Test the flow of data from various endpoint, with varying payloads, spread over several hours • You need to ensure that the cloud can sustain heavy loads, handle concurrency, and consistently deliver solid performance for all transactions. • Analyze the impact of encryption, transformation, data replication, and the various ways that data is being manipulated to address security, compliance, reliability, and scalability requirements. • This stage is critical and forms a basis for troubleshooting on any performance bottlenecks.

15. Measuring Performance • Business Logic Interactions of services and processes. • User Experience Does the application meet user needs? • First test each service individually. • Test the flow of data through the various combinations of service calls that the system is expected to perform. • Rigorous testing and measure performance for load, sustainability, concurrency, etc. • Test the system holistically and in the eyes of the end user • It would be very expensive to find performance issues from the lower levels of the architecture at this point. • This approach is iterative and agile and aims at removing performance risks earlier in the lifecycle thus reducing the risk of project delays.

16. Mind the “Security” Gaps Source: cloudsecurityalliance.org

17. Security in the Cloud • Test the security of the VMs augmented by third party security tools which provides layered security • Ensure security controls are in place internal to the VMs other than the built in hypervisor isolation — such as intrusion detection, anti-virus, vulnerability scanning, etc. • Test the security controls which are in place external to the VMs to protect administrative interfaces (web-based, APIs, etc.) exposed to the customers. • VM-specific security mechanisms embedded in hypervisor APIs must be utilized to provide granular monitoring of traffic crossing VM backplanes, which will be opaque to traditional network security controls.

18. Security in the Cloud • Administrative access and control of virtualized operating systems is crucial, and should include strong authentication integrated with enterprise identity management, as well as tamper-proof logging and integrity monitoring tools. • Ensure that VMs are segregated in separate security zones by type of usage (e.g., desktop vs. server), production stage (e.g., development, production, and testing) and sensitivity of data on separate physical hardware components such as servers, storage, etc. • Be aware of multi-tenancy situations with the VMs where regulatory concerns may warrant segregation.

19. In conclusion - Why is it Different? • Shared “multi-tenant” test environments • Security in context of the applications being off premise and potentially sharing all resources – processor, storage, network • Integration of on/ off premise systems (Public/ Private Clouds) • Performance / Stress testing is challenging since often new instances get spawned off automatically at certain thresholds • Defect isolation is not always easy • Cloud Platform is dynamic in terms of the actual versions of software running and in terms of loads and other applications sharing the same resource pool

20. Thank You anu@prakat.com