Download
1 / 59
590 likes | 742 Views
Develop a Cloud Testing Strategy for Today’s Apps. The Cloud opens new opportunities for testing that were previously impossible. Introduction.
E N D
Develop a Cloud Testing Strategy for Today’s Apps The Cloud opens new opportunities for testing that were previously impossible.
Introduction Organizations are finding it difficult to execute test cases and monitor performance of a new breed of cloud applications. Leverage cloud testing to augment traditional practices and open the door to a higher level of software testing. Organizations that are refining testing methodologies and protocols to encompass cloud technology. Application Testing Managers responsible for setting a standardized testing strategy. Software Developers tasked with testing small units of newly written code. Quality Assurance Managers looking to streamline QA processes. Realize the changes of the software testing life cycle with the introduction of cloud technology. Show you how the Cloud can significantly expand your testing capabilities. Understand the costs and benefits of cloud testing. Develop a standardized approach to testing in the Cloud. This Research Is Designed For: This Research Will Help You:
Follow the Info-Tech Software Testing Roadmap • Develop and Improve a Software Testing Strategy • Traditional software testing processes provide a concrete foundation when developing and standardizing your software testing strategy. • Develop a Cloud Testing Strategy for Today’s Apps • This solution set will provide insight into how to incorporate cloud technology into your current software testing strategy. • Learn how to align cloud testing initiatives with the development and traditional testing workflow. You are Here! • Assess the Appropriateness of Automated Functional Testing • Automating functional testing can improve software quality for the right shop, butthe cost of automation tools and ongoing maintenance can outweigh the benefits. • Vendor Landscape Plus: Tools for Functional Testing Automation • There are dozens of tools out there that can automate functional testing. Select the right tool by understanding the features and requirements associated with functional testing tools. Testing Strategy • Vendor Landscape: Software Test Management • Gain insight into which vendor best aligns with your organization’s needs and which Software Test Management solution will provide the most efficiency within your infrastructure.
Executive Summary The reality of today’s complex web and cloud applications demands a new approach to testing. Your rollout strategy needs to be in three phases. Get cloud testing into your testing portfolio mix and start reaping the benefits. • Increasingly connected apps using the Cloud at the back end along with highly fragmented endpoints at the front end make traditional combinatorial testing almost impossible to execute; there are simply too many options. • Traditional load and stress testing scaled down, simulated production environments, and forecasting the break point doesn’t hold with cloud virtualization and scalability. The focus on load and test needs to be done in the Cloud with scale if testing is to be reliable. • Today’s increasingly global apps requires a global test lab with custom stacks. The Cloud offers a cost effective means to achieve this. • Business Case: Align cloud testing strategy and execution in order to ensure both the implementation and metrics are of value to the business. • Implementation: Thankfully much of the testing and development process remains the same. Organizations need a good vendor with a toolset aligned along the business strategy, and solid knowledge of the Cloud to help the implementation go smoothly. • Metrics: Organizations need to measure testing metrics before and after cloud testing is rolled out. The ability to parallelize tests in volume with low cost can add considerable return on the investment. • Innovation is occurring in the in the young cloud testing domain. This includes such features as the ability to test gestures for multiple touch devices – something almost impossible to achieve in volume for a broad array of devices. • Cloud testing vendors are eager to gain market share in this young business model. This gives organizations a high degree of service as cloud testing vendors compete for revenue.
Testing should be where applications are hosted: in the Cloud As more applications move to the Cloud, testing with a limiting on-premise toolset introduces risk. Challenges? • Today’s applications are hosted on the web or moving towards the Cloud. Organizations require a new framework around testing cloud applications that account for on-demand scalability and self provisioning. • Expectations of application consumers (whether desktop, mobile, or touch) are continually increasing with speed-to-market and quality being the norm. • Applications reside in different locations – on-premise, private cloud, public cloud, and hybrid cloud options – and testing against such diverse deployments requires reach and scale. • Even if vendors have a cloud testing product, migrating would mean added costs of reconstructing existing test cases. Opportunities! • With cloud testing services, organizations can now more accurately represent a production environment by deploying scalable infrastructure components and application stacks cost effectively. • Long running tests can now be shortened by using cloud testing services to fan out a scalable number of functional testing units on demand during testing and executing in parallel. You need a lot servers to be able to simulate load. Cloud computing gave us access, availability, and affordability to lots of servers around the world to simulate that load. - Tom Lounibos, President & CEO, SOASTA
Adopt cloud testing or risk releasing defective applications due to budget constraints Traditional testing is costly and time consuming. Remove these burdens with a cloud testing solution. Scale on Demand Multiple Form Factors Globalization Cloud technology does not change the traditional testing processes that many organizations have accepted and adopted. It does, however, enhance traditional practices with increased replication capacity, execution speed, and compatibility through its virtual infrastructure and automated processes. For more information about traditional testing, refer to Info-Tech’s Develop and Improve Your Software Testing Strategy.
Leverage the benefits of cloud testing and avoid being stuck with traditional testing limitations Cloud testing offers two unique capabilities that are difficult to replicate with traditional testing: virtualization and compatibility with cloud services. Virtualization is the foundation for automated, on-demand, and elastic testing. Organizations can: • Create custom test labs, of both in-scope and out-of-scope systems, for various technology stacks through virtual machines and service virtualization. • Closely replicate production environments. Traditional test labs were restricted to scaled versions for capital expense. • Parallelize testing through creations of multiple test labs running concurrently such that each lab is focused on a subset of use cases. Cloud testing is compatible with today’s web and cloud paradigm. Cloud testing can reside: • In a public Cloud and spread globally to provide a more accurate picture of load testing. • In a private Cloud with access to internal resources and capability to test software-as-a-service (SaaS) cloud services in the public Cloud. Virtualization Compatibility with Cloud Services
Use cloud testing to handle test cases for cloud or web apps and avoid incomplete test runs caused by scalability barriers Cloud testing offers better options for tests that depend on scale and automation. Cloud technology has shone a new light onto traditional software testing strategies, enabling a number of unseen testing activities, such as: • High quality testing environments. • Parallel development and testing capabilities. Physical testing labs are replaced by virtual ones that can be scaled n-times. Testers are no longer required to make individual configurations to the testing infrastructure due to the Cloud’s automated process. Volatility is minimized. Stubbing is a practice of the past. Dependent systems are virtualized and replicated at a high degree of accuracy. Cloud testing offers a new testing avenue for certain tests. Development Cloud Traditional Testing Requirements Develop Test Case Develop Test Case Testing Design Code Code Fail Fail Deployment: UAT/SMOKE Test Execution Test Execution Test Eval. Test Eval. Maintenance Succeed Succeed Source: Info-Tech Research Group
Integrate cloud testing into your current development lifecycle to strengthen quality The only impact to your software development life cycle (SDLC) is within testing. The process for development and lower level unit and integration testing remain intact. Each stage before testing remains the same. Functional, load, and stress testing is forked to the cloud while other test processes remain as is. 1. Requirements 2. Design 3. Build Software Development Life Cycle (SDLC) 4. Test 6. Maintain 5. Deploy Development Workflow Source: Info-Tech Research Group
Manage current technology trends through cloud testing to avoid stretching timelines or compromising test results Cloud applications introduce new capabilities around JIT scalability and global reach. Your testing practices need to account for these new capabilities in light of recent trends. The application development and testing models are changing: Fixed infrastructure is now elastic. Mandated and organization-issued devices are now employee supplied through bring-your-own-device (BYOD). Data centers are now hosted and stored in the Cloud. Development times are shorter. Organizations need to adapt to these changes because current testing processes are limited. Today’s cloud testing needs to: Be elastic to scale up/down test clients. Accommodate test labs with unique stacks for multiple devices. Handle applications hosted on the web and in the Cloud (public and private) rather than client/server. Be available on-demand. Decreasing Cycle Time Elastic Environment Testing & Development Device Fragmentation Cloud Applications
Develop a cloud testing strategy and avoid being left with buggy applications for the modern era Follow a step by step process to ensure cloud testing satisfies the required business and IT goals. Traditional testing alone is no longer sufficient for modern applications. Cloud testing fills the gap. 1. Build a Business Case Cloud testing requires new thought patterns to handle modern web and cloud integrated apps. 2. Implement Cloud Testing Create metrics to measure the success of your cloud testing efforts. 3. Focus on the Metrics Follow this diagram to navigate your way through the storyboard.
Build a Business Case Build a Business Case Implement Cloud Testing Focus on the Metrics Understand organizational characteristics Evaluate cloud testing readiness List of business drivers Cloud testing benefits and challenges Determine the costs Establish key performance indicators
Create a cloud testing business case for your organization to avoid downstream compliance hurdles Perform an analysis to assess operational and risk factors before embarking in cloud testing. Cloud testing can have little value to an organization’s software testing needs if company policies and industrial standards restrict the full potential of the Cloud. Consider the following organizational attributes when developing a testing strategy: Cloud deployment – Applications can be hosted and tested on private, public, community, and hybrid cloud platforms. Each type of Cloud requires different testing access permissions: identify your cloud deployment on the cloud continuum. Hosting platforms – Applications can be hosted on the web, desktop (locally or virtualized), and in the Cloud. Application integration – Integration enables applications to communicate with each other directly or through a broker or bus. Application complexity – Applications with multi-tiered architecture introduces intricate application environments. Cloud Deployment Options • Private Cloud – Cloud infrastructure provisioned for exclusive use by a single organization. • Community Cloud – Cloud infrastructure provisioned for exclusive use by a specific community of consumers. • Public Cloud – Cloud infrastructure provisioned for open use by the general public. • Hybrid Cloud – Cloud infrastructure composed of two or more distinct cloud infrastructure. Public Cloud Community Cloud Hybrid Cloud Private Cloud CLOUD CONTINUUM
Conduct a readiness assessment across people, process, and technology to avoid roadblocks • Since cloud testing can affect people, process, and technology across the enterprise, build a cloud testing strategy designed around these. People Cloud testing requires people which means it will affect the development team: consider resourcing levels and the overall corporate culture, including possible change readiness. • End-user technical skill. • Willingness to learn new methods and technologies. • Frequency of communication between testing and development. • Acceptance of, and willingness to use, new tools. • Willingness to be a team player and align to corporate goals and strategies for improving application quality. Process Cloud testing works best within the context of testing processes for applications: consider the level of business and application process orientation, automation, and standardization. • Level of development process maturity and standardization. • Level of testing automation. • Level and availability of documentation. • Willingness to implement and refresh IT policies and procedures. • Frequency of process change to meet changing development strategies. Technology Cloud testing requires technology: IT needs to consider how the infrastructure, hardware, and software support match testing needs and complexity. • Age of current technology environment. • Available cloud solutions from vendors. • Type of application hosting platform. • Degree of standardization around new and existing technology. • Rate of application delivery. • Reliability, capacity, and availability of technology. • Degree of physical and virtualized testing labs.
Use Info-Tech’s Cloud Testing Readiness Assessment Tool to determine your organization’s readiness Analyze the people, process, and technology in your organization to determine cloud testing readiness. Focus on the areas for improvements in this tool and plan to deal with them before fully engaging in your cloud testing strategy. This tool will help you think about your current testing processes, which sets the stage for developing a cloud testing strategy. Answer questions regarding people, process, and technology, and realize the areas that need improvement. Info-Tech Insight
Steer your cloud testing strategy towards your business drivers to increase business buy-in Identifying business drivers is crucial to cloud testing success and is the initial point of your cloud testing strategy. Business drivers are the root of a cloud testing strategy. With well defined business drivers, your testing requirements, test design, test execution, and user and release assessments will also be well-defined – increasing cloud testing success. Involve all key business stakeholders when generating the list of important business drivers. These drivers will reveal the applications, technologies, patterns, and scenarios necessary to achieve the desired business goals. Keeping business stakeholders involved in the foundation of a cloud testing strategy drastically increases the software testing success rate. Info-Tech Insight Cloud technology is no longer a hype. Adoption has seen considerable growth in recent years and the impact of adoption will be experienced throughout IT. For more information refer to Info-Tech’s solution set, Prepare IT for Increased Cloud Adoption. Common Business Drivers for Cloud Testing • Reduce capital (CAPEX) and operational (OPEX) development expenditures. • Faster time-to-market delivery. • Transparent reporting into development status. • Development and testing standardization. • Self-service capabilities. • Global application access.
Realize technical hurdles early so that your cloud testing strategy can work around the risks Development process and technology may be cemented based on proven best practices. Cloud testing should fit closely into those practices. Identify any current hurdles within your organization’s boundaries: Technological – Applications and test cases need to be migrated to the Cloud, systems need to replicated, and environments need to be provisioned. This will likely affect your current budgeting and resources allocated to IT. Test Environment Issues – Test environments may need to be redesigned to scale up or scale down components n-times and across Cloud deployments. Response Times – When applications are hosted on third-party platforms and infrastructure, latencies are beyond the tester’s control. Testing processes need to be adjusted to different testing runtimes. Compliance and Regulations – Company standards and regulations may restrict test data from being hosted off-premise. Pay attention to privacy and compliance issues surrounding test data management. Other Notable Obstacles • Firewall exceptions, remote configurations, and other security and privacy issues. • Storage and bandwidth costs. • International requirements. • Systems that can not be virtualized. • Various approaches to cloud computing from different cloud vendors.
Include hard benefits when making the cloud testing case to make the investment a well calculated risk When executives see intangible benefits alongside dollar gains, they are more likely to commit to a formal cloud testing strategy, improving success. Benefits from a formal cloud testing strategy largely depend on how requirements are defined and how tests are designed.Include techniquesthat will bridge people, process, and technology gaps, and create quality test cases when making the strategy case. In an action plan, include the preliminary people (testers, developers), processes (automation, migration), and technology (applications, Cloud, SaaS) that will be used to fill any foreseeable gaps. Organizations With Formal Testing Strategies are Perceived To Have Greater Cloud Testing Benefits Survey: Info-Tech Research Group; N = 39 Formal cloud testing strategies receive greater benefits, as displayed above. Formal cloud testing requires well defined business drivers defined by business leadership. Let the business know its involvement is crucial to maximizing these benefits. Additional benefits include: rapid provisioning, advanced virtualization, scalable and elastic testing environments, multi-tenancies, real-time testing, accurate replication of production environment, and executing long-running tests.
Quantify how cloud testing improves the testing life cycle to surface the opportunity cost of not doing cloud testing Developers implement lower level tests in their tightly controlled workflow. Any long running test cycles will throw off the development timeline. Each software version is released for debugging and testing. When a software version passes this test, the update is added to the trunk, or parent, codebase. This serial process has two fundamental flaws: • Development can only proceed when the prior version passes debugging and testing. Inadequate testing slows software development. • Test cases need to be developed or updated when new versions branch from the trunk. Organizations with ad-hoc strategies will consume a significant amount of time and resources rerunning tests. Cloud testing enhances existing testing practices while leaving traditional development processes intact. The Cloud enables transferrable test cases, rapid test executions, and quick scalable resource provisions. This translates to shortened testing lifecycles. Info-Tech Insight Flaws of Traditional Testing Fast Test Run-time With every software update, the testing designs and scripts must be redefined. Leverage a test case catalog to use old tests as frameworks and to identify the dependencies of the updates with out-of-scope systems. Quick Scalable Resources Reusable Test Cases More Bugs Found Per Test Cycle Significant Cost Savings from Cloud Testing Added Value from Cloud Testing
Be realistic and plan to deal with cloud testing challenges or get ready to suffer with disillusionment Organizations that had a harder time dealing with cloud testing challenges experienced lower levels of success. Migrating Test Cases to the Cloud Controlling Implementation & Operational Costs Redefining Testing Processes Survey: Info-Tech Research Group; N = 36
When making the case, outline KPIs relevant to the success of the business to avoid strategic misalignment Establishing benchmarks when making the case shows stakeholders the cloud testing strategy targets relevant and attainable business goals. There is little motivation to continue IT projects if the performance and effectiveness are not evaluated. Measure your on-going key performance indicators (KPIs) against your benchmarks established by current IT standards and business drivers. Identify the necessary criterion to exit the project. Continuously update these exit-conditions according to the changes in architecture and organizational needs. At a minimum, the KPIs should be: Relevant to your business strategy. Based on your current IT metrics, at least in part. Relevant to current and proposed testing services. Metrics that exhibit software testing and development. KPI Considerations • Number of errors and defects • Testing latencies • Provisioning virtual machines and services on-demand • Performance degradation • Server and storage utilization • Time to develop and migrate test cases • Availability of SLA • Test run-time • Testing efforts • CAPEX and OPEX • Length of application development and testing lifecycles Choose metrics and pilot projects carefully. A successful pilot (as measured by your metrics) will gain political support for continuing the cloud testing initiative. Keep the business informed while establishing KPIs. Successful KPIs understood by the business are more likely to get the cloud testing initiative approved.
Benefit Scenario: An organization evaluates the cost of cloud testing Company X, a small ecommerce business, is considering testing a new business application through a cloud testing solution. Traditional Testing Cloud Testing TEST CASE • Number of test cycles in parallel: 1 • Cycle run time: 1 hour/cycle • Number of test cycles: 3 • Cycle run time: 5 hours/cycle TEST DESIGN • Test Environment Cap.: 45 tests/cycle • Number of Scenarios to test: 45 • Test Environment Cap.: 15 tests/cycle • Number of Scenarios to test: 45 TEST/LAB CONSTRUCTION • Avg. Testing Resourcing: $41/hour • Avg. Testing Resourcing: $83/hour TEST EXECUTION • Bugs found per cycle: 210 • Development Resourcing: $50/hour • Rate of bug fixing: 70 bugs per day • Bugs found per cycle: 70 • Development Resourcing: $50/hour • Rate of bug fixing: 70 bugs per day TEST REPORTING Total Cost: $1091 Total Cost: $2295 DEVELOPMENT FIXES Source: Info-Tech Research Group Bottom Line: Company X will see a net benefit of over $1000 when executing a load and functional test in the Cloud. The benefit comes from the increased scalability and execution speed from cloud testing.
Build a business case on cloud testing so that benefits are clear to the business for the specific investment Cloud testing is not just about an IT tool, it directly impacts business ROI. Your business case should incorporate four key components: 1 2 3 4 Business Drivers Cost/Benefit Challenges KPIs • Analyze the hard and soft costs and benefits across several years. The business will likely be unwilling to fund the cloud testing project if monetary gains is not immediate or significant. • Understand the challenges of cloud testing from a business perspective. These challenges should be prioritized to determine whether cloud testing should be used for greenfield development or across the board. Establish an evaluation scheme to monitor the performance of the cloud testing initiatives using KPIs. Business owners can then determine if cloud testing is risky. Identify the functionalities and capabilities the business can obtain from cloud testing. Business drivers are the foundations of a cloud testing strategy. [Business drivers] always starts from speed and urgency of the business problem… but then it eventually comes back to the old standard within the enterprises of ‘we would like greater control’. - Tom Lounibos, President & CEO, SOASTA
Case Study: Formalize testing strategies based on business drivers to increase buy-in and improve planning A client of an IT consulting company is prepared to migrate to automated testing. • Situation • Current Testing Process Drivers to Cloud Testing The testing process is based on an incremental model, utilizing a change board process. Their change board process is a series of building, analysis, and approval activities between development and testing teams. Test cases are manually executed in a step-by-step process, utilizing in-house test scripts. Their test scripts act as a pass/fail checklist, outlining how to conduct the test and what to look for in a successful test. Their detailed test scripts provide frameworks to structure and implement automated processes. Cloud testing could help automate the manual documentation and testing process. With a limited budget, maintaining testing quality would be difficult since implementing extensive regression and load tests would be limited. They would like to maintain application quality when certain regulations and standards are amended without incurring additional costs. CSC offers a range of innovative business solutions to help solve complex industry challenges. Their application portfolio includes web applications. Some applications are located on hardware, i.e. client-based. For one of their clients they used a standardized and documented manual testing process, and a formal software development life cycle process. Test cases are put into version control and are manually updated when changes are made. Profile IT and Professional Consulting Nicole Victor Test Manager CSC www.csc.com
Use Info-Tech’s Cloud Testing Project Planning and Monitoring Templateto create the project test plan Describe the objectives and scope of your testing initiatives. Identify the functions and features being tested. Specify the major activities, techniques, and tools which are used to test. Identify the major testing tasks and estimate the time required for each task. This tool will help have a clear idea of testing dependencies. Propose your testing initiatives to IT managers by outlining your testing methodologies, resource provisions, and testing schedule. Info-Tech Insight
Implement Cloud Testing Build a Business Case Implement Cloud Testing Focus on the Metrics Rollout cloud testing in a well defined process: Select a cloud testing vendor. Obtain proper trainings and support. Conduct a pilot. Measure results and KPI. Increase footprint. Standardize cloud testing.
Implement cloud testing in a controlled manner and avoid retooling chaos Follow a well defined rollout process that keeps business objectives in mind while optimizing for cloud testing. 1 2 3 4 5 6 Select the cloud testing vendor Obtain proper training and support Conduct a pilot Measure results and KPIs Increase footprint Standardize cloud testing Choose the right cloud testing vendor that will provide the right level of support and expertise to help you get your cloud testing initiative off the ground. • As with any new product, training and support are essential. Build time into the development cycle for retooling and moving down the learning curve. • Select a pilot project to start. Involve the right people from the development team who can work with the vendor to iron out any issues. Collect relevant testing metrics that show business value for the cloud testing investment. Gradually roll out cloud testing to include more applications. Prioritize based on urgency and importance. Document and implement cloud testing as the normative path for web and cloud applications development. Introducing a new variable, like cloud testing, injects risk into the development mix which the containment counter measure is rooted in calculated steps along the journey. Follow this icon to navigate your way through this section. 1
Select the right cloud testing vendor to avoid development debt during testing 1 Some key characteristics to look for in a cloud test vendor: Horizontal breadth – Ability to offer both a cloud testing product as well as a testing service. In some cases, it can be more efficient to farm out an entire suite of test cases rather than have an internal team execute and record results. Vertical depth – A full service model that includes extensive experience with application testing, migration of testing scripts, and training for new users of cloud testing. On the service side, this includes a well documented process for notification around outages, upgrades, and bug fixes. On the deployment side, this includes the ability to test web and cloud applications that include public Cloud, private Cloud, and hybrid Cloud. Easy to use – Service is easy to migrate into for testers used to popular on-premise testing suites. Allows testers to increase productivity through simple point and click decisions and the ability to generate real time reports. Global footprint – Ability to spin up test clients globally to properly conduct load and stress test from any region. Work with your cloud testing vendor to realign your current testing process to incorporate their tool efficiently and avoid costly delays. Horizontal breadth Vertical depth Global footprint Easy to use A cloud vendor must be aligned with your business needs and requirements to ensure a sustainable partnership. A wrong partner can lead to missteps that increase technical debt and force IT to deploy quickly in the future without proper testing. It may be necessary to prioritize the importance of each variable both in the short and long term. Vendors in this space include: Soasta, LoadStorm, and 360logica.
Use a scoring criteria and select the right cloud testing vendor to avoid getting caught in overselling 1 Consider your short, mid, and long term strategies to determine how the importance of scoring variables may change over time. Horizontal breadth Do they offer black box testing service? Do they offer cloud testing training and tool specific training? Do they offer services to improve customer tests? Do they offer integration services for various cloud deployment options? Do they offer advice on how to save expenses on cloud testing? Vertical depth • Do they have extensive experience with app testing? • Is there a process for notification around outages, upgrades, and bug fixes? • Will they help you migrate your test cases to their platform? • Does the tool integrate with current development processes? • Is their tool continually being enhanced? Your scoring criteria may change over time as you evolve with cloud testing. Therefore, it is imperative to revisit your scoring criteria regularly to ensure vendor fit with business goals. Global footprint • Are test endpoints available globally? • Is global support available to help troubleshoot testing discrepancies? • Can testing be scheduled across various regions at different times? • Does the tool accommodate multiple languages? • Can each region within regional testing be scaled independently? Ease of use • Is the vendor support quick and timely? • Is the tool interface simple to understand? • Does reporting enable export functionality? • Does the tool allow notifications to be sent automatically? • Is the tool scalable on demand?
Case: An organization is looking for a cloud testing vendor with simple tools and deep vertical experience 1 Horizontal breadth (Vendor Score: 2, Optimal: 1) Do they offer black box testing service? Do they offer cloud testing training and tool specific training? Horizontal breadth Vertical depth (Vendor Score: 5, Optimal: 5) • Do they have extensive experience with app testing? • Is there a process for notification around outages, upgrades, and bug fixes? • Will they help you migrate your test cases to their platform? • Does the tool integrate with current development processes? • Is their tool continually being enhanced? Vertical depth Global footprint Optimal Fit Easy to use A multi-layered graph can be generated by adding multiple vendors as an overlay to determine commonalities amongst the vendors. Vendor Score Global footprint (Vendor Score: 2, Optimal: 1) • Can testing be scheduled across various regions at different times? • Does the tool accommodate multiple languages? Mapping each criteria to a score and plotting the result provides insight into the relative strengths and weaknesses of the vendor. Ease of use (Vendor Score: 3, Optimal: 5) • Is the vendor support quick and timely? • Is the tool interface simple to understand? • Is the tool scalable on demand?
Obtain cloud testing training to ease the burden of retooling on test resources 2 To ensure ease of transition to cloud testing: Testing team should have access to training and support resources from the cloud testing vendor. Testing team should have access to a sandbox cloud testing environment where scripts can be generated and tested initially. The testing team does not change, just the tool changes. Allocate sufficient time in the cloud testing plan rollout for test resources to get trained prior to a committed project. Cloud Testing Support Team Cloud testing should not preclude the ability to test applications that reside in multiple cloud instances. The cloud testing support team should work closely with the organization’s testing team to ensure the necessary connectivity is in place. Cloud Testing Community Cloud Public Cloud Client Testing Team Insufficient time to settle a new tool could adversely affect the quality of the final application and increase overall costs caused by insufficient time to address workflow bugs. External to the organization Shared within community Hybrid Cloud Cloud Testing or apps being tested can reside anywhere Organizational Boundary Private Cloud Internal to the organization
Use the pilot to modify parts of the testing life cycle so that a full rollout will be more stable 3 A different way to design test scripts. Some vendors rely on traditional scripting while others use a visual scripting tool built into the Cloud testing service. Different reports particularly if existing reports are localized within a region. Cloud testing introduces global reporting. In some cases the reports may come back much faster especially if parallelization of test cases is used. In such cases, the development team will need to handle results coming back from the cloud testing team much sooner. Lab setup time which was traditionally done by IT, but now is done through a cloud self-service portal. A triage of testing which delegates some test cases to the Cloud and others to traditional methods. Expect to see some changes in workflow with the introduction of a new cloud testing tool. Possible Workflow Changes It is important to realize that the cloud testing workflow does not impact the entire application development process. Incoming and outgoing tasks for testing can remain intact. The only difference now is how the test team conducts their functional, load, and stress tests. TEST CASE TEST DESIGN TEST/LAB CONSTRUCTION Cloud testing leverages virtualized and automated techniques to minimize test runtime and reduce the risk of compromising live systems. Consider these capabilities in your pilot program. TEST EXECUTION With cloud testing, setup and teardown of test labs may require its own governance to ensure test clients are properly configured. One approach to handle this is “Infrastructure as Code”. TEST REPORTING DEVELOPMENT FIXES
Take advantage of the virtualization under a cloud testing approach to improve test results 3 Traditional testing can leverage virtualization techniques to scale the production environment. Typically, but not always, a non-virtualizable system component in a cloud infrastructure will bottleneck the testing throughput. The scalability factor is based on the least scalable component. • Consider virtualizing the service rather than the actual testing component to eliminate the bottleneck. Include the capabilities of cloud testing with your design of a virtual testing environment, such as: • Scale up and down test environments by a large factor. • Rapidly provision resources. • Pooling of testing assets and services. • Maintain multiple test beds based for multiple release testing. Benefits from Virtualization Through Cloud Testing Drawbacks from Virtualization Through Traditional Testing • Scalable by a large factor. Testing labs can be hosted on virtual servers off-premise, thereby eliminating the need for testing-specific hardware. • Virtualized testing assets can be accessed globally. The Cloud provides easy access to off-site users. • Scalable by a small factor. The total number of test labs is restricted by the total number of physical servers or memory. • Virtualized testing assets are only accessed locally or through unique transfer protocols. Off-site users need special access to the servers of the virtualized assets. On-Premise n virtual servers fixed virtual servers Before After
Simulate production systems through virtual data to eliminate the risk to live and inaccessible systems 3 Cloud testing enables the parallel execution of development and testing activities. Cloud environments can be virtualized images of your in-scope and out-of-scope applications and systems. • Develop an automation model to capture the relevant scenarios that are out-of-scope. • Virtual data is used to simulate the behavior of out-of-scope systems with enough realism that the in-scope system believes it is interacting with a live system. Parallel testing solutions allows teams to execute against live systems where synchronized data and functionality is available. If a component of an updated application breaks or if a new testing scenario is needed, testers can immediately switch back to virtual services. • This ability is only applicable to purely downstream live or virtual systems. Problem: Organizations may not have full access rights to the entire out-of-scope system or the input data of the in-scope system. The usual practice to alleviate this issue is stubbing. Solution: Virtualization through cloud testing accurately captures the performance of systems by working around access and privacy hurdles. Test Data Management Input Data In-Scope App. Out-of-Scope Security and regulatory concerns may prevent organizations from recording and storing live production data on third-party cloud applications, and public Clouds. Perform a risk assessment of expediting production data to external testing sites. In this case, consider developing an internal, private cloud testing strategy.
Automate tests through cloud testing to avoid exhausting systems and memory resources 3 Automation can reproduce some situations that would be virtually impossible for a person to do, such as simulating hundreds of thousands of users. However, automation under traditional testing comes with costs. Automation: • Consumes significant amounts of power from physical servers to execute. • Requires sufficient memory to support immense testing data from scalable tests. Cloud testing incorporates automation, self-service, and cost accounting (metering) features into their management stacks. • Infrastructure and platforms are virtual and off-premise. Immense computing power is not a requirement. • Testing data is stored virtually; thereby, eliminating the need to provision large quantities of memory. Cloud testing can run through numerous scenarios in a fraction of the time it would take to manually run the same tests using less resources. Cloud Testing Solution Virtual Server/User Generator Automated Iterations Repeated Requests … 1 2 3 … n Automated tests can be shared between your test group and developers, and can be triggered to run automatically every time new or changed code is checked in. If the test fails, the developer can be notified automatically and avoid involving your testing resources. For more information about automation, refer to Info-Tech’s solution set, Assess The Appropriateness of Automated Functional Testing. Repeated Simulations In-Scope Applications
Measure KPI results from the pilot to test effectiveness of a full rollout 4 Keep business strategy in focus during the pilot phase. There are several metrics for measurement that are possible for cloud testing. Faster test execution Info-Tech Insight It is not always possible to see improvement in all KPI metrics. Often there is a trade-off involved. The best way to determine whether cloud testing should move forward for the business is to prioritize the KPI metrics and focus cloud testing on the most important metrics. Cloud Testing Reporting Lower testing cost Increased automation ROI Reports from cloud testing can help determine the following metrics: • Test execution times • Test throughput • Global results Low cost of migration Decrease development time More accurate testing Testing Lifecycle Observation from the modified testing workflow can help determine the following metrics: • Cost of migration • Resource allocation for testing • Development time • Degree of automation
Take a gradual approach for migrating into cloud testing to allow time for processes to settle 5 Move testing to the Cloud over time to allow benefits of elasticity and virtualization to integrate. The benefits of a gradual rollout include: Controlled migration into cloud testing. The ability to repurpose existing scripts and to set up test templates in the Cloud for future reuse. Conformance with business priorities around application development thereby not wasting resources on those applications that will be phased out. Allowing modified testing workflow time to settle. End goal Measured improvement in testing Additional pilots Pilot 2 The risk of a gradual rollout include: • Distributed applications may be difficult to partition thereby requiring a “big bang” transition to cloud testing. • Benefits will be deferred for some applications potentially leading to higher opportunity costs of not moving to cloud testing. • Test teams will have to use two different testing tools for functional, load, and stress testing in the interim. Process settlement Pilot 1 As the rollouts continue, the maturity of cloud testing skills should increase paving the way for cloud testing as the standard for functional, load, and stress testing.
Implement cloud testing as the standard to avoid relapse into traditional testing 6 Ensure cloud testing meets business needs consistently throughout the organization to maintain business buy-in. Cloud testing has now been adopted across various teams and departments in the organizations. However, the implementation and monitoring strategies may be inconsistent within the organization. Consolidate the multiple cloud testing approaches into one focused strategy. A formal, standardized strategy will achieve several key reusability and accessibility benefits. These benefits will reduce the cost of re-provisioning resources and creating new testing assets: • Testing assets, such as virtual data, are easily accessible through the Cloud. • Multiple teams can obtain, use, and amend test cases in the Cloud. • Virtualized and simulated systems can be shared. • Consistent compliance across departments. • Standardized communication approaches with vendors. Cloud Testing Standardization Standard Practice Cloud Testing Implementation Monitor the executions of each test. Cloud testing vendors structure their performance metrics based on common factors in an industry, which may or may not be representative of your organization’s standards. Develop a performance model based on your expectations and compare your results with your vendor’s. The next section of this solution set will help you identify and utilize key performance metrics. Initial Adoption Inconsistent Testing Approaches
Case Study: Standardize traditional testing practices to minimize risks in transitioning to the Cloud “Partner with a vendor and deliver a proof-of-concept, then decide to buy something after that experience.” – Lead Testing Analyst • Situation • Current Testing Process • Cloud Testing Deployment Each team and department team use a different testing and development model, such as waterfall and agile. They use whichever model works best for that particular group. A testing center solution is used for their testing needs. They also use the vendor’s services to mimic on-premise services and use it as a comparison with tests conducted off-premise. Internal application tests are executed in-house. They use an internal toolset from the testing center for performance testing. All test scripts are easily portable to the cloud testing vendor because the Cloud and internal toolsets are similar. Testing scripts are written internally on their hardware and transferred to the cloud vendor site. This vendor gives them complete control of running the tests. The vendor leases them the resources to execute the test internally. There is no need to reach out to external testing solutions for production environment tests, due to risk management issues. This company strives to be the leading provider of electricity, natural gas, and related services across the United States. They are a high end, high security facility which restricts the use of live data for external testing or public cloud deployment. About half of their system is virtualized in the Windows space and about a third is virtualized in the Linux space. They operate their virtualized environment as a private Cloud. Profile Lead Testing Analyst Testing Analyst Group Power Generation Industry
Focus on the Metrics Build a Business Case Implement Cloud Testing Focus on the Metrics Ensure cloud vendor complies with testing requirements Amend applications according to the results of large testing datasets Evaluate the performance of cloud vendors Be aware of the changing cloud testing landscape
Evaluate the effectiveness of cloud testing over the limitations of traditional testing Cloud testing excels in two areas: scalability and rapidity. Load/stress tests and functional tests for web and cloud apps are ideal candidates to benefit from these features. Focus your metrics around scale and speed.
Ensure cloud vendors continuously comply with your standards and testing requirements Regular meetings with cloud vendors are your opportunity to address known and unknown issues as they emerge. Measure vendor response times. You are giving a critical part of your software development life cycle to a third-party testing solution provider. As such, you want to make sure that the vendor has a clear picture of your testing needs. Facilitate continuous communication between your organization and your cloud testing vendor throughout the execution of testing initiatives. Understand that dealing with consequences of failures beyond your control is the trade-off you make when leveraging testing-as-a-service (TaaS) and other cloud services. Cloud testing has shifted the roles and responsibilities of the testing and development teams due to its added capabilities As a result, operational, technical, installation, and support touch points with business stakeholders and cloud vendors have changed from traditional practices. Determine the locations of the touch point changes, and identify the new contacts at the various stages of cloud testing. Info-Tech Insight • Make sure the vendor provides notification and transparency into service levels and disruptions. • Make sure there is a point person responsible for receiving updates and notifications for each service (and delegating that responsibility as needed). • Have clear plans and processes in place so information gets from the point person to those who need it. Some Cloud Testing Vendor Relationship Factors • Satisfaction • Response times • Knowledge gaps • Resolution rate • Support documentation For additional information of cloud vendor management practices refer to Info-Tech’s Effectively Manage Cloud Vendors.
Adapt test reporting to anticipate larger data sets and realize how this benefits the overall application quality Debug, Fix, and Update Your Application • Realize that testing is now an analytics project. Testers receive numeric test results due to the black-box nature of testing practices, whether done locally or in the Cloud. • Due to the growth of integrated applications and systems, the importance of load and system compatibility significantly increases. • Cloud testing produces large testing datasets that have a higher degree of usage accuracy, and identifies more bugs than traditional testing. This translates to less testing and maintenance cycles, and fewer bug updates; thereby, reducing costs and improving application quality. • Document and measure any changes to your application and test cases in a catalog or library for future reference. Adjust test reporting to include the diverse testing scenarios from Cloud testing. Repeat Your Tests, If Necessary Design and Develop Implement • Testing is not a purely downstream workflow. If the testing needs are not satisfied in the first iteration of the testing cycle, then the application should be updated with the latest fixes, test cases redesigned, testing scripts rewritten, and the tests executed again. The [application] issues aren’t always in the design of the app. or the functionality of the app.; it’s how that app. or that workload relates or correlates to an infrastructure, a network, and millions of users around the world. - Tom Lounibos, President & CEO, SOASTA
Measure performance of cloud testing to calculate benefits for ROI Evaluate the overall performance of your cloud testing solution and establish benchmarks. Some performance benchmarks and benefits include: Volume of tests capable of running with cloud testing versus without cloud testing. Larger volume shortens testing time translating to a reduction of resource cost. Number of maintenance release cycles required using cloud testing versus without cloud testing. Fewer maintenance release cycles translates to faster time to market on subsequent product versions while reducing testing costs for each maintenance release iteration. Speed of testing cycle when using cloud testing versus without cloud testing. Parallelizing translates to shorter testing cycles translating to a reduction of resource cost. Number of memory leaks (through long running tests) found using cloud testing versus without cloud testing. Shortening long running tests translates to reduction in maintenance cycles and improved productivity of end users. Resources required to run functional tests using cloud testing versus without cloud testing. Shorter testing cycle times translates to fewer resources required over the testing lifecycle. Info-Tech Insight 25% Resource Having a clear and consistent testing strategy is key to managing ROI. Those areas important to the business should be measured to determine the benefit derived. 22% Volume 15% Time 5% ROI 30% ROI 35% ROI 30% ROI Source: Info-Tech Research Group
Be aware of cloud testing limitations so that you play on its strengths and avoid disillusionment Cloud testing is green and expanding. Prepare for the following four factors to stay ahead in the game.
Summary Build a Business Case Focus on the Metrics • Formal cloud testing strategies consider business drivers, and key business and IT stakeholders. Engaging in a formal cloud testing strategy and involving the business when aligning business goals with cloud testing values increases the success rate. • When making the case for cloud testing include a list of business drivers that support the cloud testing strategy, the benefit and costs, your organization’s unique characteristics, and the KPIs that will be used to gauge success. • Assess your cloud testing readiness by taking an inventory of the people, process, and technology your organization has access to. Take note of the hurdles and strategies to overcome them before beginning your strategy. • Evaluate the impact of cloud testing over traditional testing load/stress and functional tests. These metrics should be focused on scale and speed. • Realize the changes in the touch points from cloud testing and measure cloud vendor performance accordingly. Implement Cloud Testing • Implement cloud testing solutions in a well-defined rollout process to ensure business objectives are realized and cloud testing initiatives are optimized. • The rollout process consists of six steps: select a cloud testing vendor, obtain proper training and support, conduct a pilot, measure results and KPIs, increase footprint, and standardize cloud testing.
