Agile Testing

1. Agile Testing

2. Introduction to Agile Testing The definition of agile testing can be described as follows: “Testing practice for projects using agile technologies, considering development as per-requisite of testing and introducing a test-first design methodology. In agile development, testing process is being integrated throughout the project life-cycle, testing the software throughout its development.” This Agile is a methodology which is seeing increasingly widespread, and it is easy to understand, significantly if you consider the developer's and user's point of view. Users: They are not interested to spend much of their time wondering in detail about the exact requirement and processes to be considered for the entire system, following the review a huge specification, which they know could come back to puzzle them. Developers: They are not interested to have to follow a lined specification, without any expression of their own contribution towards creative talents, specifically if they can see a better alternative of processing the work.

3. Agile Testing Defined • Agile testing is a software testing practice that follows the principles of Agile software development. Agile development integrates testing into the development process, rather than keeping it as a separate and distinct SDLC phase. • Agile testing involves a cross-functional Agile team actively relying on the special expertise contributed by testers. • Agile teams use a “whole-team” approach to “bake in quality” to the software product. Because testing occurs in real time, this approach allows the team to collaborate actively with the development team, helping them identify and transfer issues into executable specifications that guide coding. • Before the start of testing, the development team’s detailed design documents must be approved so that test case creation can start. • Both coding and testing are performed incrementally and iteratively in Sprints (or iterations), building each feature until it delivers enough stability and adds quality to the product .

4. Common Testing Challenges on an Agile Team • Agile projects have specific problems for testers which would go beyond a lack of comprehensive documentation, and making the planning tedious. The process of testing in the agile environment is not a complex work when planned, but still there are certain common issues suffered. • Environment's Complexity • Nth iteration Testing • Performance Bottlenecks • Lack of focused testing • Ownership of test automation

5. Testing Challenges - Environment's Complexity • In most of the agile teams, there would certainly be a continuous integration and deployment. But still, continuous deployment also indicates that environment is always in a constant pace. • As a tester, it is mandatory to be having a capacity to test in a stable environment whenever required. This is however sometime drains effort. • Especially, when there is a need to promote code to staging and production, the level of confidence high in promoting the version tested. • Solving a problem always becomes a problem because every application is different and environment often has many stakeholders.

6. Testing Challenges - Nth iteration Testing • The more is delivered; there is more probability to get things wrong in an unexpected way. Scripts get thickened, and an inadequate test set of single iteration may be a poor selection in the upcoming sprints. • Fixes might introduce unexpected issues which would lead to strategic problems which results to prone to turn into unpleasant expectations. • It is also usual to see bugs when a fix is backed out since it causes unintended issues which are equivalent to, or still worse than the initial ones. • After a couple of iterations, there is a least probability to test from scratch, but the complexity of the system would have reached a prominent stage where no part of the system is genuinely immune from new emerging issues/bugs. • There might be a thought to return to code freezes, waterfall approaches and testers as bug stoppers, but this agile project has already become clumsy.

7. Testing Challenges - Performance Bottlenecks • As software becomes gets more mature, the complexity of the software naturally goes up. • This complexity increases the lines of code which would introduce performance issues if the developer is not aware on how the changes would impact the end-user performance. • In order to solve this issue, one must first know what areas of the code are causing performance issues and how performance is being impacted over time.

8. Testing Challenges - Lack of focused testing • Testing in agile often becomes little complex and therein resulting it lack of focus on testing. There are a lot of things which require some time to the testers. • Some of them is to concentrate and be prepared for testing in the later part for the future sprint with the aid of Business Analysts, discuss difficulties of the ongoing sprint with the developing team along with the Business Analysts, to ensure that the cons on the build-on CI, to also cooperate with product owner for a demo, to help the developers in extracting the test data, environment set-up or unit testing, integration test, review unit, make sure that non-functional testing is been satisfied whenever needed. • The test automation, support prod release, test in different environments and so on also to be noted in order to have an clean agile environment. This might often leave very little room for the actual testing process planned.

9. Testing Challenges - Ownership of test automation • Test automation is most important process of agile teams. There are a lot of successful and also unsuccessful automation testing efforts. • The biggest factor in the success or failure of automation effort is the ownership of testing. If test automation is being owned by a team, it has good options of adding value. It’ is really been a tough process to promote this way of work and of course it takes some time. • Mostly, the testing team in the environment do not want any chance of the testing to go wrong. They'll want to make sure that they would abandon control and still on the other side, the developers in the team, might be a little reluctant in taking additional responsibility. • So, it's a little difficult task to all the teams work accordingly, but the point to be noted is when team owns test automation of the environment, the faster would be the feedback which in turn adds more and more value.

10. Few more Changes in Agile Testing • Start of a Sprint is typically the start of Testing. • Test estimation is provided as Sprint tasks on a Sprint Planning meeting. • The product owner, Scrum master, developer and tester form the Scrum team ( usually a team of 5-7) • Entire scrum team is responsible for the quality and the result of a deliverable • A daily stand up meeting within the Scrum team spanning for 15 - 30 minutes to provide the status of each task assigned to team for the Sprint. • Maximum communication within the Scrum team. • Provide continuous feedback.

11. Pros of being Agile • Less risk of compressed test period • Test all the time, not just at the end. • Work together as one team towards common goal. • Defects are Easy to Fix • Fewer Changes Between Fixes • Flexibly Incorporate New Requirements • Early and Predictable Delivery. • Agile testing saves time and money • Less documentation when compared to waterfall modal • Regular feedback from the stake holders. • Daily meetings help in addressing the issues and getting the resolution in faster node.

12. Cons of being Agile • Agile testing proves to be the best testing methodology only if the requirements are clear to the project sponsors. • If the requirements frequently change (as allowed for in Agile), the following scenarios can occur: • Difficulty in leaving the traditional understanding of the roles • The team struggles to adapt to changes because significant effort has already gone into the initial requirements development and testing process. • Challenges in estimations and sizing requirements. Sometimes QA gets short shrift since it’s logically the last task in completing the user Story. Therefore, any delay in the prior development task risks impacting QA timelines. • QA is sometimes prevented from executing a test case for the whole iteration, leaving the team struggling to finish the task. • Not asking the right questions. It is very dangerous for QA not to ask questions, especially at the point where the user Story is picked up for implementation. Daily team meetings can avoid this problem. • Addition of new user Stories into the current iteration. QA should be included in the addition of the new user Story, to build up appropriate commitments and estimations in order to avoid misalignment and protracted timeframes

13. Roles of tester in SCRUM • Participate in Release/Sprint Planning • Support developers in Unit Testing – say them about testing view instead of development view • Test user story when completed, testers are last gate to confirm user completeness • Collaborate with customer and Product Owner to define acceptance criteria • Develop automation testing

14. Types of Testing

15. Types of Testing Topics Covered in this lesson are:

16. System Testing • Complete, integrated system is tested to evaluate it’s ability to satisfy the specified system requirements. • Broadly, system is evaluated for functional and non-functional requirements. • Non-functional tests include performance, security, compatibility, volume, load, stress.

17. Functional Testing • Functional tests are based on the functions, described in documents or understood by testers and may be performed at all test levels. • E.g. test for a component may be based on component specification. • Testing functionality can be done from two perspectives • Requirement based perspective • Business process based perspective • Specification-based techniques may be used to derive test conditions and test cases from the functionality of the software or system. • Experience based techniques can also be used. • Functional testing considers the external behavior of the software. • Test condition and test cases are derived from the functionality of the component or system.

18. IntegrationTesting • Focus on verifying the component interfaces when the units are combined to function as a single sub-system. • Goal is to see if the modules can be integrated properly. • Test activity can be considered as testing the design. • Intermediate level of testing • One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently fails in significant and mysterious ways, and it is difficult to fix it • Progressively unit tested software components are integrated and tested until the software works as a whole • Test that evaluate the interaction and consistency of interacting components.

19. AcceptanceTesting • Participate in Release/Sprint Planning • Is the process of comparing a program to its requirements • Testing the system with the intent of confirming readiness of the product and customer acceptance. • The performance and reliability of the system will be tested and confirmed. • Formal testing conducted to enable a user, customer or other authorized entity to determine whether to accept a system or component. • An acceptance test is a test that the user defines, to tell whether the system as a whole works the way the user expects.

20. RegressionTesting • Regression testing tests for changes. • Typically a regression test compares new output from a program against old output run using the same data. • Regression testing tests for changes. • Typically a regression test compares new output from a program against old output run using the same data.

21. PerformanceTesting • Determine what performance factors are most critical to this application and what degree. • Throughput • Response Time • Handling Peak-loads • Determining a good strategy to evaluate the performance capability

22. Load and Stress Testing • Load Testing • Evaluate the system’s capacity to handle large amounts of data during short time periods. • Stress Testing • Evaluate system’s capacity to handle large numbers of processing transactions during peak periods

23. Load and Stress Testing (Cont…) • Eliminates surprises and ensures that a System will be able to perform under load. • Without testing, it's impossible for to know what might happen when a site has to function under a heavy load. • Will response time degrade slowly or drop off precipitously? • At what point might a Web site crash completely? • To conduct Load Testing on a web-site : • Apply stress to a web-site by simulating real users and real activity • Monitor response time as load is increased. • Perform Capacity Testing to determine the maximum load a Web site can handle before failing • Capacity testing reveals a system ultimate limit.

24. SecurityTesting • Protected at different levels based on the security requirements in the organization • Attempts to verify that protection mechanisms built into the systems will protect it from improper penetration • The tester play the roles of the individual who desires to penetrate the system. • Acquire passwords through external clerical means. • Overwhelm the system, thereby denying the service to others • Purposely cause system errors, hoping to penetrate • Browse through insecure data, hoping to find the key to system entry • The good security testing will ultimately penetrate the system.

25. RecoveryTesting • Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly performed. • Automatic - data recovery, check pointing mechanisms and restart are evaluated for correctness. • Human Intervention - mean time to repair is evaluated to determine whether is within acceptable limits.

26. CompatibilityTesting • Testing a Web site, or Web-delivered application for compatibility with a range of the leading browsers and desktop hardware platforms • Testing an application on a range of different operating systems, and in combination with other applications • Verifying the compatibility and interoperability of a set of enterprise-class applications in a complex hardware and software environment designed to meet your specific business needs • Testing a peripheral with a wide range of PCs • Testing a server with different add-in cards, applications, and operating systems

27. ScalabilityTesting • Scalability testing differs from simple load testing in that it focuses on the performance of your Web sites, hardware and software products, and internal applications at all the stages from minimum to maximum load.

28. UsabilityTesting • “Usability is a quality attribute that assesses how easy user interfaces are to use. The word 'usability' also refers to methods for improving ease-of-use during the design process”. • “Usability is the measure of the quality of a user's experience when interacting with a product or system”

29. MutationTesting • Mutation Testing is based upon seeding the implementation with a fault (Mutating it)‏ • Also called as Error Seeding • Method used to find effectiveness of testing • When tester finds a certain percentage of seeded defects, it is an indication of the effectiveness • Determine a testing technique that identifies this fault. • Mutated program is called as a mutant. • If the testing technique (test case) identifies this mutant, it is said to kill the mutant • The idea behind this approach is such that if the test case find these small differences, it is likely to be good at finding real faults.

30. SmokeTesting • Verify the major functionality of the software without concern for finer details. • Non-exhaustive software testing, ascertaining that the most crucial functions of a program work. • In a retailing application, core processes of Order Entry, Credit Card processing and Invoicing could be tested quickly.

31. SanityTesting • Performed when cursory testing is sufficient to prove the application is functioning. • Normally includes a set of core tests of basic GUI functionality, connectivity to the database, application servers, printers. • Used prior to release as final inspection and after applying minor patches.

32. Ad hoc Testing • Testing done without any defined process. • No specified rules required. • It’s up to the tester to test the software of his/her own. • Tested by “end-users” with no knowledge of formal testing or the specific product. • Will generate new test scenarios which will exercise the product along new code paths.

33. BetaTesting • Carried out on a software product at the user’s site, in the absence of the development team. • About 400,000 beta test sites for testing Windows 95 Operating System. • Enables the development team to fix several defects in a short time, before the official launch of a product.

34. GorillaTesting • Gorilla Testing is used to test the GUI of the software package. • Assume that the software is given to the Gorilla, what does it do? • It presses the key randomly! • While we are testing GUI (particularly for game software), this type of testing is very useful.

35. StructuralTesting • Testing of Software Structure / Architecture (Structural Testing) • A third target of testing is the structure of the system or component. • Structural testing assumes that the procedural design is known to us. • Test cases are designed which can test internal logic of the program. • Disadvantages: • Does not ensure that user’s requirements are met • Does not establish if the decisions / conditions / paths / statements are insufficient

36. Testing Related to Changes • Confirmation /Retesting : • It is the process of re-executing all the failed test cases to check if the development team has really fixed the defect or not. • Regression Testing : • It is the re-execution of few or all the test cases to check: 1) Any addition to the software 2) Any deletion to the software 3) Any updating or fixing of defect does not affect the functionality of unchanged modules.

37. Globalization/ Localization Testing • Globalization testing checks proper functionality of the product with any of the culture / locale settings using every type of international input possible. • The localization process also includes translating any help content associated with the application. • Most localization teams use specialized tools that aid in the localization process by recycling translations of recurring text and resizing application UI elements to accommodate Localized text and Graphics. • Localization Testing check the quality of a product’s Localization for a particular target culture / locale.

38. BenchmarkingTesting • Benchmarking involves activities that compare an organization product with similar market leading one • The comparison is based on the following factors. • Performance Parameters • User Friendliness • Richness of Features

39. MaintenanceTesting • Why maintain software? • Model of reality: As the reality changes, the software must either adapt or die. • Pressures from satisfied users, to extend the functionality of the product. • Software is much easier to change than hardware. As a result, changes are often made to the software whenever possible. • Successful software ,survives ,well beyond the lifetime of the environment for which it was written.

40. MaintenanceTesting (Cont.…) • Types of Maintenance Testing • Corrective maintenance is maintenance performed to correct faults in hardware or software [IEEE 1990] • Determining how the existing system may be affected by changes is called impact analysis, and is used to help decide how much regression testing to do. • Adaptive maintenance is software maintenance performed to make a computer program usable in a changed environment [IEEE 1990]. • Perfective maintenance is software maintenance performed to improve the performance, maintainability, or other attributes of a computer program. [IEEE 1990] • Preventive maintenance is maintenance preformed for the purpose of preventing problems before they occur [IEEE 1990]

41. Black-box Testing

42. Test Case Design • Beyond the psychological and economic aspects of testing, the most important aspect in software testing is the design of effective test cases. • Complete testing is impossible and a test of any program must be necessarily incomplete. • Obvious strategy is to try to reduce this incompleteness as much as possible.

43. Design Strategy • Given the constraints on time, cost etc • Key Question is … • What subset of all possible test cases has the highest probability of detecting the most errors?

44. What is a test case? • A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement. • Main Approaches • Two main approaches to designing test cases are • Black-box testing • White-box testing

45. Black-box testing • Tester views the program as a Black-box. • Tester is completely unconcerned about the internal behaviour or structure of the program. • Tester is only interested in finding circumstances in which the program does not behave according to its specifications. • Test data are solely derived from the specifications.

46. Testing • As shown in the figure Black Box Testing is the process of Executing the system/ program by giving the input and checking the output of the system/ program. • We will not look inside the structure of the program.

47. Why Black Box Testing….? • Purpose of Black Box Testing is to cause failures in order to make the faults visible. • Another purpose is to assess the overall quality level of the code.

48. Why the name Black Box…? • Software tester does not have access to the source code. • Code is considered to be a “Black Box” to the tester who cannot see inside the box. • Based on the specifications, the tester knows what to outcomes to expect from the black box.

49. Black Box Testing • Black Box Testing is called as Functionality Testing or Data-Driven Testing or Closed Box Testing. • A System or Components whose inputs, outputs and general functions are known, but whose contents or implementation are unknown for Testing. • Tester only knows the inputs and what the expected outcomes should be and not how the program arrives at those outputs. • Tester does not ever examine the programming code and does not need any further knowledge of the program other than its specifications.

50. Testing Strategies / Techniques • Black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function. • Data outside of the specified input range should be tested to check the robustness of  the program. • Boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output. • Number ZERO should be tested when numerical data is to be input. • Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems. • Crash testing should be performed to see what it takes to bring the system down.