Testing and quality assurance I. Quality control and web site development

1. Testing and quality assurance I. Quality control and web site development • The importance of testing • Quality assurance II. Testing for quality assurance • Phases of testing • Types of testing III. The test plan

2. Testing and quality assurance I. Quality control and web site development • The importance of testing Testing is a process of quality control and assurance It is intended to improve the site by assessing how it works under a variety of conditions Platforms, browsers,OS, monitors, digital devices, bandwidth It ensures that the site does what it is supposed to do for the intended audience Usability and accessibility Navigation, search and retrieval

3. It is critical to gain client buy-in for testing You must be ready with a clear justification for the time and costs of testing Questions you should be ready to answer What are the costs and how will testing save money? How many people will be involved and how much time will it take? How will it improve the site? What is the client’s role in testing? This should be determined in advance The process should be clear and accounted for in the budget

4. Risks associated with testing Slippage Not allotting enough time for testing or taking time away because of problems elsewhere Coverage: how much of the original plan did you cover? Bug fixes People have to be assigned to this and time has to be set aside Typically this is at the end when people are tired and swamped Metrics These are unclear to testers

5. More risks associated with testing Outdated specifications Makes benchmarking difficult for testers Poorly trained testers Should have familiarity with the site and with the error management procedures Unclear assessment of the targeted users Leads to inappropriate testing scenarios Component, module, application system integration Testing should be taking place in a relatively stable environment

6. Quality assurance What is quality? Being able to understand and use a system to achieve desired goals and recover efficiently and gracefully from problem situations Meeting requirements and conforming to target market needs and expectations This means the requirements must be known to be valid This means those needs and expectations are seen to be realistic The delivery of a product that satisfies user expectations of functionality, performance, and security at previously agreed upon costs and timetables

7. Meanings of quality Descriptive: being able to understand and utilize a system to achieve desired goals Also, recovering efficiently and gracefully from problem situations Prescriptive: being able to meet the requirements negotiated with the client The requirements must be known to be valid It also means conforming to target market needs and expectations Those needs and expectations are known to be realistic)

8. Objective quality The degree of compliance of a process/product or its outcome with a predetermined set of criteria These criteria are presumed essential to the ultimate value the process or product provides Subjective quality The level of perceived value reported by the person(s) who benefits from a process/product or its outcome Quality is the delivery of a product that Satisfies user expectations of functionality, performance, and security The site is delivered at previously agreed upon costs and in a timely manner

9. Quality involves building applications and formulating methodologies that add value to the business and customers It is a valuation perceived by organizations, customers, and users This is based on how a product or service meets their collective and individual needs Information quality is a strategic resource for an organization and supports “managing with facts” It is an ongoing process of building and sustaining relationships by assessing, anticipating, and fulfilling stated and implied needs It encompasses business concerns (i.e., the producers and customers) and the concerns of the end-user as well

10. There is a distinction between business needs and market needs This mirrors the distinction between a “customer” and a “user” A customer will be focused on the product meeting certain objectives and including a certain amount of functionality A user will focus on issues such as ease-of-use and overall operability How does the user perceive the quality of the product? The view the customer has of the product is the view that the organization must have The bottom-line of the organization is dependent upon its view of the customer or user

11. The main goal of quality is to enable competitive advantage and business effectiveness It involves building applications and processes that add value to the business and customers Quality is a tradeoff between time and money The goal is to make the site as good as it can be given the constraints of the project The site meets the stated objectives These are laid out in the scope document It is delivered on time an budget Quality can be empirically measured This requires standards, standard methods, and accepted processes

12. Quality involves the creation and maintenance of relationships between the business and its customers In a practical sense you are trying to make quality visible Showing tangible aspects of quality by focusing on how customer experiences with a site impact the bottom line The tangible result of quality is the degree to which the stability and market viability of a product Increases the probability of outcomes desired by the customer Reduces the probability of undesired outcomes for the organization

13. Customers judge the quality according to stated and implied needs and expectations and the availability of alternatives The site should be Reliable: ease of use Responsive: allows efficient use Easier to use than the alternatives Compared to the competition

14. What is quality assurance? It is a function that identifies, documents, and reviews for improvement the processes that deliver products Systematic actions that constantly optimize productivity, communication, and value within an organization Measures attributes, properties, and characteristics of a product Nyman, J. (2002). What is Quality Assurance? http://www.globaltester.com/sp2.html Another view of QA (ISO 9001) What you need to do, to demonstrate that your product or service will satisfy your customers

15. The “practice of quality assurance” is auditing, monitoring, and testing How it functions is the means by which you do that auditing, monitoring, and testing Testing is so much a part of quality assurance that it is really inseparable Context of testing Your company is in the position of making some sort of product That product will have various projects placed against it Either to create new functionality to the product or to modify existing functionality

16. The quality focus, overall, is looking at the product lifecycle The means by which it is produced Requirements documents Design specifications, source code, executable binaries, etc. At each phase of the lifecycle, QA is going to be doing some sort of testing Testing documents (to check for things like consistency) Testing code itself (in the form of code reviews or inspections) Testing the direct execution of code without a front-end (unit testing)

17. QA testing Testing the full system, after all units have been integrated and work with a front-end (system testing) Testing to be sure that new functionality did not affect existing functionality and is not, itself, broken (regression testing) QA is also auditing and monitoring What QA is doing Establishing certain processes: defect tracking solution or a standard change control process Audit that which you establish in order to make sure that what you are doing is effective

18. What is a QA system? The organizational structure, the processes and procedures necessary to ensure that the overall intentions and direction of an organization as regards quality are met The quality of the product or service is assured ISO 9001 Quality Assurance. (2002). Home Page. http://www.iso9001qualityassurance.com A well developed ISO 9001 QA system should: Improve your product and service quality Give customers confidence that their needs will be met Standardize your business by giving it a consistent approach to its operations Improve work processes, efficiencies, morale and reduce waste

19. Testing and quality assurance I. Quality control and web site development • The importance of testing • Quality assurance II. Testing for quality assurance • Phases of testing • Types of testing III. The test plan

20. II. Testing for quality assurance • What is testing? Creating manual or automated sequences that are structured, methodical, and repeatable These methods are designed to probe suspected product or system weaknesses and to find defects This provides an indication of product reliability The degree to which the product fits the specifications Response of client and customer How well the product meets their needs Both are necessary for QA A reliable product that satisfies stakeholders and users

21. Testing involves two distinct phases Before the milestone This is verification testing The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase (IEEE/ANSI) Given the requirements, are we doing the right thing? Testing up to this point is necessary to determine which among a range of alternatives to select As new content and functionalities are added, they should be tested This is iterative or developmental testing and is intended to point out bugs

22. Phases of testing After the milestone This is validation testing The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements Given the requirements, did we do what we said we were going to do? No new content or functionality is accepted Testing after this milestone is to determine the usability and quality assurance (QA) of the site that will be launched Goal is to uncover operational or functionality problems

23. Sample metrics These determine the ideal test environment and should resemble the typical or majority user environment Browser/OS configurations Hardware requirements Software requirements Memory size requirements File and application size Bandwidth and connection speed Response times for download

24. Sample testers They don’t (and should not) care about what’s inside the black box They should just be using the site The test is based on the site specifications Tests reflect the functional requirements set out in the scope document Testers are given a representative range of scenarios This is based on typical uses of the site Testers use valid and invalid inputs They should intentionally try to break the system This provides information about error handling

25. Eight validation axioms 1. Testing can be used to show the presence of errors, but never their absence 2. One of the most difficult problems in testing is knowing when to stop 3. Avoid unplanned, non-reusable, throwaway test cases 4. A necessary part of a test case is a definition of the expected output or result 5. A comparison should be made of the actual versus the expected result 6. Test cases must be written for invalid and unexpected, as well as valid and expected, input conditions Nyman, J. (2002). What is Quality Assurance? http://www.globaltester.com/sp2_a6.html

26. 5. Invalid input Invalid input is a condition outside the set of valid conditions and should be diagnosed as such by the program being tested 6. Cases must be written to generate desired output Do not think just in terms of inputs Determine the input required to generate a pre-designed set of outputs 7. A program should not be tested (except in unit and integration phases) by the person or organization that developed it 8. The number of undiscovered errors is directly proportional to the number of errors already discovered

27. Types of testing User acceptance/usability testing To determine whether the site meets the specified requirements Whether the intended audience can use the site Useful when the pre-production prototype is finished (pre-milestone) Also when launch version is ready (post-milestone) Can use a variety of methods Observation, accompanied browsing Heuristic evaluation, card sorting, focus groups, peer review

28. Functional testing Going through the entire site to ensure that everything works as it is supposed to Compatibility test: testing configurations as described in the project document’s technical specifications # of browsers x # of operating systems x # of screen resolutions = the # of tests to be performed Start with the configuration that is likely to cause the most problems (and use Netscape) Try to break it (unusual characters in forms) Regression test: retest with the most common configuration after finishing the changes

29. Functional testing Compatibility test Elements include: Links, forms, navigation, download times, readability, content management, transactions, push functions, personalization, printing Error management What is your process for logging errors and problems uncovered in testing? How much detail should be recorded? Testers and fixers need access to this system so they they can record problems and fixes in real time

30. Functional testing Error management Resolution of the error or problem should also be logged The resolver log: incident number, date, logged by, priority, category, description (with error message), date work began, owner, comments, status, date finished Priority: 1 Fatal (crash or freeze) 2 Serious (works but not the way it should) 3 Significant (can be fixed, interferes with user experience) 4 Minor (typos…) Tester should then try to recreate the error after it has been resolved

31. Operational testing Testing the back end of the site Also the service components involved in the site Focus: delivery times, shipping accuracy, customer service, data integrity Goal is to find the problem before the customer or user does Stress/load testing Simulate possible scenarios of use Testing the stability of the programming and the capacity of the hardware Can be done with scripting that simulates multiple HTTP requests

32. Security testing Test the stability of the firewall and intrusion detection software Check the protection of sensitive data (especially passwords and credit card information) This will probably require outside expertise Proofreading Read every page carefully and make sure there are no typos and the grammar is correct Check the contact information and verify the content

33. Testing and quality assurance I. Quality control and web site development • The importance of testing • Quality assurance II. Testing for quality assurance • Phases of testing • Types of testing III. The test plan

34. III. The test plan This is a document that outline the range of tests that will be conducted throughout the project lifecycle It specifies types of risks that can be expected and the test that will be used to assess these risks It outlines procedures, timeline, and costs It defines the acceptable metrics that will be used These can be in the form of benchmarks How many seconds will it take for the home page to download? A secondary page? How long will that average search return take?

35. Structure of the test plan Scope What kinds of tests will be conducted? What types of test environments will be used? Who are the testers? What are the main metrics that will be used Types and severity of bugs Functional: problems with programming Usability: problems with navigation Creative: differences in color rendering across platforms Severity: use a simple ranking system

36. Structure of the test plan Version control Explaining the system in place Risks and assumptions What is the range of expected problems that could compromise testing? What resources are needed in order to carry out the testing? Testing schedule and cycles When they occur, how long they take, and when and how often they are repeated

37. Structure of the test plan Test cases Include the scenarios used as cases Bug reports Listing of all the bug found in testing Description of how they were handled QA summary General observations Discussion of major problems and strategies for resolving them The test plan is an important addition to the project archives