Ensuring Application Reliability and Scalability Through Effective Test Automation

1. Ensuring Application Reliability and Scalability Through Effective Test Automation By Denice Wong Technical Consultant – IBM Rational Hong Kong

2. Insufficient/Changing requirements Waterfall process Poor Unit testing Limited time for testing Poor Performance testing Inaccurate tests Late detection of errors Functional testing blocked Unable to perform full regression testing Inaccurate performance prediction on production environment Challenges Associated With Testing Effects Causes

3. a An Example: Challenges of Siebel Implementations “Customers are spending close to $½ Million on testing complex projects”1 • Surveyed 130+ projects • Results: • Typical customer spends ~25% of project costs on testing • Testing costs vary widely depending on project type and complexity 1 Source: Siebel Test Automation Datasheet

4. Do you ever have enough time to test everything??? Trade coverage for risk • Reduce scope of testing • Reduce testing cycles • Reduce configuration testing Add people to the project Introduce / enhance automation

5. Automation Adds Options! Improve human productivity • Broader test coverage Reduce redundancy • Automate repetitive tasks Enable process improvements • Increase regression testing • Expose risk areas • Tie testing to requirements

6. What Does Test Automation Mean? Do my testing for me!

7. What is Automated FunctionalTesting? Record Enhance Execute • Using automation tools to record actions taken against an application. • Test scripts are derived from these recordings which are then used for subsequent test execution.

8. What Is Automated Performance Testing? • The process of exercising an application by emulating actual users with a load generation tool for the purpose of finding system bottlenecks Controller System Under Test

9. What is Automated Test Management? Plan • Using automation tools to accelerate the process of test planning, test creation, test execution and test results analysis Analyze Create Execute

10. Functional Test

11. Challenges of Functional Testing “Our testers are not experienced in building automated tests.” • Controlling cost of test automation • Inexperienced testers face steep learning curve for tools • Test maintenance outpaces testing “Application UI changes are always breaking our test scripts.”

12. Challenges of Functional Testing “Our script language limits our ability to overcome some automation obstacles.” • Control cost of test automation • Inexperienced testers face steep learning curve for tools • Test maintenance grows with project • Testing Tools Lack Flexibility • Power users hands are tied by tool limitations “I need one tool for both my expert and novice testers.”

13. IBM Rational Functional Tester Automated regression testing for Web, J2EE and .Net applications Key Benefits Minimize test maintenance with scripts resilient to application changes Wizard enhanced automation to speed test creation for the new user Powerful scripting language and IDE for the professional tester Supports Team oriented parallel development Functional Testing with IBM Rational Functional TesterTest automation for the novice and the professional

14. Effective Test AutomationCreating an automated test is a three steps process Record Enhance Execute • Script Recording • Capture user interactions with the system under test • Script Enhancing • Modify scripts for situation specific challenges • Script Execution • Ensure reliable playback for local or remote execution

15. Recording ScriptsData Driven Testing • Data Driven Testing • Separates test data from test script • Enables a single script to run multiple tests by using multiple data sets • Wizard driven process • No programming involved • Import data from external sources

16. Recording ScriptsValidating Dynamic Data • Dynamic Data/Content Matching • Use pattern matching technique to verify dynamic data and create robust tests • E.g. Instead of validating “Order ID 230”, validate “Order ID ###” or Order ID 2##, etc. • This allows for a wide variety of acceptable responses as well as restrictions on acceptable responses when validating the application’s behavior Test Passes when comparing pattern baseline to dynamic data Test Fails when comparing static baseline to dynamic data Order ID changes with each order placed

17. Enhancing ScriptsPowerful, Professional Debugger • Functional Tester offers two development environments • Eclipse based IBM Software Development Platform • Visual Studio .net • Both environments offering powerful debugging features • Code assist editors • Step debugging • Variable watches • More…

18. Reduce Test Script MaintenanceObject Map • The object map is a collection of descriptions of test objects • It is automatically populated during test script recording • It contains properties for each test object • Recognition properties enable Rational XDE Tester to find the test object during playback • Changes to an object need only be made once – in the map Central Object Map Frame Panel Test Scripts EditBox CheckBox Button

19. TesterSees ToolSees DeterminesMatch Reduce Test Script MaintenanceReliable Playback with Script Assure Version 1.0 Version 2.0 No User Intervention Required With ScriptAssureTM Score = 75

20. Summarizing Key Features in IBM Functional Tester • Broad Environment Support • Web, Java. .net • Create data driven tests without coding • Static data and properties verification • Dynamic data validation without coding • What can’t or shouldn’t be automated? • Flexible and Extensible Scripts • Flexible coding language • Powerful, professional debugger • Object map editing flexibility • Version control of scripts • Script Execution Options • Reliable playback with ScriptAssure • Remote and local playback on various platforms

21. Performance Test

22. Challenges of Performance Testing • Complex Tools & Complex Task • Tool complexity and lack of experience intimidates many first time users “We don’t have a lot of experience building good load tests.” “Most load test tools are much more complicated than we require.”

23. Challenges of Performance Testing • Complex Tools & Complex Task • Tool complexity and lack of experience intimidates many first time users • Testing Tools Lack Flexibility • Power users hands are tied by tool limitations “We don’t have a good understanding of what exactly is occurring during a load test.” “We have highly technical performance testers that need to customize their tests.”

24. Challenges of Performance Testing • Complex Tools & Complex Task • Tool complexity and lack of experience intimidates many first time users • Testing Tools Lack Flexibility • Power users hands are tied by tool limitations • Hidden Costs • Hardware intensive tools and inability to share information with team mates increases cost of testing “Our developers have no way of performing load tests.” “We don’t have enough hardware to run our tests.”

25. IBM Rational Performance Tester Performance Testing for Web Based Applications Key Benefits Built for Day 1 Productivity Mask complexity to get the job done Advanced Data Access & Manipulation Automated data variation and synchronization Simulate real world workload model Lower Cost of Performance Testing Low resource requirements Desktop IDE integration Performance Testing with IBM Rational Performance TesterTest automation for the novice and the professional

26. Creating a Performance TestCreating a performance test is a three step process Build Scripts Execute & Analyze Schedule Workload • Script Creation Considerations • Visual test editor, varying input data & correlating server responses • Scheduling Considerations • Accurately representing a true user workload • Execute and Analyze Considerations • Validating responses & finding the bottleneck

27. Tests are represented in a tree view as a list of web pages and their elements, such as images. No programming necessary to create a test Underlying detail is accessible but hidden to simplify test creation and maintenance No-Code Tests

28. Creating a Performance TestBuilding Scripts • Automated creation of parameterized tests • Literal values used during recording are replaced with variables • At playback time, variables are populated with data from a datapool • Data for playback can be imported from external data sources • No user coding involved Session No. 100 User Name: JSmith Session No. 101 User Name: CBryson Session No. 102 User Name: TJones

29. LoadSmart scheduling Easy to use • Model complexreal world scenarios • Point and clickscheduling

30. Creating a Performance TestSchedule Workload • Powerful and flexible scheduling • Visual schedule editor for no code scheduling • Coordinate timing and dependencies • Accurately model real users workloads • Dynamically increase the load during the run

31. Find Slow Pages • Easily determine slow performing pages of your web site • Reports are available live during the run and for post run analysis

32. Find Resource Bottlenecks • Simultaneously monitor resource utilization during the test • Capture and analyze a long list of performance monitoring statistics • Determine your resource bottlenecks

33. c b Identify the Root Cause of Performance Problems IBM Performance Optimization Toolkit for Rational Performance Tester 1 2 Pinpoint performance problems in the test lab... Speed changes by providing detailed data to developers System under test Tester a UserLoad Developer Run Performance Test to Emulate Multiple Users • Isolate elusive performance problems in complex, multi-tier environments • Speed delivery of quality software Isolate Performance Bottlenecks and Submit Change Request AnalyzeApplication Response Time

34. Summarizing Key Features in Performance Tester • Built for Day 1 Productivity • Create code free tests • Mask complexity to get the job done • Advanced Data Access & Manipulation • Create code free schedules • Automated data variation and synchronization • Create real world workload model • Lower Cost of Performance Testing • Low resource requirements • Desktop IDE integration

35. Test Management

36. Why Should I Test? • Associating Test Inputs defines Why you are testing • Establishes a link between the test and the motivator • Motivator can be RequisitePro Requirements, Rose Model, Excel Spreadsheet, or custom • Establishing link enables requirements coverage reporting Test Input

37. Suspect Test Cases Within TestManager: • Suspect Test Cases show up with red slashes in the Test Inputs View AND • Suspect Test Cases show up in bold in the Test Plan View

38. Automatically Track Impact of Changing Requirements Automatic Notification of Changed Requirements that Affect Test Cases

39. Test Case Script Requirement Comprehensive Change/Defects Management

40. Test Case (Test Manager) Control Your ProcessTrack Test Thoroughness Requirements (ReqPro) Defect Tracking (ClearQuest)

41. Control Your ProcessTrack Test Thoroughness

42. Summarizing Key Features in TestManager • Speed your Test Cycles • TestManager controls, tracks and maintains all test assets • Simulate Real World Workload • Create real world workload model to make sure the test result is meaningful • Effortless Analysis • Intuitive reporting provides results analysis for accurate and up-to-date test reports

43. The IBM Rational Software Development PlatformA complete, open, modular, and proven solution for Software Quality Assurance Analyst Architect Developer Tester Deployment Manager Rational Functional & ManualTester Rational Application Developer Tivoli Configuration Manager RequisitePro • Rational Software Modeler Rational Software Architect Rational Performance Tester TivoliMonitoring Rational Web Developer CustomerExtensions 3rd PartyISV Tools ECLIPSE RequisitePro, ClearQuest, ClearCase, Test Manager Project Manager Rational Portfolio Manager Executive

44. Thank You What’s Next? • Rapid Web Application Deployment Seminar (26th May, 2006) • Please fill in the feedback form and get IBM Rational evaluation CD kit !!