Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

1. Strategic Direction for Functional Test Automation SoftTest 2005 Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD Title slide

2. Future for Functional Testing History of Functional Automation Strategic Direction Automation Frameworks in Action Impact upon You? AGENDA

3. Future for Functional Testing

4. Increased Demand………..? • Global development, with large projects having multi site, multi geography and multi suppliers to contend with. • Corporate and regulatory requirements growing all the time • Business demands new applications, faster and cheaper to obtain competitive advantage • Closer alignment of IT and business to repair lack of business confidence

5. ..Or, increased pressures? • Automation frameworks and greater business knowledge requirements, mean reduced numbers of traditional functional testers • Agile development methods mean developers undertake more unit and component testing • Growth of outsourced testing to different geographies, leading to greater competition for the roles Is the role of the functional tester (who is neither technical or business specialist) dead?

6. History of Functional Automation

7. Record and Playback • “all you need to do…….…!” • Easy to record the scripts • Extremely fragile • Expensive to maintain • On average, each test run requires at least 50% of the scripts to be recorded again Please tell me no one does this anymore!!!

8. Scripting • Use the scripting language to write scripts that do what you want • Build in as much robustness as possible • But, you’re building an application to test an application, which also needs testing! • Maintenance costs can be very high • Needs programmer skills • Cost of implementation not affordable at project level

9. Table Driven • Like scripting but more flexible and greater re-use • Remove items from the script that change, e.g. data • Reduces maintenance costs a bit • Implementation costs about the same • Still requires specialist skills to implement

10. Functional Test Automation is Broken! • Focus on technology rather than business needs • 80% of functional testing still manual • 60% to 70% of automation tools used for non-functional testing • Typically, traditional functional automation stops at 100 scripts, regardless of test coverage requirement • Critical factors; cost of implementation and maintenance, skills required and asset sharing over different technologies

11. Strategic Direction

12. Automation Frameworks • Wouldn’t it be good if…….. • Tests could be documented in common format, regardless of whether they are manual or automated • The format for the tests resulted in quicker preparation time than traditional manual tests • Both developers and business testers could understand and use the same test format • Script maintenance was no longer a requirement • Tests could use any of the test execution tools required by the underlying technology Faster planning, faster execution, and far less technical skill required

13. Business Analysts already using them, and use will grow • Home grown frameworks built within organisations to meet business demands • Niche suppliers providing frameworks; try a Google search – 512,000 hits this week • Seen a handful that appear mature • Latest review by Paul Herzlich (OVUM analyst) • Market Leaders such as Mercury developing Business Process Tester (BPT) This is the industry direction now

14. Automation Frameworks in Action

15. The project • Lloyds TSB Offshore in Channel Islands • Migration from current banking platform IBIS to Temenos Globus G13 • Tight deadlines due to EU Savings Tax Directive coming into force on 1st July 2005 • Business processes required additional Globus modules • Multiple releases leading up to deadline required substantial testing • SQS-UK contacted to provide consultancy

16. The Team • SQS-UK recognised the requirement for test automation • Joint team from Odin Technology and SQS-UK assembled • Working alongside Lloyds TSB staff • Odin Technology provided testing technology • SQS-UK provided implementation resource and testing know-how

17. Tool Evaluation Technical Implementation Support Technician Tester Test Analysis & Design Project Time Project Approach Split project roles for Automation Technicians and Testers • Three phase technical approach • Tool Evaluation & Technical Feasibility • Technical Implementation • Support • Test Analysis and Design throughout

18. Project Phase 1:Tool Evaluation and Technical Feasibility

19. Tool Evaluation • Automation isn’t always straightforward! • GUI Tools don’t work “out of the box” for all environments • They will almost certainly require some custom functionality to interact with the SUT • The Evaluation process: • Assess what custom functionality is required • Assess what involvement is required from an Automation Technician to provide the custom functionality • Assess if there are any other efficiency improvements that can be made through application of technology • This provides the scope of technical implementation phase

20. Project Phase 2: What would normally happen...

21. Typical Automation Project • Take existing Manual Scripts • Rework by hand into automated scripts & function libraries • The bulk of the work… • Building the Object Map • Providing the Mapping between the business terms used to describe the SUT Interface and the test tools requirements • Coding Test Scripts • Providing the instructions and associated data for the test tool to interact with the SUT and perform testing

22. Application Object Mapping Providing the Mapping between the business terms used to describe the SUT Interface and the test tools’ requirements

23. Example Manual Tester “The Login Window” Test Tool class: MSWDialog label: “Login.*”

24. Example 2 Manual Tester “Username” Test Tool class: HTMLEdit id: Lgn_Uid

25. Object Mapping in a typical Automation Tool • Usually implemented under different names in different tools • GUI Map • Object Repository • Object Map • Mapping an Object • Start the Object recognition tool • Point at the Object with the Mouse Pointer • Select the Object • Give the Object a meaningful business name • Make manual amendments to the properties if required

26. AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 Script 1.1 Test Script Creation Providing the instructions and associated data for the test tool to interact with the SUT and perform testing

27. Function Login(ByVal UID As String, ByVal PWD As String) Dim rc rc = wait_window(“Login”,30) if rc <> 0 Then log_msg(“Window Not Found”, FALSE) Return End if focus_window(“Login”) click_on_text (“UserName”) type(UID & “{Return}”) click_on_text (“UserName”) type(PWD & “{Return}”) End Function Scripting Approach Use the Tools Scripting Language to construct business processes

28. Object Maps Reworking of Manual Scripts Script 1.1 Script 1.1 Script 1.1 Script 1.1 Script 1.1 Application AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Test Automation Tool System Under Test

29. Project Phase 2: What was done differently… • Taught an old dog some new tricks…

30. Test Design Model A simple structured way of defining tests that is independent of the interface and the means of execution

31. Features of the Test Design Model • Easy to design tests without technical knowledge • Well structured • Can be used to describe tests for any interface • Graphical User Interfaces (GUI) • Non-UI Components (WebServices, APIs etc.) • Independent of Execution Mechanism • Manual testing • Automated testing • Not theory - proven 7 year history of implementations • In use in UK and Europe at many organisations for defining testing

32. Sub-tests Test steps A single test step Object Action Test Data Test Design Model

33. Step Enter Username “jsmith” Sub-test Login as user 1 Test Login – enter user credentials Product search – enter product ID Product details – validate product info Logout Test Model – GUI Example

34. A test step • Object + Action + Data • Action • SET enter data, navigation • GET retrieve data • VAL compare expected/actual data

35. How tests are designed • Using Microsoft Excel • Business processes are modelled • Sequences of steps and data are built up in Sub-Test Tables • Sequences of Sub-Tests model business processes

36. Interpreting the Model • There are products that can process the model • To generate Test Automation Scripts • For UI Testing using GUI Automation tools • For Non-UI Testing using Harnesses • To generate documentation for manual testing

37. Manual Scripts Non-UI Automated Scripts Automated Scripts Interpreting the Model

38. Globus A Self Descriptive Application

39. Application What is a Self-Descriptive Application? An application that, through an automated process, can provide a description of its interfaces

40. Globus Desktop Application Application A description of the interface for an application is stored in the database The Globus Desktop interprets the description and dynamically creates the user interface Windows and objects are presented to the user The Globus User Interface How it works

41. API Globus Desktop Application Custom Interpreter API can be invoked to provide the description without presenting it How Globus Describes its Interface

42. A Sample of Globus Information APP: CUSTOMER_INST FIELD CUST_SNAME TYPE INPUT LABEL Surname LENGTH 20 SEQ 1 FIELD CUST_GEN TYPE OPTION LABEL Gender LENGTH 1 SEQ 7

43. How this Information was used • The Description gives us 3 things: • The business names used for the objects • The properties the test tool required for object identification • The sequence of the objects in the window • This could provide us with two things automatically: • The Object Map • The sequence of steps for our TDM sub-tests

44. More on Self-Descriptive Apps • Requirement on development to “name” objects in business terms • Microsoft .NET Applications • Windows Forms and Web Applications can provide us with • Object Maps • Sub-tests for the Test Design Model • This can drastically reduce the cost of automation

45. Microsoft Excel Globus Test Design Model & Object Maps Script 1.1 Script 1.1 Script 1.1 AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Axe Automation Scripts Overall - The Test System Design

46. Project Phase 2:Implementation

47. The Implementation • Technical Implementation • 9 days of Test Tool Technician • Using Automatic Interface Description in Globus • 1800+ Objects Mapped automatically • 90 Sub-test components generated • 30-75 steps per Sub-test • Time to Generate: 42 seconds

48. What was left for the Testers? • Test Design! • Designing scenarios for testing the application • Providing Test Data • Sequencing Sub-Tests to form business processes

49. Designing the Tests • 4 Testers – Using the Test Design Model • No previous automation experience • 40 Days Elapsed time • 1012 Complex Banking Scenarios Automated

50. Raw Statistics – Test Execution • Single Script • Automated = 80secs • Manual = 480secs • Automated is 6x Faster • Full Regression Pack • Automated = 22 Hrs (1 Workstation) • = 5.5 Hrs (4 Workstations) • Manual = 132 Hrs • = 22 Man/Days (using 6hrs productivity per day) • 22 Man/Days Testing in 5.5 Hrs.