Towards Further Automation of the Q uality A ssurance Cycle

1. Towards Further Automation of the Quality Assurance Cycle Alan Berg: University of Amsterdam Central Computer Services (IC)‏ Group Education and Research Services

2. But first an advert QA’ ing makes you sexy

3. QA or the Sakaiger gets it

4. QA’ing helps you get the toy

5. We have only 30 minutes, so only a brief biased personal snapshot is possible

6. Agenda The potential for a QA event horizon What does the QA WG want? Current hints to the developer Where are we The Sash Tool Jameleon Conclusions

7. The QA event horizonRisk losing a deterministic QA cycle http://amazing-space.stsci.edu/resources/explorations/blackholes/teacher/grabbag.html

8. To remain deterministic we will need more automation

9. The QA event horizonRisk losing a deterministic QA cycle The Obvious Code base increasing rapidly Not just Java, but also XML, Javascript, Python,CSS and many Java frameworks Iterative process that requires defined response times Requires manpower/resources on time Would love QA as part of Continuous Integration

10. What does QA WG want? A "stack" of open source tools for load testing and nightly integration testing A load-testing repository and "cookbook" for open source tools Regression testing work flow QA Process - no tool makes it into the core (stable) set of tests QA Process - no bug in JIRA is closed without an automated test that checks for that bug. http://bugs.sakaiproject.org/confluence/display/QA/QA+Test+Automation

11. Hints to the developer

12. Supporting developers Nightly builds – Continuum [Claret] Javadocs - [nightly] Static code reviews – [UvA] Internationalisation - [LOI ] Unit testing [ developers ] – Potentially through Continuum -Mock objects from Josha Holtzman Knowledge capture [Confluence, community, conferences]

13. Hinting nightly http://qa1-nl.sakaiproject.org

14. Random Example

15. Duplicate Code

16. Do we wish to use the reports more aggressively? Are the developers taking note? Bugs reported are not always correct Bugs found not always important But duplicate code, unit test coverage, and failing to deal properly with exceptions hint strongly at quality. Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.

17. The Sash Tool

18. The Sash Tool AWSOME http://qa1-nl.sakaiproject.org:8080/

19. Realistic Proof of Concept Provision tool add/delete users,sites,members,tools,permissions upload files from file system to dropbox/resources Controlled by a property file in site resources liveTest Diagnose Report But have we enough API’s to do the job?

20. SASH TOOL (Steven Giffens)‏ Live Brilliant for learning Environment that targets sys_admin Easy to extend commands Consistent use of cover api's allow services to be easily guessed. Use to write provision, diagnose, live test tools.Just requires imagination and Sakai specific knowledge.

21. LIVE import org.sakaiproject.tool.cover.ToolManager; import org.sakaiproject.tool.api.Tool; import org.sakaiproject.component.cover.ComponentManager; import org.sakaiproject.content.api.ContentHostingService; ident="...."; all = ToolManager.findTools(null, null); counter=1; for ( Iterator i = all.iterator(); i.hasNext(); ) { tool= i.next(); print("["+counter+"] "+tool.getTitle()); print("ID : "+tool.getId()); print(ident+tool.getDescription()); print (ident+ident); counter++; } counter=1; print ("Interfaces"); inters = ComponentManager.getRegisteredInterfaces(); for ( Iterator i = inters.iterator(); i.hasNext(); ) { inter= i.next(); print("[: "+counter+"]"+inter); counter++; } Uses covers Consistent use of java.utils Trunk 2.4....Tools=104,Interfaces=568 live

22. Cover like ... import org.sakaiproject.test.cover.TestManager; import org.sakaiproject.testl.api.Test; all = TestManager.findTests(TestManager.RUNALL); counter=1; for ( Iterator i = all.iterator(); i.hasNext(); ) { test= i.next(); print("["+counter+"] "+test.getTitle()); print(test.getDescription()); try{ print(test.doTest()); }catch(FailedTestException e){ print(“Error was: “+e.getMessage()); }finally{ counter++; } }

23. Remember QA Process - no tool makes it into the core (stable) set of tests BRAINSTORM ONLY • Push tests back to the developer • Runs against a live system • Can be removed at compile time for production distributions • Non standard [Yeuch] • Consistent, and test driven • Easy to script through Sash • Only solves a certain significant class of tests

24. Jameleon http://jameleon.sourceforge.net/

25. Regression testing workflow CRITERIA Run automatically Results consistent Many roads to Rome. Selenium RC Selenium RC wrapper in Maven, Python, Ant Cost benefit ratio favourable

26. Jameleon - Proof of Concept Wrapper around Selenium RC and other tools Junit,Jiffie (Internet Explorer,HttpUnit,generic)‏ Configured via XML Generates reports Can run headless via cron or continuum Data driven Almost self documenting

27. Data driven Makes tests server generic Property file per organization uva.properties name=admin pass=admin base=qa1-nl.sakaiproject.org:8380 csv with variables defined in first line eid newbiex newbie2

28. Self documenting http://jameleon.sourceforge.net/

29. XML test cases <?xml version = '1.0'?> <testcase xmlns="jelly:jameleon"> <test-case-summary >Add user to SakaiQA</test-case-summary> <test-case-author >Alan Berg</test-case-author> <test-case-level >SMOKE</test-case-level> <functional-point-tested >Add User to Sakai ${baseUrl}</functional-point-tested> <selenium-session startSeleniumProxy="true" application="uva" beginSession="true" seleniumMultiWindowMode="true" seleniumStopProxyServerOnSessionClose="true"> <csv name="add_user_sakai" countRow="true"> <selenium-open functionId="Goto Portal" url="http://${base}/portal/tool/!gateway-710?panel=Main" /> <selenium-type keys="${eid}_FIRST" functionId="Type Firstname " locator="name=first-name" /> <selenium-type keys="${eid}" functionId="Type username ${eid}" locator="name=eid" /> <selenium-type keys="${eid}" functionId="Type password" locator="name=pw" /> <selenium-type keys="${eid}" functionId="Type verify" locator="name=pw0" /> <selenium-type keys="${eid}@dev.null" functionId="Type mail" locator="name=email" /> <selenium-type keys="${eid}_LAST" functionId="Type Lastname" locator="name=last-name" /> <selenium-click functionId="Click submit" locator="eventSubmit_doSave" functionDelay="1000" /> <selenium-select-window functionId="Select relevant frame" windowName="null" /> <selenium-click xmlns="jelly:jameleon" functionId="Click submit" locator="link=Logout" /> </csv> </selenium-session> </testcase>

30. Workflow From Cron Download and build Sakai Copy different scripts to directory Run Jameleon (can run against IE, Firefox, Opera etc)‏ Generate reports Move reports to website Can be added as a script to run to continuum

31. Results Works Selenium plug-in still beta Tests are fragile and require maintenance Will require a significant effort for basic coverage Still very promising and should be considered. HATS OFF TO THE DEVELOPERS

32. My plans I have one day a week free to: Concentrate on Sash scripts Wait until the next release of Jameleon. Beg the big brains to build a realistic test manager.

33. Questions ? ? ? ? ?