An Update on Automatic Regression Testing at Diamond

1. An Update on Automatic Regression Testing at Diamond An Update on Automatic Regression Testing at Diamond

2. An Update on Automatic Regression Testing at Diamond Contents • Motivation • Update on the Testing Framework at Diamond • Hardware Regression Test • EPICS Base Regression Test

3. An Update on Automatic Regression Testing at Diamond Introduction • Diamond is now operating for ~4500 hours per year which will increase to ~5500 hours per year over the next two years • Shutdown time is being reduced, and so are becoming increasingly busy • 2010 and 2011 see installation “Mini Beta Optics” in two long straight sections necessitate replacing pairs of SR girders • All this results in less time to test new software releases • Drive to improve operational reliability • The Control System has not had significant number of faults • Don’t want to change this

4. An Update on Automatic Regression Testing at Diamond The Grand Objective • Test all 200 support modules • Test modules and EPICS DB templates • Test device/drivers that interface to hardware • Hardware test environment • Test the versions of EPICS base we are running • Issues of changing EPICS base and /Tool chain/OS/ version • Test software against a tool chain, build and run environment • We want to ensure that operational system use the same environment

5. An Update on Automatic Regression Testing at Diamond Good Software Engineering Practice • Four major parts to software engineering • Design: Should come first! • Code: We are good at this. • Documentation: Doxygen etc. • Testing: Often manual and hard to repeat. • Testing is not an optional extra • Can be carried out by the Software Developer • Or can be carried out by the Software User • Or can be automated • Tests must be properly repeated • At least before every release and preferably more often • Tests are easier to repeat if they are completely automatic

6. An Update on Automatic Regression Testing at Diamond Test Framework • A software framework that aids the creation of automatic test suites for modules. • Based on PyUnit, the Python unit testing library. • Test reports conform to the TAP protocol and/or JUnit XML format. • Monitoring of EPICS database records for coverage reporting. • Hardware and simulation targets can use same test suite.

7. An Update on Automatic Regression Testing at Diamond The Run Script • Searches a directory tree for modules with automatic test support. • Runs tests suites it finds, logging results and output. • Can run multiple tests in parallel subject to resource constraints. • Level of diagnostic output can be specified. • Tests run can be restricted to single module, single target, single case. • “dls-run-tests -h” for command help. • The run script is designed to operate from the ‘support’ directory of the work tree.

8. An Update on Automatic Regression Testing at Diamond Directory Structure Changes support <moduleName> new directories etc test tests.py makeDocumentation makeIocs iocs

9. An Update on Automatic Regression Testing at Diamond Device Simulations • Based on existing DLS practice. • Written in Python. • Currently capable of simulating any device that communicates through serial or IP connection. • Support for instrumentation to allow protocol coverage reporting.

10. An Update on Automatic Regression Testing at Diamond Framework Class Diagram

11. An Update on Automatic Regression Testing at Diamond Example – The Test Suite from pkg_resources import require require('dls.autotestframework') from dls.autotestframework import * class Fw102TestSuite(TestSuite): def createTests(self): # Define the targets for this test suite Target("simulation", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot", epicsDbFiles="db/fw102Ex.db", simDevices=[SimDevice("controller1", 9016)], guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) Target("hardware", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex.boot", epicsDbFiles="db/fw102Ex.db", guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) # The tests CaseLocalIncrementSwitch(self) CaseLocalDecrementSwitch(self) CasePowerOffOn(self) CasePvIncrement(self) CasePvDecrement(self) CasePvMultipleIncrement(self) CasePvMultipleDecrement(self) CasePvSetPosition(self) CasePvTriggerMode(self) CasePvPolling(self)

12. An Update on Automatic Regression Testing at Diamond Example – Intermediate Class class Fw102Case(TestCase): def curDevicePos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self.command("controller1", "getpos") args = self.recvResponse("controller1", "pos", 1) if args is not None: result = int(args[0]) return result def verifyPosition(self, intended): ''' Verify that the wheel is in the intended position ''' if self.simulationDevicePresent("controller1"): self.verify(self.curDevicePos(), intended) self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended) self.verifyPv("FGZ73762:WHEEL1:POSITION", intended) self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)

13. An Update on Automatic Regression Testing at Diamond Example – A Test Case class CaseLocalIncrementSwitch(Fw102Case): def runTest(self): '''The local increment switch.''' if self.simulationDevicePresent("controller1"): # Check the current position of the wheel before = self.initialPosition() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self.command("controller1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self.sleep(2) self.diagnostic("Before=%d, after=%d" % (before, after), 1) self.verifyPosition(after) before = after

14. An Update on Automatic Regression Testing at Diamond Example - Test Report [1] 1..10 [fgz73762@pc0054 diamondtest]$ ./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102 [1] ok 1 - The local increment switch. [1] ok 2 - The local decrement switch. [1] ok 3 - Power off and on. [1] ok 4 - The PV increment command. [1] ok 5 - The PV decrement command. [1] ok 6 - Fast increment sync correction. [1] ok 7 - Fast decrement sync correction. [1] ok 8 - Movement directly to a position. [1] ok 9 - Control of the trigger mode. [1] ok 10 - Control of the polling mechanism. [1] # ============================== [1] # Passed 10/10 tests, 100.00% okay, in 260.57s [1] # [1] # ============================== [1] # Sim device controller1 coverage report: [1] # setpos: ok [1] # settrigger: ok [1] # getpos: ok [1] # gettrigger: ok [1] # [1] # ============================== [1] # EPICS database coverage report: [1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok [1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok [1] # FGZ73762:WHEEL1:RESTART(ai): not touched [1] # FGZ73762:WHEEL1:INPOS(calc): ok [1] # FGZ73762:WHEEL1:REINIT1(fanout): ok [1] # FGZ73762:WHEEL1:POSITION(mbbo): ok [1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok [1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok [1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok [1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok [1] # FGZ73762:WHEEL1:STEPBK(calcout): ok

15. An Update on Automatic Regression Testing at Diamond • Success (all actions successful) • Unstable (some tests failed) • Failed (compile error) Hudson • Use Hudson framework • For scheduling and reporting of test • A Continuous Integration Engine • Checkout from repository • Does some actions • Build, Post-build actions, (execute test script) • Determine Status • Invoked by period checks on SVN repository • Status by web and emails to developers

16. An Update on Automatic Regression Testing at Diamond

17. An Update on Automatic Regression Testing at Diamond Hardware Regression Test • Test IOC consists of: • Prodex OMS-58 8 axis servo control card • A voltage to quadrature encoder conversion card • Hytec 8001 DIO and 8515 Octal Serial • EPICS IOC test specific db • Stimulation IOC: • 8005 – digital I/0 for limit switches, home switches • 8401 – ADC input card to monitor OMS-58 servo demand output • 8402 – DAC to drive the encoder simulation • VxWorks application implemented by cut down drivers for above cards and a simulation of some ‘realish’ hardware to control • Test harness: • Wiring to join Test IOC to Stimulation IOC.

18. An Update on Automatic Regression Testing at Diamond Hardware Test Framework Test harness Switch Terminal Server Serial Stimulation Unit (vxWorks stimulations) Test Unit (EPICS IOC) Hardware under test Digital & Analog I/O Processor Processor Power strip (IP) Linux control PC Test suite control, redirection, etc Network Hudson

19. An Update on Automatic Regression Testing at Diamond EPICS Base Regression Test Suite (1) • Came out of the Codeathon at Diamond • Nick Rees, Jon Thompson and Matt Pearson • Objective • To generate automated testing of the EPICS Base project code on a diverse range of architectures, both workstation and real-time OSs. • These will be run nightly at Diamond on head of trunk and results published on web site. • Provide as service to the collaboration • This is work in progress. Only started it last week.

20. An Update on Automatic Regression Testing at Diamond EPICS Base Regression Test Suite (2) • There are established EPICS tests, but these are run manually as part of releases • Wrap the established EPICS tests in Python to make callable from test framework. • Run test on • EPICS Base, C Unit tests • Report on test coverage for each EPICS Base test • IOCs based tests

21. An Update on Automatic Regression Testing at Diamond EPICS Base Types of tests (1) • C unit tests based on epicUnitTest.h functions. • These are mostly used to test libCom functions at this point • Generate TAP output that is interpreted by the test harness • Propose to add new make targets to describe how the tests are run on embedded architectures (give a download binary and a command line function to run) • Make generates a test specification that is used by the test framework to run the tests

22. An Update on Automatic Regression Testing at Diamond EPICS Base Types of tests (2) • IOC based system tests based on Diamond module test framework • Python framework that loads IOC under test and a stimulation system (if required) • Can get, set and monitor PV’s, and/or control or monitor the IOC command line • Tests reside in a new etc/test directory in the application tree • Have a wide range of verification commands. • Many of the EPICS “soft-test” suite tests were converted to this framework during the Codeathon

23. An Update on Automatic Regression Testing at Diamond EPICS Base Platforms • Plan is to run this for popular development platforms and target architectures. • What about historic development platforms, Solaris, and latest platforms say Fedora XX or Ubuntu XX.YY and other target architectures ?

24. An Update on Automatic Regression Testing at Diamond EPICS Base Test Results • Results will be published as a web page containing: • Results of ‘make runtests’ on the target architectures • Results of running the ‘soft-tests’ on the target architectures • Code coverage report for the Linux host. • URL to be announced

25. An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

26. An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

27. An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

28. An Update on Automatic Regression Testing at Diamond Acknowledgement • This is being lead by Jon Thomson at Diamond • It’s the work of many of the Controls Group at Diamond