Automatic Regression Test Facility for Support Modules

1. Jon Thompson, Diamond Light Source Vancouver, 1 May 2009 Automatic Regression Test Facility for Support Modules

2. Introduction • Four major parts to software engineering: • Code: We are good at this. • Design: Should come first! • Document: Doxygen etc. • Test: Often manual and hard to repeat. • DSC and the US telephone network. • Google for: DSC communications software fault • Adding that weird feature to existing code.

3. The Lessons • Testing is not an optional extra. • Tests must be properly repeated at least before every release and preferably more often. • Tests are easier to repeat if they are completely automatic.

4. The Requirements • Testing of device support modules against a Python simulation of the device. Most recent stream device support modules have been implemented alongside a python simulation and an example Linux based IOC. • Testing of device support modules against the device hardware. Getting enough feedback from the hardware into the test cases may require extra hardware. • Hardware and simulation testing should utilise the same test scripts. Where there are differences (for example, getting feedback from the device), the test case must take the target device into account. • Tests scripts to be written in Python utilising standards as appropriate and available. • Automatic identification of support modules that have test scripts. This will require the establishment of standards regarding the location of test IOCs and test scripts within a module’s directory. • Cron job to start the test sequence at regular intervals. Ideally, starting in the evening with results available the following morning. • Test sequence to run without routine user intervention. The operation of, and feedback from, any hardware must all be automatic. This may require extra hardware to provide inputs and monitor outputs. • Daily test sequence results to be made available through a web page or email. • Provide information regarding the ‘quality’ or ‘coverage’ of test suites. • Entire test sequence to run in a ‘reasonable’ time. This may mean that test suites must be run in parallel with some sort of server computer starting tests and collecting the results.

5. What Do You Get • A software framework that allows the creation of automatic test suites for modules. • A script that runs test suites and collects reports. • A hardware test bench containing ‘high value’ equipment. • Software simulations of equipment.

6. Initial Prototype Hardware Software Motor Ethernet Switch Test IOC Simulation Newport XPS Geobrick Test Cases Linux PC Motor Run Script

7. The Framework • Based on PyUnit, the Python unit testing library. • Test reports conform to the TAP protocol. • Monitoring of EPICS database records for coverage reporting. • Hardware and simulation targets can use same test suite.

8. The Run Script • Searches a directory tree for modules with automatic test support. • Runs tests suites it finds, logging results and output. • Can run multiple tests in parallel subject to resource constraints. • Level of diagnostic output can be specified. • Tests run can be restricted to single module, single target, single case.

9. Device Simulations • Based on existing DLS practice. • Written in Python. • Currently capable of simulating any device that communicates through serial or IP connection. • Support for instrumentation to allow protocol coverage reporting.

10. Directory Structure • Test suites must exist in a standard place in a module so that the run script can find them. The support module’s ‘TOP’ <moduleName> dls test tests.py

11. Framework

12. Usage

13. Example – The Test Suite • class Fw102TestSuite(TestSuite): • def createTests(self): • # Define the targets for this test suite • Target("simulation", self, • iocDirectory="example", • iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot", • epicsDbFiles="db/fw102Ex.db", • simDevices=[SimDevice("controller1", 9016)], • guiCmds=['edmMain -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl'])‏ • Target("hardware", self, • iocDirectory="example", • iocBootCmd="./bin/linux-x86/stfw102Ex.boot", • epicsDbFiles="db/fw102Ex.db", • guiCmds=['edmMain -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl'])‏ • # The tests • CaseLocalIncrementSwitch(self)‏ • CaseLocalDecrementSwitch(self)‏ • CasePowerOffOn(self)‏ • CasePvIncrement(self)‏ • CasePvDecrement(self)‏ • CasePvMultipleIncrement(self)‏ • CasePvMultipleDecrement(self)‏ • CasePvSetPosition(self)‏ • CasePvTriggerMode(self)‏ • CasePvPolling(self)‏

14. Example – Intermediate Class • class Fw102Case(TestCase): • def curDevicePos(self): • ''' Get the current wheel position from the device simulation''' • result = 0 • self.command("controller1", "getpos")‏ • args = self.recvResponse("controller1", "pos", 1)‏ • if args is not None: • result = int(args[0])‏ • return result • def verifyPosition(self, intended): • ''' Verify that the wheel is in the intended position''' • if self.simulationDevicePresent("controller1"): • self.verify(self.curDevicePos(), intended)‏ • self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended)‏ • self.verifyPv("FGZ73762:WHEEL1:POSITION", intended)‏ • self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)‏ • def initialPosition(self): • ''' Initialise for a test case, return current position''' • ...

15. Example – A Test Case • class CasePvIncrement(Fw102Case): • def runTest(self): • '''The PV increment command.''' • # Check the current position of the wheel • before = self.initialPosition()‏ • # Take the wheel round twice • for i in range(12): • # Now advance the wheel using channel access • self.putPv("FGZ73762:WHEEL1:STEPFW.PROC", 1)‏ • after = before + 1 • if after > 6: • after = 1 • # Check the final position of the wheel • self.sleep(2)‏ • self.diagnostic("Before=%d, after=%d" % (before, after), 1)‏ • self.verifyPosition(after)‏ • before = after

16. Example - Simulation • class fw102(serial_device): • Terminator = "\r" • regexp = re.compile(r'\s*(\w*)([\=\?])(\w*)')‏ • def __init__(self): • '''Constructor. Remember to call the base class constructor.''' • serial_device.__init__(self, • protocolBranches = ["setpos", "settrigger", "getpos", "gettrigger"])‏ • self.position = 1 • self.trigger = 0 • def command(self, text): • '''Interface function for commands from the test suite.''' • args = text.split()‏ • if args[0] == "incr": • self.position = self.position + 1 • if self.position > 6: • self.position = 1 • elif args[0] == "decr": • self.position = self.position - 1 • if self.position < 1: • self.position = 6 • elif args[0] == "getpos": • self.response("pos %d" % self.position)‏ • else: • serial_device.command(self, text)‏ • def initialise(self): • '''Called by the framework when the power is switched on.''' • self.trigger = 0

17. Example - Simulation • def reply(self, command): • '''This function must be defined. It is called by the serial_sim system • whenever an asyn command is send down the line. Must return a string • with a response to the command or None.''' • result = None • # Parse the command • m = self.regexp.match(command)‏ • if m == None: • return result • m = m.groups()‏ • cmd = m[0] • set = m[1] == '=' • value = m[2] • # Decode and act on the command • if self.isPowerOn(): • if set: • if cmd == "pos": • self.position = int(value)‏ • result = "pos=%d" % self.position • elif cmd == "trig": • self.trigger = int(value)‏ • result = "trig=%d" % self.trigger • else: • if cmd == "pos": • result = "pos?\r%d" % self.position • elif cmd == "trig": • result = "trig?\r%d" % self.trigger • return result

18. Example - Test Report • [fgz73762@pc0054 diamondtest]$ ./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102 • [1] 1..10 • [1] ok 1 - The local increment switch. • [1] ok 2 - The local decrement switch. • [1] ok 3 - Power off and on. • [1] ok 4 - The PV increment command. • [1] ok 5 - The PV decrement command. • [1] ok 6 - Fast increment sync correction. • [1] ok 7 - Fast decrement sync correction. • [1] ok 8 - Movement directly to a position. • [1] ok 9 - Control of the trigger mode. • [1] ok 10 - Control of the polling mechanism. • [1] # ============================== • [1] # Passed 10/10 tests, 100.00% okay, in 260.57s • [1] # • [1] # ============================== • [1] # Sim device controller1 coverage report: • [1] # setpos: ok • [1] # settrigger: ok • [1] # getpos: ok • [1] # gettrigger: ok • [1] # • [1] # ============================== • [1] # EPICS database coverage report: • [1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok • [1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok • [1] # FGZ73762:WHEEL1:RESTART(ai): not touched • [1] # FGZ73762:WHEEL1:INPOS(calc): ok • [1] # FGZ73762:WHEEL1:REINIT1(fanout): ok • [1] # FGZ73762:WHEEL1:POSITION(mbbo): ok • [1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok • [1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok • [1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok • [1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok • [1] # FGZ73762:WHEEL1:STEPBK(calcout): ok • [1] # FGZ73762:WHEEL1:STOP(bo): values not covered: 0, 1 • [1] # FGZ73762:WHEEL1:OFFLINEDET(calcout): ok • [1] # FGZ73762:WHEEL1:STEPFW(calcout): ok • [1] # FGZ73762:WHEEL1:POSTRACK(calcout): ok • [1] # • [1] # ==============================

19. What Next • Coverage reports for support modules written in ‘C’ may be possible. • Coverage reports for the Python simulation. • Improvements to the resource management of the run script. • Automatic generation of dependancies. • Implement cron job systems and web result reporting. • More hardware in the test racks. • Retro-fit test suites to existing support modules