Image processing algorithm regression testing framework

1. Image processing algorithm regression testing framework Soumik Ukil

2. Testing Ground Rules • Objectives: • Testing is the process of executing a program with the intent of finding errors • A good test case is one that has a high probability of finding an error • A successful test is one that uncovers an error • However, testing cannot show the absence of defects

3. Unit Testing: • Testing to determine that individual program modules perform to specification • Basic units of software tested in isolation • Regression Testing: • Selective retesting to detect faults introduced during modification of system • Should be performed after any changes in software like bugfixes

4. Motivation • Projects like NETT and BRP • Complex algorithms which run on hundreds of datasets • Source is being constantly modified to fix bugs etc • Manual testing infeasible • Need automated regression testing

5. Testing Strategies • Dynamic Analysis • Exercise the software in its execution environment • Black Box • Treat the system as black -box, ie no knowledge of internal structure • Only need to know legal input and expected output • White Box • Depends on Internal structure of program • Test all paths through code

6. Testing Strategies • Static Analysis • Code Reviews • Walkthroughs • Inspections

7. Tests on image data • Gold-standard data used as basis for correctness of output • Black box tests: • Already know 'correct' output • Generate data from algorithm • Define tests that compare the two sets of data

8. Tests on image data • Generic Tests: • Check output is of correct datatype • Check image dimensions • Specific tests: • Depends on application being tested • For airway tree validation we may want to compare distance between branchpoints • For Lung Segmentation we need to compare volumes and/or distance between contours

9. Objectives of test framework • Different applications: • Regression testing after any changes • Validation of data • Flexible: • Testers only plug in specific tests • Test data generation and reporting of test results taken care by framework

10. Implementation • Built on top of PyUnit: • Part of standard Python library for Python 2.1 and later • Based on JUnit, a proven testing architecture • Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI mode

11. Writing tests with PyUnit: • Basic building blocks called Test Cases • Created by deriving from base class unittest.TestCase • An instance of a TestCase class is an object that can completely run a single test method • Fixtures • A set-up and tear down method for each test case • Many different test cases can use the same fixture • Test Suites • Test case instances can be grouped together according to the features they test • All tests can be executed together as part of a suite • Test Runner • A class whose instances run tests and report results • Can be used in text or GUI mode

12. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

13. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

14. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

15. Application dependent tests: • For lung segmentation we use an area overlap measure to test correctness: • Check that value is above a given threshold to 'pass' test • Similar tests can be defined with distance measures between contours • User has to plug-in appropriate tests

16. Flow diagram:

17. Sample configuration file: [DEFAULT] # Directory of gold-standard data Refdir = ../ValidationData #Directory of data generated by algorithm Maskdir = ../NewData # Output file extension MaskExt = mask.img # Name of logfile directory Logdir = ../logfiles [ALGORITHM] # Dir. of raw image data to run algorithm on Inputdir = /home/xyz/lung/Data # Path to executable for algorithm Execpath = ../lung # Input file extension InputExt = img

18. Reporting test results: • Textual output directed to log files Success: --------------------------------------------------------------------- Ran 6 tests in 88.337s OK Failure: FAIL: Left lung pixel count test (slice by slice). ---------------------------------------------------------------------- Traceback (most recent call last): File "lungsegtests.py", line 101, in test6 self.failIf(fail==1,errormsg) File "unittest.py", line 264, in failIf if expr: raise self.failureException, msg AssertionError: significant segmentation mismatch for left lung on slices: [458, 462]

19. Status: • Testing for Lung segmentation/smoothing has been implemented with the following tests: • Type checking of output masks • Checking that all expected labels are present • Slice by slice area comparison for both lungs • Can be used by segmentation algorithms on ANALYZE images which produce labeled masks • Framework can be extended to handle other formats like airway tree definitions etc

20. References: • http://pyunit.sourceforge.net • http://www.xprogramming.com/testfram.htm Documentation and source code: CVS repository: pulmonary/TestFrameWork