1 / 20

Image processing algorithm regression testing framework

Image processing algorithm regression testing framework. Soumik Ukil. Testing Ground Rules. Objectives: Testing is the process of executing a program with the intent of finding errors A good test case is one that has a high probability of finding an error

melita
Download Presentation

Image processing algorithm regression testing framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Image processing algorithm regression testing framework Soumik Ukil

  2. Testing Ground Rules • Objectives: • Testing is the process of executing a program with the intent of finding errors • A good test case is one that has a high probability of finding an error • A successful test is one that uncovers an error • However, testing cannot show the absence of defects

  3. Unit Testing: • Testing to determine that individual program modules perform to specification • Basic units of software tested in isolation • Regression Testing: • Selective retesting to detect faults introduced during modification of system • Should be performed after any changes in software like bugfixes

  4. Motivation • Projects like NETT and BRP • Complex algorithms which run on hundreds of datasets • Source is being constantly modified to fix bugs etc • Manual testing infeasible • Need automated regression testing

  5. Testing Strategies • Dynamic Analysis • Exercise the software in its execution environment • Black Box • Treat the system as black -box, ie no knowledge of internal structure • Only need to know legal input and expected output • White Box • Depends on Internal structure of program • Test all paths through code

  6. Testing Strategies • Static Analysis • Code Reviews • Walkthroughs • Inspections

  7. Tests on image data • Gold-standard data used as basis for correctness of output • Black box tests: • Already know 'correct' output • Generate data from algorithm • Define tests that compare the two sets of data

  8. Tests on image data • Generic Tests: • Check output is of correct datatype • Check image dimensions • Specific tests: • Depends on application being tested • For airway tree validation we may want to compare distance between branchpoints • For Lung Segmentation we need to compare volumes and/or distance between contours

  9. Objectives of test framework • Different applications: • Regression testing after any changes • Validation of data • Flexible: • Testers only plug in specific tests • Test data generation and reporting of test results taken care by framework

  10. Implementation • Built on top of PyUnit: • Part of standard Python library for Python 2.1 and later • Based on JUnit, a proven testing architecture • Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI mode

  11. Writing tests with PyUnit: • Basic building blocks called Test Cases • Created by deriving from base class unittest.TestCase • An instance of a TestCase class is an object that can completely run a single test method • Fixtures • A set-up and tear down method for each test case • Many different test cases can use the same fixture • Test Suites • Test case instances can be grouped together according to the features they test • All tests can be executed together as part of a suite • Test Runner • A class whose instances run tests and report results • Can be used in text or GUI mode

  12. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

  13. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

  14. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): self.labels=[20,30] # left and right lung labels def tearDown(self): self.data = None self.ref_data= None def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') def test2(self): """Test to check that correct labels are present in mask file.""" for n in self.labels: errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

  15. Application dependent tests: • For lung segmentation we use an area overlap measure to test correctness: • Check that value is above a given threshold to 'pass' test • Similar tests can be defined with distance measures between contours • User has to plug-in appropriate tests

  16. Flow diagram:

  17. Sample configuration file: [DEFAULT] # Directory of gold-standard data Refdir = ../ValidationData #Directory of data generated by algorithm Maskdir = ../NewData # Output file extension MaskExt = mask.img # Name of logfile directory Logdir = ../logfiles [ALGORITHM] # Dir. of raw image data to run algorithm on Inputdir = /home/xyz/lung/Data # Path to executable for algorithm Execpath = ../lung # Input file extension InputExt = img

  18. Reporting test results: • Textual output directed to log files Success: --------------------------------------------------------------------- Ran 6 tests in 88.337s OK Failure: FAIL: Left lung pixel count test (slice by slice). ---------------------------------------------------------------------- Traceback (most recent call last): File "lungsegtests.py", line 101, in test6 self.failIf(fail==1,errormsg) File "unittest.py", line 264, in failIf if expr: raise self.failureException, msg AssertionError: significant segmentation mismatch for left lung on slices: [458, 462]

  19. Status: • Testing for Lung segmentation/smoothing has been implemented with the following tests: • Type checking of output masks • Checking that all expected labels are present • Slice by slice area comparison for both lungs • Can be used by segmentation algorithms on ANALYZE images which produce labeled masks • Framework can be extended to handle other formats like airway tree definitions etc

  20. References: • http://pyunit.sourceforge.net • http://www.xprogramming.com/testfram.htm Documentation and source code: CVS repository: pulmonary/TestFrameWork

More Related