1 / 37

Performance Evaluation of Grouping Algorithms

Performance Evaluation of Grouping Algorithms. Vida Movahedi Elder Lab - Centre for Vision Research York University Spring 2009. Overview. Grouping and evaluation methods Region-based measures Boundary-based measures Mixed measures Alignment measure. Overview.

shea-mcconnell
Download Presentation

Performance Evaluation of Grouping Algorithms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance Evaluationof Grouping Algorithms Vida Movahedi Elder Lab - Centre for Vision Research York University Spring 2009

  2. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  3. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  4. Grouping Edge segments: Example Centre for Vision Research, York University

  5. Perceptual Organization/ Grouping • A process of assembling features into groups which are perceptually significant based on various cues (Lowe, 1985) • The problem of aggregating primitive image features that project from a common structure in the visual scene (Elder, 2002) Centre for Vision Research, York University

  6. Evaluation Measure • How good is each grouping? • Which algorithm has a better performance? • What is the best grouping that can be achieved? • Note differences with regional segmentation evaluation Centre for Vision Research, York University

  7. Evaluation Methods • Three main categories (Zhang, 1996) • Analytical methods Consider the algorithms themselves, e.g. based on the a priori knowledge they use (not based on output of the algorithms) • Empirical goodness methods Based on the outputs of the algorithms, e.g. based on the intra-region uniformity of the segments, or the inter-region contrast between the segments. • Empirical discrepancy methods A reference segmentation or ground truth is assumed, to compare the outputs with Centre for Vision Research, York University

  8. Goal: Measure Discrepancy Centre for Vision Research, York University

  9. SOD: Salient Object Dataset • Based on Berkeley Segmentation Dataset (BSD) • 300 images • 7 subjects Centre for Vision Research, York University

  10. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  11. Region-based Discrepancy • (Young, 2005),(Ge, 2006), (Goldmann, 2008) • A and B two boundaries • RB the region corresponding to a boundary B and |RB| the area of this region • 1: maximum discrepancy, • 0: maximum consistency Centre for Vision Research, York University

  12. Interpretation Centre for Vision Research, York University

  13. Evaluation by this measure • Not sensitive to spikes, wiggles, shape >= (more error) Centre for Vision Research, York University

  14. Examples of near-optimal cases Centre for Vision Research, York University

  15. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  16. Boundary-based Distance • Distance of one point a from B • Distance Signature of all a in A • One directional Hausdorff • Two directional Hausdorff Centre for Vision Research, York University

  17. Evaluation by this measure • Not sensitive to wiggles, shape • Not sensitive to the distance distribution, but only to the maximum value Centre for Vision Research, York University

  18. Euclidean vs. Geodesic Distance Geodesic Distance: the min. distance between two points a and b without cross Euclidean Distance: the min. distance between two points a and b Centre for Vision Research, York University

  19. Evaluation by this measure Almost the same by De Almost the same by De & Dg Centre for Vision Research, York University

  20. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  21. A mixture of boundary-based and region-based • Penalizing the over-detected and under-detected regions by their Euclidean or Geodesic distances pj, j=1..Nfp are pixels in the false negative region (RB-RA) qk, k=1..Nfn are pixels in the false positive region (RA-RB) Centre for Vision Research, York University

  22. Evaluation by this measure • Not penalizing effectively, e.g. narrow false positives below Centre for Vision Research, York University

  23. Correspondence Problem • The false negative and false positive regions can be very small, yet the boundaries be very different • Segments on one boundary should correspond to segments on the other Centre for Vision Research, York University

  24. Alignment • The order of matching points on the two boundaries should be monotonically non-decreasing. Centre for Vision Research, York University

  25. Correspondence (Cont.) • Note that if correspondence is maintained, De will work almost like Dg! Centre for Vision Research, York University

  26. Overview • Grouping and evaluation methods • Region-based measures • Boundary-based measures • Mixed measures • Alignment measure Centre for Vision Research, York University

  27. Alignment Distance • Main idea: We need to find the ‘alignment’ that leads to minimum total distance. • Method: • Use N samples on each boundary (equally spaced) • Find the NxN matrix of Euclidean distances. • The diagonals show correspondences with some rotations • The one with min sum of distances is the best correspondence and its sum is our measure of discrepancy. Centre for Vision Research, York University

  28. Alignment Measure (cont.) Note: Order of both samples increases clockwise Centre for Vision Research, York University

  29. Evaluation by this simple measure • Samples falling out of phase • Solution: finer sampling on one boundary >= (more error) Centre for Vision Research, York University

  30. Bimorphism • (Tagare, 2002) • A method to let correspondence of 1 to many and many to 1  symmetric Centre for Vision Research, York University

  31. A symmetric Alignment Distance • Edit cost of changing one string to another • Edit operation, cost of operation • A sequence of operations taking A to B • Symmetric: Centre for Vision Research, York University

  32. Example Centre for Vision Research, York University

  33. Cyclic shifts • Cyclic shifts • Alignment Distance • Dynamic programming • Complexity: • (Maes, 1990) Complexity: Centre for Vision Research, York University

  34. Examples Centre for Vision Research, York University

  35. Examples Centre for Vision Research, York University

  36. Evaluation by this measure • Note: If using Euclidean distance, there is no sensitivity to region Centre for Vision Research, York University

  37. References (Elder, 2002) J. H. Elder and R. M. Goldberg (2002), "Ecological statistics of Gestalt laws for the perceptual organization of contours." J Vis, vol. 2, pp. 324-353. (Zhang, 1996) Y. J. Zhang. (1996), “A survey on evaluation methods for image segmentation”, Pattern recognition 29(8), pp. 1335. (BSD) D. Martin (2001), "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proceedings of the 8th IEEE International Conference on Computer Vision, vol. 2, pp. 416-423. (Ge, 2006) F. Ge, S. Wang and T. Liu (2006), "Image-Segmentation Evaluation From the Perspective of Salient Object Extraction," Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 1146-1153. (Goldmann, 2008) L. Goldmann. (2008), Towards fully automatic image segmentation evaluation. Lecture notes in computer science 5259 LNCS, pp. 566. (Young, 2005) D. P. Young (2005), "PETS Metrics: On-line performance evaluation service," Proceedings - 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, VS-PETS, vol. 2005, pp. 317, 2005. (Huttenlocher, 1993) D. P. Huttenlocher (1993), “Comparing images using the Hausdorff distance”, IEEE transactions on pattern analysis and machine intelligence 15(9), pp. 850. (Tagare, 2002) H. D. Tagare. (2002), “Non-rigid shape comparison of plane curves in images”, Journal of mathematical imaging and vision 16(1), pp. 57. (Maes, 1990) M. Maes (1990), “On a cyclic string-to-string correction problem”, Information processing letters 35(2), pp. 73. Centre for Vision Research, York University

More Related