Leaf Classification from Local Boundary Analysis

1. Leaf Classification from Local Boundary Analysis Anne Jorstad AMSC 664 University of Maryland Spring 2008 Final Report Advisor: Dr. David Jacobs, Computer Science

2. Background • Electronic Field Guide for Plants

3. Background • Current System: • Inner-Distance Shape Context (IDSC) • Measures the shortest distance between two points on a path contained entirely within a figure • Good for detecting similarities between deformable structures

4. Background • Current System: • All shape information is compared at a global level, no specific consideration of edge types Cephalanthus occidentalis (smooth boundary) Carpinus caroliniana (serrated boundary)

5. Problem Statement Use local boundary information to make classification decisions that complement the existing system.

6. The Algorithm • Input: • Capture boundary curve:

7. The Algorithm: Wavelets • Discrete wavelet transform • In: vector of points • Out: two vectors, each half original length • Approximation coefficients: • general spatial information • Detail coefficients: • local detail information • Repeat for multiple scales

8. The Algorithm: Wavelets • Model leaf by its detail coefficients over several scales Input Approximations, continually subtracting out detail information

9. The Algorithm: Data • Forget leaves: • Each boundary point: • Lose one degree of freedom in preserving rotation invariance • For 3 wavelet scales, leaf is ~2000 5-D points • Combine data for all leaves: • #leaves x ~2000 5-D points • Group all points into meaningful clusters

10. The Algorithm: Clustering • Goal: Sort points into “buckets” to get a unique distribution for each leaf species • K-Means Clustering: group all points into 36 representative clusters

11. The Algorithm: Distribution Comparison • Distribution of individual leaf’s 2000 points over the 36 clusters represents leaf (a) (b) (c) Leaf image and corresponding histogram for (a) Corylus americana, (b) Corylus americana, different example, (c) Asimina triloba

12. The Algorithm: Distribution Comparison • Compare distributions between leaves using the chi-squared distance: where • Smallest distance defines best match • New leaf is assigned the species of the closest match

13. Validation • Training data: 20 species, 10 examples of each → 200 leaves 10 serrated species 10 smooth species

14. Validation • Test data: same 20 species, 5 new examples of each • Nearest-Neighbor Classification • Species classification: 46% correct • Serration classification: 100% correct • closest match was to species with appropriate serration

15. Validation • Test data: same 20 species, 5 new examples of each • Nearest-Neighbor Classification • Species classification: 46% correct • Serration classification: 100% correct • closest match was to species with appropriate serration Local serration information IS being captured!

16. Combining Results • Original IDSC results on same data set: • Species correct: 62% • Serration correct if species wrong: 53% • No better than chance • How to combine wavelet distances with IDSC distances?

17. Combining Results • Given and • Want to find:

18. Naïve Bayes Classification • From Bayes’ Rule: • Can now calculate all relevant probabilities from training data

19. Naïve Bayes Classification • Wavelet distances → binary serration value • Add small linear smoothing term • IDSC distances → species ranked in order from nearest to farthest • Add Gaussian smoothing term

20. Validation Results • Test on same 20 species, 5 examples of each • Adding serration information has improved overall classification results!

21. Full Data Set • 245 species, 7481 leaves • Binary serration assignment no longer makes sense:

22. Linear Optimization • Find best linear weighting of distances: • Train over previous training set % correct alpha

23. Full Data Set • Nearest-Neighbor Classification over all 7481 leaves • Wavelet alone: 20% correct • IDSC alone: 54% correct • Combined: 64% correct

24. In Practice • Electronic field guide displays top 5, 10 or 20 matches • Calculate correct % in top n matches, for n = 1, …, 20

25. In Practice % correct # matches considered

26. In Practice • Need results in near real-time • Otherwise no benefit over paper field guides • Running time • Preprocessing: (several hours) • Determine cluster centers • Determine distributions for each leaf • On the spot: (0.92 seconds) • Calculate single distribution • Compare to all distributions in system

27. Conclusions • Wavelets do capture local serration information • Wavelet + IDSC classification does a better overall job than the original IDSC alone • Calculations can be done in real time to make the system realistic to use

28. References • Gaurav Agarwal, Haibin Ling, David Jacobs, Sameer Shirdhonkar, W. John Kress, Rusty Russell, Peter Belhumeur, Nandan Dixit, Steve Feiner, Dhruv Mahajan, Kalyan Sunkavalli, Ravi Ramamoorthi, Sean White. “First Steps Toward an Electronic Field Guide for Plants”. Taxon, vol. 55, no. 3, Aug. 2006. • Cene C.-H. Chuang, C.-C. Jay Kuo. “Wavelet Descriptor of Planar Curves: Theory and Applications”. IEEE Transactions of Image Processing, Vol. 5, No. 1, January 1996. • Pedro F. Felzenszwalb, Jushua D. Schwartz. “Hierarchical Matching of Deformable Shapes”. IEEE Conference on Computer Vision and Pattern Recognition, 2007. • Haibin Ling, David Jacobs. “Using the Inner-Distance for Classification of Articulated Shapes.” CVPR, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, 2005. • Jitendra Malik, Serge Belongie, Thomas Leung, Jainbo Shi. “Contour and Texture Analysis for Image Segmentation”. International Journal of Computer Vision, vol. 34, no. 1, July 2001. • Stephane Mallat. “A Wavelet Tour of Signal Processing”. Academic Press, Chestnut Hill, Massachusetts, 1999.