Image archive and leaf classifier SPECIFIC ENABLERS

1. “ENVIROfying” the Future Internet Image archive and leaf classifier SPECIFIC ENABLERS Stuart E. Middleton, BanafsheArbab-Zavar, Stefano Modafferi, Ken Meacham and ZoheirSabeur University of Southampton IT Innovation Centre ENVIROFI specific enabler 17th January 2013

2. Image archive and leaf classifier specific enablers Overview • WP1 pilot use case • Image archive • Architecture • User interface • Leaf classifier • Architecture • Algorithms • User interface

3. Image archive and leaf classifier specific enablers WP1 pilot use case • WP1 pilot: Citizens in Tuscany • Data sources • Proof of concept • Crowd sourcing from Sir Harold Hillier Gardens, UK • http://www3.hants.gov.uk/hilliergardens • User trial • Crowd sourcing via WP1 Pilot in the Tuscany region • Image archive to record crowd-sourced leaf images • Web portal & backend service (Italian & English) • Integrated mobile phone platform • Support for general public and botanical experts • Leaf image + auxiliary images + geo-tag + metadata

4. Image archive and leaf classifier specific enablers WP1 pilot use case • Leaf classifier to label unknown images • Web portal & backend service (Italian & English) • Integrated mobile phone platform • Biodiversity ontology support • Scientific names (Latin) • Common names (Italian, English) • Domain ontology URI’s (e.g. TaxMeOn) • Natura 2000 habitat codes • Value proposition • Supporting crowd sourced leaf observations allows image data collection by volunteers at a scale beyond traditional methods

5. Image archive and leaf classifier specific enablers Image archive architecture Crowd sourcing (web upload and mobile support) Expert review of labels

6. Image archive and leaf classifier specific enablers Image archive user interface

7. Image archive and leaf classifier specific enablers Leaf classifier architecture Users request classifications (unlabelled images) Top N matches returned (leaf classifier algorithm)

8. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Classic benchmark datasets • e.g. Swedish leaf: 1,125 images, 15 species • No Shadows • limited Rotation • Crowd-sourced datasets  challenging! • e.g. Hillier Gardens (IT Innovation): 1400 images, 54 species • Shadows • Natural outdoor lighting • Arbitrary rotation

9. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Segmentation - Colour-based Expectation-Maximization • HSV colour space; discard hue due to the high level of noise • Colour-based EM algorithm for pixel classification using k-means clustering to initialize the EM algorithm (Belhumeur2008) • Three clusters are considered representing: leaf; shadow and background. P. Belhumeur, et al."Searchingthe World’s Herbaria: A System for Visual Identification of Plant Species." ECCV. 2008. 116-129.

10. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Segmentation - Colour-based Expectation-Maximization Belhumeur2008 tried segmentation with two clusters - problems handling shadows We use three clusters for leaf, shadow, background - shadows eliminated

11. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Segmentation - Examples ← The 3 clusters are re-classified based on cluster’s properties. Here, both leaf and shadow clusters were subsequently classified as leaf.

12. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Feature extraction - Inner Distance Shape Context (Ling, 2007) • Matching - fusion of two matching methods based on confidence levels: • Point-based IDSC matching • Contour matching Point correspondence between two images of the same class Inner-distance connections between sampled points Inner-distance shape context H. Ling, D. W. Jacobs. Shape Classification Using the Inner-Distance. 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, pp. 286 - 299.

13. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Distinctive classes VitexAgnus-Castus P(Best match) = 100% Confidence = 100% AlnusGlutinosa'Pyramidalis‘ P(Best match) = 100% Confidence = 99.66% Platanus ’Pyramidalis’ P(Best match) = 100% Confidence = 97.60% QuercusPolycarpa P(Best match) = 100% Confidence = 99.82% RhamnusAlpina P(Best match)=92.86% Confidence = 82.28% Acer Monspessulanum P(Best match) = 100% Confidence = 97.5% TiliaTomentosa 'Petiolaris' P(Best match) = 100% Confidence = 81.85% PopulusNigra P(Best match)=93.33% Confidence = 76.67% CornusSanguinea P(Best match)=90.32% Confidence = 74.91% FagusSylvatica 'Grandidentata' P(Best match)=90.00% Confidence = 77.78% Ulmus P(Best match)=90.00% Confidence = 66.48%

14. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Erroneous results can be caused by: • Similarity between the leaf shape of different species • Error in segmentation • Insufficient number of training samples CarpinusBetulus Acer Platanoides 'Globosum' OstryaCarpinifolia Magnolia x Loebneri Acer SaccharumsubspLeucoderme RhamnusAlpina Magnolia x Soulangeana Ulmus Platanus ’Pyramidalis’ Examples of similar shapes

15. Image archive and leaf classifier specific enablers Leaf classifier algorithms • Hillier Gardens dataset results • Current dataset: 1400 images, 54 species • Mean probability of correct first match: 85.18% • Mean confidence in correct classification: 73.88%

16. Image archive and leaf classifier specific enablers Leaf classifier user interface

17. Thank you for your attention The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement Number 284898 Stuart E. Middleton {sem}@it-innovation.soton.ac.uk www.ENVIROFI.eu twitter.com/ENVIROFI