Towards Performance Evaluation of Symbol Recognition & Spotting Systems in a Localization Context

1. Towards Performance Evaluation of Symbol Recognition & Spotting Systems in a Localization Context Mathieu Delalandre CVC, Barcelona, Spain EuroMed Meeting LORIA, Nancy city, France Monday 18th of May 2009

2. Introduction tub door skin door sofa r1 r2 r3 symbol background text Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i.e. input image) is classified as belonging to one of the relevant classes (i.e. predefined symbols) in the application domain” [Chhabra1998][Cordella1999] [Llados2002] [Tombre2005] Electricaldiagram Mechanicaldrawing Utilitymap labels learning database document database Recognition CAD file Web image scanned Spotting Query By Example (QBE) rank Symbol spotting: “a way to efficiently localize possible symbols and limit the computational complexity, without using full recognition methods” [Tombre2003] [Dosch2004] [Tabbone2004] [Zuwala2006] [Locteau2007] [Qureshi2007] [Rusinol2007]

3. Introduction dATA Results Data Results Results Data Groundtruth Groundtruth Groundtruth tub Performance evaluation: Information Retrieval [Salton1992], Computer Vision [Thacker2005], CBIR [Muller2001], DIA [Haralick2000] Case of symbol recognition & spotting: [Ezra2008][Delalandre2008] door skin door sofa Labels Training data System Ranks QBE Learning Spotting/Recognition System r1 r2 r3 Groundtruthing Region Of Interest Characterisation Groundtruth Mapping Characterization truthresults Performance evaluation Performance evaluation

4. Plan • Groundtruth and test documents • Performance characterization • Conclusions and perspectives

5. Groundtruth and test documents Overview of approaches Document Groundtruth Document Groundtruth Groundtruth Document 1. Overview of approaches 2. Existing datasets groundtruthed drawings groundtruth GT Real approach ground-truthing - - weak ++ good validation Groundtruthing drawings and alerts validation and alerts recognition results real approach evaluation test images synthetic approach Dosch and al 2006 4 5 4 5 Rusinol and al 2009 Yan and al 2004 1 3 1 0 2 3 0 2 connected parallel and overlapped

6. Groundtruth and test documents Overview of approaches Document Groundtruth Document Groundtruth Groundtruth Document 1. Overview of approaches 2. Existing datasets Synthetic approach - - weak ++ good real approach Groundtruthing Setting synthetic approach Valveny and al 2007 Aksoy 2000 Zhai and al 2003 binary noise vectorial noise

7. Groundtruth and test documents Overview of approaches symbol background Graphical documents are composed of two layers 1. Overview of approaches 2. Existing datasets Delalandre2008 - - weak ++ good real approach To use a same background layer with different symbol layers synthetic approach

8. Groundtruth and test documents Overview of approaches 1. Overview of approaches 2. Existing datasets C1 M1 c1 C2 M2 Delalandre2008 M3 c2 C3 M4 C4 - - weak ++ good real approach p synthetic approach L  p1 loaded symbol symbol model bounding box and control point p2 L1 L2 θ2 θ1 alignment

9. Groundtruth and test documents Overview of approaches Symbol Models Symbol Positioning Document Generation Positioning Constraints GT GT GT GT 1. Overview of approaches 2. Existing datasets Delalandre2008 Symbol Models (2) run - - weak ++ good Background Image Building Engine (1) edit real approach (3) display synthetic approach

10. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

11. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

12. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

13. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

14. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

15. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

16. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

17. Groundtruth and test documents Existing datasets y s [0,1] x 0 v vmax 1. Overview of approaches 2. Existing datasets 1. Random selection of a document 2. Radom selection of a symbol Groundtruth Generator of queries GREC ICPR 3. Random crop SESYD Others

18. Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GREC ICPR SESYD Others

19. Plan • Groundtruth and test documents • Performance characterization • Conclusions and perspectives

20. Performance characterization Introduction • Performance characterisation (segmented symbols) • [Valveny2004] [Dosch2006] [Valveny2007,2008a,2008b] • Recognition rate • Precision/Recall • Homogeneity • Separability Performance characterisation (real context) tub door skin door sofa Labels Ranks QBE Learning Spotting/Recognition System r1 r2 r3 Region Of Interest Groundtruth Mapping Characterization truthresults Performance evaluation

21. Performance characterization About mapping Mapping cases Single : a model line matches only with one detected line. Split : two model lines match with one detected line. Merge : a model line matches with two detected lines. False alarm : a detected line doesn't match with any model lines. Miss : a model line doesn't match with any detected lines. segmentation truthresults groundtruth Symbol spotting [Rusinol2009] Layout analysis [Antonacopoulos1999] g1 g2 Groundtruth segmentation Text/graphics separation [Wenyin1997] r Results groundtruth c1 c2 Mapping segmentation

22. Performance characterization Mapping, application to symbol Which representation ? How to define the regions ? Compatibility with recognition systems ? Lot of systems use sliding windows to detect symbols providing only points [Adam2001] [Dosh2004] [Rusinol2007] Lot of systems use sliding windows to detect symbols providing only points [Adam2001] [Dosh2004] [Rusinol2007] How to define local thresholds point the polarized pat of the capacitor belong to the symbol ? Systems providing region of interest can “tune” their results, how to limit the over segmentation cases ? the precision will depend of the model wrapper box, ellipsis groundtruth Same for the moving area of the door ? segmentation convex polygon could be of weak precision precise but comparison is time consuming concave polygon

23. Performance characterization Work in progress Comparison of some criteria System of [Qureshi’08] , 100 floorplans (2521 symbols) Signature based characterization Domain definition of the ROI Orientation sampling [0-2π] Reporting [0-2π] Rates % results groundtruth Region size dx×dy

24. Plan • Groundtruth and test documents • Performance characterization • Conclusions and perspectives

25. Conclusions and perspectives • Conclusions • Large databases of segmented symbol images exist “GREC” • Synthetic databases in real context exist “SESYD” • True-life documents and groundtruth are at the corner “EPEIRES” • Characterization tools have been proposed “SymbolRec” • Perspectives • Continue to produce other databases, using existing platforms • Mapping is the key problem today, to achieve a performance evaluation in real context

26. Thanks All the referenced papers can be found in [1] M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol Recognition and Spotting Systems: A Overview. Workshop on Document Analysis Systems (DAS), pp 497-505, 2008.