Mathieu Delalandre 1 , Ernest Valveny 1 , Tony Pridmore 2 1 CVC, Barcelona, Spain

Graphics Programming of Constraint for Object Positioning: Application to Performance Evaluation of Symbol Recognition & Spotting Systems Mathieu Delalandre1, Ernest Valveny1, Tony Pridmore2 1 CVC, Barcelona, Spain 2 SCSIT, Nottingham, UK CVC Seminar CVC, Barcelona, Spain Friday 27th of June 2008

Introduction tub door skin door sofa r1 r2 r3 symbol background text Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i.e. input image) is classified as belonging to one of the relevant classes (i.e. predefined symbols) in the application domain” [Chhabra1998][Cordella1999] [Llados2002] [Tombre2005] Electricaldiagram Mechanicaldrawing Utilitymap labels learning database document database Recognition CAD file Web image scanned Spotting Query By Example (QBE) rank Symbol spotting: “a way to efficiently localize possible symbols and limit the computational complexity, without using full recognition methods” [Tombre2003] [Dosch2004] [Tabbone2004] [Zuwala2006] [Locteau2007] [Qureshi2007] [Rusinol2007]

Introduction System Groundtruthing Results Results Results Characterisation Groundtruth Groundtruth Performance evaluation Groundtruth Performance evaluation: Information Retrieval [Salton1992], Computer Vision [Thacker2005], CBIR [Muller2001], DIA [Haralick2000]

Introduction Document Groundtruth Document Groundtruth Document Groundtruth groundtruthed drawings groundtruth Groundtruthing methods for symbol recognition/spotting: GT Real approach ground-truthing - - weak ++ good validation Groundtruthing drawings and alerts validation and alerts recognition results real approach evaluation test images synthetic approach Dosh and al 2006 Yan and al 2004 4 5 4 5 1 3 1 0 2 3 0 2 connected parallel and overlapped

Introduction Document Groundtruth Document Groundtruth Document Groundtruth Synthetic approach Groundtruthing methods for symbol recognition/spotting: - - weak ++ good real approach synthetic approach Groundtruthing Setting Valveny and al 2007 Aksoy 2000 Zhai and al 2003 binary noise vectorial noise

Introduction Key appraoch symbol background To use a same background layer with different symbol layers Production rules ‘Creasy’ but well formed drawing 2-connected 0-n - 1-connected 0-n To associate symbols to positioning constraints and to select them at random Graphical documents are composed of two layers + + 1-connected 1 1-connected 1 c1 - 2-connected O-n c2 Problematic, how to build whole synthetic documents ? First constraint, how to deal with the knowledge level ? Second constraint, how to make usable the system ? chemical symbol Database network symbols Symbol recognition people

Plan 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives • Positioning constraint • Building system • Graphics User Interface • Experiments • Conclusions and perspectives

Positioning constraint 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives (3) positioning on background control point anchor point selection How to define the constraint ? Model dependent constraint Our approach, model independent ρ = 1 ;  = 0.125 to connect to border (2) sloping Unit polar coordinate to define a control point inside a bounding box ρ  (1) computation of control point to include +1 a model constraints -1 +1 Zone -1 How to get the direction d from θ ? How to get the length l from ρ ? (3) (2) (3) (2) (1) (1) (4) ϴ d (4) (4) 1 dy (3) (2) (1) (1) L (2) d dy (4) dx 1 (3) (3) (1) (2) (4) dx (4)

Positioning constraint 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives d2 e l L p 1 2π - d1 b r  0 0,5 l = r  L d2 d1 and d2 are the two lines’ directions (a) (b) - 2 0 6 + x’,y’ 01 12 5 l dy 1  point r x,y dx 1 0,5 4 3 0 The polygon includes the point if | 01 + 12 + …… 60 | = 2π How to select the anchor point ? 3. Taken at random inside a zone 1. Just a fixed value (x,y) 2. Taken at random belong a line x0 x0+dx x0+rxdx y0 y0+rydy 4. Other shapes (arc, curve, ellipsis etc.) y0+dy 0  rx,y 1

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives constraint stack empty no yes (1) symbol and constraint selection constraint selection model selection symbol and constraint symbol factory & positioning (2) symbol factory and positioning object and constraint overflow (3) constraint checking continue full constraint building failure continue overlap (4) building checking continue continue new loop (5) Stopping criterion building end

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives c3 ws1 = 0.33 ws2 = 0.33 ws3 = 0.66 ws4 = 0.33 ws5 = 0.33 ws6 = 1.00 ps1 = 0.11 ps2 = 0.11 ps3 = 0.22 ps4 = 0.11 ps5 = 0.11 ps6 = 0.33 wc1 = 0.33 wc2 = 0.33 wc3 = 1.00 s2 s6 s1 c1 s4 s3 s5 c2 (c)onstraint and (s)ymbol constraint stack empty no yes C1 S1 (1) symbol and constraint selection C2 S2 constraint selection model selection S3 C3 S4 symbol and constraint C4 symbol factory & positioning constraintbasedselection symbol based selection s1 ci s2 object and constraint s3 ci-1 overflow ci-2 … continue full constraint building failure continue overlap continue continue new loop building end c1 c2

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives d2 2π - d1  d2 d1 and d2 are the two lines’ directions (4) positioning on background x’,y’ l dy  control point anchor point selection x,y dx end width width radius center begin center constraint stack empty no yes constraint selection model selection symbol and constraint symbol factory & positioning (2) symbol factory and positioning end trigonometric object and constraint overflow γ width   0 begin 2Π continue full constraint building failure arc is directed to  because  is lower than γ continue (3) sloping overlap • rotation scaling, morphing continue (2) computation of control point continue new loop building end

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives e L l up L L dy1 dy2 p 1 p dx1 dx2 b r right left 0 0,5 bottom l = r  L b e symbol overflows L if right or up or left or bottom is not covered by L Full constraint L covers L if e and b are overlapped (a) (b) Each constraint have a number of symbol to push - 2 0 6 + 01 12 5 P 1 L overlap p if dx1 dx2 < 0 or dy1  dy2< 0 p1 p2 point symbol overflows P if P doesn’t include p1 or p2 or p3 or p4 P 4 3 p4 The polygon includes the point if | 01 + 12 + …… 60 | = 2π p3 constraint stack empty no yes constraint selection model selection symbol and constraint symbol factory & positioning object and constraint overflow (3) constraint checking continue full constraint building failure continue overlap continue continue new loop building end

Building system 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives L dy1 dy2 b e p dx1 dx2 L overlap p if dx1 dx2 < 0 or dy1  dy2< 0 L overlaps L if e or b are overlapped Boxes are overlapped if L1 overlaps L2or L2 overlaps L1 and L3 overlaps L4or L4 overlaps L3 L1 L2 L3 L4 constraint stack empty no yes Layer 2 with overlap constraint selection model selection Layer 1 without overlap symbol and constraint symbol factory & positioning background object and constraint overflow continue full constraint building failure continue overlap (4) building checking continue continue new loop building end

ContributionsBuilding Manager 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Continue When building failures (overflow, full, overlap) become upper to number of wished symbol we stop the process constraint stack empty no yes constraint selection model selection symbol and constraint symbol factory & positioning object and constraint overflow continue full constraint building failure continue overlap continue continue new loop (5) Stopping criterion building end

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Constraint Selector Constraint Locator Constraint Setting Model Selector Building Viewer Model Viewer Building Bar Symbol Models Run Building Engine Edit View 1. Background broswing 2. Model loading 3. Model viewing 4. Constraint edit 5. Constraint linking 6. Building viewing 7. Constraint import/eport 8. Resolution setting …

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Model loading, selection and viewing Background attachment and browsing attach loading focus ‘click and well’ select ‘scroll & click’ shift ‘drag’ automatic zooming

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Link the constraints Locate the constraints select ‘click’ ‘click’ link model ‘click, move, click’ or’ ‘pressed, drag, release’ add ‘right double click’ delete ‘left double click’ ‘click(s), double click’

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives Control Slope

Graphics User Interface 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives without rotation, without scaling with rotation, without scaling without rotation, with scaling with rotation, with scaling

Experiments 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives

Conclusions and Perspectives 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives • Conclusions First valid system to build whole graphic synthetic documents • Built documents kept meaning and seem realistic • No constraint on database sizes (thousands …..) • Contents of documents can be controlled + Can be use on several backgrounds with “reasonable” work costs using the GUI (1-2 hour per background)

Conclusions and Perspectives 1. Positioning constraint 2. Building system 3. Graphics User Interface 4. Experiments 5. Conclusions and perspectives SCSIT Nottingham LITIS Rouen LORIANancy LI Tours CVC Barcelona • In progress, perspectives Workgroup Forum document tub Evaluation objectives Campaign organization Background types Model & domain scalability Sizes of image and databases Degradation models System training (learning data, QBE) Groundtruth data & file Mapping algorithm Recognition & ranking System taxonomy System profiling door skin door sofa Labels QBE Learning Spotting/Recognition System Ranks Region Of Interest Alicia Fornes, DimosthenisKaratzas, HervéLocteau, Jean-Pierre Salmon, Jean-Yves Ramel, MarçalRusinol, Philippe Dosch, Rashid Qureshi and Tony Pridmore r1 r2 r3 Groundtruth Mapping Characterization truth results

Thanks Bibliography • M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol Recognition and Spotting Systems: An Overview. Accepted paper to the Eighth IAPR Workshop on Document Analysis Systems (DAS), 2008. • M. Delalandre, T. Pridmore, E. Valveny, H. Locteau and E. Trupin. Building Synthetic Graphical Documents for Performance Evaluation. Revised Selected Papers of Workshop on Graphics Recognition (GREC), Lecture Notes in Computer Science (LNCS), vol 5046, pp 288-298, 2008.

Mathieu Delalandre 1 , Ernest Valveny 1 , Tony Pridmore 2 1 CVC, Barcelona, Spain