Loop Investigation for Cursive Handwriting Processing and Recognition

1. Loop Investigation for Cursive Handwriting Processing and Recognition By Tal Steinherz Advanced Seminar (Spring 05)

2. Outline • Background on cursive handwriting • Introduction to loops • Pattern recognition and machine learning conflicts • Feature extraction solutions • Demonstrations and experimental results

3. Cursive Handwriting (J. C. Simon) “Displacing a pen from left to right in an oscillating movement, with loops, descendants (legs), and ascendants (poles).”

4. Cursive vs. Character • Cursive – continuous concatenated set of strokes.produced by a human being in a free style. • Character – a single standalone symbol.produced by a machine subjected to numerous alternative fonts.

5. Online vs. Offline • Online – captured by pen-like devices.the input format is a two-dimensional signal of pixel locations as a function of time (x(t),y(t)). • Offline – captured by scanning devices.the input format is a two-dimensional image of gray-scale colors as a function of location I(m*n).strokes have significant width.

6. Online vs. Offline (demo)

7. A Loop (T. Steinherz) “A set of neighboring foreground pixels surrounding a hole, i.e., a connected blocked group of background pixels in the word’s image, where all foreground pixels are within stroke width distance from the hole.”

8. Ascending (Descending) Loops

9. Axial (of the middle zone) Loops

10. The importance of loops • Shared by many letters (especially a,d,e,g,o,p,q) • Byproduct of the continuous nature of cursive handwriting (like with b,f,h,j,k,l,s,t,y,z) • Elementary and prominent features • Carry additional information given by a set of descriptive parameters

11. The motivation to investigate loops • Character recognitionsupports discrimination between letters. • Writer modeling • Identification • Examination contributes to applications in forensic science and graphology.

12. The output of loop investigation • Incomplete (open) loop identification • Hidden (collapsed) loop tracking - locating blobs that correspond to online loops • Multi (encapsulated) loops understanding - distinguishing natural from artificial loops • Temporal information recovery - retracing the original path of a pen

13. The Engineering Approach(J. C. Simon & T. Pavlidis) “Requires understanding the structure of the objects to be recognized and apply the appropriate combination of (pattern recognition) techniques.”

14. Feature extraction dilemmas • Offline cursive word signal representation • Loop identification • Signal to noise ratio • Feature vector translation The difficulties consist in the feature extraction and preprocessing rather than the machine learning \ recognition engine phase.

15. Offline cursive word signal representation We use the external upper and lower contours in conjunction with the internal contour of all visible loops.

16. Loop identification Given a set of singular points, identification is provided by correlation between pieces of the same contour (around anchor points), of the opposite contours and\or in association with subsets of internal contours.

17. Signal to noise ratio In order to improve the signal’s parametric quantifiability and reduce noisy artifacts, the contour is transformed to a polygon.

18. Hidden loop tracking -the mutual distance principle

19. Hidden loop tracking -the mutual distance principle (cont.)

20. Hidden loop tracking -the mutual distance principle (cont.)

21. Multi loops understanding -the continuity principle

22. Temporal information recovery -the matching principle

23. Writer#1 Writer#2 Writer#3 Writer#4 Writer#5 Writer#6 Total Number of words 223 219 223 170 215 223 1273 Number of characters 1130 1113 1130 835 1083 1130 6421 Number of Loops (all kinds) 1039 1272 1013 745 1332 1146 6547 Hidden loop tracking -an application to ascending (descending) loops Experimental Results

24. Real Loops Online Loops Offline Loops Encapsulated Disqualified Found Total Number 1006 259 186 519 964 Rate 100% 25.7% 18.5% 51.6% 95.8% Hidden loop tracking -an application to ascending (descending) loops (cont.) Experimental Results

25. Large Loops (8<) Large Loops (6<) Online Loops Online Loops Offline Loops Offline Loops Encapsulated Encapsulated Disqualified Disqualified Found Found Total Total Number Number 856 1105 288 233 147 177 390 341 721 855 Rate Rate 100% 100% 27.2% 26.1% 17.2% 16.0% 39.8% 35.3% 77.4% 84.2% Hidden loop tracking -an application to ascending (descending) loops (cont.) Experimental Results

26. Hidden loop tracking -an application to ascending (descending) loops (cont.) Experimental Results Small Loops Total Threshold No Loops 8 180 209 389 209 340 6 131

27. Multi loops understanding -a classifier of beginning a-s Experimental Results More than 40 writers with 1-4 samples per writer.

28. Total Loops Type A Type B Error Questionable Number 81/93 32/36 26/28 16/21 7/8 Rate 100% 39%/38% 30%/32% 19%/22% 7.5%/8% Multi loops understanding -a classifier of beginning a-s Experimental Results