A Handwriting Interface to Mathematical Expressions

1. A Handwriting Interface to Mathematical Expressions May 11, 2006 Western Ontario University Masakazu Suzuki Kyushu University, InftyProject

2. Plan of the talk • About InftyProject • Resarch scope: Mathematical Information Processing • 2. Software release: InftyReader,InftyEditor, etc. • Handwriting Interface • Online handwritten math recognition / printed math recognition, • Character recognition (Voting method), • Formula structure recognition (Automatic writing method), • Adaptability (User’s script registration), • Connection to computer algebra system.

3. Plan of the talk • About InftyProject • Resarch scope: Mathematical Information Processing • 2. Software release: InftyReader,InftyEditor, etc. • Handwriting Interface • Online handwritten math recognition / printed math recognition, • Character recognition (Voting method), • Formula structure recognition (Automatic writing method), • Adaptability (User’s script registration), • Connection to computer algebra system.

4. 1. Infty Project • The beginning： • Started as a research project to help visually impaired people in scientific fields in 1995. • Current research subjects： • Recognition and understanding of math documents, • User interface and data conversion, etc. • Policy: • Priority in practical system development.

5. 1. Infty Project • URL：http;//www.inftyproject.org Go

6. 1. Infty Project • Mathematical Document Digitization • InftyReader : Mathematical document reader (Paper → XML) • Interface to Mathematics • InftyEditor : Authoring tool, Handwriting interface for math, • ChattyInfty: Voice output, Braille output, etc., • AsirPad : Handy computer algebra tool (PDA), • ActiveInfty: Math Edit Box on the Web. Software:

7. 1. Infty Project • Mathematical Document Digitization • InftyReader : Mathematical document reader (Paper → XML) • Interface to Mathematics • InftyEditor : Authoring tool, Handwriting interface for math, • ChattyInfty: Voice output, Braille output, etc., • AsirPad : Handy computer algebra tool (PDA), • ActiveInfty: Math Edit Box on the Web. Software:

8. 2. Handwriting Interface for math • Demonstration…

9. 2. Handwriting Interface for math • Printed math document recognition • Difficulty : Noise, Variation of layout and fonts, etc. • Needs speed and high accuracy (99.5% - 99.9%),. • Online recognition of handwritten math • Main Difficulty : Distortion of symbol shape, size and position, • Advantage in char recognition: - Accuracy 95% is good, 90% is already usable,- stroke information is usable, • Correction method of structure analysis error is important, Comparison with printed math recognition:

10. 2. Handwriting Interface for math • Character and Symbol recognition • Combination of two different methods (Voting), • Restriction of usable symbols, • Structure Analysis of Math Expressions • Automatic rewriting method -Accuracy of structure analysis, - Immediate correction of recognition error, • Subscript/Superscript decision algorithm. Some Features of out method:

11. 2. Handwriting Interface for math • Character and Symbol recognition • Combination of two different methods (Voting), • Restriction of usable symbols, • Adaptability (User’s script registration) --- new • Structure Analysis of Math Expressions • Automatic rewriting method -Accuracy of structure analysis, - Immediate correction of recognition error, • Subscript/Superscript decision algorithm. Some Features of out method:

12. 3. Character Recognition • To get recognition results by voting, we implemented two different methods of character recognition using: • Direction element feature, • Matching of segmented stroke sequence.

13. 3.1. Direction Element Feature • A character is separated into 3×5 blocks and each block has 8 direction elements.

14. 3.1. Direction Element Feature • That is, we consider 120 dimensional vector as a direction element feature.

15. 3.2. Matching of Segmented Stroke Sequence • A written stroke is modified and segmented as follows:

16. 3.2. Matching of Segmented Stroke Sequence • Modify small loops to cusps.

17. 3.2. Matching of Segmented Stroke Sequence • Segment the stroke at extreme points on the vertical coordinate.

18. 3.2. Matching of Segmented Stroke Sequence • Finally, segment at cusp points again.

19. 3.2. Matching of Segmented Stroke Sequence • Segmented strokes are classified into 28 patterns.

20. 3.2. Matching of Segmented Stroke Sequence • Straight lines are classified into 8 class by their direction.

21. 3.2. Matching of Segmented Stroke Sequence • Winding downstrokes are classified into 10 patterns.

22. 3.2. Matching of Segmented Stroke Sequence • Winding upstrokes are also classified into 10 patterns similarly.

23. 3.2. Matching of Segmented Stroke Sequence • The character recognition is done by the matching of the sequences of segment stroke patterns thus obtained. , , , ,

24. 3.2. Matching of Segmented Stroke Sequence To calculate the recognition cost, the following features are used: • segment strokes' aspect ratios • the positions of their bounding rectangles, • the directions of their original vectors vo and terminal vectors vt.

25. 3.3. Voting • Each of the two recognition methods returns three ordered candidates with costs. • The voting cost of the candidate =the ratio of its cost to the third candidate's. • The final recognition results is determined by the ascending order of the sum of the voting costs of the two recognition methods.

26. 3.4 Restrictions of Usable Characters

27. 4. Structure Analysis • Handwritten characters and math expressions input using pen interface have generally severe distortion.

28. 4. Structure Analysis • Handwritten characters and math expressions input using pen interface have generally severe distortion.

29. 4. Structure Analysis • Handwritten characters and math expressions input using pen interface have generally severe distortion. • An error of link structure often destroys the structure analysis of the remaining part.

30. 4. Structure Analysis • Handwritten characters and math expressions using pen interface have generally severe distortion. • An error of link structure often destroys the structure analysis of the remaining part.

31. 4. Structure Analysis • Handwritten characters and math expressions using pen interface have generally severe distortion. • An error of link structure often destroys the structure analysis of the remaining part. • The labors for the correction of this kind of errors disturb seriously the smooth input of mathematical expressions

32. 4. Structure Analysis • Automatic Rewriting Method isintroduced to overcome this difficulty.

33. 4. Structure Analysis • Automatic Rewriting Method isintroduced to overcome this difficulty. • In our system, each written character is rewritten by neat strokes in an appropriate size and position, immediately and automatically.

34. 4.1. Determination of Characters Extendable Character: F → E , C → G or d, ＝→≠, etc. Unextendable Character: A , B , E , etc. Extendable Area: The area in which the next stroke is supposed to be pushed down when it is extended to other character by adding some strokes.

35. 4.1. Determination of Characters Each time a stroke is written, the character recognition is performed by the system. The written strokes are rewritten neatly by computer in the following cases: • The recognition cost is low (good) enough and either the recognition result is an unextendable character, or the next stroke is pushed down outside of its extendable area. • An interval of onesecond has elapsed with no input of new stroke.

36. C D 4.2. Determination of Positions and Sizes of Characters For two written characters C, D,the relation pointP(C,D) is defined by P(C,D):=(h/H× 100, d/H× 100).

37. 4.2. Determination of Positions and Sizes of Characters To determine the relation between C and D, select the nearest ideal relation point.

38. 5. Adaptation to user Registration of user’s script: - Demonstration.

39. 6. Further development • Accuracy • Grammer • Bigrams • Frequently used expressions • Editing facility by pen. • Insert/Delete • Select, Cut, Paste

40. Conclusion • Brief introduction of InftyProject • Difficulty of handwritten math recognition • Sketch of our method: • Character recognition • Structure analysis (Automatic rewriting) • Future work

41. “INFTY”R&D project for math information processing Thanks you! Masakazu Suzuki Graduate School/ Faculty of Math. Kyushu University suzuki@math.kyushu-u.ac.jp http://math.kyushu-u.ac.jp/~suzuki InftyProject: http://www.inftyproject.org