Experiments with a Pen-Based Math System

1. Experiments with a Pen-Based Math System George Labahn Symbolic Computation Group University of Waterloo

2. Point of View • Tablet PC : a solution looking for a problem to solve • these do generate a great deal of interest. • however not well utilized at present • Mathematics and computer algebra systems should be one of the killer apps for Tablet PC’s. • Math on tablet is difficult and complex • (that is if you want to build a real one)!

3. Some Information on projects at UW/UWO • Funding : Microsoft Canada • a bit of a story at UW • Team leaders : • George Labahn, Mark Giesbrecht, Stephen Watt • Lots of previous work done by Stephen

4. MathBrush project at UW • People involved: • George Labahn • Mirette Marzouk • Scott Maclean • David Tausky • Ian Rutherford With help from • Yu-Hong (Maple Soft)

5. Obvious Motivation • Latex \begin{equation} \int {\frac { \left( 3\,{x}^{2}+2 \right) \sin \left( {x}^{3}+2\,x-1 \right) } { \left( \cos \left( {x}^{3}+2\,x-1 \right) \right) ^{3}+12\, \left( \cos \left( {x}^{3}+2\,x-1 \right) \right) ^{2}-3}} dx \end{equation} • Maple Int((3*x^2+2)*sin(x^3+2*x-1)/(cos(x^3+2*x-1)^3 + 12*cos(x^3+2*x-1)^2-3),x); • Mathematica Integrate[(3*x^2+2)*Sin[x^3+2*x-1]/(Cos[x^3+2*x-1]^3 + 12*Cos[x^3+2*x-1]^2-3),x]

6. Objective • Investigate the use of pen-based devices for mathematical computation and exploration • Study the key issues when combining pen-based interfaces with Computer Algebra Systems (CAS) • Build an experimental pen-based math system to allow the user to investigate the various components that make up a pen-math system e.g. Ink input, editing, recognition, computation

7. Additional Motivation • Entering mathematical concepts on a computer for presentation is tedious and difficult. Ex. Latex • The current interface of CAS, although immensely improved, constraints the user to express their thoughts in a command line form • Different CAS systems have different languages and commands • Manipulation and transformation of expressions in CAS lack intuitiveness when using only keyboard and mouse

8. Disclaimers • First pancake • No intention of creating a complete interface • We do not want to reinvent the wheel by reimplementing all the interface features in existing CAS • Ultimately what is needed is to have (invisible) pen based math windows which sit inside existing CAS • Stephens idea

9. Challenges • Text recognizers are not suitable: • work with ASCII Characters • depend on language specific dictionaries • assume input is one dimension • Mathematical symbols are more ambiguous than regular text Ex.

10. Challenges • Mathematical symbols appear with wide variation in size and do not necessarily follow baselines Ex. superscripts and subscripts • Correct symbols recognition does not lead to a unique math expression. Ex. u(x+y) u times x+y or u applied to argument x+y ? • Rendering two-dimensional math expressions with line-breaking and need for interactivity

11. MathBrush Components Display information Math Rendering Tool Ink input from user Character Recognizer + User format Bounding boxes and candidates with recognition confidence MathML representation of input expression MathML representation of results + CAS + Operation CAS Interface Tool Structural Analyzer

12. MathBrush ComponentsCharacter Recognizer • Phases of Character Recognition • Input (Strokes) Preprocessing • Stroke joining • Re-sampling • Trimming • Smoothing • Normalization • Segmentation • Heuristic Approach Combine hints to choose the number of strokes in the next symbol • Feature Extraction Approach Compare features and use Confusion Matrix to resolve conflicts Symbols DB Samples for each symbol Bounding boxes + Ranked candidates Ink input from user Character Recognizer

13. MathBrush ComponentsCharacter Recognizer • Phases of Character Recognition • Input (Strokes) Preprocessing • Segmentation • Recognition To produce final confidence the results of the following algorithms are combined • Elastic Matching • Deformable Template Matching • Structural Chain Code Matching

14. MathBrush ComponentsStructural Analyzer • Phases of Structural Analysis • Determine Layout Build an initial baseline tree Ex. • Pre-Parsing Refine candidates using expected mathematical content Ex. matching brackets, matching Integral and dx, Numbers .. Symbols DB Structural & Semantic Types Bounding boxes + Ranked candidates Structural Analyzer Presentation MathML

15. MathBrush ComponentsStructural Analyzer • Phases of Structural Analysis • Determine Layout • Pre-Parsing • Structural Grouping • Use DB information to generate structural confidence • Decide on the best candidate using character and structural confidence • Parsing • Consult a DB of likely expressions to refine the result

16. MathBrush ComponentsCAS Interface Tool Presentation MathML of input expression Math operation to execute CAS Interface Presentation MathML of results CAS to interface with CAS Systems

17. MathBrush ComponentsMath Rendering Tool • Main Features • Rendering Presentation MathML following W3C recommendations • Line breaking • Rendering short form for math expressions Presentation MathML of expression Math Rendering Tool Bounding boxes for display Requested Output format from user

18. Summary • Objective • Builds a pen-based math system while using the power of CAS. • Current Status • MathBrush a preliminary testbed for testing different modules, investigating key issues and getting feedback !! • Main Features • Modular system components • Presentation MathML • Interaction between different platforms (C#, C++, and Java)

19. Future Extensions • Allow the use of more gestures • Allow for more interaction with the results • Support new classes of expressions: matrices .. • Use guided input for better character recognition • Train the structural analyzer parameters • Interface with more math systems

20. DEMO