Pen-Centric Shorthand Interfaces

1. Pen-CentricShorthand Interfaces Charles C. Tappert Seidenberg School of CSIS, Pace University

2. Themes of Presentation • Online Handwriting Recognition and Pen Computing Tutorial • Historical Research – undertaken for the Palm-Xerox Patent Infringement Lawsuit • Recent Research - Enhanced Pen-Centric Shorthand Interfaces can have benefits • DPS dissertation could extend M.S. thesis

3. Enhanced Pen-Centric Shorthand Interfaces • Can use word/phrase shorthand to speed text input • Can provide critical infrastructure for many pen-centric applications • Can enhance natural pen-centric interactions for many applications • Will have greatest impact on the utility of applications running on small mobile devices

4. Part 1: Online (Pen-Centric)Handwriting Recognition • Written Languages and Handwriting Properties • The Fundamental Property of Writing • Handwriting Recognition Difficulties • Online(Pen-Centric) Handwriting Recognition • Onlinemore accurate than Offline Recognition • Online Info Can Complicate Recognition Process • Design Tradeoffs / Design Decisions • Computer Problems in English

5. Written Language and Handwriting Properties • Alphabet • Letters, digits, punctuation, special symbols • Writing is a time sequence of strokes • Stroke – writing from pen down to pen up • Usually complete one character before beginning the next • Spatial order – e.g., in English left to right

6. Fundamental Property of Writing • Differences between different characters are more significant than differences between different drawings of the same character • This makes handwritten communication possible • Can there be exceptions – say, different characters written identically?

7. Fundamental Property of Writingin English • Property holds within subalphabets of uppercase, lowercase, and digits, but not across them • “I”, “l”, and “1” written with single vertical stroke • “O” and “0” written similarly with an oval

8. Handwriting Recognition Difficulties • Shape, size, and slant variation • Similarly shaped characters – U and V • Careless writing • in the extreme, almost illegible writing • Resolving difficult ambiguities requires sophisticatedrecognition algorithms, syntax/semantics

9. Online(Pen-Centric) Handwriting Recognition • Electronic tablets invented in late 1950s • Digitizer and display in separate surfaces • Pen Computers arrived in 1980s • Combined digitizer and display • Brought input and output into one surface • Immediate feedback via electronic ink • Created paper-like interface

10. Tablet Digitizers – Dynamic Information • Pen down – indication of inking • X-Y coordinates as function of time • Sampling rate: 100 points per second • Resolution: 200 points per inch

11. Early Pen-Centric Interface • Different surfaces for input and output • Rand system, about 1959

12. Pen Computers • IBM vision • Paper-like interface, 1992 • Microsoft Tablet PC • Launched, 2001

13. Pen-Centric PDAs • Early Palm Pilot • Palm Tungsten T3 and Sony Clié TH55

14. Online(Pen-Centric) Handwriting Recognition • Machine recognizes the writing as the user writes • Digitizer equipment captures the dynamic information of the writing • Stroke number,order,direction,speed • A stroke is the writing from pen down to pen up

15. Online(Pen-Centric) more accurate than Offline (Static) Recognition • Can use both dynamic and static information • Can often distinguish between similarly shaped characters • E.g., 5 versus S where the 5 is usually written with two strokes and the S with one stroke

16. Online Information Can Complicate Recognition Process • Large number of possible variations • E can be written with one, two, three, or four strokes, and with various stroke orders and directions • A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)

17. Online Information Can Complicate Recognition Process • Other variations

18. Online Information Can Complicate Recognition Process • Segmentation ambiguities • character-within-character problem • lowercase d might be recognized as a cl if drawn with two strokes that are somewhat separated from one another

19. Design Tradeoffs/Decisions • No constraints on the user • Machine recognizes user's normal writing • User severely constrained • Must write in particular style such as handprint • Must write strokes in particular order, direction, and graphical specification

20. English Writing Styles • Handprint • Uppercase – about 2 strokes per letter • Lowercase- about 1 stroke per letter • Cursive Script • Usually less than 1 stroke per letter • Delayed crossing and dotting strokes

21. Computer Problems in English • Constrained Handprint • Printing one symbol per box – form filling • Printing on lines – symbols can touch or overlap • Unconstrained Handprint • No lines and symbols can touch or overlap • Cursive Script • Mixed Printing and Cursive

22. Computer Problems in English

23. Pencept Commercial Product 1980s

24. Part 2Shorthand in Pen-Centric PDAs • Famous Uses of Shorthand • Historical Shorthand Alphabets • Pen-Centric Shorthand Alphabets • Pen-Centric Word/Phrase Shorthand • Allegro/Chatroom Shorthand System • M.S. thesis that could be extended into a DPS dissertation

25. Background • Famous writings throughout history were effectively written in a style of shorthand • Cicero’s orations • Martin Luther’s sermons • Shakespeare’s and George Bernard Shaw’s plays • Samuel Pepys’ diary • Sir Isaac Newton’s notebooks

26. Historical Shorthand Alphabets • We first review the history of shorthand systems prior to pen computing • Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases” • Shorthand can be traced back to the Greeks in 400 B.C.

27. Historical Shorthand Alphabets • We focus on shorthand alphabets that might be appropriate for PDAs • We review two types of shorthand • Geometric shorthand • Small number of basic shapes • Shapes reused in multiple orientations • Non-geometric shorthand shorthand

28. Historical Shorthand Alphabets • Ancient Greeks – 400 BC • Tironian Alphabet – 63 BC • John Willis’s Stenography – 1602 • Gabelsberger Alphabet – 1834 • Moon Alphabet – 1845

29. Tironian Alphabet, 63 B.C.Non-Geometric

30. Stenography Alphabet, 1602

31. Stenography Alphabet, 1602 • Basic Shapes and Orientations

32. Gabelsberger Cursive-Style, 1834 Non-Geometric Alphabet

33. Moon Geometric Alphabet, 1845

34. Other Historical Shorthand Systems • Phonetic alphabets • Pitman (1837), was popular in UK • Gregg (1888), was popular in USA • Systems for the blind • Braille(1821)

35. Pen-Centric Shorthand Alphabets • Some of the earliest were for CAD/CAM • symbols represent graphical items and commands • Others developed for text input on small consumer devices like PDAs that have limited computing power • We review geometric and non-geometric shorthands appropriate for small devices

36. Pen-Centric Shorthand Alphabets • Historical alphabets presented above could be used for machine recognition • symbols drawn with a single stroke (except “K” in Tironian and “+” in Stenography) • In addition to shape and orientation, online systems can use stroke direction to differentiate among symbols

37. Pen-Centric Shorthand Alphabets • Geometric Pen-Centric Shorthands • Organek – 1991 • Allen – filed 1991, patent 1993 • Goldberg (Xerox) – filed 1993, patent 1997 • Non-Geometric Pen-Centric Shorthands • Graffiti (Palm Computing) – 1995 • Allegro (Papyrus) – 1995

38. Organek Alphabet, 1991

39. Organic Alphabet, 1991 Basic Shapes and Orientations One shape in 4 orientations. This gives 8 directions that together with 3 lengths provide 24 symbols. A second wheel provides additional symbols.

40. Allen patent, filed 1991

41. Allen patent, filed 1991 Basic Shapes and Orientations

42. Goldberg patent, filed 1993(“unistroke symbols”)

43. Goldberg patent, filed 1993 Basic Shapes and Orientations

44. Goldberg patent, filed 1993 5 Basic shapes 4 Orientations 2 Stroked Directions 40 Possible Symbols • Designed for Speed of Input and Maximum Symbol Separation

45. Shorthand Alphabet Design • How would you design a shorthand alphabet? • What would be the design criteria?

46. Design of Graffiti Alphabetfor the Palm Pilot • Small alphabet • Uppercase, digits, special symbols • One stroke per symbol to avoid segmentation difficulty • Separate writing areas for letters and digits to avoid same-shape confusions

47. Graffiti Alphabet, 1995

48. Graffiti Mimics Keyboard Input • Character by character input • Mode shifts for • Uppercase • Special characters • Eyes can focus on application’s insertion point rather than on input area

49. Graffiti Alphabet Design • What was the additional design criterion?

50. Graffiti Alphabet Design • Designed for ease of learning • 20 letters exactly match the Roman alphabet • 6 remaining ones match partially