Dialog Design - Gesture & Pen Interfaces

1. Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: November 2010.

2. Dialog Styles 1. Command languages 2. WIMP - Window, Icon, Menu, Pointer 3. Direct manipulation 4. Speech/natural language 5. Gesture & pen UI Design - Georgia Tech

3. Many Types of Gesture-based UIs • Purpose • Text input vs. selecting objects and actions on objects • Gestures constrained to 2D surface vs. not constrained (i.e., 3D) • Single vs. multiple points • One finger vs. multi-touch • Sensed via something in/on user’s hand vs. not • Stylus on tablet vs. finger on touchpad • Data glove vs. video for hand gestures • Size of sensing area • iPhone – iPad – tablet computer – table-top – wall - room • Type/scale of user motor control • Finger movements vs. hand movements vs. body movements • Work area used by one user vs. more than one UI Design - Georgia Tech

4. Agenda • Look at several points in the gesture design space • PDAs such as iPhone, iPad, Palm Pilot • Text input with some more general uses • Scroll, resize, rotate • Stylus or fingers (for multi-touch) • Multi-touch, shared work space • 3d Gestures, video recognition UI Design - Georgia Tech

5. PDAs – the New Swiss Army Knife UI Design - Georgia Tech

6. What is a Personal Digital Assistant? • A multi-function, portable device supporting activities such as • Communications by voice, text, video • Listen, read, watch – audio, news, books, video • Web browsing • Calendaring • Gaming • Social networking • Navigating from point A to point B • Managing personal information of all sorts • Etc etc etc • Aka iPhone, iPad – a small but powerful computer with pen or gesture input UI Design - Georgia Tech

7. Personal Digital Asst. (PDA) Been around for a longggg time Palm VII Palm IIIc Handspring Visor Circa 2000 HP Jornada Apple Newton (1993) UI Design - Georgia Tech

8. Contemporary PDAs iPhone Nokia BlackBerry Acer Android Tablet Motorola iPad UI Design - Georgia Tech

9. Interacting with a PDA • Text input methods • Soft keyboard • Free-form ink – no recognition • Stylized characters to facilitate recognition • Superimposed on keyboard • Block printing recognition • Script recognition • Stylus or finger UI Design - Georgia Tech

10. Soft Keyboards Pros/cons of soft keyboards vs. hard keyboards? Finger vs. stylus soft keyboards? UI Design - Georgia Tech

11. Free-form Ink Ink is the data, take as is Human is responsible forunderstanding andinterpretation Like a sketch pad UI Design - Georgia Tech

12. Examples • Digital Ink - CMU • Video, CHI ‘98 • View it at www. …….. • Flatland - Xerox PARC • Video, CHI ‘99 • View it at www. …. UI Design - Georgia Tech

13. Gesture Recognition - Graffiti • Stylized characters to facilitate recognition • Graffiti - Unistroke alphabet on Palm PDA • Unistroke = one stroke UI Design - Georgia Tech

14. Gesture on Keyboard - Cirrin Developed by Jen Mankoff (GT Ph.D -> Berkeley -> CMU) Word-level unistroke technique UIST ‘98 paper Use stylus to go from one letterto the next -> UI Design - Georgia Tech

15. Gesture on Keyboard - Quikwriting Developed by Ken Perlin UIST ‘98 paper UI Design - Georgia Tech

16. Quikwriting p l e Said to be as fast as graffiti, but have to learn more http://mrl.nyu.edu/projects/quikwriting/ UI Design - Georgia Tech

17. Gesture Recognition - Shapewriter Aka Shark Live demo on iPad UI Design - Georgia Tech

18. Gesture Recognition – ShapeWriter/Shark Video UI Design - Georgia Tech

19. Recognizing Block Printing and Script • Recognizing letters and numbers and special symbols • Lots of commercial systems • English, kanji, etc. • Not perfect, but people aren’t either! • People - 96% handprinted single characters • Computer - >97% is really good • OCR (Optical Character Recognition) UI Design - Georgia Tech

20. Recognition Issues • Off-line vs. On-line • Off-line: After all writing is done, speed not an issue, only quality. • Work with either a bit map or vector sequence • On-line: Must respond in real-time - but have richer set of features - acceleration, velocity, pressure • Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity and to correct errors • 1 I l • On-line: show choices or best guess UI Design - Georgia Tech

21. More Issues • Boxed vs. Free-Form input • Sometimes encounter boxes on forms • Printed vs. Cursive • Cursive is much more difficult to impossible • Letters vs. Words • Cursive is easier to do in words vs individual letters, as words create more context • Similar to typo correction on phones and tablets UI Design - Georgia Tech

22. Recognizing Printed Characters UI Design - Georgia Tech

23. Beyond Text: Gestures – single point • Might mean delete • Insert • Paragraph Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system UI Design - Georgia Tech

24. Brown’s Line-o-grammer UI Design - Georgia Tech

25. Beyond Text: Gestures – Multiple Points Mac Trackpad: up to four points of input For power users UI Design - Georgia Tech

26. Pen Use Modes • Inputting text vs. selecting vs. action • On iPhone/iPad • Enter text mode by selecting a text entry box • Single touch => select • Single touch + depress => move action • Multi touch => other actions • Other ways to switch modes? • Mode icon(s) – for instance to draw lines vs. print text • Others with which you are familiar?? UI Design - Georgia Tech

27. Creative Uses of Multi-touch Surfaces • Liquid Text – Craig Tashman • On my Mac • LiquidText | Multitouch Document Manipulation.flv • Jeff Han • http://www.ted.com/talks/jeff_han_demos_his_breakthrough_touchscreen.html • BrailleTouch UI Design - Georgia Tech

28. Wall-size Single Touch => Single User Complete with pie menu  UI Design - Georgia Tech

29. Multi-touch => Multi-User • Multi-touch permits multi-users • Two users can simultaneously operate on two areas or display or on two objects • But can’t determine which user is doing what UI Design - Georgia Tech

30. Wall-size, Multi-Touch => Multi-User UI Design - Georgia Tech

31. Table-size, Multi-Touch: MS Surface Simultaneous user multi-touch No user ID No pressure Modest price UI Design - Georgia Tech

32. Microsoft Surface ‪ UI Design - Georgia Tech

33. Table-size, Multi-Touch: Mitsubishi Diamond Touch Simultaneous user multi-touch Pressure!! User ID!! Expensive!!!! UI Design - Georgia Tech

34. Diamond Touch Video • On my Mac • Diamond Touch CeBit 2011 – YouTube.flv UI Design - Georgia Tech

35. 3D Gestures, Video Recognition • Imaginary • On my Mac • ‪interactive ui in minority report‬‏.flv UI Design - Georgia Tech

36. 3D Gestures, Video Recognition • This one is real • On my Mac • ‪Hands Free 3D_ Second Life Object Editing Demo‬‏.flv UI Design - Georgia Tech

37. A Different Application Signature verification But not with a mouse :) UI Design - Georgia Tech

38. Pen-less Gestures • As in iPhone & iPad – 2D • Minority Report • http://www.youtube.com/watch?v=NwVBzx0LMNQ&feature=related • Toshiba • http://www.youtube.com/watch?v=MDUN01U--jE&feature=fvsr • Multi-touch • http://www.youtube.com/watch?v=ok1n0lwY1ZA&feature=related • Put that there – gesture & speech • http://www.youtube.com/watch?v=RyBEUyEtxQo UI Design - Georgia Tech

39. The End UI Design - Georgia Tech