Delimiters for Selection-Action Pen Gesture Phrases in Scriboli

1. Delimiters for Selection-Action Pen Gesture Phrases in Scriboli Ken Hinckley Patrick Baudisch Gonzalo Ramos Francois Guimbretiere Microsoft Research

2. The Traveling (pen) SalesmanProblem • Copy + Paste: 1. Select Lasso Mode

3. The Traveling (pen) SalesmanProblem • 2. Circle objects

4. The Traveling (pen) SalesmanProblem • 3. Invoke Edit Menu

5. The Traveling (pen) SalesmanProblem • 4. Choose Copy Command

6. The Traveling (pen) SalesmanProblem • 5. Invoke Edit Menu again

7. The Traveling (pen) SalesmanProblem • 6. Choose Paste

8. The Traveling (pen) SalesmanProblem • 7. Acquire the copy

9. The Traveling (pen) SalesmanProblem • 8. Drag copy to desired position.

10. The Traveling (pen) SalesmanProblem • The well traveled pen.

11. The Traveling (pen) SalesmanProblem • Waste time moving the pen • Focus attention on small targets • Same steps every time • But what did I forget?

12. The Traveling (pen) SalesmanProblem • 9) Forgot to reselect PEN mode. Oops.

13. Pen Computing Now?

14. Recognize Pen Gestures? • Classic problem: “ink” or “gesture” ? • Hard problem: any ‘gesture’ could be ink • All we need to solve this is ONE BIT of info • Scriboli: Make it a physical skill instead of guessing game • physical button on bezel • nonpreferred hand, no trip to “lasso mode” • Next paper: Li, Hinckley, Guan & Landay • works really well

15. Gestures for Nouns & Verbs • Lasso selection Marking menus • Crossing selection? Abandon lasso selection? • How to combine scope + command efficiently?

16. Delimiters for scope selection + marking • A delimiter is “something different” • lexical structure of stroke • e.g. self-crossing gesture stroke • VIDEO :delimiter techniques

17. Experiment Design • {Pigtail, Handle, Timeout, Button} X • Practice: 32 trials • Main Block: • All 8 Marking Directions (N, NE, …) X • 2 Selection Types (single, or row/column) X • 5 trials • = 80 main block trials • Repeated Invocation (RI) Block • Lasso one item, select E from menu 24x

18. Results – Completion Time • Left bar – main block; right bar – repeated block • Dotted bars are Pigtail2 design iteration Button Handle Pigtail Pigtail2 Timeout

19. Results – Learning Effects • For the 24 trials of Repeated Invocation Block

20. Results – Error Rate • Error rate of selecting wrong marking direction Button Handle Pigtail Pigtail2 Timeout

21. Improving Pigtail • Problems • 1. Pigtail Size • No upper bound

22. Pigtail Design Iteration • Problems • 1. Pigtail Size • No upper bound • 2. Pretzels

23. Pigtail Design Iteration • Problems • 1. Size Limit • A<5600 no good • 2. Pretzels • 1st intersection? No.

24. Pigtail Design Iteration • Problems • 1. Size Limit • A<5600 no good • 2. Pretzels • 1st intersection? No. • 2nd intersection? Yep.

25. Pigtail2 Preliminary Results • No errors at all in Repeated Invocation block! Button Handle Pigtail Pigtail2 Timeout

26. Scriboli DEMO • Direct manipulation phase • Scope, command, & manip in one stroke • Self-revelation techniques • Disjoint Selections • Lassos, Crossing, Tapping • Circles of exclusion

27. Conclusion • Performance data on delimiter techniques • Data suggest Handle has slight advantage • But handle was large at 48x48 pixels • pen up/ pen down + spatial mux • Can’t be used e.g. in paper interface • Pigtail offers design alternative • Performs almost the same, lasso tough case? • Seems to offer some unique design advantages

28. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Future Work • ‘grammar’ of building blocks for pen UI • What are alternative designs for each part? • More complex phrases possible?

29. Questions?

30. Handle vs.Pigtail • Handle can get in the way • Adds some visual clutter • Pigtail handles more than one scope elegantly

31. Handwriting has a Deep Structure • Learning cursive • fast, brief, repetitive, pre-programmed • variations on oscillatory movements • The scribbling of Scriboli

32. Qualitative Results • Favorite: 7 Handle, 4 Pigtail, 4 Timeout • Handle • two strokes {disruptive, natural} • Clutter + box makes part of screen inaccessible • Timeout • good for beginners… but I hate waiting for it. • I felt like I was almost too accurate with it • Pigtail • Felt just like handwriting… once I learned it, I could open up & fly! • Pigtail was trouble. I got a little mad at it. • Takes a little bit of practice.

34. Marking Menus • Blind Prompted • Scale-independent in blind mode (333ms) • Visually-guided in prompted mode

35. Preserves scale-independence • Scale-ind. only matters for blind marking • Dragging naturally lasts longer marking starts visual menu pops up cross outer boundary check for crossing dragging blind phase 333 ms

36. GEdit – Kurtenbach & Buxton Delete one obj. Copy – add final “C” to move Delete group – end stroke inside lasso • One of the first works considering pen phrases • Every stroke is gesture • Only 3 cmds • Group vs. single obj. inconsistent Move – end stroke outside. Objects move on PenUp.

37. FlowMenu – Francois Guimbretiere • Integrates command selection with continuous parameter manipulation • But no “blind marking,” size of gesture matters Crossing inout chooses Item… submenu Crossing outin chooses Move command Dragging continues

38. Design Philosophy of Scriboli • RAPID • Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention • UNAMBIGUOUS • No guessing games • No reco unless I tell the system otherwise • (There is almost no reco in Scriboli) • EXPRESSIVE • Supports many commands, easy to grow design • Mechanisms for a wide span of pen apps, not restricted to specific niche application

39. Other Delimiter Techniques • Handle • 2 strokes • Box makes part of screen inaccessible • Timeout • 500 ms dwell(stroke-and-hold) • Button • Hit when want to bring up menu • too early / too late …

40. Questions?