InkSeine: i n situ search for active note taking

1. Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tan InkSeine: in situ search for active note taking oronto

2. In situ search for digital ink • InkSeine : • Ink + • Seine • -n: a fishing net • -v: to fish with a seine • = insanely cool ink search! • InkSeine lets the user fish for useful information directly from ink notes. Fishermen catching salmon on the Columbia River using a seine.(courtesy Wikipedia)

3. Active note taking • XLibris—pen-based system for active reading(reading + critical thinking + learning) • InkSeine combines pen-based note taking with searching, linking, collecting & sensemaking: ActiveNote Taking

4. Core interaction pattern: Ink-Search-Gather • Ink – excels at capturing user’s thoughts & ideas with minimum distraction from formatting issues Ink Search Gather

5. Ink-Search-Gather interaction pattern • Search – digging through the file system on a tablet is really a pain • Active note taking benefits from low overhead access to supporting materials & related docs • Focus on personal search, but support Web as well Ink Search Gather

6. Ink-Search-Gather interaction pattern • Gather – Make it easy to use what you find by pulling content directly into unstructured notes • Thumbnail hyperlinks to documents • Clippings from documents • Queries that can be revisited later Ink Search Gather

7. Ink-Search-Gather interaction pattern • InkSeine unifies these three elements in a fluid user experience that is integrated directly with inking Ink Search Gather

8. In situ search • Leverage preexisting ink to initiate search • No context switch to “search app” – stay in the flow • No tedious copying of information to a “search box” • Queries as first class objects, commingled with ink notes • Flag some ink for search for later • Visible and salient search history as part of notes • Queries can be copied, pasted, moved, etc. • Interleave inking, searching & gathering • No barrier between inking / searching – note ideas while searching, side-by-side searching, span app boundaries for content gathering • Tightly couple queries with application content • Queries naturally persist in originating context • Easy re-visitation of prior queries • Results of queries become new content

9. Rich tradeoffs in cost structure of sensemaking • Continuum of time cost – can defer at any point: • 1 second: Capture thought “I should search on this” • 5 seconds: Trigger search & see initial results list • 10s: Open a document from result list, or drag result into notes • 10s-30s: • Inspect details of a few results • Apply filters to results • Add a word to the query • Grab a snapshot from a short document • Correct occasional reco errors – but Tablet’s reco is amazingly good! • Longer: sufficient depth for common info needs; not mired by last 10% of complex queries that require full featured search tools • Scroll through longer results lists • Inspect result documents to see if they meet information needs • Sideways searches based on Title, Date, or Author of a result

10. InkSeine system • VIDEO

12. Paper Prototype

13. Paper Prototype

14. Paper Prototype

15. Findings • Existing desktop search tool difficult to use • Difficult text entry • Enter and correct text

16. Findings • Existing desktop search tool difficult to use • Difficult text entry • Enter and correct text • Annoying context switch and flow interruption • Switch among apps and context

17. Findings • Existing desktop search tool difficult to use • Difficult text entry • Enter and correct text • Annoying context switch and flow interruption • Switch among apps and context • Tedious information gathering • Difficult to link to documents and gather pieces of information

18. Notebook Control

19. In Situ Search

20. Breadcrumb Breadcrumb

21. Breadcrumb

22. Trigger Personal Search

23. Search Panel

24. Query Area QueryArea

25. Result List Result List

26. Result List Result List Focus Result

27. File and Timeline Filters Filters

28. Gestures for Filters Cross – Multi-select Tap - Select Cross & Turn Right – Exclude Disjunctive cross – Multi-select

29. Work Flow & Occlusion Selection Correction Search

30. Work Flow & Occlusion

31. Work Flow & Occlusion

32. Crossing Widgets

33. Resize Search Panel

34. Examine Document

35. Document & Tracking Menu

36. Multiple Search Panels

37. First Round Observations • Gesture discovery problem

38. First Round Observations • Gesture discovery problem • Context preservation & occlusion problem

39. First Round Observations • Gesture discovery problem • Context preservation & occlusion problem • Recognition problem for out-of-vocabulary word UIST ? List?, Dist?, Gist?, Mist?, Hist?

40. Second Round Observations • Highlighter hints • Smart positioning of search panel • Personalized indexer

41. Summary • In situ search experience • Optimum workflow / maximum flexibility • Enable rich tradeoffs • Gather content • Minimize search screen real estate • Span application boundaries • Tailored to pen input

42. Future Directions • Pull in materials for searches from notes

43. Future Directions • Pull in materials for searches from notes • Support vector-based searches

44. Questions?