HCI Issues in eXtreme Computing

1. HCI Issues in eXtreme Computing James A. Landay Endeavour-DARPA Meeting, 9/21/99

2. HCI in the eXtreme Computing Era • Future computing devices won’t have the same UI as current PCs • wide range of devices • small or embedded in environment • often w/ “alternative” I/O & w/o screens • special purpose applications • “information appliances” • lots of devices per user • all working in concert • How does one design for this environment? • UC Berkeley Endeavour Project •

3. Design Challenges • Design of good appliances will be hard • how do you design cross-appliance “applications”? • e.g., calendar app.: one speech based & one GUI based • Hard to make different devices work together • multiple devices, UIs & modes, which to “display”? • How to build UIs for a physical or virtual space? • take advantage of the resources as the user moves • Information overload is a major problem • how to just extract what is relevant? • UC Berkeley Endeavour Project •

4. Key Technologies • Tacit information analysis algorithms • Design tools that integrate • “sketching” & other low-fidelity techniques • immediate context & tacit information • interface models • UC Berkeley Endeavour Project •

5. Our Approach • Evaluate rough prototypes in target domains • learning • high-speed decision making • Build • novel applications on existing appliances • e.g., on the Palm PDA & CrossPad • new information appliances • e.g., SpeechCoder (w/ ICSI) • Evaluate in realistic settings • Iterate • use the resulting experience to build • more interesting appliances • better design tools & analysis techniques • UC Berkeley Endeavour Project •

6. Domains of Focus • Group-based learning • groups of students teach themselves material • “teachers” give structure, diagnose problems, & respond • shown successful outcomes, but doesn’t scale well • key idea: use ubiquitous sensors & activity data to allow • teachers to stay aware of activities as class size scales • groups to find expertise among other groups • Emergency response decision making • respond to fires, earthquakes, floods, hurricanes, ... • quickly allocate resources • situation awareness is paramount • key idea: use activity data to discover & exploit tacit structure • user expertise & information quality • informal work teams & hierarchies • UC Berkeley Endeavour Project •

7. Analyze Tacit Activity: Find People & Info • The real world • who is talking? who are they looking at? what else is happening? • The digital environment • who reads (or writes) what and when? • who communicates with whom and when? with what tools? • Goal: Describe an information ecology • people w/ various expertise, backgrounds & roles • quickly find human experts (e.g., how to restart pumps…) • documents with content, authority, intended audience… • structures: groups, communities, hierarchies, etc. • visualization that provides awareness without overload • feed this information back to the infrastructure • Challenge: recognize/compute from sensor/activity data • UC Berkeley Endeavour Project •

8. Tacit Information Analysis Methods • Social Networks • centrality measures for estimating authority • Clustering • discovering tacit groups, and related documents • UC Berkeley Endeavour Project •

9. Use Context: Improve Interaction • Services to discover available devices • there is a wall display -> use it for my wearable • Choose interaction modes that don’t interfere • UC Berkeley Endeavour Project •

10. Use Context: Improve Interaction • Services to discover available devices • there is a wall display -> use it for my wearable • Choose interaction modes that don’t interfere • context understanding services • people are talking -> don’t rely on speech I/O • user’s hands using tools -> use speech I/O & visual out • use context as a way to search data collected by ubiquitous archiving services -> UI design tools should understand context & support multimodal I/O • UC Berkeley Endeavour Project •

11. Multimodal Interaction • Benefits • take advantage of more than 1 mode of input/output • computers could be used in more situations & places • UIs easier and useful to more people • Building multimodal UIs is hard • often require immature “recognition” technology • single mode toolkits recently appeared (“good enough”) • hard to combine recognition technologies • few toolkits & no prototyping tools -> experts required • this was the state of GUIs in 1980 • UC Berkeley Endeavour Project •

12. Multimodal Design Tools Should Support • Rapid production of “rough cuts” • don’t handle all cases • informal techniques • sketching/storyboarding • “Wizard of Oz” • iterative design • user testing/fast mods • Generate initial code • UIs for multiple devices • designer adds detail & improves interaction • programmers add code • UC Berkeley Endeavour Project •

13. Model Approach: Sketches & Models • Infer models from design “sketches” • model is an abstraction of appliance’s UI design • Use models to • semi-automatically generate UIs • dynamically adapt apps UI to changing context • UC Berkeley Endeavour Project •

14. Specifying UI Elements w/ “Sketches” • UC Berkeley Endeavour Project •

15. Combining the Physical & the Virtual • UC Berkeley Endeavour Project •

16. Combining the Physical & the Virtual • UC Berkeley Endeavour Project •

17. Specifying Non-Visual Elements • How do designers do this now? • speech • scripts or grammars (advanced designers only) • flowcharts on the whiteboard • “Wizard of Oz” -> fake it! • gestures • give an example & then tell programmer what it does • We can do the same by demonstration • UC Berkeley Endeavour Project •

18. Specifying Non-Visual Events (Speech) • UC Berkeley Endeavour Project •

19. Plan for Success • Year 1 • evaluate context-aware prototypes in target domains (op6) • test & refine authority mining algorithms (op5) • Year 2 • design & implement multimodal UI design tool (op7) • implement tacit mining algorithms using sensing data for (op5) • expert locator & query-free retrieval • providing visual awareness of group & task clustering • create new applications using the tools for (op6) • learning • high-speed decision making • Year 3 • evaluate tools & applications • integrate with S/W & H/W design tools • UC Berkeley Endeavour Project •

20. HCI Issues in eXtreme Computing James A. Landay Endeavour-DARPA Meeting, 9/21/99

21. State of the Art • Traditional tools & methodologies (paper, VB, …) • no support for multimodal UIs (especially speech) • do not allow targeting one app to platforms w/ varying I/O capabilities (assume like a PC) • Model-based design tools • force designers to think abstractly about design • Context-aware widgets • how do devices communicate high-level contexts? • XML or UIML • still need to understand what should be expressed • UC Berkeley Endeavour Project •

22. In-Class Group Learning • Participatory learning: Students work in groups of 4-7; communicate via pen or keyboard chat • each group has one main note-taker; others add their own comments or questions to the transcript • students can mark up a group transcript, the lecturer’s notes, or a private window • one student per group works as facilitator or TA, posing questions to the others • UC Berkeley Endeavour Project •

23. Emergency Decision-Making • Tacit activity mining (from ubiquitous sensing) • determines where people are, what they are working on, what they know, etc. • quickly find human experts (e.g., how to restart pumps…) • automatic authority mining (quality of information) • visualization that provides awareness without overload • Challenge is to recognize and compute structure • we borrow ideas from social network theory • UC Berkeley Endeavour Project •