Ubiquitous Computing at Intel Research Seattle: The Where, What, & How

1. Ubiquitous Computing at Intel Research Seattle: The Where, What, & How James A. Landay Director, Intel Research Seattle Professor, University of Washington February 9, 2004 @ GVU Center, Georgia Tech

2. Intel Research Seattle Overview • Mission: Develop & evaluate new usage models, applications, & underlying technology for ubiquitous computing • Current project focus on location-enhanced computing & human activity inferencing • Size: 12 researchers, growing to 20 full-time researchers • community of 35 (w/ interns, visitors, & campus collaborators) • Work in collaboration with the University of Washington • Universities the “Radar” in Intel Research’s Open Innovation Paradigm • Intel + free flow of university research (e.g., interns) • Mostly “open collaborative research” • Also collaborating with labs in Berkeley, Pittsburgh, & Cambridge, England

3. Location Stack Middleware for multi-sensor location Explicit representation of uncertainty Statistically combines GPS & WiFi beacons for “best of both worlds” indoor & outdoor positioning Particle filters unify the error models Ph.D. work of Jeff Hightower, UW Place Lab Lowering cost & barriers to ubiquitous location on the Web Positions using existing ($0) RF beacon infrastructure (e.g. Wi-Fi) eventually + GPS at additional $$ Privacy managed by personal device doing client positioning Collaborating with Universities to location enhance their campuses Location-Enhanced ComputingWhere are people’s devices? = + High density of Wi-Fi in cities (Seattle at left) plus powerful personal devices (middle) makes private, client-side positioning possible opening the way to the Location-Enhanced WWW (right)

4. Human Activity InferencingWhat are people doing? • Guide • Combines object-identification with machine learning to infer human activities • Glove-based sensor records use of RFID tagged objects (eventually use a bracelet) • Mines web to build models about the world • e.g., what objects are used in brushing teeth • Seeing objects & ordering gives activity • Example Application: CareFrame • A picture frame UI that helps inform members of an elder’s care network about specific events & activities of the elder • e.g., mom knitted today or took her medication • motivated by Mynatt, et. al.’s Digital Family Portrait

5. Overview • Introduction to Intel Research Seattle • Place Lab • Human Activity Inferencing • Design Pre-Patterns for Ubicomp

6. I am here The Goal of Place Lab • Enable widely available location-enhanced computing • Build a positioning system that is • Wide-scale, indoor & outdoor • Can be used everywhere • Privacy observant, low barrier to participation • Can be used by everyone

7. How Will it Work? • Exploit wide-scale WiFi deployment • Urban areas have dense WiFi coverage • WiFi base stations broadcast unique IDs • Position WiFi devices using map of base-station-IDs to location

8. open closed WEP Building up the Database • Build a global user-contributed RF-location mapping service • Leverage “war drivers” and WiFi clubs • Collect trace logs of regular users

9. Access points ((( ((( ((( ))) ))) ))) The Place Lab Architecture Location-enhanced network services Location-enhanced requests Service data Distributed Mapper Service Store maps of WiFi MAC Address ↔ GPS coordinates WiFi maps End Users Usage logs War-driving logs from Netstumbler, Kismet, Personal server War Drivers

10. Making Location-Enhanced Application Development Easier Program with place rather than location • Coordinates are not meaningful to people • “Hey Joe! You’ll never guess who I bumped into at 47.232, -122.454!” • Give higher level abstraction to developers • Develop techniques for mapping coordinates to meaningful places • E.g., 47.232, -122.454 → {‘bank’, ‘Wells Fargo’, ‘private property’, ‘indoor quiet space’} • Make place a first-class entity in APIs

11. 1500 1st Street Commercial A&M Real Estate Barnes and Noble Bookstore “School hangout” “Quiet place” Moving from Location to Place • Solving this problem requires • Discovering and mapping boundaries of place • Learning place labels • From the WWW • Mine place boundaries using GIS maps • Extract place labels from GIS, Yellow Pages, … • From the RF beacon data • Infer place boundaries from behavior and usage • From the user • Direct entry of place labels “Café”

12. Privacy in Place Lab is Different • Many location-enhanced computing services have a computation trust problem… • “Who can figure out where I am?” • Computation trust in Place Lab: • Eased by client-side computation of location • Clients cache mapping data locally • Clients listen for beacons using passive monitoring (like GPS)

13. Understanding Notions of Place & Privacy • Explore users’ notions of place & privacy with experience sampling method • Give participants WiFi-enabled PDAs and periodically interrupt them • Ask questions of the form: where are you? are you with other people? is it ok to reveal your current location? • Bonus: WiFi traces allow us to evaluate our location-to-place mapping

14. Place Lab Status • Have built version .5 of beacon database infrastructure & cross-platform clients • Distributed early toolkit to graduate course on location-aware computing at UW • Distributing to more campuses in March – get involved! • Planning out user studies for notions of place & privacy (summer intern) • Designing initial application to help push on toolkit & privacy issues (Ambush?)

15. Overview • Introduction to Intel Research Seattle • Place Lab • Human Activity Inferencing • Design Pre-Patterns for Ubicomp

16. Monitoring Human Activity Is… • … an essential aspect of human life • Monitoring others • caregiving, training, managing, directing, surveillance • Monitoring oneself • reminding, prompting • … difficult and expensive • Tedious, intrusive • … not helped by conventional computing techniques • Fine-grained sensor-based reasoning has proved difficult

17. Four scaling challenges: Designing model schema Designing low-level sensing for observables Extracting models Solving models fast activities Activity Recognition observables Object Detection Feature Selection Segmentation Object Detection Feature Selection Segmentation Object Detection Feature Selection Segmentation raw sensor data The Traditional Architecture sensors

18. Four Ideas for Scaling • One simple model schema suffices for many activities • Unconventional sensors can obviate custom low-level sensing • Models can be extracted automatically from text • Sample-based methods can reason about sparse, irregular distributions fast but accurately

19. Making tea Simple, expressive modelExample: Activity Model for Making Tea • Semi-Markov Model (Markov model w/ time durations) w/ linear sequence • Two kinds of observables • Object involvement: touch, proximity, motion • Duration of steps • Simple belief structure

20. Sensors detect observables directly isTouching, isMoving, isNear activities Activity Recognition Activity Recognition observables observables Object Detection Feature Selection Segmentation Object Detection Feature Selection Segmentation Object Detection Feature Selection Segmentation sensors raw signals sensors 2. Avoiding custom low-level sensingDirect-Association Sensing

21. 1 cm Radio Frequency ID (RFID) Sensors Stick RFID tags on objects • 10-45c/tag • postage-stamp size stickers • no battery Read tags with ambient readers • 3cm-6m range • 100mW-25W power • 1-40 tags/second Tags return globally unique ID when read

22. iGlove: Detecting isTouching iReader • Skytek short-range reader • iMote radio • USB rechargeable power supply + copper/mylar antenna Object tagged on grasping surfaces reliably detected

23. 3. Extracting models from textHuman Descriptions of Activities • how-tos e.g. ehow.com • recipes e.g. epicurious.com • training manuals e.g. Intel manufacturing

24. Boil Pasta 1.  Fill a large pot ¾ full with cold water 2.  Place the pot on the stove and turn the burner to High 3.  Wait for the water to boil then add salt … Parse from concrete syntax • Noun-phrase extraction • Object extraction, synonym identification (WordNet) Boil Pasta 1.  pot water 2.  burner pot stove 3.  pasta salt water ... Generate priors using Google <activity> <title>boil pasta</title> <tasklist> <task> <object probability=0.8>pot</object> <object probability=0.9>water</object> </task> ... Mining Models from Text

25. Generating Priors for Object Involvement • Mine priors from the web • Mirror assumptionIf object o co-occurs frequently with activity a, term o co-occurs frequently with term a in human discourse • Google Conditional Probability (GCP) Pr(o | A) = google_count (o . A) / google_count A

26. 4. Fast, Approximate Sample-based SolversSequential Monte-Carlo Solution Initialize: Scatter particles uniformly Update, given observables: • Compute new probabilities of states • Repopulate space by sampling from particles as per new probabilities

27. Evaluation: ADL Form Filling • Tagged real home with 108 tags • 14 subjects each performed 12 ADLs • Used glove-based reader • Recorded tasks as performed • Given trace, automatically recreate sequence of activities

28. Legend General solution, Quantitative* results Point solution, Quantitative results Low-res general solution, Anecdotal results Point solution, Anecdotal results Results 95/84 Precision / recall avg. 88% / 73% (make a claim / something happening)

29. Human Activity Inferencing Status • Underlying machine learning works well • Improving to handle interleaved tasks & improve recognition rates (e.g., better NLP in building models) • Adding support for other sensors • e.g., rough audio to infer social interactions • Designing initial application to help push on toolkit & underlying machine learning • support for anesthesiology training

30. Overview • Introduction to Intel Research Seattle • Place Lab • Human Activity Inferencing • Design Pre-Patterns for Ubicomp

31. Design = Solutions • Design is about finding solutions • unfortunately, designers often reinvent • hard to know how thingswere done before • why things were done a certain way • how to reuse solutions • Norman recently said “… as computer technology moves into other fields, they're repeating the same mistakes. Each time, people think everything is new. It takes about five years to sort that out …”

32. Design Patterns • Design patterns communicate common design problems and solutions • first used in architecture [Alexander] • ex. How to create a beer hall where people socialize?

33. Design Patterns • Not too general & not too specific • use a solution “a million times over, without ever doing it the same way twice” • Design patterns are a shared language • a language for “building and planning towns, neighborhoods, houses, gardens, and rooms.” • ex. BEER HALL is part of a CENTER FOR PUBLIC LIFE… • ex. BEER HALL needs spaces for groups to be alone…

34. A Web of Design Patterns (8) Mosaic of Subcultures Cities & Towns (31) Promenade (33) Night Life (90) Beer Hall Local Gatherings (95) Building Complex Interiors (179) Alcoves (181) The Fire

35. Web Design Patterns • Now used in Web design • Communicate design problems & solutions • how to create navigation bars for finding relevant content… • how to create a shopping cart that supports check out… • how to make sites where people return & buy…

36. NAVIGATION BAR (K2) • Problem: Customers need a structured, organized way of finding the most important parts of your Web site

37. First-level navigation Link to home Second-level navigation NAVIGATION BAR (K2) • Solution • captures essence on how to solve problem

38. Design Pre-Patterns for Ubicomp • Can patterns actually lead design? • Pattern purists will say no – so let’s call ‘em pre-patterns • Can we find patterns from the most popular apps? • Mobile communication via cell phone • CONTEXT SENSISTIVE I/O (D6) – Use appropriate input & output modalities for current environment • e.g., in movie theater don’t use sound – use vibration/visual cues • e.g., in a car use sound • Can we find patterns that are found across several different ubicomp (research) applications?

39. 45 Ubicomp Pre-Patterns in 4 Groups

40. Images

42. Images

43. Ubicomp Pre-Patterns Study • Participants: 9 pairs of designers • 5 w/ high experience ( 6 person-years) • 4 w/ low experience • Two conditions: with & without patterns • patterns groups e-mailed patterns 2 days prior • Tasks • heuristic eval. of location-enhanced bus locator (30 min) • design location-enhanced service for mall customers (80 min) • present design to us (10 min) • Judging • 3 judges watched taped presentation & rated designs on creativity, completeness, & quality

45. Ubicomp Pre-Patterns Study Results • Pattern participants rated usefulness of patterns • 5 out of 8: very useful • 3 out of 8: not useful (main complaint: not enough time to absorb patterns) • Judging results • higher creativity & completeness for patterns groups • higher quality for patterns groups (except high-exp. groups) • not statistically significant (low inter-rater reliability) • Notable observations • designers often communicated ideas by pointing at a pattern • designers w/o patterns revisited & fixed design decisions more • some non-patterns pairs struggled w/ issues patterns address • patterns did not help much with privacy • In process of re-running study w/ slight changes (pre-test & structured judging)

46. Design Pre-Patterns Status • Very new work • Looking for 1-2 interns to help develop this idea further • Possibly develop patterns for the home domain • Can users use patterns to configure their home Ubicomp systems?

47. Conclusions • Intel Research Seattle is working on some of the key problems to Ubicomp • Location-enhanced computing • Human activity inferencing • Privacy in Ubicomp systems • Design patterns for Ubicomp • Our mission includes collaboration with universities • Especially interested in Georgia Tech students for internships & permanent positions

48. Backup Slides

49. Dumbo: Detecting isNear Map tagged objects • Scale to hundreds of objects • <1 m precision desirable Long-range reader is expensive, imprecise (6m range) • Use mobile reader • Simultaneously localize and map • Fuse tag location readings

50. RF/Motion: Detecting isMoving • Monitor received signal strength (RSS) from long-range reader • RSS sensitive to distance, orientation • Confounding factors: • Multiplexing between tags • Occlusion • Coupling with ambient sources 9 ft