The Vision and Reality of Ubiquitous Computing

1. The Vision and Reality of Ubiquitous Computing Prof. Henning Schulzrinne Dept. of Computer Science Columbia University (with Arezu Moghadam, Ron Shacham, Suman Srinivasan,Xiaotao Wu and other IRT members as well as the IETF GEOPRIV and ECRIT WGs) UCSB

2. Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Beyond connectivity: 7DS • CS technology into reality UCSB

3. Ubiquitous computing  ubiquitous communications • “It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.” • Weiser’s original vision (“Nomadic Issues in Ubiquitous Computing”, 1996) • “one person, many computers” • many computers embedded in environment • dynamic ownership • PC phonebooth • “IR use will grow rapidly” • Updated version, 2007 • not physically invisible, but transparent • emphasis on communications, not computing • most devices are mobile (or nomadic) • cheap electronics personal devices • radio (channelized and UWB) UCSB

4. Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Universality: 7DS • CS technology into reality UCSB

5. User challenges vs. research challenges • Are we addressing real user needs? • Engineering vs. sports scoring • My guesses ease of use no manual reliability no re-entry no duplication integration phishing data loss cost limited risk UCSB

6. Example: Email configuration • Application configuration for (mobile) devices painful • SMTP port 25 vs. 587 • IMAP vs. POP • TLS vs. SSL vs. “secure authentication” • Worse for SIP... UCSB

7. Example: SIP configuration partially explains • highly technical parameters, with differing names • inconsistent conventions for user and realm • made worse by limited end systems (configure by multi-tap) • usually fails with some cryptic error message and no indication which parameter • out-of-box experience not good UCSB

8. Mobile why’s • Not research, but examples of real annoyances • Why does each mobile device need its own power supply? • Why do I have to adjust the clock on my camera each time I travel? • Why do I have to know what my IMAP server is and whether it uses TLS or SSL? • Why do I have to type in my address book? • Why do I have to “synchronize” my PDA? • Why do I have to manually update software? • Why is connecting a laptop to a projector a gamble? • Why do we use USB memory sticks when all laptops have 802.11b? UCSB

9. Consumer wireless & mobile devices Prius key Garage door opener TAN display Water leak alarm SPOT watch wireless door bell MSN Direct weather UCSB

10. Mobile systems - reality GPS • idea: special purpose (phone) --> universal communicator • idea is easy... • mobile equipment: laptop + phone • sufficiently different UI and capabilities • we all know the ideal (converged) cell phone • difficulty is not technology, but integration and programmability • (almost) each phone has a different flavor of OS • doesn’t implement all functionality in Java APIs • no dominant vendor (see UNIX/Linux vs. Microsoft) • external interfaces crippled or unavailable • e.g., phone book access location data UCSB

11. Example: displays and speakers UCSB

12. The mobile ubiquitous challenge Mobile phone Mobile Internet access Interconnected devices “Internet of things” UCSB

13. What do we need? • Standards, not new technology • Radio connectivity • 802.11a/b/g/n, 802.15.4  • better discovery of networks • Location information everywhere • Discovery: devices & services • network-local discovery via Bonjour (mDNS)  • missing: location-based discovery • Advanced mobility: session, personal, service • Event notification • Data formats • location, sensor events, ... UCSB

14. Examples of “invisible” behavior • MP3 player in car automatically picks up new files in home server • A new email with vcard attachment automatically updates my cell phone address book • The display of my laptop appears on the local projector • without cable or configuration • I can call people I just met at UCSB • without exchanging business cards • My car key opens my front door • My cell phone serves as a TAN (one-time password) generator • My cell phone automatically turns itself off during a lecture • My camera knows where the picture was taken UCSB

15. An interconnected system opens doors generates TAN incoming call updates location time, location address book alert, events any weather service school closings acoustic alerts UCSB

16. Thinking beyond 802.11 and UMTS • Many interesting networks beyond those covered in conferences • ease of access by researchers vs. importance • 90% of papers on 802.11b and maybe GPRS, BlueTooth • New wireless networks • broadcast instead of unicast -- useful for many ubiquitous applications • S5 for low-rate sensors (city scale) • Zigbee (802.15.4) for local sensors (20 - 250 kb/s) • FM subcarrier (not really new) -- MSN Direct • FM Radio Data System -- TMC • Sirius / XM • HD radio • paging UCSB

17. Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Universality: 7DS • CS technology into reality UCSB

18. Application-layer mobility • terminal mobility • one terminal, multiple network addresses • Personal mobility • one person, multiple terminals • e.g., Grandcentral • session mobility • one user, multiple terminals in sequence or in parallel • service mobility • services move with user UCSB

19. Session mobility • Walk into office, switch from cell phone to desk phone • call transfer problem  SIP REFER • related problem: split session across end devices • e.g., wall display + desk phone + PC for collaborative application • assume devices (or stand-ins) are SIP-enabled • third-party call control UCSB

20. How to find services? • Two complementary developments: • smaller devices carried on user instead of stationary devices • devices that can be time-shared • large plasma displays • projector • hi-res cameras • echo-canceling speaker systems • wide-area network access • Need to discover services in local environment • SLP (Service Location Protocol) allows querying for services • “find all color displays with at least XGA resolution” • slp://example.com/SrvRqst?public?type=printer • SLP in multicast mode • SLP in DA mode • Apple Bonjour • Need to discover services before getting to environment • “is there a camera in the meeting room?” • SLP extension: find remote DA via DNS SRV • LoST to find services by geographic location UCSB

21. Session mobility Local Devices Transcoder Internet SLP DA SLP UA SLP SA SIP SM SIP UA SIP UA Correspondent Node (CN) SLP SIP RTP SIP SM SIP UA SLP UA UCSB Mobile Node (MN)

22. Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Universality: 7DS • CS technology into reality UCSB

23. Context-aware communication • context = “the interrelated conditions in which something exists or occurs” • anything known about the participants in the (potential) communication relationship • both at caller and callee UCSB

24. GEOPRIV and SIMPLE architectures rule maker DHCP XCAP (rules) target location server location recipient notification interface publication interface GEOPRIV SUBSCRIBE presentity presence agent watcher SIP presence PUBLISH NOTIFY caller callee SIP call INVITE INVITE UCSB

25. Presence data architecture presence sources PUBLISH raw presence document privacy filtering create view (compose) depends on watcher XCAP select best source resolve contradictions XCAP privacy policy composition policy (not defined yet) draft-ietf-simple-presence-data-model UCSB

26. Presence data architecture candidate presence document raw presence document post-processing composition (merging) watcher filter remove data not of interest SUBSCRIBE difference to previous notification final presence document watcher NOTIFY UCSB

27. Presence data model “calendar” “cell” “manual” person (presentity) (views) alice@example.com audio, video, text r42@example.com video services devices UCSB

28. RPID = rich presence • Provide watchers with better information about the what, where, how of presentities • facilitate appropriate communications: • “wait until end of meeting” • “use text messaging instead of phone call” • “make quick call before flight takes off” • designed to be derivable from calendar information • or provided by sensors in the environment • allow filtering by “sphere” – the parts of our life • don’t show recreation details to colleagues UCSB

29. Rich presence • More information: for (authorized) people and applications • automatically derived from • sensors: physical presence, movement • electronic activity: calendars • Rich information: • multiple contacts per presentity • device (cell, PDA, phone, …) • service (“audio”) • activities, current and planned • sphere (home vs. work) • current user mood • surroundings (noise, privacy, vehicle, …) • contact information • composing (typing, recording audio/video IM, …) UCSB

30. Presence and privacy • All presence data, particularly location, is highly sensitive • Basic location object (PIDF-LO) describes • distribution (binary) • retention duration • Policy rules for more detailed access control • who can subscribe to my presence • who can see what when <tuple id="sg89ae"> <status> <gp:geopriv> <gp:location-info> <gml:location> <gml:Point gml:id="point1“ srsName="epsg:4326"> <gml:coordinates>37:46:30N 122:25:10W </gml:coordinates> </gml:Point> </gml:location> </gp:location-info> <gp:usage-rules> <gp:retransmission-allowed>no </gp:retransmission-allowed> <gp:retention-expiry>2003-06-23T04:57:29Z </gp:retention-expiry> </gp:usage-rules> </gp:geopriv> </status> <timestamp>2003-06-22T20:57:29Z</timestamp> </tuple> UCSB

31. Events as missing Internet capability • aka PUB/SUB • Used across applications, e.g., • email and voicemail notification • presence • replace RSS (= poll!) • web service completion • emergency alerts (“reverse 9-1-1”) • network management • home control • data synchronization • Rich research history • but too complex, optimize the wrong thing • XMPP and SIP as likely transport candidates UCSB

32. Local Switch Automatic Number Identification Automatic Location Identification Collaboration between local phone providers and local public safety agencies UCSB

33. 911 technology failures • NY Times (“An S O S for 911 Systems in Age of High-Tech”), 4/6/07: • “40% of … counties, most of them rural or small-town …, cannot yet pinpoint the location of the cellphone callers, though the technology to do so has been available for at least five years.”“In … Okmulgee, Okla., last November, 4-year-old Graciella Mathews-Tiger died in a house fire after a 911 operator who lacked the technology to pinpoint the call misheard the address.” • Phase II wireless; billions of dollars spent • In Mississippi, only 1 of out 5 counties • “As it ages, it is cracking, with problems like system overload, understaffing, misrouted calls and bug-ridden databases leading to unanswered calls and dangerous errors.” • operator (CAMA) trunks, with 8-digit number delivery • MSAG and ALI databases UCSB

34. Location delivery GPS HELD HTTP wire map DHCP LLDP-MED UCSB

35. Location, location, location, ... Voice Service Provider (VSP) sees emergency call but does not know caller location ISP/IAP knows user location but does not handle call UCSB

36. Options for location delivery • Wireless • GPS • S5 wireless (active sensors + triangulation) • Skyhook (802.11 in urban areas) • HDTV • L2: LLDP-MED (standardized version of CDP + location data) • periodic per-port broadcast of configuration information • L3: DHCP for • geospatial (RFC 3825) • civic (RFC 4676) • L7: proposals for retrievals: HELD, SIP, … • for own IP address or by third party (e.g., ISP to infrastructure provider) • by IP address • by MAC address • by identifier (conveyed by DHCP or PPP) • HTTP-based UCSB

37. Locating Caller using LLDP-MED LLDP-MED stands for: * Link Layer Discovery Protocol “a vendor-neutral Layer 2 protocol that allows a network device to advertise its identity and capabilities on the local network.” Media Endpoint Discovery “an enhancement to the LLDP that allows discovery of other things including location“ * From Wikipedia “I am LLDP-MED Capable. I can process location information.” “Your location is: 500 W 120TH st. New York NY 10027” UCSB

38. Location determination options UCSB

39. SIP message for Location Info. ------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY= MIME-Version: 1.0 Content-Type: application/pidf+xml Content-Transfer-Encoding: 8bit <?xml version="1.0" encoding="ISO-8859-1"?> <presence xmlns="urn:ietf:params:xml:ns:pidf" xmlns:gp="urn:ietf:params:xml:ns:pidf:geopriv10" xmlns:cl=" urn:ietf:params:xml:ns:pidf:geopriv10:civilLoc" xmlns:gml="urn:opengis:specification:gml:schema-xsd:feature:v3.0" entity="sip:calltaker_ny2@irt.cs.columbia.edu"> <tuple id="28185"> <status> <gp:geopriv> <gp:location-info> <cl:civilAddress> <cl:country>us</cl:country> <cl:A1>ny</cl:A1> <cl:A2>new york</cl:A2> <cl:A3>new york</cl:A3> <cl:A6>amsterdam</cl:A6> <cl:HNO>1214</cl:HNO> </cl:civilAddress> </gp:location-info> <gp:method>Manual</gp:method> </gp:geopriv> </status> <contact priority="0.8">sip:eddie@160.39.54.70:5060</contact> <timestamp>2005-09-26T15:57:34-04:00</timestamp> </tuple> </presence> ------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY=-- INVITE urn:service:sos SIP/2.0 request line To: urn:service:sos Call-ID: 763782461@192.168.1.106 Via: SIP/2.0/TCP 192.168.1.106:4064;rport Content-Type: multipart/mixed; boundary From: sip:caller@irt.cs.columbia.edu Contact: <sip:eddie@160.39.54.70:5060> CSeq: 1 INVITE Content-Length: 1379 header fields ------ =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY= MIME-Version: 1.0 content-Type: application/sdp Content-Transfer-Encoding: 8bit v=0 o=eddie 1127764654 1127764654 IN IP4 192.168.1.106 s=SIPC Call c=IN IP4 160.39.54.70 t=0 0 m=audio 10000 RTP/AVP 0 3 m=video 20000 RTP 31 SDP PIDF-LO UCSB

40. UCSB

41. Tracking UCSB

42. Data formats: location • Civic (street) • jurisdictional & postal • Geo (longitude & latitude) • point, polygon, circle, … • see GeoRSS for simple example UCSB

43. Civic as well as geospatial queries civic address validation Recursive and iterative resolution Fully distributed and hierarchical deployment can be split by any geographic or civic boundary same civic region can span multiple LoST servers Indicates errors in civic location data  debugging but provides best-effort resolution Can be used for non-emergency services: directory and information services pizza delivery services, towing companies, … ECRIT: LoST Functionality <findService xmlns="urn:…:lost1"> <location profile="basic-civic"> <civicAddress> <country>Germany</country> <A1>Bavaria</A1> <A3>Munich</A3> <A6>Neu Perlach</A6> <HNO>96</HNO> </civicAddress> </location> <service>urn:service:sos.police</service> </findService> UCSB

44. LoST: Location-to-URL Mapping VSP1 cluster serving VSP1 replicate root information cluster serves VSP2 123 Broad Ave Leonia Bergen County NJ US LoST root nodes NJ US NY US sip:psap@leonianj.gov search referral Bergen County NJ US Leonia NJ US UCSB

45. LoST Architecture G tree guide G G G broadcast (gossip) T1: .us T2: .de G resolver T2 (.de) seeker 313 Westview Leonia, NJ US T3 (.dk) T1 (.us) Leonia, NJ  sip:psap@leonianj.gov UCSB

46. Left to do: event notification • notify (small) group of users when something of interest happens • presence = change of communications state • email, voicemail alerts • environmental conditions • vehicle status • emergency alerts • kludges • HTTP with pending response • inverse HTTP --> doesn’t work with NATs • Lots of research (e.g., SIENA) • IETF efforts starting • SIP-based • XMPP UCSB

47. Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Universality: 7DS • CS technology into reality UCSB

48. Problems with Wide Area Wireless • 802.11 currently hard to deploy across city or large area • Problem: How can mobile devices / gadgets get information while on the move? • Use local peer-to-peer wireless networks to exchange information • Peers can get information they do not have from another peer Solution: 7DS! UCSB

49. How 7DS Works • When devices are in the same BSS (Basic service set) of 802.11 ad-hoc network, they discover each other using service discovery of Zeroconf zeroconf UCSB

50. How 7DS Works • If there is no Internet connection, the devices can communicatewith each other to exchange information Internet UCSB