The Vision and Reality of Ubiquitous Computing

The Vision and Reality of Ubiquitous Computing Prof. Henning Schulzrinne Dept. of Computer Science Columbia University (with Arezu Moghadam, Ron Shacham, Suman Srinivasan,Xiaotao Wu and other IRT members; parts in cooperation with DoCoMo Eurolabs) Mobiquitous 2007

Overview • The original vision of ubiquitous computing • User challenges • Beyond terminal mobility • Location as new core service • Universality: 7DS Mobiquitous 2007

Ubiquitous computing  ubiquitous communications • “It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.” • Weiser’s original vision (“Nomadic Issues in Ubiquitous Computing”, 1996) • “one person, many computers” • many computers embedded in environment • dynamic ownership • PC phonebooth • “IR use will grow rapidly” • Updated version, 2007 • not physically invisible, but transparent • emphasis on communications, not computing • most devices are mobile (or nomadic) • cheap electronics personal devices • radio (channelized and UWB) Mobiquitous 2007

User challenges vs. research challenges • Are we addressing real user needs? • Engineering vs. sports • My guesses ease of use no manual reliability no re-entry no duplication integration phishing data loss cost limited risk Mobiquitous 2007

Example: Email configuration • Application configuration for (mobile) devices painful • SMTP port 25 vs. 587 • IMAP vs. POP • TLS vs. SSL vs. “secure authentication” • Worse for SIP... Mobiquitous 2007

Example: SIP configuration partially explains • highly technical parameters, with differing names • inconsistent conventions for user and realm • made worse by limited end systems (configure by multi-tap) • usually fails with some cryptic error message and no indication which parameter • out-of-box experience not good Mobiquitous 2007

Mobile why’s • Not research, but examples of real annoyances • Why does each mobile device need its own power supply? • Why do I have to adjust the clock on my camera each time I travel? • Why do I have to know what my IMAP server is and whether it uses TLS or SSL? • Why do I have to type in my address book? • Why do I have to “synchronize” my PDA? • Why do I have to manually update software? • Why is connecting a laptop to a projector a gamble? • Why do we use USB memory sticks when all laptops have 802.11b? Mobiquitous 2007

Consumer wireless & mobile devices Prius key Garage door opener TAN display Water leak alarm wireless door bell MSN Direct weather Mobiquitous 2007

Mobile systems - reality GPS • idea: special purpose (phone) --> universal communicator • idea is easy... • mobile equipment: laptop + phone • sufficiently different UI and capabilities • we all know the ideal (converged) cell phone • difficulty is not technology, but integration and programmability • (almost) each phone has a different flavor of OS • doesn’t implement all functionality in Java APIs • no dominant vendor (see UNIX/Linux vs. Microsoft) • external interfaces crippled or unavailable • e.g., phone book access location data Mobiquitous 2007

Example: displays and speakers Mobiquitous 2007

The mobile ubiquitous challenge Mobile phone Mobile Internet access Interconnected devices “Internet of things” Mobiquitous 2007

What do we need? • Standards, not new technology • Radio connectivity • 802.11a/b/g/n, 802.15.4  • better discovery of networks • Location information everywhere • Discovery: devices & services • network-local discovery via Bonjour (mDNS)  • missing: location-based discovery • Advanced mobility: session, personal, service • Event notification • Data formats • location, sensor events, ... Mobiquitous 2007

Examples of “invisible” behavior • MP3 player in car automatically picks up new files in home server • A new email with vcard attachment automatically updates my cell phone address book • The display of my laptop appears on the local projector • without cable or configuration • I can call people I just met at Mobiquitous • without exchanging business cards • My car key opens my front door • My cell phone serves as a TAN (one-time password) generator • My cell phone automatically turns itself off during a lecture • My camera knows where the picture was taken Mobiquitous 2007

An interconnected system opens doors generates TAN incoming call updates location time, location address book alert, events any weather service school closings acoustic alerts Mobiquitous 2007

Thinking beyond 802.11 and UMTS • Many interesting networks beyond those covered in conferences • ease of access by researchers vs. importance • 90% of papers on 802.11b and maybe GPRS, BlueTooth • New wireless networks • broadcast instead of unicast -- useful for many ubiquitous applications • S5 for low-rate sensors (city scale) • Zigbee (802.15.4) for local sensors (20 - 250 kb/s) • FM subcarrier (not really new) -- MSN Direct • FM Radio Data System -- TMC • Sirius / XM • HD radio • paging Mobiquitous 2007

Application-layer mobility • terminal mobility • one terminal, multiple network addresses • Personal mobility • one person, multiple terminals • e.g., Grandcentral • session mobility • one user, multiple terminals in sequence or in parallel • service mobility • services move with user Mobiquitous 2007

Session mobility • Walk into office, switch from cell phone to desk phone • call transfer problem  SIP REFER • related problem: split session across end devices • e.g., wall display + desk phone + PC for collaborative application • assume devices (or stand-ins) are SIP-enabled • third-party call control R. Shacham, H. Schulzrinne, S. Thakolsri, W. Kellerer, “Ubiquitous device personalization and use:The next generation of IP multimedia communications”, ACM TOMCCAP, May 2007 Mobiquitous 2007

How to find services? • Two complementary developments: • smaller devices carried on user instead of stationary devices • devices that can be time-shared • large plasma displays • projector • hi-res cameras • echo-canceling speaker systems • wide-area network access • Need to discover services in local environment • SLP (Service Location Protocol) allows querying for services • “find all color displays with at least XGA resolution” • slp://example.com/SrvRqst?public?type=printer • SLP in multicast mode • SLP in DA mode • Apple Bonjour • Need to discover services before getting to environment • “is there a camera in the meeting room?” • SLP extension: find remote DA via DNS SRV • LoST to find services by geographic location Mobiquitous 2007

Session mobility Local Devices Transcoder Internet SLP DA SLP UA SLP SA SIP SM SIP UA SIP UA Correspondent Node (CN) SLP SIP RTP SIP SM SIP UA SLP UA Mobiquitous 2007 Mobile Node (MN)

Context-aware communication • context = “the interrelated conditions in which something exists or occurs” • anything known about the participants in the (potential) communication relationship • both at caller and callee Mobiquitous 2007

GEOPRIV and SIMPLE architectures rule maker DHCP XCAP (rules) target location server location recipient notification interface publication interface GEOPRIV SUBSCRIBE presentity presence agent watcher SIP presence PUBLISH NOTIFY caller callee SIP call INVITE INVITE Mobiquitous 2007

Presence data architecture presence sources PUBLISH raw presence document privacy filtering create view (compose) depends on watcher XCAP select best source resolve contradictions XCAP privacy policy composition policy (not defined yet) draft-ietf-simple-presence-data-model Mobiquitous 2007

Presence data architecture candidate presence document raw presence document post-processing composition (merging) watcher filter remove data not of interest SUBSCRIBE difference to previous notification final presence document watcher NOTIFY Mobiquitous 2007

Presence data model “calendar” “cell” “manual” person (presentity) (views) alice@example.com audio, video, text r42@example.com video services devices Mobiquitous 2007

RPID = rich presence • Provide watchers with better information about the what, where, how of presentities • facilitate appropriate communications: • “wait until end of meeting” • “use text messaging instead of phone call” • “make quick call before flight takes off” • designed to be derivable from calendar information • or provided by sensors in the environment • allow filtering by “sphere” – the parts of our life • don’t show recreation details to colleagues Mobiquitous 2007

Rich presence • More information: for (authorized) people and applications • automatically derived from • sensors: physical presence, movement • electronic activity: calendars • Rich information: • multiple contacts per presentity • device (cell, PDA, phone, …) • service (“audio”) • activities, current and planned • sphere (home vs. work) • current user mood • surroundings (noise, privacy, vehicle, …) • contact information • composing (typing, recording audio/video IM, …) Mobiquitous 2007

Presence and privacy • All presence data, particularly location, is highly sensitive • Basic location object (PIDF-LO) describes • distribution (binary) • retention duration • Policy rules for more detailed access control • who can subscribe to my presence • who can see what when <tuple id="sg89ae"> <status> <gp:geopriv> <gp:location-info> <gml:location> <gml:Point gml:id="point1“ srsName="epsg:4326"> <gml:coordinates>37:46:30N 122:25:10W </gml:coordinates> </gml:Point> </gml:location> </gp:location-info> <gp:usage-rules> <gp:retransmission-allowed>no </gp:retransmission-allowed> <gp:retention-expiry>2003-06-23T04:57:29Z </gp:retention-expiry> </gp:usage-rules> </gp:geopriv> </status> <timestamp>2003-06-22T20:57:29Z</timestamp> </tuple> Mobiquitous 2007

Events as missing Internet capability • aka PUB/SUB • Used across applications, e.g., • email and voicemail notification • presence • replace RSS (= poll!) • web service completion • emergency alerts (“reverse 9-1-1”) • network management • home control • data synchronization • Rich research history • but too complex, optimize the wrong thing • XMPP and SIP as likely transport candidates Mobiquitous 2007

Local Switch Automatic Number Identification Automatic Location Identification Collaboration between local phone providers and local public safety agencies Mobiquitous 2007

911 technology failures • NY Times (“An S O S for 911 Systems in Age of High-Tech”), 4/6/07: • “40% of … counties, most of them rural or small-town …, cannot yet pinpoint the location of the cellphone callers, though the technology to do so has been available for at least five years.”“In … Okmulgee, Okla., last November, 4-year-old Graciella Mathews-Tiger died in a house fire after a 911 operator who lacked the technology to pinpoint the call misheard the address.” • Phase II wireless; billions of dollars spent • In Mississippi, only 1 of out 5 counties • “As it ages, it is cracking, with problems like system overload, understaffing, misrouted calls and bug-ridden databases leading to unanswered calls and dangerous errors.” • operator (CAMA) trunks, with 8-digit number delivery • MSAG and ALI databases Mobiquitous 2007

Location delivery GPS HELD HTTP wire map DHCP LLDP-MED Mobiquitous 2007

Location, location, location, ... Voice Service Provider (VSP) sees emergency call but does not know caller location ISP/IAP knows user location but does not handle call Mobiquitous 2007

Options for location delivery • Wireless • GPS • S5 wireless (active sensors + triangulation) • Skyhook (802.11 in urban areas) • HDTV • L2: LLDP-MED (standardized version of CDP + location data) • periodic per-port broadcast of configuration information • L3: DHCP for • geospatial (RFC 3825) • civic (RFC 4676) • L7: proposals for retrievals: HELD, SIP, … • for own IP address or by third party (e.g., ISP to infrastructure provider) • by IP address • by MAC address • by identifier (conveyed by DHCP or PPP) • HTTP-based Mobiquitous 2007

Locating Caller using LLDP-MED LLDP-MED stands for: * Link Layer Discovery Protocol “a vendor-neutral Layer 2 protocol that allows a network device to advertise its identity and capabilities on the local network.” Media Endpoint Discovery “an enhancement to the LLDP that allows discovery of other things including location“ * From Wikipedia “I am LLDP-MED Capable. I can process location information.” “Your location is: 500 W 120TH st. New York NY 10027” Mobiquitous 2007

Location determination options Mobiquitous 2007

SIP message for Location Info. ------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY= MIME-Version: 1.0 Content-Type: application/pidf+xml Content-Transfer-Encoding: 8bit <?xml version="1.0" encoding="ISO-8859-1"?> <presence xmlns="urn:ietf:params:xml:ns:pidf" xmlns:gp="urn:ietf:params:xml:ns:pidf:geopriv10" xmlns:cl=" urn:ietf:params:xml:ns:pidf:geopriv10:civilLoc" xmlns:gml="urn:opengis:specification:gml:schema-xsd:feature:v3.0" entity="sip:calltaker_ny2@irt.cs.columbia.edu"> <tuple id="28185"> <status> <gp:geopriv> <gp:location-info> <cl:civilAddress> <cl:country>us</cl:country> <cl:A1>ny</cl:A1> <cl:A2>new york</cl:A2> <cl:A3>new york</cl:A3> <cl:A6>amsterdam</cl:A6> <cl:HNO>1214</cl:HNO> </cl:civilAddress> </gp:location-info> <gp:method>Manual</gp:method> </gp:geopriv> </status> <contact priority="0.8">sip:eddie@160.39.54.70:5060</contact> <timestamp>2005-09-26T15:57:34-04:00</timestamp> </tuple> </presence> ------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY=-- INVITE urn:service:sos SIP/2.0 request line To: urn:service:sos Call-ID: 763782461@192.168.1.106 Via: SIP/2.0/TCP 192.168.1.106:4064;rport Content-Type: multipart/mixed; boundary From: sip:caller@irt.cs.columbia.edu Contact: <sip:eddie@160.39.54.70:5060> CSeq: 1 INVITE Content-Length: 1379 header fields ------ =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY= MIME-Version: 1.0 content-Type: application/sdp Content-Transfer-Encoding: 8bit v=0 o=eddie 1127764654 1127764654 IN IP4 192.168.1.106 s=SIPC Call c=IN IP4 160.39.54.70 t=0 0 m=audio 10000 RTP/AVP 0 3 m=video 20000 RTP 31 SDP PIDF-LO Mobiquitous 2007

Mobiquitous 2007

Tracking Mobiquitous 2007

Data formats: location • Civic (street) • jurisdictional & postal • Geo (longitude & latitude) • point, polygon, circle, … • see GeoRSS for simple example Mobiquitous 2007

Civic as well as geospatial queries civic address validation Recursive and iterative resolution Fully distributed and hierarchical deployment can be split by any geographic or civic boundary same civic region can span multiple LoST servers Indicates errors in civic location data  debugging but provides best-effort resolution Can be used for non-emergency services: directory and information services pizza delivery services, towing companies, … ECRIT: LoST Functionality <findService xmlns="urn:…:lost1"> <location profile="basic-civic"> <civicAddress> <country>Germany</country> <A1>Bavaria</A1> <A3>Munich</A3> <A6>Neu Perlach</A6> <HNO>96</HNO> </civicAddress> </location> <service>urn:service:sos.police</service> </findService> Mobiquitous 2007

LoST: Location-to-URL Mapping VSP1 cluster serving VSP1 replicate root information cluster serves VSP2 123 Broad Ave Leonia Bergen County NJ US LoST root nodes NJ US NY US sip:psap@leonianj.gov search referral Bergen County NJ US Leonia NJ US Mobiquitous 2007

LoST Architecture G tree guide G G G broadcast (gossip) T1: .us T2: .de G resolver T2 (.de) seeker 313 Westview Leonia, NJ US T3 (.dk) T1 (.us) Leonia, NJ  sip:psap@leonianj.gov Mobiquitous 2007

Left to do: event notification • notify (small) group of users when something of interest happens • presence = change of communications state • email, voicemail alerts • environmental conditions • vehicle status • emergency alerts • kludges • HTTP with pending response • inverse HTTP --> doesn’t work with NATs • Lots of research (e.g., SIENA) • IETF efforts starting • SIP-based • XMPP Mobiquitous 2007

Problems with Wide Area Wireless • 802.11 currently hard to deploy across city or large area • Problem: How can mobile devices / gadgets get information while on the move? • Use local peer-to-peer wireless networks to exchange information • Peers can get information they do not have from another peer Solution: 7DS! Mobiquitous 2007 S. Srinivasan, A. Moghadam, S.G. Hong, H. Schulzrinne, “7DS - Node Cooperation and Information Exchange in Mostly Disconnected Networks”, ICC '07. June 2007.

How 7DS Works • When devices are in the same BSS (Basic service set) of 802.11 ad-hoc network, they discover each other using service discovery of Zeroconf zeroconf Mobiquitous 2007

How 7DS Works • If there is no Internet connection, the devices can communicatewith each other to exchange information Internet Mobiquitous 2007

The Vision and Reality of Ubiquitous Computing