Max Mühlhäuser

Preface: Systematic Approach to Ubiquitous Computing and its Application to Real-Time Enterprises Max Mühlhäuser

P-0: Outline - What this chapter covers P-1 A first introduction • ‘Definition’: What is Ubiquitous Computing? • Importance: (why) is Ubiquitous Computing (UC) so important? First glance P-2 UC challenges – a first approach • The challenge of humane UC • The challenge of integrated UC P-3 The S.C.A.L.E. classification of Ubiquitous Computing • Intro: broad vs. deep: what should a treatment of UC cover – and why • Explanation of & introduction to the S.C.A.L.E classification • Relation to major Computer Science research fields P-4 Reader’s Digest • Relation of S.C.A.L.E. classification to further chapters • Digest for Readers with different background/goals

P-1.1 Motivation & Definition: What is UC? • Ubiquitous Computing describes the 3rd “era” of computing • Era 1: Mainframe Computing MC - many users use one computer • Era 2: Personal Computing PC - one user uses one computer • Era 3: Ubiquitous Computing UC - one user uses many computers • Note, for Era 3: • some computers are worn (mobile phone, PDA; body sensors, life recorder…) • other computers are encountered on-the-move, more or less consciously(cf. info kiosk / data projection versus badge-reader / surveillance cam) • UC means “networked computers everywhere” • µC’s were embedded in VCRs, washing machines etc. since decades already! • BUT: now, with UC, the embedded computers become • networked, i.e. part of a “web” of interacting computers surrounding us • “programmable”: changing firmware / SW is easier, less noticeable to user(good: cf. bug fixes; bad: cf. undesired functionality, e.g. spying)

P-1.1 Definition: What is UC? • UC connects general & special purpose computers • the distinction is more a spectrum than two distinct classes • the power / resource spectrum (CPU, memory, bandwidth, …) becomes huge! How to ‘define’ Ubquitous Computing? A simple definition is given above: UC is the third era of computing according to the aforementioned categorization Our more detailed and self-sustained definition: Ubiquitous Computing is the dawning era of computing, in which individuals are surrounded by many networked, spontaneously yet tightly cooperating computers, some of them worn or carried, some of them encountered on the move, many of them serving dedicated purposes as part of physical objects, all of them used in an intuitive, hardly noticeable way with limited attention”.

P-1.1 Importance: Sample Nodes in UC networks Tiny, rather general purpose computers acting, e.g., as nodes in sensor networks (cf. UC Berkeley’s motes, Particle Computer GmbH’s particles) Wearables and mobile computing devices. cf. handhelds for ware- house picking, washable computer jackets, or companions like the “lovegetty” pictured: stores user’s profile, beeps when a “compatible” person – with lovegetty – appears Smart labels such as RFIDs (above) and active badges(below) identify (option: locate, characterize) objects, humans, animals; benefits and privacy threats widely discussed in press Networked appliances, e.g., “smart vending machines” that transmit fill status, errors etc. to head offices; press rumors about adaptive pricing caused protests

P-2.3 Imporance: Exploding # of nodes log (capacity) Internet #nodes, wired speed 1010 Appli- ances 109 Billions of wirelessInternet 108 mobile non tech users „Web“ 107 Zillions of everyday things eMail 106 P2P P2M M2M People  Machines time time 2000 issue #1: on the networking side, the exponential growth (# of nodes connected in the Internet) continues way beyond the world’s populationconsequence: scalability becomes even more crucial

P-2.3 Importance: exploding # of nodes, consequence wired-net bandwidth (bps) graphic cards (#polygons/sec) Internet (# of nodes) CPU (Moore‘s Law) log (capacity) unpredictable: wireless-net bandwidth (bps) battery capacity ?? Impact on ... • distributed computing: • disconnected-op? • mobile code? • server architectures time 2000 issue #2:different doubling rates for networks, CPUs (Moore’s law) etc.; impact on: system architectures, algorithms, etc. (e.g., RPC becomes cheaper, mobile computing maybe more expensive?)

P-1.2 Importance: Conclusion At the 30,000 ft. level, UC is “the next key technology to change our society” for 3 reasons: • UC describes the next era of computing • therefore, all of computer science / IT is potentially impacted • … it will be difficult to be selective in designing this course/book(but we will see that a course / book on UC still makes a lot of sense) • UC has potential impact on every facet of our lives • computing is no more “what we do when we sit at the computer” +“what is encapsulated/hidden deep inside VCRs etc.” • see the more / less comparison on next slide • UC is impossible and inevitable at the same time • components (“gadgets”) are developed and deployed, cf.slide “UC sample nodes” above • more and more UC scenarios become profitable  industry will push • many issues of “the whole” i.e. integrated UC systems still unresolved

P-2.1 From Importance to Challenges A first glance (not yet structured, non exclusive) at challenges… Anytime/anywhere presence of networked computers means: • more sensitivity  less protection (cf. security) • more dependence  less perfection (cf. reliability) • more obtrusion  less attention (cf. HCI) • more garrulity  less throughput (cf. networking) In other words: as computers become ubiquitous … • there is a risk that “the whole may be way less than the sum of the parts” i.e. the desired integral functionality may lack way behind • some known problems aggravate considerably – we’ve got to care! • in short, UC is a problem of complex integrated systems, not of gadgetscompare to: new Airbus, Space Shuttle, not better rocket-fuel, headlights

P-2.1 Humane & Integrated UC - Overview features of UC „gadgets“ are not the problem … … but making them „humane“ and integrating them into a useful whole Outlook (see below): Scalability Connectivity Adaptability Liability (=Security ++) Ease-of-Use Top Level Challenges: A. Systems Integration(Strength of Europe) B. Humane Computing(Chance for Europe)

P-2.2 Challenges: Humane UC Mainstream UC Research emphasizes two issues: • Context-Aware Computing ill defined; basically, the quest for adaptive software that … • understands the „situation in which the user currently works“ • adapts to that situation by • reducing complexity (e.g., adjusting UI to current needs & options) • automating input (e.g., retrieving data from „environment“, not user) • optimizing functionality • Multiple Modalities again, ill defined; modality is a „way of doing I/O“, often aninput and/or output „channel“ (what is that?) like graphics, voice,keyboard/mouse, gesture (note: examples from different categories!) • UC applications on appliances (cell phone, PDA… today, more tomorrow) •  Multiple modalities to be supported • truly multimodal UI: flexible combination of „channels“, user‘s choice • We will put these two issues in a larger context

P-2.3 Challenges: UC Integration Satellite Regional Area High-tier Low-tier ? LocalArea Wide Area Low Mobility High Mobility Pertinent Application:Real-Time Enterprise.full coverage:production/ supply chain, retail, post-sales Global Internet: seamless context-aware (service!) network UC World UC Gadgets

P-3.1: S.C.A.L.E. Overview: Breadth vs. Depth • Problem: UC touches almost every area of Computer Science, and more! • readers need background for understanding why what will be emphasized later • a categorization of issues (taxonomy) is needed for organizing the topics and for getting oriented  cf. the S.C.A.L.E. taxonomy • So, is it a research/teaching subject in its own right? • yes, compare to Distributed Systems: • there is distributed simulation, programming, algorithms, databases, AI, … “distributed everything”! • yet, there are courses on Distributed Systems • but: how to master breadth vs. depth? • be broad first, organize “problem space”  S.C.A.L.E., see below • then emphasize pertinent problems • either “new” aspect in UbiComp • or “much more relevant” aspect with UbiComp than without • standard computer scientist should hardly need add’l background • … but will sometimes want to dig deeper in some subtopic … which is normal • But then, is course/book going to be “complete”? • no, but more complete than any other book/course before • … and more well organized, due to S.C.A.L.E. scheme, see below

P-3.1: S.C.A.L.E. - Rationale for Taxonomy UC is novel subject  reader needs overview of problem space “Problem Space” approached in two ways: • Holistic view of pertinent research fields • S.C.A.L.E. classification scheme devised for that purpose • explained below, used throughout for structuring •  major “guide” through residual chapters • Holistic view of “the global UC network=system” • Mundo Reference Architecture devised for that purpose • imagine as rough floorplan for an “entire” UC system • shows differences to Distributed Systems Ref-Arch’s and Definitions • Mundo not supposed to be accepted worldwide • (might, at best, influence a future Ref.Arch. standard) • it’s purpose is to help readers in understanding & organizing UC field

P-3.1 - S.C.A.L.E. Scheme Major UC challenges, on a very high level: • S – SCALABILITY • how to support cooperation of “zillions” of components? • how to support nomadic users around the globe? • C – CONNECTIVITY • how to “easily” connect these zillions? Several levels of abstraction: • wireless networks – a blessing and a curse (unreliable!) • most issues come above wired/wireless net (note: overlap with scalability): • how to find/understand your peers? How to enable zero configuration • how to design networks for zillions2 of connections, without central-server bottleneck • A – ADAPTABILITY • usage during daily work, surrounded by 100s of components: need minimal interaction • major approach: context-aware computing – use it to automate tasks & reduce options • why “adaptability”? adapting-to-particular-user (user modeling) focussed beyond context-awareness • L- LIABILITY • term indicates: we must go beyond today’s IT security solutions (not goals), since • today’s solutions do not scale (centralized components?), are not “humane” (see below), … • … & don’t flexibly consider conflicting (privacy, traceability) & related goals (dependability etc.) • E – EASE-OF-USE • adaptability permits “minimal” …, ease-of-use means “optimal” interaction (related!) • issue: optimal use & combination of modalities, advancement of specific modalities • issue: “understanding” natural input: a) xxx-to-text; b) text-to-meaning; “intelligence”?

P-3.1 S.C.A.L.E - Scalability • S – SCALABILITY (is a “top priority” challenge  reflected in acronym: scale) • Mathematical Scalability (in the mathematical sense) UC leads to (potential) cooperation of “zillions” of devices • thus, solutions need to work efficiently with zillions of components • comes to mind: complexity of the made  the born (K. Kelly!)  learn from the born most relevant areas (basically, alternatives for addressing technical scalability): • bionics i.e. bio-analog computing • neural networks, cooperating robots: huge fields, only marginal importance for UC • ant colonies – often simulated / executed on single computer today; swarms, autonomous computing, …; • what else? plants as examples? brain-like modeling human memory: example given in main chapter “S” • socionics:learning from complex societies … is still in its infancies wrt. UC • Future communication/cooperation (pub/sub; P2P, Grid) – see further below (C – connectivity) • Technical-EconomicalScalability (in thesense of global interoperability) • human-attached  encountered components need to cooperate globally • standards come to mind, but global standards are not sufficient & (often) not realistic:e.g., interface definition languages (cf. remote method/procedure call etc.): • they define syntax (types/name of operations/operands exported @ interface • they usually don’t define, e.g.: valid call sequences, preconditions, cost/quality etc. (approaches exist) • they don’t define semantics: what does an operation actually do? approach??: ontologies & service discovery most relevant areas (to be combined into an overall solution): • WebServices and Business Processes • Ontologies a holy grail for making services understand each other (cf. what does the op’ do) ? • Service Discovery that should take into account SLA (service level agreements), QoS (quality of Service – in the business sense of “get what you pay for”), and more

P-3.1 S.C.A.L.E: Connectivity Global interconnection of UC components is related to the following(and to “scalability” above): • Scaleable Communication • wireless networks: (often!) a prerequisite for higher layers  needbasic understanding of technology [ZigBee … WiMax] and beyond • event based communication: praised as the UC connectivity approach • means “push” paradigm, a prerequisite for scaleable open cooperation of components (supersedes client/server!); note relation to “S” • maybe one day superseded by bionic/socionic approach; today: best offer • remaining problems (advertising, openness, integration of other paradigms) • plus: what else is needed in UC middleware (services? tools? which ones?) • Scaleable Cooperation • Overlay Nets: special subnetworks in Internet; at least 3 classes: • Peer-to-Peer Networks: no centralized bottleneck, scale well? • Opportunistic Networks: ad hoc net (node proximity) parallels global human “network” • Grid Networks: dynamically available resources parallel fluctuating resource demands • Smart Item Networks: AutoID and beyond: towards scaleable networks of “machines” • Service Discovery: prerequisite for zero configuration • component  environment, makes itself known & learns about environment • sounds like solution to “economic scalability” above, but isn’t: many assumptionsabout components are made  rather a connectivity issue

P-3.1 S.C.A.L.E: Adaptability • Context Awareness (adaptation to “situation of use”) • sensed context: what sensors can measure (temP-, shock, location, …) • modeled context: info held in “other” software/DB: tasks & activities etc. • inferred context: built from (several?) sensed or modeled contexts, e.g.,GPSstreet … co-located chemicals  dangerous!, is always modeled • note: context “ages”, is “probabilistic”/maybe contradictory (sensor imprecise? calender entry gives different location than GPS sensor?) • most investigated context: location • may be absolute (“X is room 253”) or relative (“voltmeter is with Mr. X”) • absolute: “2x2” technologies to survive? outdoor GPS & cell?; indoor UWB & IR? • relative: technologies for tags (RFID, RF+, IR, cell, …) will coexist? • User Awareness (adapt to user(s) -and, mid term, provider?) • technology? “usual suspects”: user models, profiles&preferences, user agents • great challenge: the huge crowd of “new” users • unexperienced, hands/eyes free, little attention • understand their actions  support them! [TK] • for UC: ubiquitous user models? UC components contribute&use only fractions

P-3.1 S.C.A.L.E: Liability Liability :== protection of actors and those concerned by actions (peers, third parties, society) through “right” mix of protection/prosecution – all in presence of zillions of peers; liability = “security and beyond” • Scaleable security: old goals (privacy, authentication, …), but new aspects: • machine2machine communication, ad hoc “encounters”: PKI not viable • cannot check certificate chain with (central!) roots zillions of times per second! • even if so: a party may change (e.g., due to virus) zilliseconds after check • maybe: not always/reliably connected! • early approaches: resurrecting duckling++, TCI, … :  viable overall solution? • there ain’t no end2end encryption (cf. https-connection vs. “frontend” password-spyware) • Anthropomorphic liability: UC in everyday life  conforms with human intuition? • intuitive models: “human-understandeable” models of trust, risk, recommendation etc. • user friendliness: security measures accepted  observed by users? • novel concepts, e.g.: minimal trusted, concealed entity (“ME”, see below)? • “Social” compliance: conflicting goals  conflicting forces (the core of liability!!) • e.g., privacy often traded off with public interest, peer’s interest (cf. traceability, non repudiation) • required: max. flexibility, “lever” for tradeoff (different place/time/parties: change mix) • Collateral goals most important: dependability / reliability / correctness • above terms: overlapping definitions; security added  “trustworthiness” • classical dependability: possible in light of zillions of bugs of MS Windows alone? • Aspect Trusted Computing (TCI): no a priori trust in peers  rely on TCI? (doubts!) • idea: virtual machine layers “underneath my SW” (CPU, Bios, OS, VM) “frozen” via digital signature • Aspect Accounting & Contracts • user: I will get what I pay for; provider: I will get paid for what I provide

P-3.1 S.C.A.L.E: Ease-of-Use remember: UI-adaptability may be considered a subset of ease-of-use (UI reducted to “what makes sense”)issue: optimal use & combination of modalities, advancement of specific modalities • Consider variety of I/O devices in UC world  “multimodal interaction” simple distinction: hands&eyes- vs. mouth&ears-interaction  • advanced hands&eyes interaction: GUIs predominant, but further developments needed • examples: focus+context-displays, 3rd dimension (VR), 4th dimension (dynamic displays), immersion, narration • mouth&ears interaction: voice underdeveloped today, great potential!  nomadic (hands-/eyes free) operation • but: needs speech understanding, is transient  difficult • integration of HCI and SWE: the grand challenge! • today: HCI before & after SWE – but “incompatible” • multimodality … : “beyond transcoding (program output XML, XSLT produces device-specific XML variant)” • quest for “abstract interaction layer”: shields modalitity details from “core” program yet leverages off modality advantages • true multimodality: parallel use of modalities, lots of freedom  “fusion” of inputs? “fission” of outputs? • integration with other advanced UIs? (“context-sensitive, adapting/proposing to user [AI techniques?]) … & federated devices: optimal use of multiple I/O devices currently surrounding user (includes “liability” issues) • Approach human “intuition”: new post-desktop metaphors – a cross cutting concern • post-desktop metaphors: cross cutting wrt. all aspects above • example: today, eMail folders  directories  …; tomorrow: all app’s adapt MY favorite structure? • methaphors invented for graphical UIs  approproate ones for voice, etc.? • … and related to scalability: humans must understand scalable = probabilistic behaviour in UC as inherent characteristic • example mass data (cf. DigiCam recordings swamping user disks) • options today:reduce/delete or: classify or: accept chaos • tomorrow: model brain  classify / rank automatically! ( prioritize/summarize/reduce; even: forget i.e. delete?) • Approach ease of human-to-human dialogues • natural dialogue:natural language processing NLP  intelligent user interfaces • grammar and semantics – possible for limited domain / world knowledge? simple/full grammar? beyond phrases: understand dialog? • formal knowledge: structured, semantically annotated documents <-> “informal” knowledge (eMails, Forums, WebSites, Blogs…) • even worse: electronic communication / informal knowledge ruled by abbreviated expression • understanding “natural” input • means: handwriting, speech, gesture, mimics (emotion!, “true” meaning) --- to be combined with NLP 

P-3.2 Relation to Computer Science Research HCI Distributed Systems context aware computing Dependability Bionics user modelling Socionics IT Security Computer Networks AI (/NLP/KM) Dependability Quality-of-Service, SLA, Accounting Software Engineering

P-4.1 SCALE Taxonomy As Introduced

P-4.1 SCALE Taxonomy vs. Chapters 1 27 28 26

P-4.2 Reader’s Digest 1 27 28 26 Introductory Chapters: 1,2,3,7,14,22 Sufficient for anOverview of UC; experts might want to leave out “their” intro chapter Readers who want to focus on Distributed Systems: read intro chapters + 8-13, add 15-17 and 5+6 according to interests For Ubiquitous Security, read intro chapters and chapters 19-21 For HCI aspects, read intro chapters and chapters 23-28, plus 17, 15-16, 4-6 according to interests Practitioners: read chapters “SAP pilots” chapters at the end of the book

Max Mühlhäuser