The Endeavour Expedition: Charting the Fluid Information Utility

1. The Endeavour Expedition:Charting the Fluid Information Utility Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley Berkeley, CA 94720-1776

2. Why “Endeavour”? • DARPA BAA 99-07: Information Technology Expeditions • To strive or reach; a serious determined effort (Webster’s 7th New Collegiate Dictionary); British spelling • Captain Cook’s ship from his first voyage of exploration of the great unknown of his day: the southern Pacific Ocean (1768-1771). • These voyages brought brought more land and wealth to the British Empire than any military campaign. • Cook’s lasting contribution: comprehensive knowledge of the people, customs, and ideas that lay across the sea • “He left nothing to his successors other than to marvel at the completeness of his work.”

3. Expedition Goals • Enhancing human understanding through information technology • Dramatically more convenient for people to interact with information, devices, and other people • Supported by a “planetary-scale” Information Utility • Stress tested by challenging applications in decision making and learning • New methodologies for design, construction, and administration of systems of unprecedented scale and complexity • Figure of merit: how effectively we amplify and leverage human intellect • A pervasive Information Utility, based on “fluid systems technology” to enable new approaches for problem solving & learning

4. Expedition Assumptions • Human time and attention, not processing or storage, are the limiting factors • Givens: • Vast diversity of computing devices (PDAs, cameras, displays, sensors, actuators, mobile robots, vehicles); No such thing as an “average” device • Unlimited storage: everything that can be captured, digitized, and stored, will be • Every computing device is connected in proportion to its capacity • Devices are predominately compatible rather than incompatible (plug-and-play enabled by on-the-fly translation/adaptation)

5. Expedition Challenges • Personal Information Mgmt is the Killer App • Not corporate processing but management, analysis, aggregation, dissemination, filtering for the individual • People Create Knowledge, not Data • Not management/retrieval of explicitly entered information, but automated extraction and organization of daily activities • Information Technology as a Utility • Continuous service delivery, on a planetary-scale, on top of a highly dynamic information base • Beyond the Desktop • Community computing: infer relationships among information, delegate control, establish authority

6. Expedition Approach • Information Devices • Beyond extrapolated desktop devices to MEMS-sensors/actuators plus capture/display to yield enhanced activity spaces • Information Utility • “Fluid”, Network-Centric System Software • Paths/Streams: process/store/manage information • “Movable” Processing and Storage • Partitioned/distributed functionalityThin-Clients/Fat-Infrastructure • Nomadic Data • Negotiation-based Interfaces • Always-Available Functionality • Wide-area distributed coordination and control on scalable servers

7. Expedition Approach • Information Applications • High Speed/Collaborative Decision Making, Learning • Augmented “Smart” Spaces: Rooms, Vehicles • Design Methodology • HW/SW Co-design • Formal Methods • Decomposable and Reusable Components • User-centered Design

8. High Speed Decision Making Learning Classroom E-Book Vehicles Applications Collaboration Spaces Info Appliances Human Activity Capture Generalized UI Support Event Modeling Transcoding, Filtering, Aggregating Statistical Processing/Inference Proxy Agents Negotiated APIs Self-Organizing Data Information Utility Interface Contracts Wide-area Search & Index Nomadic Data & Processing Wide-Area Data & Processing Automated Duplication Movement & Positioning Distributed Cache Management Stream- and Path-Oriented Processing & Data Mgmt Non-Blocking RMI Soft-/Hard-State Partitioning Laptop PDA Wallmount Display Camera Information Devices Smartboard MEMS Sensor/Actuator/Locator Handset

9. Needed Expedition Expertise • MEMS and hardware devices • Scalable computing architectures • Networked-oriented operating systems • Distributed file systems • Data management systems • Security/privacy • User interfaces • Collaboration applications • Intelligent learning systems • Program verification • Methodologies for HW/SW design/evaluation

10. Alex Aiken, PL Eric Brewer, OS John Canny, AI David Culler, OS/Arch Joseph Hellerstein, DB Michael Jordan, Learning Anthony Joseph, OS Randy Katz, Nets John Kubiatowicz, Arch James Landay, UI Jitendra Malik, Vision George Necula, PL Christos Papadimitriou, Theory David Patterson, Arch Kris Pister, Mems Larry Rowe, MM Alberto Sangiovanni-Vincentelli, CAD Doug Tygar, Security Robert Wilensky, DL/AI Interdisciplinary, Technology-Centered Expedition Team

11. D e s I g n M e t h o d o l o g y Applications Rapid Decision Making, Learning, Smart Spaces: Collaboration Rooms, Classrooms, Vehicles Information Utility Fluid Software, Cooperating Components, Diverse Device Support, Sensor-Centric Data Mgmt, Always Available, Tacit Information Exploitation (event modeling) Base Program Information Devices Option 1: Sys Arch for Diverse Devices MEMS Sensors/Actuators, Smart Dust, Radio Tags, Cameras, Displays, Communicators, PDAs Option 2: Oceanic Data Utility Option 3: Capture and Re-Use Option 4: Negotiation Arch for Cooperation Option 5: Tacit Knowledge Infrastructure Option 6: Classroom Testbed Option 7: Scalable Heterogeneous Component-Based Design Organization: The Expedition Cube

12. Base Program: Leader Katz • Broad but necessarily shallow investigation into all technologies/applications of interest • Primary focus on Information Utility • No new HW design: commercially available information devices • Only small-scale testbed in Soda Hall • Fundamental enabling technologies for Fluid Software • Partitioning and management of state between soft and persistent state • Data and processing placement and movement • Component discovery and negotiation • Flexible capture, self-organization, info re-use • Limited Applications • Methodology: Formal Methods & User-Centered Design

13. Option 1: “System Architecture for Vastly Diverse Devices”Leader Culler • Distributed control & resource management: data mvmt & transformation, not processing • Path concept for information flow, not the thread • Persistent state in the infrastructure, soft state in the device • Non-blocking system state, no application state in the kernel • Functionality not in device is accessible thru non-blocking remote method invocation • Extend the Ninja concepts (thin client/fat infrastructure) beyond PDAs to MEMS devices, cameras, displays, etc.

14. Option 2: Implementation & Deploy-ment of Oceanic Data Info UtilityLeader Kubiatowicz • Nomadic Data Access: serverless, homeless, freely flowing thru infrastructure • Opportunistic data distribution • Support for: promiscuous caching; freedom from administrative boundaries; high availability and disaster recovery; application-specific data consistency; security • Data Location and Consistency • Overlapping, partially consistent indices • Data freedom of movement • Expanding search parties to find data, using application-specific hints (e.g., tacit information)

15. Option 3: Sensor-Centric Data Management for Capture/ReuseLeader Hellerstein • Integration of embedded MEMS with software that can extract, manage, analyze streams of sensor-generated data • Wide-area distributed path-based processing and storage • Data reduction strategies for filtering/aggregation • Distributed collection and processing • New information management techniques • Managing infinite length strings • Application-specific filtering and aggregation • Optimizing for running results rather than final answers • Beyond data mining to “evidence accumulation” from inherently noisy sensors

16. Option 4: Negotiation Architecture for Cooperating ComponentsLeader Wilensky • Cooperating Components • Self-administration through auto-discovery and configuration among confederated components • Less brittle/more adaptive systems • Negotiation Architecture • Components announce their needs and services • Service discovery and rendezvous mechanisms to initiate confederations • Negotiated/contractural APIs: contract designing agents • Compliance monitoring and renegotiation • Graceful degradation in response to environmental changes

17. Option 5: Tacit Knowledge Infra-structure/Rapid Decision MakingLeader Canny • Exploit information about the flow of information to improve collaborative work • Capture, organize, and place tacit information for most effective use • Learning techniques: infer communications flow, indirect relationships, and availability/participation to enhance awareness and support opportunistic decision making • New collaborative applications • 3D “activity spaces” for representing decision-making activities, people, & information sources • Visual cues to denote strength of ties between agents, awareness levels, activity tracking, & attention span

18. Option 6: Info Mgmt for Intelligent ClassroomsLeader Joseph • Electronic Problem-based Learning • Collaborative learning enabled by information appliances • Enhanced Physical and Virtual Learning Spaces • Wide-area, large-scale group collaboration • Capture interaction once for replay • Preference/task-driven information device selection • Service accessibility • Device connectivity • Wide-area support • Iterative evaluation

19. Option 7: Safe Component Design and UI Design ToolsLeader Sangiovanni • Information Appliances as an application of hardware/software codesign • Co-design Finite State Machines (CFSMs) • Formal methods to verify safety from faults • Safe partitioning of components into communicating subcomponents placed into the wide-area • Model-based User Interface Tools • Information device user interfaces • Multimodal interface design for variety of devices

20. Option 8: Scaled-up Field TrialsLeader Katz • Testbed Rationale • Study impact on larger/more diverse user community • Higher usage levels to stress underlying architecture • Make commitment to true utility functionality • Increasing Scale of Testbeds • Building-Scale • Order 100s individuals • Campus-Scale • Order 1000s individuals • City-Scale • Order 100000 individuals

21. Putting It All Together Devices Utility Applications 1. Diverse Devices 2. Data Utility 3. Capture/Reuse 4. Negotiation 5. Tacit Knowledge 6. Classroom 7. Design Methods 8. Scale-up Component Discovery & Negotiation Fluid Software Info Extract/Re-use Self-Organization Group Decision Making Learning

22. Letters of Support • AT&T Labs, Research: Dr. Hamid Ahmadi, Networking and Distributed Systems Research Vice President • Cadence: Dr. Patrick Scaglia, VP Research, Cadence Laboratories • Hewlett Packard: Dr. Steve Rosenberg, Manager, External Research, HP Labs • IBM: Dr. William Cody, Manager, Exploratory Database Systems • Intel: Dr. Richard Wirt, Director, Intel Microcomputer Laboratory • Lucent/Bell Labs: Dr. William M. Coughran, Jr., Bell Labs Research Silicon Valley Vice President

23. Letters of Support • Microsoft: Dr. Daniel Ling, Director, Microsoft Research • Motorola: Dr. John Barr, Director, System of Systems Architecture, Personal Information Networking Division • Nortel Networks: Dr. Daniel Pitt, VP Technology and Director Bay Architecture Lab • Sprint: Dr. Frank Denap, Director, Advanced Technology Labs • Sun Microsystems: Dr. Greg Papadopoulos, Vice President and Chief Technology Officer • Xerox: Dr. Mark Weiser, Chief Technologist, Palo Alto Research Center

24. Letters of Support

25. Discussion