Reliable Instrumented CyberPhysical Spaces Project Kickoff Meeting

1. Reliable Instrumented CyberPhysical Spaces Project Kickoff Meeting Feb 7th, 2011 – UC Irvine UCI:Nga Dang, Nikil Dutt, Leila Jalali, Xu Jie, Dmitri Kalashnikov, Zhijing Li, Sharad Mehrotra, Kazuyuki Tanimura,Ronen Vaisenberg, Nalini Venkatasubramanian, Xiujuan Yi, Liyan Zhang SRI: Grit Denker, Minyoung Kim, Carolyn Talcott

2. Agenda 10:00 - 10:15 Welcome and Introduction 10:15 -10:45 Project Introduction and scope (Nalini) 10:45- 12:00 Presentations -UCI SAFire/FireTalk (Sharad/Dmitri) - SRI talks 12:00-1:30 Lunch and poster presentations (by students) 1:30 - 2:45 Scenario Discussion (surveillance, fire perspective) 2:45-3:00 Break 3:15-4:30 Project plans (research, system building)

3. What are CyberPhysical Systems (NSF) • The term "cyber-physical systems" refers to the tight conjoining of and coordination between computational and physical resources.  We envision that the cyber-physical systems of tomorrow will far exceed those of today in terms of adaptability, autonomy, efficiency, functionality, reliability, safety, and usability.  • Research advances in cyber-physical systems promise to transform our world with systems that respond more quickly (e.g., autonomous collision avoidance), are more precise (e.g., robotic surgery and nano-tolerance manufacturing), work in dangerous or inaccessible environments (e.g., autonomous systems for search and rescue, firefighting, and exploration), provide large-scale, distributed coordination (e.g., automated traffic control), are highly efficient (e.g., zero-net energy buildings), augment human capabilities, and enhance societal wellbeing (e.g., assistive technologies and ubiquitous healthcare monitoring and delivery).

4. Instrumented CyberPhysical Spaces nearby sensors Event: shooter on campus events of interest Shooter location: UCI#outdoors/(300,506) Applications: surveillance & monitoring ! Applications: Situation Awareness Shooter location: UCI#outdoors/(300,506)

5. The Irvine Sensorium (Responsphere) Testbed • Campus-Scale sensing, communication, storage, computing infrastructure • - 200+ video cameras, Motes, sun spots, RFID, mobile cameras, gas sensors, • Mesh routers, WiFi, power-line network, zigbee • storage & compute clusters

6. ICPS applications Indoor Localization Framework Human as Sensor System SAFIRE- situational awareness System Privacy Preserving Surveillance System Occupancy Forecasting System SATWARE: semantic middleware for sentient spaces nearby sensors ! Bren Hall Inauguration RF-ID tracking Calit2 Recycling Monitor

7. Architecture SATWARE – Semantic Middleware for Sentient Spaces Powerful programming environment for sentient applications hides heterogeneity, errors, complexity Native support for privacy Adaptive data collection Exploiting semantics

8. SATWARE Infrastructure Layer Video

9. Challenges in Building ICPS • Sensors • Indoor localization still unsolved • Sensor & platform diversity makes programming very complex. • Robustness of sensors remains elusive • Calibration, sensitivity to ambient conditions, resilience in extreme situations • Multimodal Sensors, e.g., audio, video • both an opportunity and a challenge • Semantic Sensing untapped potential despite benefits • E.g., human speech, observations, blogs

10. Challenges in Building ICPS • Infrastructure • Resource demands • Increasing sensor complexity  increasing network bandwidth • Application-driven constraints on quality, timeliness properties • Video, audio data with real-time constraints • Battery based power supply impose significant restrictions • limits the transmission protocols that can be used by sensors

11. Challenges in Building ICPS • Privacy in sentient spaces that monitor human activities ( E.g., surveillance systems, smart buildings/infrastructure) • Trust in infrastructure • Policy languages to express privacy policies • Who (can access), what (data), when and under which context , and for what (purpose). • Policy enforcement requires inference control • must ensure that inference made on data released does not violate privacy policies. • Tradeoffs between loss of privacy and utility • Observer utility versus loss of privacy of target

12. What can go wrong? • Infrastructure component errors/failures • Device Failures • Network Failures • Congestion and Overloads • Data Interpretation errors/failures • Uncertainty in Processing • E.g. Speech/video/image processing • Contextual errors • E.g. occlusions/obstructions to a light sensor

13. Network Unreliability: WiFi Mesh Commercial mesh routers not good enough • 5X improvement with • new antenna technology • Better signal coverage better building penetration • Some Setup effort required • Not always feasible • Vulnerable to hardware failures

14. (Un) Reliability of Wi-Fi Networks • Varying traffic load • Varying level of contentions and congestions • Varying inter-device distance Ad-hoc 1hop > Ad-hoc 2 hops > Private AP >>> Public AP • Increased bandwidth share • Reduced contentions/collisions • Less interferences • Distributed Beaconing • No background traffic • Controllable configuration

15. Proprietary EMF transmission Polar T31 Heart rate strap transmitter Polar Heart Rate Module IMU (5 degrees of freedom) SAFIRE Mote Sensor Deployment Heart Rate Crossbow MIB510 Serial Gateway Crossbow MDA 300CA Data Acquisition board on MICAz 2.4Ghz Mote Inertial positioning IEEE 802.15.4 (zigbee) To SAFIRE Server Carbon monoxide Temperature, humidity Carboxyhaemoglobin, light

16. Sensing Unreliability ↑Mobility↓Reliability Network convergence, gateway availability Calibration is essential static Frequency matters!! mobile ↑Density↑Reliability Topology matters!! ↑Size↓Reliability

17. Approaches to enable reliable onsite networking Exploit multiple networks that together provide connectivity (Mobiquitous 2005, WCNC 2007, INFOCOM 2009) WiFi mesh – direct connectivity to a mesh router MANETS – hop by hop connectivity to gateway nodes Zigbee adhoc – connect to WiFi backbone through gateway node Exploit mobility when disconnected (PERCOM 09, SECON 2010) Store-and-forward networks (Delay Tolerant Networking) mobile nodes ferry data to gateway node Combine connected network clouds and disconnected networks(PWN 10)

18. Note: Reliable infrastructure ≠Reliable Data Collection • Sensing Errors Occur • Visibility Readings vary • Occlusions etc. • Spikes in SpCO readings due to FF movement • Read errors due to misaligned sensor strip • Reliability at application level is also needed needed • Sensor Calibration (MMCN08) • Heart-rate, CO exposure • Exploitation of Semantics, prediction • Exploit application tolerance to errors

19. Information Processing (Unreliability) • Speech Recognition quality bottleneck • Poor recognition quality in noisy & realistic environments Speech Speech Recognizer Output This is a bed sun tan “This is a bad sentence” 20

20. RCPS Project Goals The goal of this project is to develop a semantic foundation, cross-layer system architecture and adaptation services to improve dependability in instrumented cyberphysical spaces (ICPS) based on the principles of “computation reflection”.

21. Traditional Fault Tolerance Aspects • Failure Modeling • Impact Analysis • Failure Prevention/Avoidance • Failure Detection • Failure Recovery

22. RCPS Main Contributions (Planned) • Develop a digital state representation that guides a range of adaptations at different layers of the ICPS (i.e. networking, sensing, applications, cross-layer) to achieve end-to-end dependability at both the infrastructure and information levels. • Techniques for reliable information delivery over multi-networks, quality aware data collection, semantic sensing and reconfiguration using overlapping capabilities of heterogeneous sensors. • Adaptations driven by formal-methods based runtime analysis of system components, resource availability and application dependability needs. • Responsphere, a real-world ICPS infrastructure on the University of California at Irvine campus, will serve as a testbed for development and validation of the overall “reflective” approach and the cross-layer adaptation techniques to achieve dependability.

23. RCPS System Development

24. Architecture RCPS System Plan – extend SATWARE Powerful programming environment for sentient applications hides heterogeneity, errors, complexity Native support for privacy (ADD RELIABILITY) Adaptive data collection Exploiting semantics

25. Urgent Tasks • Name • Target Scenario(s) • Surveillance • Fire Response

27. Related Project: SAFIRE

28. SAFIRE Project • Goal • Improve the safety of firefighters by providing decision makers with greatly improved situational awareness during response activities. • UC Irvine • Sharad Mehrotra • Nalini Venkatasubramanian • Chris Davison • Dmitri Kalashnikov • Jay Lickfett • Jeffrey Xiu • Ronen Vaisenberg • Stefano Bonetti • Imagecat Inc. • Paul Amyx • Charlie Huyck • Ron Eguchi • Deltin Corporation • Ron Cabrera • Fire Fighter Forum • County of LA Fire Dept. • Newport Beach Fire • Orange County Fire Authority • City of Ontario

29. Main Deliverable – SAFIREStreams A software framework to create situational awareness from heterogeneous multimodal sensor streams • Captures/ingests data from heterogeneous SA sensors • Personal - Physiological (heart rate, blood CO, accelerometer) and location (WiFi, Ultrasonic, RFID, GPS) • Environmental– temperature, humidity, CO, light, sound • Multimodal - Video cameras, Speech sensors (radio communication amongst responders) • External data sources - (via Ebox technology) • Transforms raw sensor data to situational information • Declarative programming language for rapid application prototyping • Provides core SA services • Alerts, archival, replay functionalities • Powerful UI for situation monitoring - Displays dynamic sensor data, overlay of contextual information

30. Highlight: An End-to-end SA Tool for ICs SAFIREStreams Visualization & Decision Support Services (Alerts, Queries, Replay, Triggers) Sensor Data Ingest Unit Sensor Stream Processing Module Firefighter Status Dashboard Sensor Data Collection Virtual Sensors for Media Level events Sensor Fusion Available GIS layers Mapping and Localization Multisensor Event Extraction Temperature humidity, visibility Multimedia Data Collection Receive /display alert messages. Ambient CO Weather Image/Video Raw Sensor data (sensors, speech, video) Raw Data DB Event DB HAZMAT Semantically Enriched Event Data CAD Systems Audio/speech Demographics SpCO, light, inertial, RFID, heart rate, .. Floor plans Occupancy Sensor/Incident Storage& Archival Ebox External Data Access Programmed in Java/C++, executes on Windows based platforms

31. SAFIREStreams Programming Programming Execution

32. Highlight –Drills & Experiments to Validate role of Sensors in Creating SA at Crisis Site • HazMat, casualties, First Response drill and SAFIRE Deployment (16 SEP 08) • Live Burn & CO Sensing Study: OCFA, LA County Fire (23 FEB 09) • HazMat drill (with multiple casualties) and SA Study (12 MAY 09) • Tabletop exercise IC Usability Experiment (15 MAY 09) to determine role of sensor based awareness for decision making • Analysis of in-field data collected to determine reliability of sensor data capture at crisis sites (IEEE PERNEMS 2010, IEEE IQ2S 2010)

33. SAFIRE / FICB Usability Study • Goal • Test usability of SAFIRE technology for creating SA for ICs • Methodology • Table Top Exercise based on technology drill • Drill Scenario stopped at 6 “freeze points” to assess Situational Awareness and impact on decision making • Results • Usability and decision-making impact significantly correlated with SAFIRE technology among ICs. • Qualitative feedback overwhelmingly positive. Also, suggestions for improvement.

34. Highlight – New Research Directions • SAFIRE Project led to new research directions • New Sensing Modalities: Radio communications between responders as a source of situational information • Initiated project with ICSI and SRI (two leading groups in speech processing and understanding) • Robust networking: crisis site communication requires “opportunistic networking” using multi-networks • Initiated work on crisis site multi-networks with SRI researchers • “On-the-fly” Information Integration: Ebox concept that provides dynamic access to site-specific information through web services interface and integrates it with SA tools for FF. • In discussion with DHS for proof of concept development

35. Highlight – Technology Transfer • Formation of the HALO Consortium to explore • Technology transfer opportunities within fire community • Strategies to bring SAFIREStreams to real deployment • Planning for continuity of research and development beyond AFG Grant • Participants • Deltin Corporation • ImageCat Inc. • University of California, Irvine • Advisory Board of Fire Agencies • Steps taken • Transfer of code base out of UCI to Deltin and Imagecat • Adding reliability to the code base (Summer 2010) • Customizing SAFIRE streams to illustrate impact on specific FF casualty scenarios • Demonstration of system to target FF groups.

36. Lessons Learned • Significant opportunities exist to improve FF safety through improved SA. • Enabling technologies for creating SA systems exist / can be engineered • Sensors, networking, data management, UI • Key challenge is transforming raw sensor feeds into actionable SA to support decision making • Heart rate versus danger to health • SAFIRE stream system is a step in that direction • Multi-sensor integration / fusion to improve quality/certainty of observations. • Easily programmable interfaces to detect complex multi-sensor events.

37. Related Project: FireTalk