As IoT applications become more pervasive, there is a real-time big data explosion

1. Smart Data Analysis for IoT (Internet of Things) ApplicationsKun-Lung Wu, Ph.D., Manager Data-Intensive Systems & Analytics Group (IBM T. J. Watson Research Center)InfoSphere Streams Language & Research (IBM SWG)

2. As IoT applications become more pervasive, there is a real-time big data explosion Almost anything can be equipped and connected to the Internet Internet of Things Everything They can generate, in real-time, streams and streams of data Real-Time Big Data Explosion Real-time data analysis is an integral part of many IoT applications

3. Examples of IoT Applications • Smart cities • Traffic control, emergency management, etc • Health care • Aiding the elderly, ICU alert management, health monitoring via wearable devices, etc • Agriculture & food • Precision farming, cold chain management, etc • Industrial applications • Manufacturing process monitoring, engine monitoring, etc • Environmental monitoring • Water, Waste, Air Quality, etc • Retail applications

4. What is different in IoT data?There are many extremes There are greater amounts of data Process and act on data more quickly in real time Volume Velocity Use more typesdata Useuncertain data Variety Veracity

5. Traditional versus IoT Big Data Traditional Approach IoT Big Data Approach AnalyzedInformation Available Information Analyze ALL Available Information Analyze Small Subsets of Information Analyze All Information Leverage more of the data being captured

6. Traditional versus IoT Big Data Traditional Approach IoT Big Data Approach AnalyzedInformation AnalyzedInformation A Small Amount of Carefully Cleansed Information A Very Large Amount of Messy Information Carefully Cleanse Information Before Any Analysis Analyze Information As Is, Cleanse As Needed Reduce effort required to leverage data

7. Traditional versus IoT Big Data Traditional Approach IoT Big Data Approach Analyze data AFTER it has been processed and landed in a Warehouse or Mart Analyze data IN MOTION as it is generated, in real-time Leverage data as it is captured

8. RE-

9. From data at rest to data in motion Data at Data in

10. IBM InfoSphere Streams Delivers Real-Time Analytics For Big Data In Motion Real time delivery Environment Monitoring ICU Monitoring Powerful Analytics Telco Churn Prediction Algorithmic Trading Smart Grid Cyber Security Government / Law enforcement Millions of events per second Microsecond Latency Traditional / Non-traditional data sources Example Streaming Data Sources: Video, audio, networks, social media

11. Big Data in Real Time with Stream Processing Filter / Sample Modify Annotate Analyze Fuse Classify Windowed Aggregates Score

12. InfoSphere Streams: For superior real time analytic processing Streams Processing Language (SPL) built for Streaming applications: Reusable operators Rapid application development Continuous “pipeline” processing Compile groups of operators into single processes: Efficient use of cores Distributed execution Very fast data exchange Can be automatic or tuned Scaled with push of a button Use the data that gives you a competitive advantage: Can handle virtually any data type Use data that is too expensive and time sensitive for traditional approaches Easy to extend: Built in adaptors Users add capability with familiar C++ and Java Dynamic analysis: Programmatically change topology at runtime Create new subscriptions Create new port properties Flexible and high performance transport: Very low latency High data rates Easy to manage: Automatic placement Extend applications incrementally without downtime Multi-user / multiple applications 12

13. What Are People Doing With Streams? • Stock market • Impact of weather on securities prices • Analyze market data at ultra-low latencies • Telephony • CDR processing • Social analysis • Churn prediction • Geomapping • Law Enforcement, • Defense & Cyber-Security • Real-time multimodal surveillance • Situational awareness • Cyber security detection • Transportation • Intelligent traffic management • Fraud prevention • Detecting multi-party fraud • Real-time fraud prevention • Smart Grid & Energy • Transactive control • Phasor Monitoring Unit • e-Science • Space weather prediction • Detection of transient events • Synchrotron atomic research • Health & Life • Sciences • Neonatal ICU monitoring • Epidemic early warning system • Remote healthcare monitoring • Other • Manufacturing • Text Analysis • Who’s Talking to Whom? • ERP for Commodities • FPGA Acceleration • Natural Systems • Wildfire management • Water management

14. Asian telco reduces billing costs and improves customer satisfaction Problem: Call volume increased to the point that batch processing in a warehouse no longer worked 1) Too expensive, 2) too slow, and 3) no capacity left for BI Solution: Real-time mediation and analysis of 8B CDRs per day Data processing time reducedfrom12 hrs to 1 sec Hardware cost reduced to 1/8th Further enabled: Proactively addressing issues impacting customer satisfaction, real time offers based on usage 14

15. Harnessing the Largest Predictive Focus Group in the World • Purpose • Understand public sentiment towards an event: movie trailers • Deeply understand the potential customer profile: gender, occupation, intent to watch • Alter marketing launch plans based on insight • Background • 1.1 Billion Tweets analyzed • 5.7 Million blogs/forum posts • 3.5 million messages • Also: Facebook, Google+, Tumblr, Flickr

16. University of Ontario Institute of Technology (UOIT) Detects Neonatal Patient Symptoms Sooner • Performing real-time analytics using physiological data from neonatal babies • Continuously correlates data from medical monitors to detect subtle changes and alert hospital staff sooner • Early warning gives caregivers the ability to proactively deal with complications “Helps detect life threatening conditions up to 24 hours sooner”

17. Challenges and opportunities • Approach overload • Is there a convergence of approaches? • Is there a “write once, use any technology” approach across tool types • Skills to apply techniques • Reduce the skill required? • More people who can be data scientists, developers, and business/domain savvy? • Uncertain data • Confidence levels need to follow data and decisions • New analytic algorithms • Real time learning and adaptation? • More automation • Availability • What does it mean for in-memory systems? • How should disaster recovery work? • Cloud • Security of Data • Data movement • Data governance, security, and privacy • What new problems can we solve?

18. To Learn more Resources Streams: streamsDev IBM Big Data: ibm.com/bigdata IBMBigDataHub.com BigDataUniversity.com Books / analyst papers

19. Try Stream Processing http://Ibm.co/streamsqs 2 download options! 19