1 / 30

Challenges and Opportunities with Big Data

Challenges and Opportunities with Big Data. Dr Hammou Messatfa IBM Europe Government CTO Member of the IBM Academy of Technology. Agenda. What is big data and why is it such a popular topic and why now? Implications on skills Organizations are extracting value from big data

dyani
Download Presentation

Challenges and Opportunities with Big Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Challenges and Opportunities with Big Data Dr Hammou Messatfa IBM Europe Government CTO Member of the IBM Academy of Technology

  2. Agenda What is big data and why is it such a popular topic and why now? Implications on skills Organizations are extracting value from big data Implications on Research IBM’s big data journey 2

  3. Data is the new Oil. Data is just like crude. It’s valuable, but if unrefined it cannot really be used. – Clive Humby, DunnHumby We have for the first time an economy based on a key resource [Information] that is not only renewable, but self-generating. Running out of it is not a problem, but drowning in it is. – John Naisbitt 3

  4. The number of organizations who see analytics as a competitive advantage is growing. 70% 63% 57% 2011 2012 2010 BUSINESS IMPERATIVE IQ business initiative

  5. Helps detect life threatening conditions up to 24 hours sooner Reduced Improper Payment Cut serious crimeby $16 30% Billion Smarter CrimePrevention Analytics is Progressing from the Possible to the Proven Tax Agency Smarter Revenue Management Smarter Healtcare Analytics

  6. Big data embodies new data characteristics created by today’s digitized marketplace Big data characteristics Characteristics of big data Source: IBM methodology 6

  7. Organizations are evolving their big data journey What skills and processes do I need to add or modify to be successful? PEOPLE & PROCESS RESEARCH 7

  8. An acute shortage of skills threatens our ability to address emerging opportunities and risks Among organizations worldwide today… have major skill gaps in mobile, business analytics, and security has all the skills it needs to be successful applying advanced technology* for business benefit report a skills shortagein the ability to manage information * Includes business analytics, mobile computing, social business, and cloud computing Sources: IBM Tech Trends report 2012, Enterprise Strategy Group, CompTIA 8

  9. Big data requires a broad set of skills "By 2015, big data demand will reach 4.4 million jobs globally, but only one-third of those jobs will be filled." Source: Gartner "Gartner's Top Predictions for IT Organizations and Users, 2013 and Beyond: Balancing Economics, Risk, Opportunity and Innovation" 19 Oct 2012 Math and Operations Research Expertise Develop analytic algorithms Data Experts Data architecture, management, governance, policy Decision Making Executive and Management Apply information to solvebusiness issues Tool Developers Mask complexity and analytics to lower skillsboundaries Industry Vertical Domain Expertise Develop hypothesis, identifyrelevant business issues, ask the right questions Visualization Expertise Interpret data sets, determine correlations andpresent in meaningful ways 9

  10. Sample critical job roles Data Policy is fastest growing job role! http://www.cutter.com/bia/fulltext/reports/2013/02/index.html http://ibmdatamag.com/2013/02/i-am-an-information-strategist 10

  11. IBM Academic Initiative August 14, 2013 IBM and Universities Team Up to Close a 'Big Data' Skills Gap By Lee Gardner • Key offering areas • Business Analytics • Big Data • Security • Software Engineering • Mobile Development IBM Corporation's skills program focused on partnering with university faculty • Our mission • Partner with academic institutions to better educate millions of students for a smarter planet and more competitive IT workforce www.ibm.com/academicinitiative 11

  12. IBM Academic Collaboration Capabilities Map FACULTY-CENTRIC STUDENT-CENTRIC PARTNERSHIP MANAGEMENT Page 12 of 17 September 2014

  13. Organizations are evolving their big data journey What are the key business issues or opportunities that big data can help me to address? STRATEGY & VALUE 13

  14. Patterns of organizational behavior are consistent across four stages of big data adoption Big data adoption Big data adoption When segmented into four groups based on current levels of big data activity, respondents showed significant consistency in organizational behaviors Total respondents n = 1061 Totals do not equal 100% due to rounding 14

  15. Big data is a business priority – inspiring new models and processes for organizations, and even entire industries 15

  16. Organizations are evolving their big data journey What are the essential analytics capabilities we need to ensure we have in place? RESEARCH 16

  17. But there’s still room for research! Improve individual tools Handle particular data types better Make it easier to find entities in data Make it easier to compose analyses from existing models Improve the environment for exploring massive data? Pre-integrated data sets to provide context Powerful infrastructure for data management and analytics Rich collection of analytics and tools for analysis Expertise in all aspects of the process A great user experiencethrough automation and intelligent guidance Leverage tools and environment to solve important problems for people, industry and the world at large 17

  18. The Big Data Approach to Analytics is Different Traditional AnalyticsStructured & RepeatableStructure built to store data Big Data AnalyticsIterative & ExploratoryData is the structure Business Users Determine Questions IT Team Delivers Data On Flexible Platform AnalyzedInformation AnalyzedInformation Analyze ALL Available Information Available Information Capacity constrained down sampling of available information Whole population analytics connects the dots Business Users Explore and Ask Any Question Analyze information as is & cleanse as needed & existing repeatable IT Team Builds System To Answer Known Questions AnalyzedInformation Carefully cleanse a small information before any analysis 18

  19. The Big Data Approach to Analytics is Different AnalyzedInformation All Information Traditional AnalyticsStructured & RepeatableStructure built to store data Big Data AnalyticsIterative & ExploratoryData is the structure Hypothesis Question Data Exploration ? Answer Data Correlation Actionable Insight Start with hypothesis Test against selected data Data leads the way Explore all data, identify correlations Analyze after landing… Analyze in motion… 19

  20. Big data: This is just the beginning 9000 100 Sensors & Devices Percentage of uncertain data 6000 Volume in Exabytes Percent of uncertain data Social Media 50 You are here Volume Veracity VoIP 3000 Enterprise Data Variety 0 2010 2015 Source: IBM Global Technology Outlook 2012 IBM source data is based on analysis done by the IBM Market Intelligence Department. IBM Market Intelligence data is provided for illustrative purposes and is not intended to be a guarantee of future growth rates or market opportunity 20

  21. Preparing data for analysis itself requires analytics job changes Event Loan bankruptcies, merger/acquisitions borrower, lender employment, director, officer Person Company insider, 5% owner, 10% owner holdings, transactions subsidiaries, insider, 5%, 10% owner, banking subsidiaries issuer holdings, transactions Security Q: “How exposed am I to my borrowers?” Midas Flow SEC Search UI Financial Companies & Key People Extract Resolve Map & Fuse Crawl Temporal Analysis FDIC Reports 21

  22. Sample Application – Real Time Lead Generation Go for the best, DP-2000 Buying a DSLR today ! Buying DSLR today! Thrza gr8 deal on ZX-550 @ the mall Entity Extraction, Fact Discovery, Intent & Sentiment Prior Business Transactions Social Data Influencers Intent 250M tweets/day Millions of tweets yield one company-specific fact Customer ready to buy a DSLR camera today, possibly at a nearby mall Michael’s online friends offer lots of advice Text Analytics used to extract intent from Social Media Married, Male, Spouse Birthdate, Gift Type, Intent to Purchase, Timeframe Wifey’s birthday tomorrow, looking for a killer dslr Sarcasm,Wishful Thinking Maybe I should buy her that purple roadster, while I’m at it. ;-) lol Intent to Purchase, Gift Type? PotentialLocations and Activity In NYC area this w/e, any good malls nearby? Region & City Location, Timeframe, Intent to Shop • Resultant fact base contains billions of facts, and is incrementally updated • Fact segmentation or clustering is rapid enough to drive a business decision 22 22

  23. Deriving actionable consumer insights from social media Leverage social media and computational models to to predict intrinsic traits that influence consumer behavior

  24. Leading IBM: Eras of computing Cognitive Systems Era Programmable Systems Era Tabulating Systems Era Computer Intelligence Time

  25. Putting the pieces together at point of impact can be game changing can be life changing difficulty swallowing Pat. History Symptoms Fam. History Symptoms Family History Medications Medications Findings fever Findings Patient History Diagnosis Models Confidence dry mouth thirst Symptoms anorexia frequent urination A 58-year-old woman presented to her primary care physician after several days of dizziness, anorexia, dry mouth, increased thirst, and frequent urination. She had also had a fever and reported that food would “get stuck” when she was swallowing. She reported no pain in her abdomen, back, or flank and no cough, shortness of breath, diarrhea, ordysuria Her medications were levothyroxine, hydroxychloroquine, pravastatin, and alendronate. Renal Failure A 58-year-old woman complains of dizziness, anorexia, dry mouth, increased thirst, and frequent urination. She had also had a fever. She reported no pain in her abdomen, back, and no cough, or diarrhea. A urine dipstick was positive for leukocyte esterase and nitrites. The patient given a prescription fo ciprofloxacin for a urinary tract infection. 3 days later, patient reported weakness and dizziness. Her supine blood pressure was 120/80 mm Hg, and pulse was 88. dizziness UTI Her family history included oral and bladder cancer in her mother, Graves' disease in two sisters, hemochromatosis in one sister, and idiopathic thrombocytopenic purpura in one sister Her history was notable for cutaneous lupus, hyperlipidemia, osteoporosis, frequent urinary tract infections,a left oophorectomy for a benign cyst, and primary hypothyroidism, diagnosed a year earlier noabdominal pain no back pain Diabetes no cough no diarrhea Influenza FamilyHistory Oral cancer Bladder cancer Hemochromatosis Hypokalemia Purpura Graves’ Disease (Thyroid Autoimmune) Esophagitis PatientHistory cutaneous lupus osteoporosis • Extract Medications • Use database of drug side-effects • Together, multiple diagnoses may best explain symptoms • Extract Findings: Confirms that UTI was present • Identify negative Symptoms • Reason with mined relations to explain away symptoms (thirst is consistent w/ UTI) • Extract Symptoms from record • Use paraphrasings mined from text to handle alternate phrasings and variants • Perform broad search for possible diagnoses • Score Confidence in each diagnosis based on evidence so far • Extract Patient History • Extract Family History • Use Medical Taxonomies to generalize medical conditions to the granularity used by the models hyperlipidemia frequent UTI Most Confident Diagnosis: Diabetes Most Confident Diagnosis: UTI Most Confident Diagnosis: Esophagitis Most Confident Diagnosis: Influenza hypothyroidism Medications Alendronate pravastatin levothyroxine hydroxychloroquine Findings urine dipstick: leukocyte esterase supine 120/80 mm HG heart rate: 88 bpm 25 urine culture: E. Coli

  26. Watson enables three classes of cognitive services • Ask • Leverage vast amounts of data • Ask questions for greater insights • Natural language inquiries • e.g. - Next generation Chat • Discover • Find the rationale for given answers • Prompt for inputs to yield improved responses • Inspire considerations of new ideas • e.g. - Next generation Search Discovery • Decide • Ingest and analyze domain sources, info models • Generate evidence based decisions with confidence • Learn with new outcomes and actions • e.g. - Next generation Apps  Probabilistic Apps

  27. IBM Research: The World is Our Lab IBM Research labs Labs added since 2010 Other IBM Research presence Dublin China Zurich Haifa Almaden Watson Tokyo India Austin Brazil Melbourne

  28. More than $16B in Acquisitions Since 2005 • More than 10,000 Technical Professionals • More than 7,500 Dedicated Consultants • LargestMath Department in Private Industry • More than 27,000 Business Partner Certifications • Partner with more than 1000 Universities on Analytics IBM building strength and leadership in big data and analytics Building the most comprehensive Business Analytics & Optimization portfolio & partnerships 2013 Talent Acquisition Social Analytics/Consumer Insight Workload Optimized Systems Advanced Case Management Content Analytics Decision Management Stream Computing Pervasive Content pureScale pureXML Deep Compression Developer Productivity Autonomic Operations Innovation that Matters 2005 29

  29. 30

More Related