1 / 16

[[The Wikibon Project]]

[[The Wikibon Project]]. Big Data and Hadoop: Key Drivers, Ecosystem and Use Cases November 2011. What is Big Data?. Big Data n Data sets whose size, type and/or speed make them impractical to process and analyze with traditional database technologies and related data management tools.

avani
Download Presentation

[[The Wikibon Project]]

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. [[The Wikibon Project]] Big Data and Hadoop: Key Drivers, Ecosystem and Use Cases November 2011

  2. What is Big Data? Big Datan Data sets whose size, type and/or speed make them impractical to process and analyze with traditional database technologies and related data management tools.

  3. Why is Big Data Important? Big Data is the new definitive source of competitive advantage across industries … … For those organizations that embrace Big Data, the possibilities for innovation, improved agility, and increased profitability are nearly endless.

  4. Three Key Big Data Drivers Volume, Variety, Velocity Hardware Commoditization Cloud Computing

  5. Characteristics of Big Data

  6. Sources of Big Data

  7. Hadoop Open source framework for processing, storing and analyzing Big Data. Fundamental concept: Rather than banging away at one, huge block of data with a single machine, Hadoop breaks up Big Data into multiple parts so each part can be processed and analyzed in parallel.

  8. Hadoop: The Pros and Cons First the pros … Hadoop is a time- and cost-effective approach to store, process and analyze large volumes of unstructured data allowing for new and unprecedented types of analytics. • Now the cons … Hadoop is complex and difficult to deploy and manage; there’s a dearth of Hadoop-savvy engineers and Data Scientists on the job market; the risk of forking and vendor lock-in remains.

  9. Hadoop: The Pros and Cons cont. More pros … Many bright minds contributing to Hadoop resulting in rapid development and an ecosystem of vendors emerging to make Hadoop enterprise-ready.

  10. The Big Data Ecosystem

  11. Big Data Pioneers • Largest Hadoop instance on the planet … 40,000 nodes handling 200+ PB of data. • Used to support research for ad systems and Web search. • Match ads with users, detect spam in Yahoo! Mail, pick relevant top stories.

  12. Big Data Pioneers cont. • Two major clusters processing and storing over 30 PB of data. • Uses HDFS to store copies of internal log and dimension data. • Developed Hive to perform large-scale analytics on user data. • Using HBase to store, manage and retrieve Facebook Messenger data.

  13. Big Data Pioneers cont. • Uses Hadoop to support “People You May Know” feature. • Tailors its search engine to return most relevant results for recruiters, employers and job seekers. • Created a visualization tool to allow users to explore their professional network to discover hidden patterns.

  14. Big Data in Financial Services • Over 30,000 databases and 15,000 applications spread across 7 business units. • Using Hadoop as the basis of its Common Data Platform. • Looking to establish 360 degree view of customer for upsell and cross-sell opportunities.

  15. Big Data in Financial Services cont. • Risk management and analysis to understand financial exposure. • Detecting fraudulent transactions and potentially criminal activity. • Conduct sentiment analysis on social media data.

  16. Thank You Jeffrey F. Kelly Principal Research Contributor The Wikibon Project jeff.kelly@wikibon.org @jeffreyfkelly www.wikibon.org www.siliconangle.com

More Related