1 / 17

Powering Next-Generation Data Architectures with Apache Hadoop

Powering Next-Generation Data Architectures with Apache Hadoop. Shaun Connolly, Hortonworks @ shaunconnolly September 25, 2012. Big Data: Changing The Game for Organizations. Transactions + Interactions + Observations = BIG DATA. BIG DATA. Mobile Web. Petabytes. Sentiment.

erol
Download Presentation

Powering Next-Generation Data Architectures with Apache Hadoop

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Powering Next-Generation Data Architectures with Apache Hadoop Shaun Connolly, Hortonworks @shaunconnolly September 25, 2012

  2. Big Data: Changing The Game for Organizations Transactions + Interactions + Observations = BIG DATA BIG DATA Mobile Web Petabytes Sentiment SMS/MMS Speech to Text User Click Stream Social Interactions & Feeds Terabytes WEB Web logs Spatial & GPS Coordinates A/B testing Sensors / RFID / Devices Behavioral Targeting CRM Gigabytes Business Data Feeds Dynamic Pricing Segmentation External Demographics Search Marketing Customer Touches User Generated Content ERP Affiliate Networks Megabytes Support Contacts Purchase detail Purchase record Payment record HD Video, Audio, Images Dynamic Funnels Offer details Product/Service Logs Offer history Increasing Data Variety and Complexity

  3. Connecting Transactions + Interactions + Observations Audio, Video, Images Business Transactions & Interactions Docs, Text, XML Web Logs, Clicks Web, Mobile, CRM, ERP, SCM, … Big Data Platform Classic ETL processing 1 Social, Graph, Feeds 3 Deliver refined data and runtime models 2 Sensors, Devices, RFID Business Intelligence & Analytics Capture and exchange multi-structured data to unlock value Spatial, GPS Retain runtime models and historical data for ongoing refinement & analysis Retain historical data to unlock additional value 5 4 Events, Other Dashboards, Reports, Visualization, …

  4. optimize optimize optimize Goal: Optimize Outcomes at Scale optimize optimize optimize optimize optimize optimize optimize Source: Geoffrey Moore. Hadoop Summit 2012 keynote presentation.

  5. Customer: UC Irvine Medical Center Optimizing patient outcomes while lowering costs Current system, Epic holds 22 years of patient data, across admissions and clinical information • Significant cost to maintain and run system • Difficult to access, not-integrated into any systems, stand alone Apache Hadoop sunsets legacy system and augments new electronic medical records • Migrate all legacy Epic data to Apache Hadoop • Replaced existing ETL and temporary databases with Hadoop resulting in faster more reliable transforms • Captures all legacy data not just a subset. Exposes this data to EMR and other applications • Eliminate maintenance of legacy system and database licenses • $500K in annual savings • Integrate data with EMR and clinical front-end • Better service with complete patient history provided to admissions and doctors • Enable improved research through complete information • UC Irvine Medical Center is ranked among the nation's best hospitals by U.S. News & World Report for the 12th year • More than 400 specialty and primary care physicians • Opened in 1976 • 422-bed medical facility

  6. Emerging Patterns of Use Big Data Transactions + Interactions + Observations Refine Explore Enrich $ Business Case $

  7. Operational Data RefineryHadoop as platform for ETL modernization Refine Explore Enrich Capture • Capture new unstructured data along with log files all alongside existing sources • Retain inputs in raw form for audit and continuity purposes Process • Parse the data & cleanse • Apply structure and definition • Join datasets together across disparate data sources Exchange • Push to existing data warehouse for downstream consumption • Feeds operational reporting and online systems DB data Unstructured Log files Refinery Capture and archive Parse & Cleanse Structure and join Upload Enterprise Data Warehouse

  8. “Big Bank” Key Benefits • Capture and archive • Retain 3 – 5 years instead of 2 – 10 days • Lower costs • Improved compliance • Transform, change, refine • Turn upstream raw dumps into small list of “new, update, delete” customer records • Convert fixed-width EBCDIC to UTF-8 (Java and DB compatible) • Turn raw weblogs into sessions and behaviors • Upload • Insert into Teradata for downstream “as-is” reporting and tools • Insert into new exploration platform for scientists to play with

  9. Big Data Exploration & VisualizationHadoop as agile, ad-hoc data mart Refine Explore Enrich Capture • Capture multi-structured data and retain inputs in raw form for iterative analysis Process • Parse the data into queryable format • Explore & analyze using Hive, Pig, Mahout and other tools to discover value • Label data and type information for compatibility and later discovery • Pre-compute stats, groupings, patterns in data to accelerate analysis Exchange • Use visualization tools to facilitate exploration and find key insights • Optionally move actionable insights into EDW or datamart DB data Unstructured Log files Explore Capture and archive Structure and join Categorize into tables upload JDBC / ODBC Optional EDW / Datamart Visualization Tools

  10. “Hardware Manufacturer” Key Benefits • Capture and archive • Store 10M+ survey forms/year for > 3 years • Capture text, audio, and systems data in one platform • Structure and join • Unlock freeform text and audio data • Un-anonymize customers • Categorize into tables • Create HCatalog tables “customer”, “survey”, “freeform text” • Upload, JDBC • Visualize natural satisfaction levels and groups • Tag customers as “happy” and report back to CRM database

  11. Application EnrichmentDeliver Hadoopanalysis to online apps Refine Explore Enrich Capture • Capture data that was once too bulky and unmanageable Process • Uncover aggregate characteristics across data • Use Hive Pig and Map Reduce to identify patterns • Filter useful data from mass streams (Pig) • Micro or macro batch oriented schedules Exchange • Push results to HBase or other NoSQL alternative for real time delivery • Use patterns to deliver right content/offer to the right person at the right time DB data Unstructured Log files Enrich Capture Parse Derive/Filter Scheduled & near real time NoSQL, HBase Low Latency Online Applications

  12. “Clothing Retailer” Key Benefits • Capture • Capture weblogs together with sales order history, customer master • Derive useful information • Compute relationships between products over time • “people who buy shirts eventually need pants” • Score customer web behavior / sentiment • Connect product recommendations to customer sentiment • Share • Load customer recommendations into HBase for rapid website service

  13. Hadoop in Enterprise Data Architectures Existing Business Infrastructure Web New Tech Datameer Tableau Karmasphere Splunk Web Applications IDE & Dev Tools ODS &Datamarts Applications & Spreadsheets Visualization & Intelligence Operations EDW Low Latency/NoSQL Discovery Tools Existing Custom Templeton WebHDFS Sqoop Flume HCatalog HBase Pig Hive MapReduce HDFS Ambari Oozie HA ZooKeeper Big Data Sources (transactions, observations, interactions) CRM ERP financials Social Media Exhaust Data logs files

  14. Hortonworks Vision & Role We believe that by the end of 2015, more than half the world's data will be processed by Apache Hadoop. 1 Be diligent stewards of the open source core 2 Be tireless innovators beyond the core 3 Provide robust data platform services & open APIs 4 Enable vibrant ecosystem at each layer of the stack 5 Make Hadoop platform enterprise-ready & easy to use

  15. What’s Needed to Drive Success? • Enterprise tooling to become a complete data platform • Open deployment & provisioning • Higher quality data loading • Monitoring and management • APIs for easy integration • Ecosystem needs support & development • Existing infrastructure vendors need to continue to integrate • Apps need to continue to be developed on this infrastructure • Well defined use cases and solution architectures need to be promoted • Market needs to rally around core Apache Hadoop • To avoid splintering/market distraction • To accelerate adoption www.hortonworks.com/moore

  16. Next Steps? • Expert role based training • Course for admins, developers and operators • Certification program • Custom onsite options Download Hortonworks Data Platform hortonworks.com/download 1 Use the getting started guide hortonworks.com/get-started 2 Learn more… get support 3 Hortonworks Support • Full lifecycle technical support across four service levels • Delivered by Apache Hadoop Experts/Committers • Forward-compatible hortonworks.com/training hortonworks.com/support

  17. Thank You! Questions & Answers Follow:@hortonworks & @shaunconnolly

More Related