1 / 26

Cloud Computing @Yahoo!

Cloud Computing @Yahoo!. Dekel Tankel Director, Product Management Yahoo! Cloud Computing dekel @yahoo- inc.com. IGT, June 2009. What we’ll cover today…. Why Cloud? Scale and Abstraction; Quality and Agility Yahoo!’s unique footprint Yahoo!’s Cloud Strategy

zaza
Download Presentation

Cloud Computing @Yahoo!

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cloud Computing @Yahoo! Dekel Tankel Director, Product Management Yahoo! Cloud Computing dekel@yahoo-inc.com IGT, June 2009

  2. What we’ll cover today… • Why Cloud? • Scale and Abstraction; Quality and Agility • Yahoo!’s unique footprint • Yahoo!’s Cloud Strategy • Overview of the Yahoo! Cloud vision and portfolio • Deep dive on Horizontal & Functional Cloud Services • The Yahoo! Open Strategy • Marrying Yahoo!’s “Open Strategy”, its platforms and ethic with external Cloud services

  3. Why Cloud? Benefits for Yahoo! Higher Agility & Stability while maintaining Scale • Abstraction • Enable developers to focus on their applications, not infrastructure • Accelerating innovation • Adding new features and products at an ever faster rate • Increasing Scale & Availability • More robustly, more globally, more completely, for a given budget Cloud is pushing up the Operation Excellence Curve Agility & Innovation Quality & Stability

  4. Yahoo!’s Unique Cloud: Unprecedented Scale • Massive user base and engagement • 500M+ unique users per month • Hundreds of petabyte of storage • Hundreds of billions of objects • Hundred of thousands of requests/sec • Global • Tens of globally distributed data centers • Serving each region at low latencies • Challenging Users • Rapidly extracting value from voluminous data • Downtime is not an option (outages cost $millions) • Variable usage patterns

  5. Yahoo! Cloud Services Users Applications Functional CloudServices Horizontal Cloud Services Physical Layer ROI & Innovation Y!OS, BOSS, YQL, APT, Analytics, … Storage, Batch, Edge Serving,…

  6. Yahoo! Cloud Services: Focus on PaaS offerings Users Applications Functional CloudServices Horizontal Cloud Services Physical Layer ROI & Innovation SaaS PaaS IaaS

  7. From Infrastructure to Shareholders benefit • Horizontal Cloud • Focus on open source and collaborative R&D with industry, academia and government • Functional Cloud • Focus on developing "open strategy" frameworks, tools and services for developers (at Yahoo! and beyond) • Combined Together • Leverage our unique scale, assets and data to drive disruptive innovations in the market and expand Yahoo!’s competitive differentiation

  8. Yahoo! Cloud Strategy in Action:The Front Page Case Study • Horizontal Cloud – Storage & Hadoop • Analyze extremely large content data sets • Functional Cloud – Content Optimization • Rate content items based on various parameters • Applications – Yahoo’s Front Page • Display “high rating” items to the right users • Benefit consumers and advertisers and grow Yahoo!’s revenue

  9. Yahoo! Cloud Strategy in Action:The Inquisitor Case Study • Horizontal Cloud – Hadoop • Analyze large search-index data sets • Functional Cloud - BOSS • Expose the data in a structured, open, flexibleand “cloud like” way • Applications - iPhoneTM Inquisitor • Leverage BOSS to provide innovative consumer experience • Benefit consumers and grow Yahoo!’s revenue

  10. Horizontal Cloud Services Users Applications Functional CloudServices Horizontal Cloud Services Physical Layer ROI & Innovation

  11. Horizontal Cloud Services • Optimized for Yahoo!-scale • Yahoo!-internal focus • Data processing and serving environments • Drive faster innovation and agility • Shorter product development cycles • Reduce labor and costs for infrastructure • Multi-year effort • Strategic investment across the company

  12. Horizontal Cloud Services: Conceptual View Simple API’s Operational StorageStructured, unstructured Batch Storage & ProcessingHadoop, PIG Edge Content ServicesCaching, Proxies Online Serving Web, Data Security and Authentication Metering, Billing Monitoring & QoS ID & Account Management Provisioning & Virtualization (Xen) Shared Infrastructure Common Approaches to QA, Production Engineering, Performance Engineering, Datacenter Management, and Optimization

  13. Horizontal Cloud Services: Use Cases Search Index Content Optimization Machine Learning (e.g. Spam filters) Ads Optimization Attachment Storage Image/Video Storage & Delivery

  14. Yahoo! Distribution of Hadoop • Hadoop in a nutshell • Open source distributed file system & parallel execution environment to process massive amounts of data • Started in 2005, became top-level Apache project in 2008 • Simple Design for Horizontal Scaling on commodity HW • Yahoo! Distribution of Hadoop • Source distribution of Yahoo!’s implementation of Hadoop(Based entirely on code found in the Apache Hadoop) • Tested and deployed at Yahoo!’s massive scale • Benefit the larger ecosystem , Increase pace of innovation • http://developer.yahoo.com/hadoop

  15. Yahoo! runs the largest Hadoop Clusters in the World • 25,000+ nodes • Clusters of up to 4,000 nodes • 4 Tiers of clusters • Development & Testing, POCs, Science & Research, Production • Terasort Benchmarks • 62 seconds to sort One Terabyte (run on 1,500 nodes) • 16.25 hours to sort One Petabyte (run on 3,700 nodes) • Webmap application • ~490 TB shuffling • ~280 TB output

  16. Case Study - Search Assist™ • Database for Search Assist™ is built using Hadoop. • 3 years of log-data, 20-steps of map-reduce • Leverage Hadoop’s scalability, load balancing and resiliency • Simplified access, flexibility for rapid innovation (from C++ to Python)

  17. Functional Cloud Services Users Applications Functional CloudServices Horizontal Cloud Services Physical Layer ROI & Innovation

  18. Functional Cloud Services • Provides functional capabilities for applications • Help developers to accomplish integrated web experiences in a faster and easier way • Provides common set of functional “building blocks” • “Powered by” the horizontal cloud services • Abstracts infrastructure services from the Application • E.g. Storage, Compute, Serving, Robustness and Scalability • Self-Served, Global, Managed, Elastic and Metered

  19. Functional Cloud Services: YQL & BOSS Build your Own Search Service Yahoo! Query Language A single endpoint service that enables developers to query, filter and combine data across Yahoo! and beyond http://developer.yahoo.com/yql/console/ Providing Yahoo! Search infrastructure and technology to developers and companies to help them build their own search experiences http://developer.yahoo.com/search/boss/

  20. Build your Own Search Service (BOSS) • Yahoo!'s open search web services platform • Serving hundreds of millions of users across the Web. • Goal: foster innovation in the search industry • Build and launch web-scale search products that utilize the entire Yahoo! Search index. • Access to Yahoo!'s investments in crawling and indexing, ranking and relevancy algorithms

  21. Yahoo! Query Language (YQL) • Single endpoint service to query, filter and combine data across Yahoo! and beyond • The “Internet API” • SQL-like SELECT syntax for getting the right data • Quickly discover available data sources and structure • Combined data from a single web browser • Easy-to-use Consol • http://developer.yahoo.com/yql/console/

  22. Y!OS and Cloud

  23. Yahoo! Open Stagey (Y!OS): Goals

  24. Y!OS and Cloud Strategy CLOUD SERVICES 24

  25. Open Collaborations around the globe • M45 - Yahoo!’s supercomputing cluster • 4,000 cores, 3 TB RAM, 1.5 PB disks, 27 teraflops! • Operational since November 2007, 4 major Universities • Focus on highly parallel computing • Open Cirrus™ with HP & Intel • A global, multi-data center, open source test bed • Target to advance cloud computing research & education • Simulates a real-life, Internet-scale environment • 9 Global sites, more than 50 research projects

  26. Questions? Dekel Tankel Director, Product Management Yahoo! Cloud Computing dekel@yahoo-inc.com

More Related