1 / 24

A Framework for Data-Intensive Computing with Cloud Bursting

A Framework for Data-Intensive Computing with Cloud Bursting. †. Tekin Bicer David Chiu Gagan Agrawal Department of Compute Science and Engineering The Ohio State University School of Engineering and Computer Science Washington State University. †. 1. Outline. Introduction Motivation

aquintero
Download Presentation

A Framework for Data-Intensive Computing with Cloud Bursting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Framework for Data-Intensive Computing with Cloud Bursting † Tekin Bicer David Chiu Gagan Agrawal Department of Compute Science and Engineering The Ohio State University School of Engineering and Computer Science Washington State University † Cluster 2011 - Texas Austin 1

  2. Outline • Introduction • Motivation • Challenges • MATE-EC2 • MATE-EC2 and Cloud Bursting • Experiments • Conclusion Cluster 2011 - Texas Austin 2

  3. Data-Intensive Computing Cluster 2011 - Texas Austin • Large amounts of data, i.e. Big data • Parallel Processing and Data Parallelism • Local clusters or Supercomputers • High performance interconnects • Local resources might be exhausted • Storage • Computation

  4. Cloud Computing Cluster 2011 - Texas Austin • Computing as a utility • Driving properties • Pay-as-you-go • Elasticity • Data storage • Computation • Different Service Types • IaaS, SaaS, PaaS

  5. From Both Sides Cluster 2011 - Texas Austin • Data-Intensive Computing • Need for large storage, processing and bandwidth • Traditionally on supercomputers or local clusters • Limited resources • Cloud Environments • Availability of elastic storage and processing • e.g. Amazon S3, Amazon EC2 • Unavailability of high performance inter-connect • Cluster Compute Instances, Cluster GPU instances

  6. Cloud Bursting - Motivation • In-house dedicated machines • Workload might vary in time • Demand for more resources • Cloud resources • Collaboration between local and remote resources • Local resources: base workload • Cloud resources: extra workload from users Cluster 2011 - Texas Austin 6

  7. Cloud Bursting - Challenges • Cooperation of the resources • Minimizing the system overhead • Distribution of the data • Job assignments • Determining workload • Time and Cost constraints • Future work Cluster 2011 - Texas Austin 7

  8. Outline • Introduction • Motivation • Challenges • MATE • MATE-EC2 and Cloud Bursting • Experiments • Conclusion Cluster 2011 - Texas Austin 8

  9. MATE vs. Map-Reduce Processing Structure • Reduction Objectrepresents the intermediate state of the execution • Reduce func. is commutative and associative • Sorting, grouping.. overheads are eliminated with red. func/obj. Cluster 2011 - Texas Austin 9

  10. MATE on Amazon EC2 • Data organization • Metadata information • Three levels: Buckets/Files, Chunks and Units • Chunk Retrieval • S3: Threaded Data Retrieval • Local: Cont. read • Selective Job Assignment • Load Balancing and handling heterogeneity • Pooling mechanism Cluster 2011 - Texas Austin 10

  11. MATE-EC2 Processing Flow for AWS S3 Data Object Computing Layer T T T C C C Job Pool Job Scheduler T 2 1 0 n 5 0 3 EC2 Master Node EC2 Slave Node Retrieve chunk pieces and Write them into the buffer Pass retrieved chunk to Computing Layer and process Request another job Request Job from Master Node C0 is assigned as job C5 is assigned as a job Retrieve the new job

  12. System Overview for Cloud Bursting (1) Cluster 2011 - Texas Austin • Local cluster(s) and Cloud Environment • Map-Reduce type of processing • All the clusters connect to a centralized node • Coarse grained job assignment • Consideration of locality • Each clusters has a Master node • Fine grained job assignment • Job Stealing

  13. System Overview for Cloud Bursting(2) Cluster 2011 - Texas Austin

  14. Experiments • 2 geographically distributed clusters • Cloud: EC2 instances running on Virginia • Local: Campus cluster (Columbus, OH) • 3 applications with 120GB of data • Kmeans: k=1000; Knn: k=1000; PageRank: 50x10 links w/ 9.2x10 edges • Goals: • Evaluating the system overhead with different job distributions • Evaluating the scalability of the system 6 8 Cluster 2011 - Texas Austin 14

  15. System Overhead: KNN Cluster 2011 - Texas Austin 15

  16. System Overhead: K-Means Cluster 2011 - Texas Austin 16

  17. System Overhead: PageRank Cluster 2011 - Texas Austin 17

  18. Scalability: KNN Cluster 2011 - Texas Austin 18

  19. Scalability: K-Means Cluster 2011 - Texas Austin 19

  20. Scalability: PageRank Cluster 2011 - Texas Austin 20

  21. Related Work Cluster 2011 - Texas Austin The Cost of Doing Science on the Cloud (Deelman et. Al.; SC’08) Data Sharing Options for Scientific Workflow on Amazon EC2 (Deelman et. Al.; SC’10) Amazon S3 for Science Grids: A viable solution? (Palankar et. al.; DADC’08) Evaluating the Cost Benefit of Using Cloud Computing to Extend the Capacity of Clusters. (Assuncao et. al.; HPDC’09) Elastic Site: Using Clouds to Elastically Extend Site Resources (Marshall et. al.; CCGRID’10) Towards Optimizing Hadoop Provisioning in the Cloud. (Kambatla et. Al.; HotCloud’09)

  22. Future Work Cluster 2011 - Texas Austin • Cloud bursting can answer user requirements • (De)allocate resources on cloud • Time constraint • Given time, minimize the cost on cloud • Cost constraint • Given cost, minimize the execution time

  23. Conclusion • MATE-EC2 is a data intensive middleware developed for Cloud Bursting • Hybrid cloud is new • Most of Map-Reduce implementations consider only one cluster; no known system for cloud bursting • Our results show that • Inter-cluster comm. overhead is low in most data-intensive app. • Jobs can be distributed affectively • Overall slowdown is modest even the disproportion in data dist. increases; our system is scalable

  24. Thanks Any Questions? Cluster 2011 - Texas Austin 24

More Related