1 / 16

Twister: A Runtime for Iterative MapReduce

Twister: A Runtime for Iterative MapReduce. HPDC – 2010 MAPREDUCE’10 Workshop, Chicago, 06/22/2010. Jaliya Ekanayake Community Grids Laboratory, Digital Science Center Pervasive Technology Institute Indiana University. Acknowledgements to:. Co authors:

nakia
Download Presentation

Twister: A Runtime for Iterative MapReduce

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Twister: A Runtime for Iterative MapReduce HPDC – 2010 MAPREDUCE’10 Workshop, Chicago, 06/22/2010 Jaliya Ekanayake Community Grids Laboratory, Digital Science Center Pervasive Technology Institute Indiana University

  2. Acknowledgements to: Co authors: Hui Li, Binging Shang, Thilina Gunarathne Seung-Hee Bae, Judy Qiu, Geoffrey Fox School of Informatics and Computing Indiana University Bloomington Team at IU

  3. Input Input map iterations Input Motivation map map Pij Data Deluge Output MapReduce Classic Parallel Runtimes (MPI) reduce reduce Data Centered, QoS Efficient and Proven techniques Experiencing in many domains Expand the Applicability of MapReduce to more classes of Applications Iterative MapReduce More Extensions Map-Only MapReduce

  4. Features of Existing Architectures(1) • Google, Apache Hadoop, Sector/Sphere, • Dryad/DryadLINQ (DAG based) Programming Model MapReduce (Optionally “map-only”) Focus on Single Step MapReduce computations (DryadLINQ supports more than one stage) Input and Output Handling Distributed data access (HDFS in Hadoop, Sector in Sphere, and shared directories in Dryad) Outputs normally goes to the distributed file systems Intermediate data Transferred via file systems (Local disk-> HTTP -> local disk in Hadoop) Easy to support fault tolerance Considerably high latencies

  5. Features of Existing Architectures(2) Scheduling A master schedules tasks to slaves depending on the availability Dynamic Schedulingin Hadoop, static scheduling in Dryad/DryadLINQ Naturally load balancing Fault Tolerance Data flows through disks->channels->disks A master keeps track of the data products Re-execution of failed or slow tasks Overheads are justifiable for large single step MapReduce computations Iterative MapReduce

  6. Iterate Static data Configure() A Programming Model for Iterative MapReduce User Program Map(Key, Value) δ flow • Distributed data access • In-memory MapReduce • Distinction on static data and variable data (data flow vs. δ flow) • Cacheable map/reduce tasks (long running tasks) • Combine operation • Support fast intermediate data transfers Reduce (Key, List<Value>) Combine (Map<Key,Value>) Close()

  7. runMapReduce(..) Iterations Worker Nodes Twister Programming Model configureMaps(..) configureReduce(..) Local Disk while(condition){ Cacheable map/reduce tasks May send <Key,Value> pairs directly Map() Reduce() Combine() operation Communications/data transfers via the pub-sub broker network updateCondition() Two configuration options : Using local disks (only for maps) Using pub-sub bus } //end while close() User program’s process space

  8. B B B B Twister Driver Master Node Worker Pool Worker Pool Local Disk Local Disk Pub/sub Broker Network Twister Architecture Main Program One broker serves several Twister daemons Twister Daemon Twister Daemon map reduce Cacheable tasks Scripts perform: Data distribution, data collection, and partition file creation Worker Node Worker Node

  9. Node 0 Node 1 Node n Input/Output Handling Data Manipulation Tool A common directory in local disks of individual nodes e.g. /tmp/twister_data Partition File Data Manipulation Tool: Provides basic functionality to manipulate data across the local disks of the compute nodes Data partitions are assumed to be files (Contrast to fixed sized blocks in Hadoop) Supported commands: mkdir,rmdir, put,putall,get,ls, Copy resources Create Partition File

  10. Partition File Partition file allows duplicates One data partition may reside in multiple nodes In an event of failure, the duplicates are used to re-schedule the tasks

  11. map task queues The use of pub/sub messaging Map workers Broker network Reduce() Intermediate data transferred via the broker network Network of brokers used for load balancing Different broker topologies Interspersed computation and data transfer minimizes large message load at the brokers Currently supports NaradaBrokering ActiveMQ

  12. Scheduling Twister supports long running tasks Avoids unnecessary initializations in each iteration Tasks are scheduled statically Supports task reuse May lead to inefficient resources utilization Expect user to randomize data distributions to minimize the processing skews due to any skewness in data

  13. Fault Tolerance Recover at iteration boundaries Does not handle individual task failures Assumptions: Broker network is reliable Main program & Twister Driver has no failures Any failures (hardware/daemons) result the following fault handling sequence Terminate currently running tasks (remove from memory) Poll for currently available worker nodes (& daemons) Configure map/reduce using static data (re-assign data partitions to tasks depending on the data locality) Re-execute the failed iteration

  14. Performance Evaluation Hardware Configurations We use the academic release of DryadLINQ, Apache Hadoop version 0.20.2, and Twister for our performance comparisons. Both Twister and Hadoop use JDK (64 bit) version 1.6.0_18, while DryadLINQ and MPI uses Microsoft .NET version 3.5.

  15. Pagerank – An Iterative MapReduce Algorithm Partial Adjacency Matrix Current Page ranks (Compressed) M Partial Updates R Well-known pagerank algorithm [1] Used ClueWeb09 [2] (1TB in size) from CMU Reuse of map tasks and faster communication pays off Partially merged Updates C Iterations [1] Pagerank Algorithm, http://en.wikipedia.org/wiki/PageRank [2] ClueWeb09 Data Set, http://boston.lti.cs.cmu.edu/Data/clueweb09/

  16. Conclusions & Future Work Twister extends the MapReduce to iterative algorithms Several iterative algorithms we have implemented K-Means Clustering Pagerank Matrix Multiplication Multi dimensional scaling (MDS) Breadth First Search Integrating a distributed file system Programming with side effects yet support fault tolerance

More Related