170 likes | 267 Views
Introduction to Scalable Programming using Makeflow and Work Queue. Dinesh Rajan and Mike Albrecht University of Notre Dame October 24 and November 7, 2012. Go to: http://nd.edu/~ccl. Click “Tutorial: Introduction to Scalable Programming”.
E N D
Introduction to Scalable Programming using Makeflow and Work Queue Dinesh Rajan and Mike Albrecht University of Notre Dame October 24 and November 7, 2012
Go to: http://nd.edu/~ccl Click “Tutorial: Introduction to Scalable Programming”
I have a standard, debugged, trusted application that runs on my laptop. One simulation runs in an hour. I have to run 100. Then I have to analyze the results, tweak the simulation, and run 100 more. Can I get a single result faster? Can I get more results in the same time?
Last year, I heard about this grid thing. This year, I heard about this cloud thing. What do I do next?
Should I port my program to MPI or Hadoop? Learn C / Java Learn MPI / Hadoop Re-architect Re-write Re-test Re-debug Re-certify
I can get as many machineson the cloud as I want!How do I organize my applicationto run on those machines?
An Old Idea: Makefiles part1 part2 part3: input.data split.py ./split.py input.data out1: part1 mysim.exe ./mysim.exe part1 >out1 out2: part2 mysim.exe ./mysim.exe part2 >out2 out3: part3 mysim.exe ./mysim.exe part3 >out3 result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result
Makeflow = Make + Workflow • Provides portability across batch systems. • Enable parallelism (but not too much!) • Fault tolerance at multiple scales. • Data and resource management. Makeflow Local Condor SGE Work Queue http://www.nd.edu/~ccl/software/makeflow
Makeflow Language - Rules part1 part2 part3: input.data split.py ./split.py input.data out1: part1 mysim.exe ./mysim.exe part1 >out1 out2: part2 mysim.exe ./mysim.exe part2 >out2 out3: part3 mysim.exe ./mysim.exe part3 >out3 result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result • Each rule specifies: • a set of target files to create; • a set of source files needed to create them; • a commandthat generates the target files from the source files. out1 :part1 mysim.exe mysim.exe part1 > out1
Makeflow + Batch System CRCSGE Cluster Private Cluster Makefile makeflow –T sge Work Queue Makeflow Work Queue Campus Condor Pool Public Cloud Provider makeflow –T condor Local Files and Programs
Drivers • Local • Condor • SGE • Batch • Hadoop • WorkQueue • Torque • MPI-Queue • XGrid • Moab
How to run a Makeflow • Run a workflow locally (multicore?) • makeflow -T local sims.mf • Clean up the workflow outputs: • makeflow –c sims.mf • Run the workflow on SGE: • makeflow –T sgesims.mf
Hands On http://nd.edu/~ccl/software/tutorials/ndtut12/mf-tutorial.php
Practice Problems http://nd.edu/~ccl/software/tutorials/ndtut12/mf-hw.php • Construct a makeflow to render a short movie featuring a Rubik’s cube • Launch the makeflow on both your laptop and SGE • Consider ways you might use Makeflow for your research