Partitioning and Divide-and-Conquer Strategies in Cluster Computing

1. Partitioning and Divide-and-Conquer Strategies Cluster Computing, UNC-Charlotte, B. Wilkinson, 2007.

2. Partitioning Partitioning simply divides the problem into parts and then compute the parts and combine results. 1. data partition or domain partition: divide the data and operate upon the divided data concurrently. 2. functional decomposing: divide into independent functions and execute them concurrently. 4.1

3. Divide and Conquer Characterized by dividing problem into sub-problems of same. Further divisions into still smaller sub-problems, usually done by recursion. Recursive divide and conquer amenable to parallelization because separate processes can be used for divided parts. Also usually data is naturally localized. 4.1

4. Partitioning/Divide and Conquer Examples Many possibilities. • Operations on sequences of number such as simply adding them together • Several sorting algorithms can often be partitioned or constructed in a recursive fashion • Numerical integration • N-body problem 4.2

5. Partitioning a sequence of numbers into parts and adding the parts 4.3

6. Possible parallel programs: • 1. Numbers are sent from the master to the slaves. Master will only send the specific numbers to the slaves. Slaves do the summation and send the results back. • 2. Master broadcasts the whole list to slaves. The specifics of the broadcast mechanism would need to be known in order to decide the merits. Usually, a broadcast will have a single startup time rather than separate startup times. • 3. Use scatter and reduce.

11. Speedup factor: Computation and communication ratio:

12. Problems Seen • Speedup factor is low if small n and large m because the slaves are idle during the fourth phase. • Ideally, we want all the processes to be active all of the time, which cannot be achieved by this formulation of problem. • So what strategy is better??

13. Divide and Conquer • Recursive method continually divide a problem until the tasks cannot be broken down into smaller parts. Simple task are performed and results are combined with larger and larger tasks. • A recursive formulation:

14. Tree construction A well balanced tree is created. We can assign each node a separate processor . We will need 2^k -1 processors, where k is the total level number 4.4

15. Dividing a list into parts 4.5

16. Partial summation 4.6

17. Divide and Conquer • Can you figure out the pattern as which processor to communicate during both the dividing and combination phase given binary representation of each processor ID? • Processor communicate with processors whose addresses differ in one bit, starting with the most significant bit for the division phase and with the least significant bit for the combining phase.

19. Complexity Analysis

20. M-ary Divide and Conquer • A task is divided into more than two parts at each stage. Achieve a higher level of parallel processing. • For example, a tree can have four children at each node. • A digitized image could be divided into four quadrants and each of the four quadrants divided into four subquadrants and so on. • An octtree is a tree in which each node has eight children. It is used in three dimension dataset.

22. Quadtree

23. Divide an image

24. Octree Image from wiki

25. Numerical Integration • Computing the “area under the curve.” • Parallelized by divided area into smaller areas (partitioning). • Can also apply divide and conquer -- repeatedly divide the areas recursively.

26. Numerical Integration

27. A better approximation Each region calculated using an approximation given by rectangles: Aligning the rectangles so the midpoint of the rectangle intersect with the curve. : 4.15

28. Numerical integration using trapezoidal method Trapezoid is a quadrilateral 4.16

29. Static Assignment

30. Adaptive Quadrature Number of intervals is fixed. However, we could increase the number of intervals until two rounds of computation give similar results. We can keep doubling the number of interval until computation converges. Solution adapts to shape of curve. Use three areas, A, B, and C. Computation terminated when largest of A and B sufficiently close to sum of remain two areas . 4.17

31. Adaptive quadrature with false termination. Some care might be needed in choosing when to terminate. Might cause us to terminate early, as two large regions are the same (i.e., C = 0). 4.18

32. Gravitational N-Body Problem Finding positions and movements of bodies in space subject to gravitational forces from other bodies, using Newtonian laws of physics. 4.25

33. Gravitational N-Body Problem Equations Gravitational force between two bodies of masses maand mbis: G is the gravitational constant and r the distance between the bodies. Subject to forces, body accelerates according to Newton’s 2nd law: F = ma m is mass of the body, F is force it experiences, and a the resultant acceleration. 4.26

34. Details Let the time interval be t. For a body of mass m, the force is: New velocity is: where vt+1 is the velocity at time t + 1 and vtis the velocity at time t. Over time interval Dt, position changes by where xtis its position at time t. Once bodies move to new positions, forces change. Computation has to be repeated. 4.27

36. Sequential Code Overall gravitational N-body computation can be described by: for (t = 0; t < tmax; t++) /* for each time period */ for (i = 0; i < N; i++) { /* for each body */ F = Force_routine(i); /* compute force on ith body */ v[i]new = v[i] + F * dt / m; /* compute new velocity */ x[i]new = x[i] + v[i]new * dt; /* and new position */ } for (i = 0; i < N; i++) { /* for each body */ x[i] = x[i]new; /* update velocity & position*/ v[i] = v[i]new; } 3.28

37. Parallel Code The sequential algorithm is an O(N2) algorithm (for one iteration) as each of the N bodies is influenced by each of the other N - 1 bodies. Not feasible to use this direct algorithm for most interesting N-body problems where N is very large. 4.29

38. Time complexity can be reduced approximating a cluster of distant bodies as a single distant body with mass sited at the center of mass of the cluster: M =sum (mi); and C = 1/M * sum(mi * ci) 4.30

39. Barnes-Hut Algorithm Start with whole space in which one cube contains the bodies (or particles). • First, this cube is divided into four subcubes. • If a subcube contains no particles, subcube deleted from further consideration. • If a subcube contains one body, subcube retained. • If a subcube contains more than one body, it is recursively divided until every subcube contains one body. 4.31

40. Recursive division of 2-dimensional space 4.34

41. Barnes-Hut Algorithm • 1) Build the QuadTree using QuadTreeBuild = O( N * maximum dept of Quad_Tree) • … already described, cost = O( N log N) • 2) For each node = subsquare in the QuadTree, compute the • CM and total mass (TM) of all the particles it contains • … “post order traversal” of QuadTree, cost = O(N log N) • 3) For each particle, traverse the QuadTree to compute the force on it, • using the CM and TM of “distant” subsquares • … core of algorithm • … cost depends on accuracy desired but still O(N log N)

42. Creates an octtree - a tree with up to eight edges from each node. The leaves represent cells each containing one body. After the tree has been constructed, the total mass and center of mass of the subcube is stored at each node. 4.32

43. Force on each body obtained by traversing tree starting at root, stopping at a node when the clustering approximation can be used, e.g. when: where is a constant typically 1.0 or less. Constructing tree requires a time of O(nlogn), and so does computing all the forces, so that overall time complexity of method is O(nlogn). 4.33

44. Orthogonal Recursive Bisection (For 2-dimensional area) First, a vertical line found that divides area into two areas each with equal number of bodies. For each area, a horizontal line found that divides it into two areas each with equal number of bodies. Repeated as required. 4.35

45. Astrophysical N-body simulation By Scott Linssen (UNCC student, 1997) using O(N2) algorithm. 4.36

46. Astrophysical N-body simulation By David Messager (UNCC student 1998) using Barnes-Hut algorithm. 4.37

47. Mandelbrot dynamic load assignment /*variable declaration and initialization*/ int ibuf[BUFSIZE]={0}, jbuf[BUFSIZE]={0}, kbuf[BUFSIZE]={0}; long int time1, time2, total_time; ….. …… MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if (myid == 0) { /* connect to Xserver */ if ( (display = XOpenDisplay (display_name)) == NULL ) { fprintf (stderr, "drawon: cannot connect to X server %s\n", XDisplayName (display_name) ); exit (-1); } Int terminated = 0; /*get screen size, set window position, create opague window, create graphics context etc*/ ….. ……

48. Master node keeps check result, receive and send task while( !terminated) { for(id = 1; id < numprocs; id++) { // Check for any messages waiting to be received MPI_Iprobe(id, 0, MPI_COMM_WORLD, &flag, &status); if(flag == 1) { // If you find one, receive it MPI_Recv(&ibuf, bufsize, MPI_INT, id, 0, MPI_COMM_WORLD, &status); MPI_Recv(&jbuf, bufsize, MPI_INT, id, 1, MPI_COMM_WORLD, &status); MPI_Recv(&kbuf, bufsize, MPI_INT, id, 2, MPI_COMM_WORLD, &status); /*draw points*/ XSetForeground (display, gc, kbuf[i]); XDrawPoint (display, win, gc, jbuf[i], ibuf[i]); // Search for next available task, If a task is found, Send the task say row number with data_tag to be 1 to slave with id MPI_Send(&row, 1, MPI_INT, id, 1, MPI_COMM_WORLD); } //If NO task is marked Available or Busy then all tasks are DONE terminated= 1; }

49. Slave node keeps receiving task, computing • while( (myid!=0) && (flag2 != 1) ){ • MPI_Iprobe(0, 1, MPI_COMM_WORLD, &flag, &status); // Waiting to receive from 0 with • tag=1 • if(flag == 1) { // If you find one, receive it • MPI_Recv(&i1, 1, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); • for(i = low; i < high; i++) • for(j=0; j < Y_RESN; j++) { // for each pixel loop the computation • z.real = z.imag = 0.0; • c.real = ((float) j - 400.0)/200.0; • c.imag = ((float) i - 400.0)/200.0; • k = 0; • do { /* iterate for pixel color */ • temp = z.real*z.real - z.imag*z.imag + c.real; • z.imag = 2.0*z.real*z.imag + c.imag; • z.real = temp; • lengthsq = z.real*z.real+z.imag*z.imag; • k++; • } while (lengthsq < 4.0 && k < 100); • ibuf[counter1++] = i; • jbuf[counter2++] = j; • kbuf[counter3++] = k; • } // match for loop

50. Slave node keeps receiving task, computing continues MPI_Send(&ibuf, bufsize, MPI_INT, 0, 0, MPI_COMM_WORLD); // sending x-coords MPI_Send(&jbuf, bufsize, MPI_INT, 0, 1, MPI_COMM_WORLD); // sending y-coords MPI_Send(&kbuf, bufsize, MPI_INT, 0, 2, MPI_COMM_WORLD); // sending color } //match if (flag==1) MPI_Iprobe(0, 3, MPI_COMM_WORLD, &flag2, &status); // check if master sends termination with data_tag to be 3 if( flag2 == 1 ) // termination received, slave should stop probing by flag2==1 MPI_Recv(&i1, 1, MPI_INT, 0, 3, MPI_COMM_WORLD, &status); }