1 / 27

Adaptive, hybrids, oblivious : what do those terms mean ?

SIAM Parallel Processing’2006 - Feb 22 Mini Symposium Adaptive Algorithms for Scientific computing. Adaptive, hybrids, oblivious : what do those terms mean ? Taxonomy of autonomic computing [Ganek & Corbi 2003] : Self-configuring / self-healing / self-optimising / self-protecting

naeva
Download Presentation

Adaptive, hybrids, oblivious : what do those terms mean ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SIAM Parallel Processing’2006 - Feb 22 Mini SymposiumAdaptive Algorithms for Scientific computing • Adaptive, hybrids, oblivious : what do those terms mean ? • Taxonomy of autonomic computing [Ganek & Corbi 2003] : • Self-configuring / self-healing / self-optimising / self-protecting • Objective: towards an analysis based on the algorithm performance 9h45 Adaptive algorithms - Theory and applicationsJean-LouisRoch &al. AHA Team INRIA-CNRS Grenoble, France 10h15 Hybrids in exact linear algebraDave Saunders U. Delaware, USA 10h45 Adaptive programming with hierarchical multiprocessor tasksThomas Rauber, Gudula Runger, U. Bayreuth, Germany 11h15 Cache-Oblivious algorithmsMichael Bender, Stony Brook U., USA

  2. Adaptive algorithmsTheory and applications Van Dat Cung, Jean-Guillaume Dumas, Thierry Gautier, Guillaume Huard, Bruno Raffin, Jean-Louis Roch, Denis Trystram IMAG-INRIA Workgroup on “Adaptive and Hybrid Algorithms” Grenoble, France • Contents • I. Some criteria to analyze adaptive algorithms • II. Work-stealing and adaptive parallel algorithms • III. Adaptive parallel prefix computation

  3. Choices in the algorithm • sequential / parallel(s) • approximated / exact • in memory / out of core • … An algorithm is « hybrid » iff there is a choice at a high level between at least two algorithms, each of them could solve the same problem Why adaptive algorithms and how? Resources availability is versatile Input data vary Measures on resources Measures on data Adaptation to improve performances • Scheduling • partitioning • load-balancing • work-stealing • Calibration • tuning parameters block size/ cache choice of instructions, … • priority managing

  4. Adaptationto choose algo_fj for each call to f Modeling an hybrid algorithm • Several algorithms to solve a same problem f : • Eg : algo_f1, algo_f2(block size), … algo_fk : • each algo_fk being recursive algo_fi ( n, … ) { …. f ( n - 1, … ) ; …. f ( n / 2, … ) ; … } • E.g. “practical” hybrids: • Atlas, Goto, FFPack • FFTW • cache-oblivious B-tree • any parallel program with scheduling support: Cilk, Athapascan/Kaapi, Nesl,TLib… .

  5. How to manage overhead due to choices ? • Classification 1/2 : • Simple hybrid iff O(1) choices [eg block size in Atlas, …] • Baroque hybrid iff an unbounded number of choices [eg recursive splitting factors in FFTW] • choices are either dynamic or pre-computed based on input properties.

  6. Choices may or may not be based on architecture parameters. • Classification 2/2. : an hybrid is • Oblivious: control flow does not depend neither on static properties of the resources nor on the input [eg cache-oblivious algorithm [Bender] • Tuned : strategic choices are based on static parameters [eg block size w.r.t cache, granularity, ] • Engineered tuned or self tuned[eg ATLAS and GOTO libraries, FFTW, …][eg [LinBox/FFLAS] [ Saunders&al] • Adaptive : self-configuration of the algorithm, dynamlc • Based on input properties or resource circumstances discovered at run-time[eg : idle processors, data properties, …] [eg TLib Rauber&Rünger]

  7. Examples • BLAS libraries • Atlas: simple tuned (self-tuned) • Goto : simple engineered (engineered tuned) • LinBox / FFLAS : simple self-tuned,adaptive [Saunders&al] • FFTW • Halving factor : baroque tuned • Stopping criterion : simple tuned • Parallel algorithm and scheduling : • Choice of parallel degree : eg Tlib [Rauber&Rünger] • Work-stealing schedile : baroque hybrid

  8. Adaptive algorithmsTheory and applications Van Dat Cung, Jean-Guillaume Dumas, Thierry Gautier,Guillaume Huard, Bruno Raffin, Jean-Louis Roch, Denis Trystram INRIA-CNRS Project on“Adaptive and Hybrid Algorithms” Grenoble, France • Contents • I. Some criteria to analyze for adaptive algorithms • II. Work-stealing and adaptive parallel algorithms • III. Adaptive parallel prefix computation

  9. Work-stealing (1/2) « Work » W1= #total operations performed «Depth » W = #ops on a critical path (parallel time on resources) • Workstealing = “greedy” schedule but distributed and randomized • Each processor manages locally the tasks it creates • When idle, a processor steals the oldest ready task on a remote -non idle- victim processor (randomly chosen)

  10. Work-stealing (2/2) « Work » W1= #total operations performed «Depth » W = #ops on a critical path (parallel time on resources) • Interests : -> suited to heterogeneous architectures with slight modification [Bender-Rabin02] -> with good probability, near-optimal schedule on p processors with average speeds aveTp < W1/(p ave) + O ( W/ ave ) NB : #succeeded steals = #task migrations < p W [Blumofe 98, Narlikar 01, Bender 02] • Implementation: work-first principle [Cilk, Kaapi] • Local parallelism is implemented by sequential function call • Restrictions to ensure validity of the default sequential schedule - serie-parallel/Cilk - reference order/Kaapi

  11. Work-stealing and adaptability • Work-stealing ensures allocation of processors to tasks transparently to the application with provable performances • Support to addition of new resources • Support to resilience of resources and fault-tolerance (crash faults, network, …) • Checkpoint/restart mechanisms with provable performances [Porch, Kaapi, …] • “Baroque hybrid” adaptation: there is an -implicit- dynamic choice between two algorithms • a sequential (local) algorithm : depth-first (default choice) • A parallel algorithm : breadth-first • Choice is performed at runtime, depending on resource idleness • Well suited to applications where a fine grain parallel algorithm is also a good sequential algorithm [Cilk]: • Parallel Divide&Conquer computations • Tree searching, Branch&X … -> suited when both sequential and parallel algorithms perform (almost) the same number of operations

  12. But often parallelism has a cost ! • Solution: to mix both a sequential and a parallel algorithm • Basic technique : • Parallel algorithm until a certain « grain »; then use the sequential one • Problem : W increases also, the number of migration … and the inefficiency ;o( • Work-preserving speed-up[Bini-Pan 94] = cascading [Jaja92] Careful interplay of both algorithms to build one with both W small and W1 = O( Wseq ) • Divide the sequential algorithm into block • Each block is computed with the (non-optimal) parallel algorithm • Drawback : sequential at coarse grain and parallel at fine grain ;o( • Adaptive granularity: dual approach : • Parallelism is extracted at run-time from any sequential task

  13. SeqCompute SeqCompute Extract_par LastPartComputation Self-adaptive grain algorithm Based on the Work-first principle : Executes always a sequential algorithm to reduce parallelism overhead => use parallel algorithm only if a processor becomes idle by extracting parallelism from a sequential computation Hypothesis : two algorithms : • - 1 sequential : SeqCompute- 1 parallel : LastPartComputation : at any time, it is possible to extract parallelism from the remaining computations of the sequential algorithm • Examples : - iterated product [Vernizzi 05] - gzip / compression [Kerfali 04] - MPEG-4 / H264 [Bernard 06] - prefix computation [Traore 06]

  14. Adaptive algorithmsTheory and applications Van Dat Cung, Jean-Guillaume Dumas, Thierry Gautier,Guillaume Huard, Bruno Raffin, Jean-Louis Roch, Denis Trystram INRIA-CNRS Project on“Adaptive and Hybrid Algorithms” Grenoble, France • Contents • I. Some criteria to analyze for adaptive algorithms • II. Work-stealing and adaptive parallel algorithms • III. Adaptive parallel prefix computation

  15. * * * * Prefix of size n/2 13 … n * * * 24 … n-1 Prefix computation : an example where parallelism always costs1 = a0*a1 2=a0*a1*a2…n=a0*a1*…*an • Sequential algorithm:for (i= 0 ; i <= n; i++ ) [ i ] = [ i – 1 ] * a [ i ] ; • Parallel algorithm [Ladner-Fischer]: W1= W = n a0 a1 a2 a3 a4 … an-1 an W =2. log n butW1= 2.n Twice more expensive than the sequential …

  16. Adaptive prefix computation Any (parallel) prefix performs at least W1 2.n - W ops Strict-lower bound on p identical processors: Tp 2n/(p+1) block algorithm + pipeline [Nicolau&al. 2000] Application of adaptive scheme : One process performs the main “sequential” computation Other work-stealer processes computes parallel « segmented » prefix Near-optimal performance on processors with changing speeds :Tp < 2n/((p+1). ave) + O ( log n / ave) lower bound

  17. 0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 Main Seq.  Steal request  Work-stealer 1  Work-stealer 2 Adaptive Prefix on 3 processors 1

  18. 0 a1 a2 a3 a4 Main Seq.  1 2 Steal request  a5 a6 a7 a8 a9 a10 a11 a12  6 Work-stealer 1 i=a5*…*ai  Work-stealer 2 Adaptive Prefix on 3 processors 3 7

  19. 0 a1 a2 a3 a4 Main Seq.  1 2 3 4 8 8 Preempt 4  a5 a6 a7 a8  6 7 Work-stealer 1 i=a5*…*ai a9 a10 a11 a12  10 Work-stealer 2 i=a9*…*ai Adaptive Prefix on 3 processors 8

  20. 0 a1 a2 a3 a4 8 Main Seq.  1 2 3 4 11 Preempt 11 8  a5 a6 a7 a8  6 7 Work-stealer 1 i=a5*…*ai a9 a10 a11 a12  10 Work-stealer 2 i=a9*…*ai Adaptive Prefix on 3 processors 8 5 6 8 9 11

  21. 0 a1 a2 a3 a4 8 11 a12 Main Seq.  1 2 3 4  a5 a6 a7 a8  6 7 Work-stealer 1 i=a5*…*ai a9 a10 a11 a12  10 Work-stealer 2 i=a9*…*ai Adaptive Prefix on 3 processors 8 11 12 5 6 7 8 9 10 11

  22. 0 a1 a2 a3 a4 8 11 a12 Main Seq.  1 2 3 4 8 11 12  a5 a6 a7 a8  5 6 6 7 7 8 Work-stealer 1 i=a5*…*ai a9 a10 a11 a12  9 10 10 11 Work-stealer 2 i=a9*…*ai Adaptive Prefix on 3 processors Implicit critical path on the sequential process

  23. Parallel Parallel Adaptive Adaptive Adaptive prefix : some experiments Join work with Daouda Traore Prefix of 10000 elements on a SMP 8 procs (IA64 / linux) External charge Time (s) Time (s) #processors #processors Multi-user context Adaptive is the fastest15% benefit over a static grain algorithm • Single user context • Adaptive is equivalent to: • - sequential on 1 proc • - optimal parallel-2 proc. on 2 processors • - … • - optimal parallel-8 proc. on 8 processors

  24. Adaptative 8 proc. Parallel 8 proc. Parallel 7 proc. Parallel 6 proc. Parallel 5 proc. Parallel 4 proc. Parallel 3 proc. Parallel 2 proc. Sequential The Prefix race: sequential/parallel fixed/ adaptive On each of the 10 executions, adaptive completes first

  25. Conclusion Adaptive : what choices and how to choose ? Illustration : Adaptive parallel prefix based on work-stealing - self-tuned baroque hybrid : O(p log n ) choices - achieves near-optimal performance processor oblivious Generic adaptive scheme to implement parallel algorithms with provable performance

  26. Mini SymposiumAdaptive Algorithms for Scientific computing • Adaptive, hybrids, oblivious : what do those terms mean ? • Taxonomy of autonomic computing [Ganek & Corbi 2003] : • Self-configuring / self-healing / self-optimising / self-protecting • Objective: towards an analysis based on the algorithm performance 9h45 Adaptive algorithms - Theory and applications Jean-Louis Roch &al. AHA Team INRIA-CNRS Grenoble, France 10h15 Hybrids in exact linear algebra Dave Saunders, U. Delaware, USA 10h45 Adaptive programming with hierarchical multiprocessor tasks Thomas Rauber, U. Bayreuth, Germany 11h15 Cache-Obloivious algorithms Michael Bender, Stony Brook U., USA

  27. Questions ?

More Related