1 / 59

Uintah Runtime System Tutorial

www.uintah.utah.edu. Uintah Runtime System Tutorial. Outline of Runtime System How Uintah achieves parallelism Uintah Memory management Adaptive Mesh refinement and Load Balancing Executing tasks – Task Graph and Scheduler Heterogeneous architectures GPUs & Intel MIC.

knox
Download Presentation

Uintah Runtime System Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. www.uintah.utah.edu Uintah Runtime System Tutorial • Outline of Runtime System • How Uintah achieves parallelism • Uintah Memory management • Adaptive Mesh refinement and Load Balancing • Executing tasks – Task Graph and Scheduler • Heterogeneous architectures GPUs & Intel MIC Alan Humphrey, Justin Luitjens,* Dav de St. Germain and Martin Berzins *NVIDIA here as an ex-Uintah Researcher/Developer DOE ASCI (97-10), NSF , DOE NETL+NNSA ARL NSF , INCITE, XSEDE

  2. Uintah OverviewDav de St. GermainUniversity of Utah

  3. Dav de St. Germains slides go here • Outline of runtime system • Uintah parallelism • DataWarehouse, TaskGraph, etc

  4. Adaptive Mesh Refinement & Load BalancingMartin BerzinsUniversity of Utah Martin – This will be your section. I have changed nothing here. Alan H., 07/23/14

  5. Scalable AMR Regridding Algorithm Berger-Rigoutsos Recursively split patches based on histograms Histogram creation requires all-gathers O(Processors) Complex and does not parallelize well Irregular patch sets Two versions – version 2 uses less cores in forming histogram

  6. AMR regridding algorithm (Berger-Rigoutsos) Number of points in each column Box is split into 2 3 3 3 1 1 3 3 2 Tagged points Initial box Process is repeated on the two new boxes (1) tag cells where refinement is needed (2) create a box to enclose tagged cells (3) split the box along its long direction based on a histogram of tagged cells This step is repeated at least log(B) times where B is the number of patches (4) fit new boxes to each split box and repeat the steps as needed.

  7. Berger Rigoutsos Communications costs • Every processor has to pass its part of the histogram to every other so that the partitioning decision can be made • The is done using the MPI_ALLGATHER function - cost of an allgather on P (cores) processors is Log(P) Version 1: (2B-1)log(P) messages Version 2: Only certain cores take part [Gunney et al.] 2P –log(P)-2 messages Alternative: use Berger Rigoutsos locally on each processor

  8. Scalable AMR Regridding Algorithm Tiled Algorithm Tiles that contain flags become patches Simple and easy to parallelize Semi-regular patch sets that canbe exploited Example: Neighbor finding Each core processes subset of refinement flags in parallel-helps produce global patch set [Analysis to appear in Concurrency]

  9. EXAMPLE – MESH REFINEMENT AROUND A CIRCULAR FRONT Global Berger- Rigoutsos Local Berger- Rigoutsos Tiled Algorithm More patches generated by tiled but noneed to split them to load balance

  10. Theoretical Models of Refinement Algorithms C = number of mesh cells in domain F = number of refinement flags in domain B = number of mesh patches in the domain Bc = number of coarse mesh patches in the domain P = number of processing cores M = number of messages T: GBRv1 = c1 (F/P) log(B) + c2 M(GBRv1) T: LBR = c4 (F/P) log(B/Bc) T: Tiled = c5 C/P T: Split = c6 B/P + c7 log (P) - is the time to split large patches

  11. Strong Scaling of AMR Algorithms GBR is Global Berger-Rigoutsos – problem size fixed as number of cores increases should lead to decreasing execution time. No strong scaling Strong Scaling T: GBRv1 = c1 (Flags/Proc) log(npatches ) + c2 M(GBRv1) T: Tiled = c5 Nmeshcells / Proc T: LBR = GBR on a compute node Dots are data and lines are models

  12. Performance Improvements Improve Algorithmic Complexity Identify & Eliminate O(Processors) O(Patches) via Tau, and hand profiling, memory usage analysis Improve Task Graph Execution Out of order execution of task graph e.g.  Improve Load Balance Cost Estimation Algorithms based on data assimilation Use load balancing algorithms based on patches and a new fast space filling curve algorithm  Hybrid model using message passing between nodes and asynchronous threaded execution of tasks on cores Partial Ice task graph

  13. Load Balancing Weight Estimation Algorithmic Cost Models based on discretization method Vary according to simulation+machine Requires accurate information from the user Time Series Analysis Used to forecast time for execution on each patch Automatically adjusts according to simulation and architecture with no user interaction Need to assign the same workload to each processor. Er,t: Estimated Time Or,t: Observed Time α: Decay Rate Er,t+1 =αOr,t + (1 -α) Er,t =α (Or,t-Er,t) + Er,t Error in last prediction Simple Exponential Smoothing:

  14. Comparison between Forecast Cost Model FCM & Algorithmic Cost Model Particles + Fluid code FULL SIMULATION

  15. UintahRuntime SystemAlan HumphreyUniversity of Utah

  16. Uintah Task Scheduler (De-centralized) • One MPI Process per Multicore node (MPI + Pthreads) • All threads access task queues and process their own MPI • Tasks for one patch may run on different cores • Shared data warehouse and task queues per multicore node • Lock-free data warehouse enables fast mem access to all cores

  17. Uintah Runtime System Data Management Network 3 2 Data Warehouse (one per-node) Thread 1 Execution Layer (runs on multiple cores) MPI_IRecv Internal Task Queue Task Graph recv Select Task & Post MPI Receives Comm Records Check Records & Find Ready Tasks valid MPI_Test External Task Queue get Select & Execute Task put MPI_ISend Post Task MPI Sends Shared Objects (one per node) send

  18. New Hybrid Model Memory Savings: Ghost Cells MPI: Thread/MPI: Raw Data: 49152 doubles 31360 doubles MPI buffer: 28416 doubles 10624 doubles Total: 75K doubles40Kdoubles Local Patch Ghost Cells (example on Kraken, 12 cores/node, 98K core 11% of memory needed

  19. Uintah Threaded Scheduler – Task Selection (per thread) // Each thread executes this method, getting work to do concurrently void UnifiedScheduler::runTasks(intt_id) { while (numTasksDone < ntasks) { while (!havework) { /* Run a CPU task that has its MPI communication complete. These tasks * get in the external ready queue automatically when their receive count * hits 0 in DependencyBatch::received, called when a MPI msg is delivered. * * NOTE: This is where the execution cycle begins for each GPU-enabled Task. */ if (dts->numExternalReadyTasks() > 0) { readyTask = dts->getNextExternalReadyTask(); if (readyTask != NULL) { havework = true; numTasksDone++; break; ......... // Other types of work to check for }

  20. Uintah Threaded Scheduler – Task Execution (per thread) /* If we have an internally-ready CPU task, initiate its MPI receives, * preparing it for CPU external ready queue. The task is moved to the CPU * external-ready queue in the call to task->checkExternalDepCount(). */ else if (dts->numInternalReadyTasks() > 0) { initTask= dts->getNextInternalReadyTask(); if (initTask != NULL) { if (initTask->getTask()->getType() == Task::Reduction || initTask-> getTask()->usesMPI()) { // delay reduction until end of TS phaseSyncTask[initTask->getTask()->d_phase] = initTask; ASSERT(initTask->getRequires().size() == 0) initTask= NULL; } else if (initTask->getRequires().size() == 0) { // no ext. dependencies, then skip MPI sends initTask->markInitiated(); // where tasks get added to external ready queue initTask->checkExternalDepCount(); initTask= NULL; } else { havework = true; break; } } } // otherwise process MPI recvs - call pendingMPIRecvs();

  21. Uintah Threaded SchedulerOverhead and Challenges • Multiple MPI communicators • Each thread has one communicator • Allow multiple global tasks run simultaneously • No tags for MPI_Reduce, MPI_Allgather (Even in MPI3) • Third party libs need communicator : PETSc • Modify existing task and infrastructure code • Thread-safe MPI libraries: MPI_THREAD_MULTIPLE • Thread-safe tasks: “Most” Uintah tasks are now thread-safe • Tasks from Component (user) code must be thread-safe • Overhead for acquiring locks becomes significant with high thread counts. Need to move more sections of infrastructure to Mutex-free, concurrent access data structures. • Multi-threaded coding complexity and often painful debugging. Many bugs have only manifested at high core counts with many threads per node.

  22. Uintah Threaded Scheduler – Thread Execution void UnifiedScheduler::execute(inttgnum /*=0*/, intiteration /*=0*/) { // signal worker threads to begin executing tasks if (!d_isInitTimestep && !d_isRestartInitTimestep) { for (inti = 0; i < numThreads_; i++) { t_worker[i]->resetWaittime(Time::currentSeconds()); // reset wait time // sending signal to threads to wake them up t_worker[i]->d_runmutex.lock(); t_worker[i]->d_idle = false; t_worker[i]->d_runsignal.conditionSignal(); t_worker[i]->d_runmutex.unlock(); } } // main thread also executes tasks (tid 0) runTasks(0); . . . . . . . } void UnifiedSchedulerWorker::run() { Thread::self()->set_affinity(d_id + 1); while (true) { //wait for main thread signal d_runsignal.wait(d_runmutex); d_runmutex.unlock(); d_scheduler->runTasks(d_id + 1); //signal main thread for next group of tasks . . . . . . . }

  23. Lock-Free Data Structures Global scalability depends on the details of nodal run-time system. Change from Jaguar to Titan – more faster cores and faster communications broke our Runtime System which worked fine with locks previously • Using atomic instruction set • Variable reference counting • fetch_and_add, fetch_and_subcompare_and_swap • both read and write simultaneously • Data warehouse • Redesigned variable container • Update: compare_and_swap • Reduce: test_and_set

  24. Uintah DWDatabase – Using Atomics template<class DomainType> intDWDatabase<DomainType>::decrementScrubCount(constVarLabel* label, intmatlIndex, constDomainType* dom) { … intrt = __sync_sub_and_fetch(&(scrubs[idx]),1); … } template<class DomainType> Void DWDatabase<DomainType>::putVar(constVarLabel* label, intmatlIndex, constDomainType* dom, RVariable* var, boolinit) { newdi->var = var; … DataItem* olddi = __sync_lock_test_and_set(&vars[idx], 0); … newdi = __sync_lock_test_and_set(&vars[idx], olddi); }

  25. Scalability Improvement De-centralized MPI/Thread Hybrid Scheduler (with Lock-free Data Warehouse) And new Threaded runtime system Original Dynamic MPI-only Scheduler Out of memory on 98K cores Severe load imbalance • Achieved much better CPU Scalability

  26. Schedule Global Sync Task R1 R2 • Synchronization task • Update global variable • e.g. MPI_Allreduce • Call third party library • e.g. PETSc, hypre • Out-of-order issues • Deadlock • Load imbalance • Task phases • One global task per phase • Global task runs last • In phase out-of-order R2 R1 Deadlock Load imbalance

  27. Multi-threaded Task Graph Execution • 1) Static: Predetermined order • Tasks are Synchronized • Higher waiting times Task Dependency Execution Order • 2) Dynamic: Execute when ready • Tasks are Asynchronous • Lower waiting times • 3) Dynamic Multi-threaded • Task-Level Parallelism • Memory saving • Decreases Communication and Load Imbalance Multicorefriendly Support GPU tasks (Future)

  28. Scalability is at least partially achieved by not executing tasks in order e.g. AMR fluid-structure interaction Straight line represents given order of tasks Green X shows when a task is actually executed. Above the line means late execution while below the line means early execution took place. More “late” tasks than “early” ones as e.g. TASKS: 1 2 3 4 5 1 4 2 3 5 Early Late execution

  29. UintahRuntime System GPUS

  30. NVIDIA Fermi Outline Host memory to Device memory is max 8GB/second, BUT … Device memory to cores is 144GB/second with less than half of this being achieved on memory bound applications ( Blas level 1,2) 40GFlops

  31. Hiding PCIe Latency • Nvidia CUDA Asynchronous API • Asynchronous functions provide: • Memcopies asynchronous with CPU • Concurrently execute memcopy with kernel(s) • Stream - sequence of operations that execute in order on GPU • Operations from different streams can be interleaved Page-locked Memory Normal Data Transfer Data Transfer Kernel Execution Kernel Execution

  32. Extend Scheduler to Support GPUs? ?

  33. Uintah Heterogeneous Runtime System

  34. GPU Task and Data Management Pin this memory with cudaHostRegister() Framework Manages Data Movement Host   Device Call-back executed here (kernel run) existing host memory • Use CUDA Asynchronous API • Automatically generate CUDA streams for task dependencies • Concurrently execute kernels and memory copies • Preload data before task kernel executes • Multi-GPU support cudaMemcpyAsync(H2D) hostRequires devRequires Page locked buffer computation Result back on host hostComputes devComputes cudaMemcpyAsync(D2H) Free pinned host memory Automatic D2H copy here

  35. Multistage Task Queue Architecture • Overlap computation with PCIe transfers and MPI communication • Uintah can “pre-load” GPU task data • Scheduler queries task-graph for a task’s data requirements • Migrate data dependencies to GPU and backfill until ready

  36. Uintah Threaded Scheduler – GPU Task Selection void UnifiedScheduler::runTasks(intt_id) #ifdefHAVE_CUDA /* If it's a GPU-enabled task, assign it to a device, initiate its * H2D computes and requires data copies. This is where the execution cycle * begins for each GPU-enabled Task. */ if (readyTask->getTask()->usesDevice()) { readyTask->assignDevice(currentDevice_); currentDevice_++; currentDevice_ %= numDevices_; gpuInitReady = true; } /* Check if highest priority GPU task's asynchronous H2D copies are * completed. If so, then reclaim the streams used for these operations, * execute the task and put it into the GPU completion-pending queue. */ else if (dts->numInitiallyReadyDeviceTasks() > 0) { readyTask= dts->peekNextInitiallyReadyDeviceTask(); if (readyTask->checkCUDAStreamComplete()) { // cudaStreamQuery() // All of this GPU task's H2D copies are complete, prepare to run. readyTask = dts->getNextInitiallyReadyDeviceTask(); gpuRunReady = true; havework = true; break; } } #endif

  37. Uintah Threaded Scheduler – GPU Task Execution void UnifiedScheduler::selectTasks(int t_id) ……… #ifdefHAVE_CUDA else if (gpuInitReady) { // initiate all asynchronous H2D memory copies for this task's requires readyTask->setCUDAStream(getCudaStream(readyTask->getDeviceNum())); postH2DCopies(readyTask); preallocateDeviceMemory(readyTask); for (inti = 0; i < (int)dws.size(); i++) { dws[i]->getGPUDW(readyTask->getDeviceNum())->syncto_device(); } dts->addInitiallyReadyDeviceTask(readyTask); } else if (gpuRunReady) { // otherwise run the task runTask(readyTask, currentIteration, t_id, Task::GPU); postD2HCopies(readyTask); dts->addCompletionPendingDeviceTask(readyTask); } else if (gpuPending) { // recycle this task's D2H copies streams reclaimCudaStreams(readyTask); } #endif

  38. GPU DataWarehouse Async H2D Copy • Automatic, on-demand variable movement to-and-from device • Implemented interfaces for both CPU/GPU Tasks Host Device CPU Task GPU Task MPI Buffer CPU Task dw.get() dw.put() MPI Buffer Hash map Flat array Async D2H Copy

  39. Uintah Threaded Scheduler – GPU DataWarehouse API // Host-side interface GPUDataWarehouse* old_gdw = old_dw->getGPUDW()->getdevice_ptr(); // Device-side interface __global__ void rayTraceKernel(patchParams patch, curandState* randNumStates, GPUDataWarehouse* old_gdw, GPUDataWarehouse* new_gdw) { // calculate the thread indices inttidX = threadIdx.x + blockIdx.x * blockDim.x + patch.loEC.x; inttidY = threadIdx.y + blockIdx.y * blockDim.y + patch.loEC.y; ......... GPUGridVariable<double> divQ; new_gdw->get( divQ, "divQ", patch.ID, matl); for (int z = patch.lo.z; z < patch.hi.z; z++) { // loop through z slices gpuIntVector c = make_int3(tidX, tidY, z); divQ[c] = c.x + c.y + c.z; } divQ[origin] = 4.0 * M_PI * abskg[origin] * ( sigmaT4OverPi[origin] - (sumI/RT_flags.nDivQRays) ); } __host__ __device__ T& operator[](const int3& idx) { // get data - global index return d_data[idx.x - d_offset.x + d_size.x * (idx.y - d_offset.y + (idx.z - d_offset.z) * d_size.y)]; }

  40. Performance and Scaling Comparisons • Uintah strong scaling results when using: • Multi-threaded MPI • Multi-threaded MPI w/ GPU • GPU-enabled RMCRT • All-to-all communication • 100 rays per cell 1283cells • Need port Multi-level CPU tasks to GPU Cray XK-7, DOE Titan Thread/MPI: N=16 AMD Opteron CPU cores Thread/MPI+GPU: N=16 AMD Opteron CPU cores + 1 NVIDIA K20 GPU

  41. UintahRuntime System Xeon Phi (Intel MIC)

  42. Xeon Phi Execution Models

  43. Uintah on Stampede: Host-only Model Incompressible turbulent flow • Intel MPI issues beyond 2048 cores (seg faults) • MVAPICH2 required for larger core counts Using Hypre with a conjugate gradient solver Preconditioned with geometric multi-grid Red Black Gauss Seidel relaxation - each patch

  44. Uintah on Stampede: Native Model • Compile with –mmic • cross compiling • Need to build all Uintah required 3p libraries * • libxml2 • zlib • Run Uintah natively on Xeon Phi within 1 day • Single Xeon Phi Card * During early access to TACC Stampede

  45. Uintah on Stampede: Offload Model • Use compiler directives (#pragma) • Offload target: #pragma offload target(mic:0) • OpenMP: #pragma ompparallel • Find copy in/out variables from task graph • Functions called in MIC must be defined with __attribute__((target(mic))) • Hard for Uintah to use offload mode • Rewrite highly templated C++ methods with simple C/C++ so they can be called on the Xeon Phi • Less effort than GPU port, but still significant work for complex code such as Uintah

  46. Uintah on Stampede: Symmetric Model • Xeon Phi directly calls MPI • Use Pthreads on both host CPU and Xeon Phi: • 1 MPI process on host – 16 threads • 1 MPI process on MIC – up to 120 threads • Currently only Intel MPI supported mpiexec.hydra-n 8 ./sus – nthreads16 : -n 8./sus.mic –nthreads 120 • Challenges: Different floating point accuracy on host and co-processor • Result Consistency • MPI message mismatch issue: Control related FP operations

  47. Scaling Results on Xeon Phi • Multi MIC Cards (Symmetric Model) • Xeon Phi card: 60 threads per MPI process, 2 MPI processes • host CPU :16 threads per MPI process, 1 MPI processes • Issue: load imbalance - profiling differently on host and Xeon Phi Multiple MIC cards

  48. Summary • DAG abstraction important for achieving scaling. Powerful abstraction for solving challenging engineering problems • Clear applicability to new architectures - accelerators and co-processors • Provides convenient separation of problem structure from data and communication - application code vs. runtime • Layered approach very important for not needing to change applications code. Shields applications developer from complexities of parallel programming • Scalability still requires engineering of the runtime system. Scalability still a challenge even with DAG approach – which does work amazingly well • GPU development ongoing • The approach used here shows promise for very large core and GPU counts but using these architectures is an exciting challenge • DSL approach very important very future • Waiting for Access to in-socket MICs (Knight’s Landing)

  49. DELETED SLIDES

More Related