Parallel Concept and Hardware Architecture CUDA Programming Model Overview

1. Parallel Concept and Hardware ArchitectureCUDA Programming Model Overview Yukai HungDepartment of MathematicsNational Taiwan University

2. Parallel Concept Overview making your program faster

3. Parallel Computing Goals • Solve the problems in less time • - divide one problem into smaller pieces • - solve smaller problems concurrently • - allow to solve more bigger problems • Prepare to parallelize one problem • - represent algorithms as Directed Acyclic Graphs • - identify dependencies in the problems • - identify critical paths in the algorithms • - modify dependencies to shorten the critical paths 3

4. Parallel Computing Goals • What is parallel computing? 4

5. Amdahl’s Law • Speedup of a parallel program is limited by amount of serial works 5

6. Amdahl’s Law • Speedup of a parallel program is limited by amount of serial works 6

7. Race Condition • Consider the following parallel program • - threads are almost impossibly executed at the same time 7

8. Race Condition • Scenario 1 • - the result value R is 2 if the initial value R is 1 8

9. Race Condition • Scenario 2 • - the result value R is 2 if the initial value R is 1 9

10. Race Condition • Scenario 3 • - the result value R is 3 if the initial value R is 1 10

11. Race Condition with Lock • Solve the race condition by Locking • - manage the shared resource between threads • - avoid the deadlock or unbalanced problems 11

12. Race Condition with Lock • Guarantee the executed instruction order is correct • - the problem is back to the sequential procedure • - lock and release procedure have high overhead 12

13. Race Condition with Semaphore • Solve the race condition by Semaphore • - multi-value locking method (binary locking extension) • - instructions in procedure P and V are atomic operations 13

14. Instruction Level Parallelism • Multiple instructions are executed simultaneously • - reorder the instructions carefully to get efficiency • - compilerreorders the assemble instructions automatically step 1 step 2 step 3 14

15. Data Level Parallelism • Multiple data operations are executed simultaneously • - computational data is separable and independent • - single operation repeated over different input data sequential procedure parallel procedure 15

16. Flynn’s Taxonomy • Classification for parallel computers and programs 16

17. Flynn’s Taxonomy • Classification for parallel computers and programs SISD SIMD 17

18. Flynn’s Taxonomy • Classification for parallel computers and programs MISD MIMD 18

19. CPU and GPU Hardware Comparison

20. CPU versus GPU 20

21. CPU versus GPU Intel Penryn quad-core 255mm2 in 0.82B transistors NVIDIA GTX280 >500mm2 in 1.4B transistors 21

22. CPU versus GPU computing GFLOPS memory bandwidth 22

23. CPU versus GPU GPU CPU comparison control/cache size and number of ALUs comparison the clock rates/core numbers/executing latency 23

24. General Purpose GPU Computation 24

25. General Purpose GPU Computation algorithm conversion requires the knowledge of graphic APIs (OpenGL and DirectX) 25

26. General Purpose GPU Computation convert from pixel data to required data restrict the general usage of algorithms 26

27. Start from Traditional GPU Architecture to Nowadays

28. Simplified Graphic Pipeline sorting stage z-buffer collection 28

29. Simplified Graphic Pipeline maximum depth z-cull feedback 29

30. SimplifiedGraphic Pipeline scale up some units 30

31. Simplified Graphic Pipeline add framebuffer access bottleneck is the FBI unit for memory management 31

32. SimplifiedGraphic Pipeline add programmability by ALU units add programmable geometry and pixel shaders 32

33. Simplified Graphic Pipeline consider two similar units special case on the pipeline (1) one trigonometry but lots of pixels: pixel shader is busy (2) lots of trigonometry but one pixel: geometry shader is busy 33

34. Iterative Graphic Pipeline combine two units to unified shader scalable between geometries and pixels memory resource management becomes important 34

35. Graphic Pipeline Comparison software pipeline hardware pipeline 35

36. SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP TF TF TF TF TF TF TF TF L1 L1 L1 L1 L1 L1 L1 L1 Host Input Assembler Setup / Rstr / ZCull Vtx Thread Issue Geom Thread Issue Pixel Thread Issue Thread Processor L2 L2 L2 L2 L2 L2 FB FB FB FB FB FB Unified Graphic Architecture • Switch on two modes: Graphic and CUDA mode 36

37. Texture Texture Texture Texture Texture Texture Texture Texture Texture Host Input Assembler Thread Execution Manager Parallel DataCache Parallel DataCache Parallel DataCache Parallel DataCache Parallel DataCache Parallel DataCache Parallel DataCache Parallel DataCache Load/store Global Memory Unified Graphic Architecture • Switch on two modes: Graphic and CUDA mode Load/store Load/store Load/store Load/store Load/store 37

38. Thread Streaming Processing no data communication issue which is suitable for traditional graphical issue 38

39. Shader Register File/Cache • Separate register files • - strict streaming processing mode • - no data sharing at instruction-level • - dynamical allocation and renaming • - not exist memory addressing order • - registers overflow into local memory 39

40. Thread Streaming Processing communication issues between different threads (in the same shader) 40

41. Shader Register File/Cache • Shared register files • - extra memory hierarchy between • shader registers and global memory • - share data in the same shader • or threads in the same block • - synchronize all threads in the shader 41

42. Unified Graphic Architecture 42

43. CPU and GPU Chip Architecture 43

44. CPU and GPU Chip Architecture 44

45. Nowadays GPU Applications • Traditional game rendering • - clothing demo and star tales benchmark • - real time physical phenomenon rendering • - mixed mode for physical phenomenon simulation • Scientific computational usage • - molecular dynamic simulation • - protein folding and nbody simulation • - medical images and computational fluid dynamics • CUDA community showcase 45

46. Nowadays GPU Features • GPUs are becoming more programmable than before • - only standard C extension on unified scalable shaders • GPUs now support 32-bit and 64-bit floating point operations • - almost IEEE floating point compliant except for some specials • - lack mantissa denormalizationfor small floating point number • GPUs have much higher memory bandwidth than CPUs • - multiple memory bank driven by need of high-performance graphics • Massive data-level parallel architecture • - hundreds of thread processors on the chip • - thousands of concurrent threads on the shader • - lightweight thread switch and long latency memory 46

47. General Purpose GPU Environment • CUDA: Compute Unified Device Architecture • - realistic hardware and software GPGPU solution • - minimal set of standard C language extensions • - tool set includes compiler and software development kits • OpenCL: Open Computing Language • - similar to CUDA from GPGPU points of view • - support both CPU and GPU hardware architecture • - execute across heterogeneous platform resources 47

48. CUDA Programming Overview

49. . . . . . . CUDA Programming Model • Integrated host and device application C program • - serial or modestly parallel parts in host C code • - highly parallel parts in device C extension code 49

50. CUDA Programming Model • What is the computed device? • - coprocessor to the host part • - have its own device memory space • - run many active threads in parallel • What is the difference between CPU and GPU threads? • - GPU threads are extremely lightweight • - GPU threads have almost no creating overhead • - GPU needs more than 1000 threads for full occupancy • - multiple core CPU can execute or create only a little threads 50