Introduction

2. Introduction • What class is this? • CS 484 – Parallel Processing • Who am I? • Quinn Snell • 2218 TMCB • snell@cs.byu.edu • 422-5098 • When are office hours? • By appointment

3. Course and Text • Homepage (Everything is here) • http://dna.cs.byu.edu/CS484 • Text: • Online tutorials & papers

4. Course Objectives • Students will understand and demonstrate the ability to develop • Shared memory parallel programs (OpenMP & Pthreads), • Distributed memory parallel programs (MPI), and • Data parallel programs (CUDA) • Students will understand and be able to implement several basic parallel algorithms and load balancing techniques.

5. Lectures • Follow schedule on the homepage • We will move quickly. • READ BEFORE CLASS!!!!! • There will be in-class quizzes - no makeup • I will post lecture notes on the web.

6. What is expected of you? • READ!!! • Assignments • C/C++ on the Supercomputer & CS Open Labs • All assignments include a report. The report IS the assignment. The program is what you did to accomplish the assignment. - I will grade your writing. • Exams • In the testing center • 8 1/2 x 11 sheet of notes allowed

7. What is expected of you? • Get an account on the university supercomputers • Go to marylou.byu.edu • Register • Do it today!!! • Read through the batch jobs tutorial for marylou • Schedule a simple hello world using a PBS script. • copy ~qos/hello/hello.c and ~qos/hello/hello.pbs • edit hello.pbs • mpicc hello.c • qsub hello.pbs

8. Grading • Grade Distribution • Quizzes 5 % • Homework & labs 55 % • Midterm 15 % • Final 15 % • Project(tutorial) 10% • Grade Scale • 94 % - Above A • 90 % - 93.9 % A- • 80 % - 89.9 % B-, B, B+ • 65 % - 79.9 % C-, C, C+ • I expect everyone to get a good grade!

9. Policies • Late Work • I don’t like late work. However….. • 10% off per school day – limited to 70% off • Programming Assignments • To get better than a C-, you must complete ALL labs • If you don’t complete all the labs, your grade ceiling is a C-

10. Other Policies • Honor Code: • I expect you to follow the honor code including the dress and grooming standards. • You can work together in groups on the homework and laboratories from a conceptual perspective, but the answers that you give and the programs that you write should be your own, not copies of other students work. • Your reports should focus on your own ideas and things you have learned from your experimentation.

11. Other Policies • Cheating • Cheating in any form will NOT be tolerated. This includes copying any part of a homework assignment or programming lab. • Any assignment turned in that is not your work will be given a negative full value score. Any cases of multiple offense or cheating on a test will result in failure of the class. • Systems Abuse Policy • Abuse in any form will result in immediate suspension of your accounts(s).

12. Other Policies • Preventing Sexual Harassment • Title IX of the Education Amendments of 1972 prohibits sex discrimination against any participant in an educational program or activity receiving federal funds. Title IX covers discrimination in programs, admissions, activities, and student-to-student sexual harassment. • BYU's policy against sexual harassment extends not only to employees but to students as well. • If you encounter unlawful sexual harassment or gender based discrimination, please talk to your professor; contact the Equal Employment Office at 377-5895 or 367-5629 (24 hours); or contact the Honor Code Office at 378-2847.

13. Other Policies • Students With Disabilities • Brigham Young University is committed to providing a working and learning atmosphere that reasonably accommodates qualified persons with disabilites. If you have any disability, which may impair your ability to complete this course successfully, please contact the Services for Students with Disabilities Office (378-2767). Reasonable academic accommodations are reviewed for all students who have qualified documented disabilites.

14. Other Policies • Children in the Classroom • The study of Computer Science requires a degree of concentration and focus that is exceptional. Having small children in class is often a distraction that degrades the educational experience for the whole class. Please make other arrangements for child care rather than bringing children to class with you. If there are extenuating circumstances, please talk with your instructor in advance.

15. What will we do in this class? • Focus on shared memory and message passing programming • PThreads • OpenMP • MPI • Data parallel programming with CUDA • Write code for the supercomputers • Study parallel algorithms and parallelization

16. What is Parallelism? • Multiple tasks working at the same time on the same problem.

17. Parallel Computing • What is a parallel computer? • A set of processors that are able to work cooperatively to solve a computational problem • Examples • Parallel supercomputers • Clusters of workstations • Symmetric multiprocessors • Multiple core processors

18. Won't serial computers be fast enough? • Moore's Law • Double in speed every 18 months • Predictions of need • British government in 1940s predicted they would only need about 2-3 computers • Market for Cray was predicted to be about 10 • Problem • Doesn't take into account new applications.

19. Applications Drive Supercomputing • Traditional • Weather simulation and prediction • Climate modeling • Chemical and physical computing • New apps. • Collaborative environments • DNA • Virtual reality • Parallel databases • Games • Photoshop

20. Application Needs • Graphics • 109 volume elements • 200 operations per element • Real-time display • Weather & Climate • 10 year simulation involves 1016 operations • Accuracy can be improved by higher resolution grids which involves more operations.

21. Cost-Performance Trend 1990s 1980s 1970s Performance 1960s Cost Serial computers

22. What does this suggest? • More performance is easy to a point. • Significant performance increases of current serial computers beyond the saturation point is extremely expensive. • Connecting large numbers of microprocessors into a parallel computer overcomes the saturation point. • Cost stays low and performance increases.

23. Computer Design • Single processor performance has been increased lately by increasing the level of internal parallelism. • Multiple functional units • Pipelining • Higher performance gains by incorporating multiple "computers on a chip.” • Gigahertz race is over (Intel won) • Multiple cores is where performance will come from

24. Computer Performance TFLOPS 1e12 IBM SP-2 Cray C90 Cray X-MP Cray 1 CDC 7600 1e7 IBM 7090 IBM 704 Eniac 1e2 1950 1975 2000

25. Communication Performance • Early 1990s Ethernet 10 Mbits • Mid 1990s FDDI 100 Mbits • Mid 1990s ATM 100s Mbits • Late 1990s Fast Ethernet 100 Mbits • Late 1990s Gig Ethernet 100s Mbits • Soon 1000 Mbits will be commonplace

26. Performance Summary • Applications are demanding more speed. • Performance trends • Processors are increasing in speed. • Communication performance is increasing. • Future • Performance trends suggest a future where parallelism pervades all computing. • Concurrency is key to performance increases.

27. Parallel Processing Architectures • Architectures • Single computer with lots of processors • Multiple interconnected computers • Architecture governs programming • Shared memory and locks • Message passing

28. Shared Memory Computers • Uniform Memory Access • All access the same memory • can lead to memory bottleneck • Non-uniform Memory Access • different speed access • more scalable

29. Message Passing Computers

30. Distributed Shared Memory Interconnection Network Processors Memory Modules

31. Message Passing Architectures • Requires some form of interconnection • The network is the bottleneck • Latency and bandwidth • Diameter • Bisection bandwidth

32. Message Passing Architectures • Line/Ring • Mesh/Torus

33. Message Passing Architectures • Tree/Fat Tree

34. Message Passing Architectures • Hypercube

35. Message Passing Architectures • Switched • Nice, but also has limitations

36. Parallel Programming Properties • Concurrency • Performance should increase by employing multiple processors. • Scalability • Performance should continue to increase as we add more processors. • Locality of Reference • Performance will be greater if we only access local memory. • Architecture affects each of the above!

37. Parallel Programming Models Threads

38. Parallel Programming Models Message Passing

39. Parallel Programming Models Data Parallel

40. So What? In this class you will learn to program using each of the parallel programming models. We will talk about advantages/disadvantages of each model We will also learn about common parallel algorithms and techniques