1 / 35

Parallel Processing: Architecture Overview

This overview provides an introduction to parallel processing, discussing its history, motivating factors, and different types of parallel systems. It also explores the architecture and programming paradigms of parallel processing.

dominickl
Download Presentation

Parallel Processing: Architecture Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WW Grid Parallel Processing: Architecture Overview Grid Computing and Distributed Systems (GRIDS) Lab. The University of MelbourneMelbourne, Australiawww.gridbus.org Rajkumar Buyya

  2. Q Please Serial Vs. Parallel COUNTER 2 COUNTER COUNTER 1

  3. Overview of the Talk • Introduction • Why Parallel Processing ? • Parallel System H/W Architecture • Parallel Operating Systems

  4. Threads Interface Microkernel Multi-Processor Computing System . . P P P P P P P Processor Process Thread Computing Elements Applications Programming paradigms Operating System Hardware

  5. Architectures System Software/Compiler Applications P.S.Es Architectures System Software Applications P.S.Es Commercialization R & D Commodity Two Eras of Computing Sequential Era Parallel Era 1940 50 60 70 80 90 2000 2030

  6. History of Parallel Processing • The notion of parallel processing can be traced to a tablet dated around 100 BC. • Tablet has 3 calculating positions capable of operating simultaneously. • From this we can infer that: • They were aimed at “speed” or “reliability”.

  7. Motivating Factors • Just as we learned to fly, not by constructing a machine that flaps its wings like birds, but by applying aerodynamics principles demonstrated by the nature... • Similarly parallel processing has been modeled after those of biological species. • Aggregated speed with which complex calculations carried out by (billions of) neurons demonstrate feasibility of PP. • Individual neuron response speed is slow (ms) –

  8. Why Parallel Processing? • Computation requirements are ever increasing -- visualization, distributed databases, simulations, scientific prediction (earthquake), etc. • Silicon based (sequential) architectures reaching physical limits in processing limits as they are constrained by: • the speed of light, thermodynamics

  9. Human Architecture! Growth Performance Vertical Horizontal Growth 5 10 15 20 25 30 35 40 45 . . . . Age

  10. Computational Power Improvement Multiprocessor Uniprocessor C.P.I 1 2 . . . . No. of Processors

  11. Why Parallel Processing? • Hardware improvements like pipelining, superscalar are not scaling well and require sophisticated compiler technology to exploit performance out of them. • Techniques such as vector processing works well for certain kind of problems.

  12. Why Parallel Processing? • Significant development in networking technology is paving a way for network-based cost-effective parallel computing. • The parallel processing technology is mature and is being exploited commercially.

  13. Parallel Programs • Consist of multiple active “processes” simultaneously solving a given problem. • And the communication and synchronization between them (parallel processes) forms the core of parallel programming efforts.

  14. Tightly Couple Systems: Shared Memory Parallel Smallest extension to existing systems Program conversion is incremental Distributed Memory Parallel Completely new systems Programs must be reconstructed Loosely Coupled Systems: Clusters Built using commodity systems Centralised management Grids Aggregation of distributed systems Decentralized management Types of Parallel Systems

  15. Processing Elements Architecture

  16. Processing Elements • Flynn proposed a classification of computer systems based on a number of instruction and data streams that can be processed simultaneously. • They are: • SISD (Single Instruction and Single Data) • Conventional computers • SIMD (Single Instruction and Multiple Data) • Data parallel, vector computing machines • MISD (Multiple Instruction and Single Data) • Systolic arrays • MIMD (Multiple Instruction and Multiple Data) • General purpose machine

  17. Instructions Processor Data Output Data Input SISD : A Conventional Computer • Speed is limited by the rate at which computer can transfer information internally. Ex: PCs, Workstations

  18. Instruction Stream A Instruction Stream B Instruction Stream C Processor A Data Output Stream Data Input Stream Processor B Processor C The MISD Architecture • More of an intellectual exercise than a practical configuration. Few built, but commercially not available

  19. Instruction Stream Data Output stream A Data Input stream A Processor A Data Output stream B Processor B Data Input stream B Data Output stream C Processor C Data Input stream C SIMD Architecture Ex: CRAY machine vector processing, Thinking machine cm* Intel MMX (multimedia support) Ci<= Ai * Bi

  20. MIMD Architecture Instruction Stream A Instruction Stream C Instruction Stream B Unlike SISD, MISD, MIMD computer works asynchronously. Shared memory (tightly coupled) MIMD Distributed memory (loosely coupled) MIMD Data Output stream A Data Input stream A Processor A Data Output stream B Processor B Data Input stream B Data Output stream C Processor C Data Input stream C

  21. MEMORY MEMORY MEMORY BUS BUS BUS Shared Memory MIMD machine Processor A Processor B Processor C Comm: Source PE writes data to GM & destination retrieves it • Easy to build, conventional OSes of SISD can be easily be ported • Limitation : reliability & expandibility. A memory component or any processor failure affects the whole system. • Increase of processors leads to memory contention. Ex. : Silicon graphics supercomputers.... Global Memory System

  22. MEMORY MEMORY MEMORY BUS BUS BUS Memory System A Memory System B Memory System C Distributed Memory MIMD IPC channel IPC channel Processor A Processor B Processor C • Communication : IPC (Inter-Process Communication) via High Speed Network. • Network can be configured to ... Tree, Mesh, Cube, etc. • Unlike Shared MIMD • easily/ readily expandable • Highly reliable (any CPU failure does not affect the whole system)

  23. S log2P P Laws of caution..... • Speed of computation is proportional to the square root of system cost. i.e. Speed = Cost • Speedup by a parallel computer increases as the logarithm of the number of processors. • Speedup = log2(no. of processors) C S

  24. Caution.... • Very fast development in network computing and related area have blurred concept boundaries, causing lot of terminological confusion : concurrent computing, parallel computing, multiprocessing, supercomputing, massively parallel processing, cluster computing, distributed computing, Internet computing, grid computing, etc. • At the user level, even well-defined distinctions such as shared memory and distributed memory are disappearing due to new advances in technology. • Good tools for parallel program development and debugging are yet to emerge.

  25. Caution.... • There is no strict delimiters for contributors to the area of parallel processing: • computer architecture, operating systems, high-level languages, algorithms, databases, computer networks, … • All have a role to play.

  26. Operating Systems forHigh Performance Computing

  27. Shared Memory Parallel Smallest extension to existing systems Program conversion is incremental Distributed Memory Parallel Completely new systems Programs must be reconstructed Clusters Slow communication form of Distributed Types of Parallel Systems

  28. Operating Systems for PP • MPP systems having thousands of processors requires OS radically different from current ones. • Every CPU needs OS : • to manage its resources • to hide its details • Traditional systems are heavy, complex and not suitable for MPP

  29. Operating System Models • Frame work that unifies features, services and tasks performed • Three approaches to building OS.... • Monolithic OS • Layered OS • Microkernel based OS • Client server OS • Suitable for MPP systems • Simplicity, flexibility and high performance are crucial for OS.

  30. Application Programs Application Programs User Mode Kernel Mode System Services Hardware Monolithic Operating System • Better application Performance • Difficult to extend Ex: MS-DOS

  31. Layered OS Application Programs Application Programs User Mode • Easier to enhance • Each layer of code access lower level interface • Low-application performance Kernel Mode System Services Memory & I/O Device Mgmt Process Schedule Hardware Ex : UNIX

  32. Application Programs Application Programs Traditional OS User Mode Kernel Mode OS Hardware OS Designer

  33. New trend in OS design Servers Application Programs Application Programs User Mode Kernel Mode Microkernel Hardware

  34. User Kernel Microkernel/Client Server OS(for MPP Systems) • Tiny OS kernel providing basic primitive (process, memory, IPC) • Traditional services becomes subsystems • Monolithic Application Perf. Competence • OS = Microkernel + User Subsystems Client Application Thread lib. File Server Network Server Display Server Microkernel Send Reply Hardware Ex: Mach, PARAS, Chorus, etc.

  35. Few Popular Microkernel Systems • MACH, CMU • PARAS, C-DAC • Chorus • QNX, • (Windows)

More Related