540 likes | 1.16k Views
Overview of Parallel Architecture Yulia Newton CS 147, Fall 2009 SJSU What is it? “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia Humans
E N D
Overview of Parallel Architecture Yulia Newton CS 147, Fall 2009 SJSU
What is it? • “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia
Humans • Human body is a parallel processing architecture • Human body is the most complex machine known to man • We perform parallel processing all the time
Why? • We want to do MORE • We want to do it FASTER
Limitations of Single Processor • Physical • Heat • Electromagnetic interference • Economic • Cost
Parallel Computing • Divide bigger tasks into smaller tasks • Reminds you Divide and Conquer concept? • Perform calculations of those tasks simultaneously (in parallel)
With Single CPU • Parallel processing is possible with single-CPU, single-core computers • Requires very sophisticated software called distributed processing software • Most parallel computing is done with multiple CPU’s
Speedup from parallelization • More processors = faster computing? • Speed-up from parallelization is not linear Can do single task in a unit of time Can do twice as many tasks in same the amount of time Can do four times as many tasks in the same amount of time
Speedup • Definition: Speedup is how much a parallel algorithm is faster than a corresponding sequential algorithm.
Gene Myron Amdahl • Norwegian American computer architect and hi-tech entrepreneur
Amdahl's Law • Formulated in 1960s • Gives potential speed-up on parallel computing system Small portion of components/elements that cannot be parallelized will limit speed-up possible from parallelizable components/elements.
Amdahl's Law (cont’d) where : • S is the overall speed-up of the system • P is the fraction that is parallelizable
Amdahl's Law Alternative Form • where • f – fraction of the program that is enhanced • s – speedup of the enhanced portion
Amdahl's Law Example Program • Program has four parts Code 1 Part 1 = 11% Loop 1 Part 2 = 18% Loop 2 Part 3 = 23% Code 2 Part 4 = 48%
Amdahl's Law Example (cont’d) • Let’s say: • P1 is not sped up, so S1 = 1 or 100% • P2 is sped up 5×, so S2 = 500% • P3 is sped up 20×, so S3 = 2000% • P4 is sped up 1.6×, so S4 = 160%
Amdahl's Law Example (cont’d) • Using P1/S1 + P2/S2 + P3/S3 + P4/S4:
Amdahl's Law Example (cont’d) • 0.4575 is a little less than ½ of the original running time, which is 1 • Overall speed boost is 1 / 0.4575 = 2.186 or a little more than double the original speed • Notice how the 20× and 5× speedup don't have much effect on the overall speed boost and running time when 11% is not sped up, and 48% is sped up by 1.6×
Amdahl's Law Limitations • Based on fixed work load or fixed problem size (strong or hard scaling) • Implies that machine size is irrelevant
John L. Gustafson • American computer scientist and businessman
Gustafson's Law • Formulated in 1988 • Closely related to Amdahl’s Law • Addresses the shortcomings of Amdahl's law, which cannot scale to match availability of computing power as the machine size increases
Gustafson's Law (cont’d) • where P is the number of processors, S is the speedup, and α the non-parallelizable part of the process
Gustafson's Law (cont’d) • Proposes a fixed time concept which leads to scaled speed up for larger problem sizes • Uses weak or soft scaling
Gustafson's Law Limitations • Some problems do not have fundamentally larger datasets • Example: processing one data point per world citizen gets larger at only a few percent per year • Nonlinear algorithms may make it hard to take advantage of parallelism "exposed" by Gustafson's law
Parallel Architectures • Superscalar • VLIW • Vector Processors • Interconnection Networks • Shared Memory Multiprocessors • Distributed Computing • Dataflow Computing • Neural Networks • Systolic Arrays
Superscalar Architecture • Implements instruction-level parallelism (multiple instructions are executed simultaneously in each cycle) • Net effect is the same as pipelining • Additional hardware is required • Contains specialized instruction fetch unit (retrieves multiple instructions simultaneously from memory) • Relies on both hardware and compiler
Superscalar Architecture • Analogous to adding a lane to a highway – more physical resources are required but in the end more cars can pass through
Superscalar Architecture • Examples: • Pentium x86 processors • Intel i960CA • AMD 29000-series
VLIW Architecture • Stands for Very Long Instruction Word • Similar to Superscalar architecture • Relies entirely on compiler • Pack independent instructions into one long instruction • Bigger size of compiled code
VLIW Architecture • Examples: • Intel’s Itanium, IA-64 • Floating Point Systems' FPS164
Vector Processors • Supercomputers are vector processor systems • Specialized, heavily pipelined processors • Efficient operations on entire vectors and matrices
Vector Processors • Heavily pipelined so that arithmetic operations can be overlapped
Vector Processors (example) • Instructions on a traditional processor: For I = 0 to VectorLength V3[i] = V1[i] + V2[i] • Instructions on a vector processor: LDV V1, R1 LDV V2, R2 ADDV R3, R1, R2 STV R3, V3
Vector Processors • Applications: • Weather forecasting • Medical diagnoses systems • Image processing
Vector Processors • Examples: • Cray series of cupercomputers • Xtrillion 3.0
Interconnection Networks • MIMD (Multiple instruction stream, multiple data stream) type architecture • Each processor has its own memory • Processors are allowed to access other processors’ memory via network
Interconnection Networks • Can be: • Dynamic – allow paths between two components to change from one communication to another • Static – do not allow a change of path between communications
Interconnection Networks • Can be: • Non-blocking – allow new connections in the presence of another simultaneous connections • Blocking – do not allow simultaneous connections
Interconnection Networks • Sample interconnection networks:
Shared Memory Architecture • MIMD (Multiple instruction stream, multiple data stream) type architecture • Single memory pool accessed by all processors • Can be categorized by how the memory access is performed: • UMA (Uniform Memory Access) – all processors have equal access to entire memory • NUMA (Non-Uniform Memory Access) – each processor has access to its own piece of memory
Shared Memory Architecture • Diagram of a Shared Memory system:
Distributed Computing • Very loosely coupled multicomputer system, connected by buses or through a network • Main advantage is cost • Main disadvantage is slow communication due to the physical distance between computing units
Distributed Computing • Example: • SETI (Search for Extra Terrestrial Intelligence) group at UC Berkley analyzes data from radio-telescopes • To help this project PC users can install SETI screen saver on their home computer • Data will be analyzed during the processor’s idle time • Half a million years of CPU time accumulated in about 18 months!
Data Flow Computing • Von Neumann machines exhibit sequential control flow; data and instructions are segregated • Dataflow computing – control of the program is directly tied to the data • Order of the instructions does not matter • Each instruction is considered a separate process • Tokens are executed rather than instructions
Data Flow Computing • Data Flow Graph for y=(c+3)*f
Data Flow Computing • Data flow system diagram:
Neural Networks • Good for massively parallel applications, fault tolerance and adapting to changing circumstances • Based on parallel architecture of human brain • Composed of large number of simple processing elements • Trainable
Neural Networks • Model human brain • Perceptron is the simplest example
Neural Networks • Applications: • Quality control • Financial and economic forecasting • Oil and gas exploration • Speech and pattern recognition • Health care cost reduction • Bankruptcy prediction
Systolic Arrays • Networks of processing elements that rhythmically compute data by circulating it through the system • Variations of SIMD architecture