High Performance Computing Systems and Enabling Platforms

Master Program (Laurea Magistrale) in Computer Science and NetworkingAcademicYear 2009-2010 High Performance ComputingSystemsand EnablingPlatforms Marco Vanneschi Departmentof Computer Science, Universityof Pisa

Master Program (Laurea Magistrale) in Computer Science and Networking High Performance ComputingSystems and EnablingPlatforms Marco Vanneschi CourseIntroduction

Myactivity • Research area • Computer Architecture • Parallel and Distributed Processing, High Performance Computing • ParallelProgrammingModels and Tools • Programmabilityofvarious HPC platforms • Multiprocessor, Cluster, GridComputing, Multi-core, Pervasive Computing • Coordinationof some National and EuropeanProjects (basicresearch and industrial research) • Researchgroup • Co-leadedwith Prof. Marco Danelutto • LaboratoryofParallelArchitectures • Strong relationshipresearch - teaching MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Thiscourse (acronym: SPA) My Personal Page:www.di.unipi.it/vanneschsection: Teaching Link in DidaWikitomy Personal Page: http://www.cli.di.unipi.it/doku/doku.php/magistraleinformaticanetworking/spa/start • FundamentalcourseofLaurea Magistrale in Computer Science and Networking, 1st Year • In common withLaurea Magistrale in Informatica • complementarycourse (studyplan in DistributedSystems) • ASE (vecchio ordinamento, 9 CFU): 6 CFU di SPA + 3 CFU di integrazione (vedi in seguito) MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

SPA 6 CFU = 48 hours (12 weeks) CREDIT DEFINITION: 1 CFU = 25 hours = 8 hours for lectures / class activities (lab, practical, etc) + 17 hours for individual study MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Contents • Objectives, motivations, approach • An informalpresentationof some concepts and technologies • Background and prerequisites • Courseprogram • Course material/notes • Exammodality and workingapproach MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Courseobjectives • Provide a solidknowledgeframeworkofconcepts and techniques in high-performance computer architecture • Organization and structureofenablingplatformsbased on parallelarchitectures • Supporttoparallelprogrammingmodels and software developmenttools • Performance evaluation (costmodels) • Methodologyforstudyingexisting and future systems • Technology: state-of-the-art and trends • Parallelprocessors • Multiprocessors • Multicore / manycore / … / GPU • Sharedvsdistributedmemoryarchitectures • Programmingmodels and theirsupport • General-purposevsdedicatedplatforms MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Motivations • Basically, the samemotivationdiscussed in “DistributedSystems: Paradigms and Models” (Prof. Marco Danelutto): • evolutionof computer technologytowardsparallelism and HPC • Multi/manycore • Large cluster, cloud, … • Heterogeneouslarge-scaleenabling ICT platforms • Embedding • Increasingmaturitywithrespectto “hardware-software relationship” • BothTechnologyPushand Technology Pull • Language-drivenarchitecturalapproaches • Concurrency and parallelismasfirst-classcitizens in applicationdevelopment • In our Master: HPC is a fundamentalmethodology and technologyforintegrated ICT infrastructures and applications MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Courseapproach: a Computer Science approach • The concept of Enabling Platform: strong relationship and integration between architectures and applications • Computing architectures are NOT “boxes and wires” methodological knowledge technological knowledge This approach does not imply that concrete technologies (of current and future) computing platforms are neglected – far from it: our goal is to fully understand and to utilize existing architectures at best, and even to define new ones. • Computer Science approach • Computingarchitecturehasitsownconcepts, principles, models, and techniques • Conceptualframework in common with the otherdisciplinesof Computer Science: • Programminglanguages, algorithms, computability and complexity, … MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Main Memory and Cache Levels CPU CPU CPU CPU CPU HPC enablingplatforms • Sharedmemorymultiprocessors • Varioustypes (SMP, NUMA, …) Main Memory and/or Cache levels . . . • From simple to very sophisticated memory organizations • Impact on the programming model and/or process/threads run-time support MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

2100 2100 2100 2100 2100 2100 2100 2100 CPU CPU CPU HPC enablingplatforms: shared and distributedmemoryarchitectures InstructionLevelParallelism CPU (pipeline, superscalar, multithreading, …) M CPU . . . Shared memory multiprocessor “Limited degree” Interconnection Network (“one-to-few”) Distributed memory multicomputer: PC cluster, Farm, Data Centre, … MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

L2 cache L1 cache L1 cache P P P P P P P P CPU chip – SMP multiprocessor CPU technologyevolution and multicore Multiprocessor on single chip • “Dramatic revolution” for ITC industry: programmability issues • Computer providers support : from sequential programming to parallel programming • Also: NETWORK PROCESSORS with multicore technology MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Multicoretechnologyexamples SUN Niagara 3 MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Multicoretechnologyexamples IBM Power 7 MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Multicoretechnologyexamples IBM CELL BE (out of production …) MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 2100 Administrative Domain Adm. domain Adm.domain Linux / Pentium Power PC / MacOS Seemless global system view SUN / Solaris HPC enablingplatforms Heterogeneous Clusters Homogeneous Clustes,in general with multiprocessor/multicore nodes (SMP, NUMA, …) Virtual Private Networks: Farms, Data Centres Large Scale Platforms (LAN, MAN, WAN):Grids, Clouds, … MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Added value: Quality of Service (QoS) Distributed/Web Computing: Objects / Component Software Technology High-performance Computing, Cluster Computing Knowledge Management and Data Intensive Computing Cooperation and Virtual Organizations, Pervasive Computing Large scale platforms (Grids and much more) MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Cluster Cluster Data & Knowledge Server Workstation Workstation Workstation Workstation Exampleofheterogeneousdistributed HPC platform: Pervasive Grid An integratedsystem composedofcentralservers and services, fixed and mobile decentralizednodes, variouskindofnetworks. A distributedapplication, e.g. emergency management,mustbeableto exploit all the processing andcommunicationresources at best. PDA PDA PDA PDA PDA PDA PDA Variouskindoffixed, mobile, and ad-hocnetworks Meteo services GIS services MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Exampleof Pervasive Gridapplication Flood management Data- and computation-intensiveactivities: OFF-LINE and REAL-TIME Water level, speed, soil status, … HydrologicalModel: floodwave Data Mining Along the river , … Visualization Post-processing(non trivial) Precipitation in time and space, fromsatellites, meteo radars, raingauges, … Autorities, supervisors, observers, rescuers, police, firemens, … MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Cluster Cluster Data & Knowledge Server Workstation Workstation Workstation Workstation Off-line vs real-time adaptive processing The tipicaloff-line, routinelytasks involve centralservers and some predefinednetworks. Mobile remote devices are usedmainlyforcommunication and visualization. In emergencysituations, tasks can bere-allocatedto remote nodes/devices and mobile networks in real-time (e.g. centralresources are disconnected or communicationisinefficient). Are remote nodes/devices and networksabletoprocesshigh-performancetaks? PDA PDA PDA PDA PDA PDA PDA Variouskindoffixed, mobile, and ad-hocnetworks Meteo services GIS services MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

M P P P P P P P P The impact ofmulticore on next-generationdedicated and mobile technology PDA Embedding into mobile and /or wearable intelligent devices PDA PDA PDA Data & Knowledge Server Cluster On-chip Multiprocessor High-performance computing on a distributed collection of “simple” remote nodes/devices is feasible (and can be very efficient) PDA MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Basic background and prerequisites • An undergraduate-levelcourse on structured computer architecture • Firmwarelevelstructuring • Assemblerlevel, CPU architecture, compiling • Memoryhierarchies and caching • Interrupt handling, exceptionhandling • Processlevel, addressingspace, low levelscheduling, interprocesscommunication • Input/Output processing • Fewbooksadopt a structuredapproach: • Tanenbaum: in principle, some partsonly • Patterson-Hennessy: mainlydescriptionofexistingtechnologies (fewconcepts) • In Pisa: course “Computer Architecture” • Book: M. Vanneschi, “Architettura degli Elaboratori”, PLUS (Pisa University Press), 2009 – in Italian! • Some initiallectureswillreview the basicconceptsof the structuredapproach • Students are stronglyinvitedtoattendthis part in a verycriticalmanner MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Background fromothercoursesof MCSN • Course by Prof. Marco Danelutto (Distributed Systems: Paradigms and Models) • Structures of parallel computations • Performance measures and cost models • Service time / bandwidth, latency, completion time, efficiency, scalability • Basic mechanisms for process cooperation (messages, shared variables) • Parallelism forms / paradigms, structured parallelism • Stream-parallel pipeline • Stream-parallel farm • Data-flow • Data-parallel map, reduce, parallel prefix • Data-parallel with stencils • Client-server computations • Impact of service-time and latency on client service-time • Course of Advanced Programming • Basic elements of Queueing Theory • Basic elements of Networking MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

CourseProgram PART 1( 1/4) • Prerequisites revisited • Firmware structuring; processors, memory hierarchies and caching; assembler level and compiler optimizations, performance parameters; process cooperation and implementation • Run-time support to concurrency mechanisms • Structured interpretation of process communication and sharing • Instruction level parallelism • Elements of pipeline and superscalar CPUs, cost models, compiler optimizations PART 2( 3/4) • Shared memory architectures • SMP, NUMA, …, interconnection networks, support to concurrency mechanisms, cost models, static and dynamic optimizations, parallel application benchmarks • Distributed memory architectures • Cluster, MPP, …, interconnection networks, support to concurrency mechanisms, cost models, static and dynamic optimizations, parallel application benchmarks • Multicore architectures • Current status and trends of single-chip shared/distributed memory architectures MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Further MCSN courses on thesesubjects • ComplementsofDistributedEnablingPlatforms (CAP) • ProgrammingToolsforParallel and DistributedSystems (SPD) • For the currentyearonly (free-choiceexam): mergedinto the samecourse (formally: SPD) • Grid and Cloud, • DistibutedOperatingSystems, • Tools and LibrariesforParallel and DistributedMachines (MPI and otherstandards or commercial products; ASSIST–Universityof Pisa and possiblyotherresearchtools) • Virtualization, • Scheduling. • Nextyear (studyplan): 2 distinctcourses (CAP: 6 CFU, SPD: 9 CFU) • Parallel and DistributedAlgorithmsNextyear Toincrease the knowledgeof some notableapplicationparadigms: • NumericalTechniques and Applications (TNA) • Network OptimizationMethodsNextyear • Data MiningTechniquesNextyear MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

The lectures • Slides and blackboard • Slides for • (part of) course material • lecture outline • Blackboard •  for each slide, where necessary or convenient: further explanation / discussion using the blackboard MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Course Material • Mypage: • www.di.unipi.it/vanneschsection: Teaching (*) • Link in DidaWiki: http://www.cli.di.unipi.it/doku/doku.php/magistraleinformaticanetworking/spa/start • Lecture Notes • Slides (*) • Documents (*) • Papers and selected book chapters • M. Vanneschi, “Architettura degli Elaboratori”, PLUS, 2009 • Part IV • English version: nextyear • Some partswillbytranslated in English during the course (*) • ReferenceBooks • D.A.Patterson, J.H.Hennessy, “Computer Organization and Design: the Hardware/Software Interface”, Morgan Kaufman Publishers Inc. • D.E.Culler, J.P.Singh, A. Gupta, “Parallel Computer Architecture: a Hardware/Software Approach”, Morgan Kaufman Publishers Inc. MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Exammodality Forallstudents: Written test + oral test (in English or in Italian) Written test: explanation/discussionofconcepts and techniquesof the course, notnecessarilyfocused on smallexercizes. Emphasiswillbe put on the knowledgeofmethodologies and theirapplication, aswellas on the synthesiscapability and on the student’s abilitytoestablish the properrelationshipsbetween the variouspartsof the course. Optional: a report on a specifictopic • individualwritten report, maximum 2 persons • No intermediate tests • Report tobesubmitted a certaintime in advancewrt the exam date. • ASE: see a subsequent slide. • Registrationto the exam on the Official Site of Corso di Laurea: http://compass2.di.unipi.it/didattica, section Laurea Magistrale in Informatica e Netwoking, subsection “orari” MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Exammodality Report • Some literature material (e.g. one/some papers) isassignedto the student • existingparallelmachines / multicore, or existingprojects, • specifictechniques and/or technologies on topicsof interest . • The assigned material mustbestudied and interpretedaccordingto the coursecontents, methodology and approach • The report mustbewritten in a didactic style, asitwere a book chapterforstudents (“student-proof”) • “ ifanauthorisnotabletoexplain a certainthing in anunderstandable and complete manner, thencertainlysuchthingisnotclearto the authorhimself ” • Literatureassignedduring the first 2-3 weeksof the course MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Architetture Parallele e Distribuite (ASE) Esame di ASE (9 CFU), laurea specialistica vecchio ordinamento: • SPA (6 CFU) • + integrazione 3 CFU sulla parte delle metodologie di parallelizzazione (Libro Vanneschi, Cap. X) • Modalità di esame: scritto tradizionale e orale • All’atto dell’iscrizione: indicare ASE MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Workingapproach • As in any other course, it is fundamental to acquire skills and capabilities in concepts and principles, besides knowing the technologies. • Critical aptitude must be properly developed. • Interaction with the teacher is strongly encouraged • Questions during the lectures • Question time (“orario di ricevimento”) (in Italian for Italians) • Wednesday, 14:30 – 17:30, in my room • or by appointment in case of collision with other courses. MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Question Time 14:30 -17:30, my room (Dept) or by appointment in case the student attends the Wednsday afternoon lectures MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

Good Luck ! MCSN - M. Vanneschi: High Performance Computing Systems and Enabling Platforms

High Performance Computing Systems and Enabling Platforms