840 likes | 977 Views
Open Source Middleware for Grid and Parallelism:. Denis Caromel, et al. http://ProActive.ObjectWeb.org OASIS Team INRIA -- CNRS - I3S -- Univ. of Nice Sophia-Antipolis, IUF. GRIDs and Parallelism Active Objects + Groups Components Deployment GUI Applications (BLAST).
E N D
Open Source Middleware for Grid and Parallelism: Denis Caromel, et al. http://ProActive.ObjectWeb.org OASIS Team INRIA -- CNRS - I3S -- Univ. of Nice Sophia-Antipolis, IUF • GRIDs and Parallelism • Active Objects + Groups • Components • Deployment • GUI • Applications (BLAST) Denis Caromel
ProActive Parallel Suite (1) Open Source + PROFESSIONAL SUPPORT Denis Caromel
1. The GRIDs PCs : 1 Billion in 2002 (25 years) Forecast: 2 Billions in 2008 Denis Caromel
The Grid Concept • GRID = Electric Network • Computer Power (CPU cycles) Electricity • Can hardly be stored, if not used Lost • Global management, • Mutual sharing of the resource 2 important aspects : 1. Diversity 2. Computational + Data Grid Denis Caromel
Enterprise Grids Internet Apache Servlets EJB Databases 5 Denis Caromel
Scientific Grids Large Equipment Internet Parallel Machine Clusters 6 Denis Caromel
Internet Grids Internet Job management for embarrassingly parallel application (e.g. SETI) 7 Denis Caromel
Intranet Grids - Desktop Grids Internet Using Spare Cycles of Desktops Potentially: - Non-embarrassingly parallel application - Across several sites How to Program ? How to Deploy ? 8 Denis Caromel
The multiple GRIDs Scientific Grids : Enterprise Grids : Internet Grids, (miscalled P2P grid): Intranet Desktop Grids Strong convergence needed and somehow in progress! 2 New Disturbing Factors: End of Moore Law and Multi-Cores For SBGrid: The Multiple Scientific GRIDs 2D and 3D graphics SB workstations EM laboratory Clusters Global grids and other federally funded equipments 9 Denis Caromel
Grid Computing with ProActive Shanghai Nice Boston Hierarchical Deployment Challenges: Programming Model, Scale, Latency, Heterogeneity, Versatility (FT, protocols, firewalls, etc.) Beijing 10 Denis Caromel
2. Distributed and ParallelObjectsProActiveProgramming 11 Denis Caromel
ProActive Parallel Suite Denis Caromel
ProActive Parallel Suite Denis Caromel
ProActive : Active objects JVM A A WBN! A ag =newActive (“A”, […], VirtualNode) V v1 = ag.foo (param); V v2 = ag.bar (param); ... v1.bar(); //Wait-By-Necessity JVM ag v2 v1 V Wait-By-Necessity is a Dataflow Synchronization Java Object Active Object Req. Queue Future Object Proxy Thread Request 14 Denis Caromel
ProActive: Explicit Synchronizations Explicit Synchronization: - ProActive.isAwaited (v); // Test if available - .waitFor (v); // Wait until availab. Vectors of Futures: - .waitForAll (Vector); // Wait All - .waitForAny (Vector); // Get First A ag =newActive (“A”, […], VirtualNode) V v = ag.foo(param); ... v.bar(); //Wait-by-necessity 15 Denis Caromel
Wait-By-Necessity: First Class Futures V= b.bar () b v v Futures are Global Single-Assignment Variables a b c.gee (V) c c 16 Denis Caromel
Proofs in GREEK Denis Caromel
Standard system at Runtime:No Sharing NoC: Network On Chip 18 Denis Caromel
TYPED ASYNCHRONOUS GROUPS 19 Denis Caromel
Creating AO and Groups A V A ag =newActiveGroup (“A”, […], VirtualNode) V v = ag.foo(param); ... v.bar(); //Wait-by-necessity JVM Group, Type, and Asynchrony are crucial for Cpt. and GRID Typed Group Java or Active Object 20 Denis Caromel
Broadcast and Scatter ag JVM c3 c3 c3 c3 c3 c3 c3 c1 c1 c1 c1 c1 c1 c1 c2 c2 c2 c2 c2 c2 c2 JVM JVM s s JVM • Broadcast is the default behavior • Use a group as parameter, Scattered depends on rankings cg ag.bar(cg); // broadcast cg ProActive.setScatterGroup(cg); ag.bar(cg); // scatter cg 21 Denis Caromel
Dynamic Dispatch Group c4 c4 c4 c6 c6 c6 c5 c5 c5 c7 c7 c7 c8 c8 c8 c0 c0 c0 c9 c9 c9 c3 c3 c3 c1 c1 c1 c2 c2 c2 JVM Slowest ag cg JVM Fastest JVM ag.bar(cg); JVM 22 Denis Caromel
ProActive Parallel Suite Denis Caromel
ProActive Parallel Suite Denis Caromel
Parallel, Distributed, Hierarchical3. Componentsfor the GridComposing, Wrapping code to make them fit for the Grid 25 Denis Caromel
A Provide/Require Component Component interface My Business Component Interface Reference OFFERED REQUIRED Event sinks Event sources Attributes Courtesy of Philippe Merle, Lille, OpenCCM platform 26 Denis Caromel
Building Component Applications =Assembling Components with XML ADL Provide + Use, but flat assembly 27 Denis Caromel
Objects to Distributed Components (1) ComponentIdentity Cpt =newActiveComponent (params); A a = Cpt … .getFcInterface ("interfaceName"); V v = a.foo(param); IoC: Inversion Of Control (set in XML) A Example of component instance V Truly Distributed Components Typed Group Java or Active Object JVM 28 Denis Caromel
P A B A A C B D C Group proxy Group proxy Groups in Components (1) A parallel component! Broadcast at binding, on client interface At composition, on composite inner server interface 29 Denis Caromel
GCM:Grid Component Model A Strong Grid Standard with ETSI: European Telecommunications Standards Institute Denis Caromel
GCM Scopes and Objectives: Grid Codes that Compose and Deploy No programming, No Scripting, … No Pain Innovation: Abstract Deployment Composite Components Multicast and GatherCast MultiCast GatherCast
GCM features provided in ProActive: • GCM Fractal ADL (XML) • (Architecture Description Language) • GCM Management (Java, C, WSDL API) • GCM Application Description (XML) • GCM Interoperability Deployment (XML) Denis Caromel
Ho to use GCM & Components for SBGrid ? • Description of a piece of code to make it fit for the Grid (& scheduling): • Required: Hardware, OS, Configuration • Input needed • Output Produced • Optional, on a case by case basis: • Wrapping the code in a portable manner, adding necessary control • Description of Services Offered, and Required (CP, + Services) • Making it possible to compose SBGrid software Denis Caromel
ProActive Parallel Suite Denis Caromel
ProActive Parallel Suite Denis Caromel
ProActive Parallel Suite (1) Denis Caromel
ProActive Parallel Suite (1) Denis Caromel
4. Deployment Deploying Denis Caromel
Large Equipment Internet Internet Internet Internet Servlets EJBs Databases Job management for embarrassingly parallel application (e.g. SETI) Parallel Machine Clusters How to deploy on the Various Kinds of Grid Denis Caromel
Abstract Deployment Model • Problem: • Difficulties and lack of flexibility in deployment • Avoid scripting for: configuration, getting nodes, connecting, etc. • A key principle: Virtual Node (VN) + XML deployment file • Abstract Away from source code, and Wrapping code: • Machines • Creation Protocols • Lookup and Registry Protocols • Protocols and infrastructures: • Globus, ssh, rsh, LSF, PBS, … Web Services, WSRF, ... Denis Caromel
GCM XML Deployment Descriptor Virtual architecture Abstract deployment model • Separates design from deployment infrastructure • Virtual Nodes (VN) • Dynamic enactment of the deployment, from the application Source code Runtime Physical infrastructure computer2 Host names Creation protocols Lookup, Registration Multi-Cores JVM2 VN1 computer3 JVM3 VN2 JVM4 JVM5 Denis Caromel
Data management: File transfer • Integrated with Deployment: • Resource acquisition file transfer when available: unicore, nordugrid, Globus • Other non-deployment protocols: scp, rcp, … • ProActive own protocol when other fails: ProActive Failsafe File Transfer • ProActive supports the following transfers: • Push: To a remote node • Pull: From a remote node • Triangle: Triggered from node A, occurring between node B C • It can be used at: • Deployment time (XML) • Execution at any time (API) • Retrieval time (XML) Denis Caromel
File Transfer in Deployment Descriptors Denis Caromel
API: Asynchronous File Transfer with Futures • Immediately returns a File Future to the caller (subject to Wait-By-Necessity) • VN RETRIEVE: pullFile is invoked, and an array of futures (File[]) is returned • Futures: increase the performance when transferring files between peers: • Outperforms rcp when Pushing while Pulling a file! Denis Caromel
Performance Comparison100Mbit LAN network with a 0.25[ms] ping,network configured at 10[Mbitsec] duplex • First, ProActive failsafe and native scp perform the same • Second, ProActive deploys the files in parallel to the nodes, faster than scp: • several invocations of push, on a set of nodes, • are executed in parallel by those nodes Denis Caromel
GUI in ProActive Parallel Suite Denis Caromel
GUI in ProActive Parallel Suite Denis Caromel
5. IC2D Interactive Control & Debug for Distribution Eclipse GUI for the GRID Denis Caromel
Programmer Interface for Analyzing and Optimizing Denis Caromel