1 / 25

Plan

Plan. Introduction computation, clusters, high-performance communications Framework PM², RPC, communication interfaces Proposal Madeleine 2 Implementation and evaluation results on top of TCP, VIA, SCI Conclusion. Introduction.

cira
Download Presentation

Plan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Olivier Aumage

  2. Plan • Introduction • computation,clusters, high-performance communications • Framework • PM², RPC, communication interfaces • Proposal • Madeleine 2 • Implementation and evaluation • results on top of TCP, VIA, SCI • Conclusion Olivier Aumage

  3. Introduction • What do academic scientist and industrial engineers need: • complex computations on huge data • high-performance computation machines: clusters of workstations • distributed parallel programming environments for clusters: PM² distributed RPC-based multithreaded environment • efficient communication support suited for such environments Olivier Aumage

  4. RPC • RPC: • Remote Procedure Call • PM² basis paradigm • A communication library for RPC should meet the following requirements: • portability • efficiency • zero-copy data transfer Olivier Aumage

  5. Low Level Interfaces • Characteristics: • direct network access • high efficiency • low portability • Examples: • BIP • Fast-Messages • VIA • SCI Olivier Aumage

  6. High-Level Interfaces: MPI • MPI - Message Passing Interface: • widely used in parallel computation applications • efficient portable implementations:MPI-CH • available on top of BIP and FM • But: • interface is not flexible enough for efficient RPC support Olivier Aumage

  7. Madeleine • Characteristics: • communication library used by PM² • portability • efficiency • dedicated to RPC paradigm • But: • heavily message passing oriented • suboptimal support of recent protocols : SCI, VIA, ... • limited interface Olivier Aumage

  8. Madeleine II Olivier Aumage

  9. User / Madeleine 2 Contract • Interface of Madeleine 2: • semantical specification of data transfer options • Consequences: • semantically adequate with user requirements • more power to the library: • more liberty of movement • more efficiency Olivier Aumage

  10. Example Example of a program using the Madeleine 2 communication library: Sending side : Receiving side : . . . . . . len = strlen("Madeleine 2"); conn = begin_packing(chan, remote); conn = begin_unpacking(chan); pack(conn, &len, sizeof(int), unpack(conn, &len, sizeof(int), send_CHEAPER, receive_EXPRESS); send_CHEAPER, receive_EXPRESS); str = malloc(len + 1); pack(conn, "Madeleine 2", len + 1, unpack(conn, str, len + 1, send_CHEAPER, receive_CHEAPER); send_CHEAPER, receive_CHEAPER); end_packing(conn); end_unpacking(conn); . . . . . . Olivier Aumage

  11. Communication Channels • Multi-channel support: • separate independant communication channels • important for easy modular application development • Multi-protocol support: • one protocol per channel • optimal use of available resources Olivier Aumage

  12. Adaptative Mechanisms • Objective: • support of protocols with multiple paradigms • dynamic paradigm selection for each data block • Solution: • use of a choice functioncriteria: • size • pack/unpack mode • anything else if needed Olivier Aumage

  13. Multi-Paradigm Structure • Connection: • contains one or more links • Link: • symbolizes a given data transfer method • allows full use of protocol dependant characteristics and optimizations • data ordering transparently enforced by Mad2’s high level generic layer • increased efficiency Olivier Aumage

  14. Buffer Management • Two kinds of buffers: • dynamic buffers • static buffers • Responsabilities of the pack function: • allocation of dynamic buffers • static buffer filling • buffers steering to adequate links • buffer group management for burst data transfer Olivier Aumage

  15. Olivier Aumage

  16. Implementation • Features of the implementation of Madeleine 2: • drivers available on top of: • VIA : message passing + RDMA write • SCI : remote write, DMA support planned • TCP • SBP • MPI and BIP support to be ready soon • 100 % operational environment Olivier Aumage

  17. Performance on top of TCP The following tests were realized using 2 PC Pentium II / 450MHz interconnected by a Fast-Ethernet network Olivier Aumage

  18. Performance on top of VIA The following tests were realized using 2 PC Pentium II / 450MHz interconnected by a Fast-Ethernet network Olivier Aumage

  19. Performance on top of SCI The following tests were realized using 2 PC Pentium II / 450MHz interconnected by a SCI network Olivier Aumage

  20. Dynamic Paradigm Selection Olivier Aumage

  21. Conclusion • Madeleine 2 : • more functionalities: multi-channel, multi-protocol, multi-paradigm • increased overall efficiency • full operational system • support for SCI, VIA, TCP, SBP • first high-performance adaptative communication library Olivier Aumage

  22. Future Work Olivier Aumage

  23. Multi-Cluster Support • Objectives : • network-heterogeneous multi-cluster support • metacomputing capabilities • Features : • inter-cluster channel • user-friendly configuration scheme • script-based applications and clusters description • one-step multi-cluster application compilation and launch • support for protocol specific loaders Olivier Aumage

  24. Example Example of cluster and application description files for Madeleine 2 : Cluster description : Application description : cluster popc: application gauss: { { hosts: popc0--11; executable: mad_gauss; domain: popc-private; path: "~/pm2/mad_applis/"; gateways: {domain: ens-lyon; host: popc0;}; clusters: adapters: {name: popc; hosts: popc0--3;}, {alias: TCP0; protocol: tcp;}, {name: sci; hosts: sci0--4;}; {alias: TCP1; protocol: tcp; suffix: "-t";}, channels: {alias: BIP; protocol: bip;}; {alias: Control; } adapter: sci.TCP0, popc.TCP0;}, {alias: Calc; adapter: sci.SCI, popc.BIP;}; } Olivier Aumage

  25. Clusters of cluster : • dynamic protocol selection • multi-protocol routing • architecture and network heterogeneity support Computation/communication scheduling : • optimized communication management • multiple reception policies support Olivier Aumage

More Related