1 / 18

A record and replay mechanism using programmable network interface cards

This research explores a record and replay mechanism using programmable network interface cards (NICs) to capture and replay non-deterministic parallel program behavior. The approach minimizes intrusion, eliminates irreproducibility, decreases the probe effect, and enables monitoring without user knowledge.

smontoya
Download Presentation

A record and replay mechanism using programmable network interface cards

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A record and replay mechanism using programmable network interface cards Laurent Lefèvre INRIA / LIP (UMR CNRS, INRIA, ENS, UCB) Laurent.lefevre@inria.fr Dieter Kranzlmüller GUP - Joh. Kepler Univ. Linz Kranzlmueller@gup.jku.at PDCN 2005 - Innsbruck - Feb. 2005 This research is partly supported by French “Programme d’Actions Intégrées Amadeus” funded by the French Ministery of Foreign Affairs and the Austrian Exchange Service (OAD), WTZ Program Amadeus under contract no. 13/2002

  2. Nondeterministic parallel program behavior • Parallel program • Same code • Same platform • Same input data • Different runs • ==> Different results ! • Reasons ? • Scheduling decisions of processor/ OS • Cache contents, cache conflicts • Memory access patterns • Network conflicts • Non determinism in the network

  3. Example : MPI applications • MPI_ANY_SOURCE • Wilcard receive • Race condition

  4. Nondeterminism • Irreproducibility problem • Cannot repeat a particular execution • No debugging actions possible • Completeness problem • Cannot observe some errors • Impossible to test all possible executions • Probe effect • Monitoring actions influence program

  5. Monitoring … … influences the observed program in • Time • Events are delayed due to monitoring overhead • Ordering of events is perturbed • Space • Storing monitoring data requires memory space

  6. Our approach : Monitoring optimizations • Minimization of monitor overhead through minimal invasive instrumentation • Minimization of monitor overhead through exploitation of additional hardware • Usage of clusters with programmable network hardware

  7. Myrinet clustering • Myrinet NICs • Link Cables • Fiber to 200m • Myrinet Switches • Software • Host • NIC firmware Desktop Hosts In-CabinetServer Clusters 2+2Gbits/s EmbeddedClusters Courtesy of Myricom Inc

  8. Programmable network cards • Myrinet NIC • Processor on board (Lanai 9.2 RISC 200 Mhz) • Memory (2 MB) • Communications between host CPU and NIC: • Programmed Input/Output (PIO) :dedicated commands • Access memory locations • Extract NIC status • Direct memory access (DMA) • Transfert between host and NIC CPU • Idenpendant from host • GM software • Software library • Kernel module • Myricom Control Program (MCP) Courtesy of Myricom Inc

  9. Myrinet NICs = Protocol Offload Engines Myrinet NICs : processor, memory, and firmware. SerDes & Transceiver SerDes & Transceiver X port X port Lanai 2XP packetinterface packetinterface copy & CRC32 engine CPU SRAMinterface x72 SRAM JTAGinterface EEPROMinterface PCI-Xinterface Courtesy of Myricom Inc

  10. Myrinet Software Interfaces Applications UDP TCP MPI Sockets OtherM'ware In theHostOS IP OS bypass Ethernet driver Myrinet driver Firmware in the Myrinet NIC One or more 2+2 Gbit/sMyrinet ports Ethernet NIC Courtesy of Myricom Inc

  11. Monitoring on Programmable network cards • We deploy Record actions from CPU host to NIC • Architecture based on 3 steps : • Preparation and instrumentation • Recording execution • Repeated replay phases

  12. Preparation and instrumentation • Loading modified MCP onto NIC • Instrumentation of MPI program by including modified MPI header file • Compiling application with modified MPICH library

  13. Recording execution • NIC buffer used to store order of incoming messages • Critical step • Optimizing based on semantics of MPI : • Delivery between 2 nodes arrive in the same order than generated by sender • We only trace messages on the receiver side

  14. Recording execution • Upon initialization of MPI program : memory reservation on NIC to store order of incoming messages • If buffer full : transfer asynchronously to host memory during execution • After execution : file generation of monitoring information extracted from NIC

  15. Replaying • To increase amount of observation data • To perform program analysis • Only hosts are involved • Using dedicated graphical environments (DeWiz)

  16. Replaying Debugging tool DeWiz screenshot with events collected on programmable card

  17. Time graph, counter analysis

  18. Conclusion and current work • Advantages: • Minimal intrusion of during initial record phase • Eliminating irreproducibility effect • Decreasing the probe effect • Monitoring without user knowledge • Tools to manipulae events graph • Adding QoS functionality on the NIC to filter monitoring actions • Deploying record and replay mechanisms inside programmable switch

More Related