70 likes | 227 Views
First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro. Infiniband in brief. Storage Target. Channel based message passing. TCA. Internet Intranet. Mem Cntlr. CPU. 1. HCA. Link. Link. Link. Link. Switch. Mem Cntlr. CPU. HCA. 2. Link. Link.
E N D
First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro
Infiniband in brief Storage Target Channel based message passing TCA Internet Intranet MemCntlr CPU 1 HCA Link Link Link Link Switch MemCntlr CPU HCA 2 Link Link Link TCA N/W Target MemCntlr CPU HCA 3 Link Link 1000’s node per subnet Same network to transport low latency ipc, storage I/O and network I/O Link Link xCA Link speed 1x 2.5 Gbps 4x 10.0 Gbps . 12x 30 Gbps MemCntlr Router CPU HCA n 1 x Link 4 x Link 12 x Link
Infiniband Transport Protocols • IBA has been developed with Virtual Interface in mind. VIPL 2.0 includes IBA extensions and RDMA operations. • SCSI RDMA Protocol (SRP). It is a T10 standard. • SRP defines mapping to IBA architecture • it is the transport protocol over IBA • SRP is based on VI • Direct Access Files System (DAFS) • Direct Access Socket (DAS) • TCP/IP functionality over VI/IB fast, low latency network storage fast, low latency TCP/IP sockets File Access Block Access SRP DAFS DAS Virtual Interface over IB IBA Host Channel Adapter
LNL Infiniband Test Bed Leaf Switch 32 1x (2.5 Gbps) ports in 1 U chassis IBA Host Channel Adapter PCI-X (max ~ 380 Mbyte/s) Supermicro P4DL6
Status of Infiniband test bed • All the hardware has been provided by Infiniswitch (1 switch + 4 HCA) • All the hardware is up and running • First p2p measurements have been performed • Software • Virtual Interface Library (VIPL) as provided by Infiniswitch • Send/Receive over reliable connections • RDMA over reliable connections • Sourceforge has a infiniband project over Linux • VIPL source is available. Compiled and works! • Performance as the infiniswitch VIPL (probably they are the same code) • Results • Round trip time small buffers ~ 40 msec (latency 20 msec) • P2p ~ 80% link saturation
First IB p2p tests Link Saturation 220 ~ 40 Mbyte/s Mbyte/s Buffer Size (Bytes)
Work Plan • 2 months IB hardware evaluation (started end of May) • Mainly focus on functionality tests both of the hardware and software available: • SRP – SCSI remote protocol – already got, to be tested • DAS and DAFS if early version will be available • Contacts started with major companies (IBM, Dell, Fujitsu/Siemens) to understand their IB products road map