1 / 17

RDDP Is Coming Soon

Remote Direct Memory Access (RDMA) over IP PFLDNet 2003, Geneva Stephen Bailey, Sandburst Corp., steph@sandburst.com Allyn Romanow, Cisco Systems, allyn@cisco.com. RDDP Is Coming Soon. “ST [RDMA] Is The Wave Of The Future” – S Bailey & C Good, CERN 1999 Need: standard protocols host software

weston
Download Presentation

RDDP Is Coming Soon

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Remote Direct Memory Access (RDMA) over IPPFLDNet 2003, GenevaStephen Bailey, Sandburst Corp., steph@sandburst.comAllyn Romanow, Cisco Systems, allyn@cisco.com

  2. RDDP Is Coming Soon “ST [RDMA] Is The Wave Of The Future” – S Bailey & C Good, CERN 1999 • Need: • standard protocols • host software • accelerated NICs (RNICs) • faster host buses (for > 1G) • Vendors are finally serious: Broadcom, Intel, Agilent, Adaptec, Emulex, Microsoft, IBM, HP (Compaq, Tandem, DEC), Sun, EMC, NetApp, Oracle, Cisco & many, many others

  3. Overview • Motivation • Architecture • Open Issues

  4. CFP SigComm Workshop • NICELI SigComm 03 Workshop Workshop on Network-I/O Convergence: Experience, Lessons, Implications • http://www.acm.org/sigcomm/sigcomm2003/workshop/niceli/index.html

  5. High Speed Data Transfer • Bottlenecks • Protocol performance • Router performance • End station performance, host processing • CPU Utilization • The I/O Bottleneck • Interrupts • TCP checksum • Copies

  6. What is RDMA? • Avoids copying by allowing network adapter under control of application to steer data directly into application buffers • Bulk data transfer or kernel bypass for small messages • Grid, cluster, supercomputing, data centers • Historically, special purpose fabrics – Fibre Channel, VIA, Infiniband, Quadrics, Servernet

  7. application Ethernet/ IP Storage Network (Fibre Channel) A Machine Database Intermachine Network (VIA, IB, Proprietary) Traditional Data Center The World Servers

  8. Why RDMA over IP? Business Case • TCP/IP not used for high bandwidth interconnection, host processing costs too high • High bandwidth transfer to become more prevalent – 10 GE, data centers • Special purpose interfaces are expensive • IP NICs are cheap, volume

  9. The Technical Problem- I/O Bottleneck • With TCP/IP host processing can’t keep up with link bandwidth, on receive • Per byte costs dominate, Clark (89) • Well researched by distributed systems community, mid 1990’s. Industry experience. • Memory bandwidth doesn’t scale, processor memory performance gap– Hennessy(97), D.Patterson, T. Anderson(97), • Stream benchmark

  10. Copying Using IP transports (TCP & SCTP) requires data copying 1 NIC Packet Buffer 2 Packet Buffer User Buffer Data copies

  11. Why Is Copying Important? • Heavy resource consumption @ high speed (1Gbits/s and up) • Uses large % of available CPU • Uses large fraction of avail. bus bw – min 3 trips across the bus 64 KB window, 64 KB I/Os, 2P 600 MHz PIII, 9000 B MTU

  12. What’s In RDMA For Us? Network I/O becomes `free’ (still have latency though) 1750 machines using 0% CPU for I/O 2500 machines using 30% CPU for I/O

  13. Approaches to Copy Reduction • On-host – Special purpose software and/or hardware e.g., Zero Copy TCP, page flipping • Unreliable, idiosyncratic, expensive • Memory to memory copies, using network protocols to carry placement information • Satisfactory experience – Fibre Channel, VIA, Servernet • FOR HARDWARE, not software

  14. RDMA over IP Standardization • IETF RDDP Remote Direct Data Placement WG • http://ietf.org/html.charters/rddp-charter.html • RDMAC RDMA Consortium • http://www.rdmaconsortium.org/home

  15. ULP RDMA control DDP Transport IP RDMA over IP Architecture Two layers: • DDP – Direct Data Placement • RDMA - control

  16. Upper and Lower Layers • ULPs- SDP Sockets Direct Protocol, iSCSI, MPI • DAFS is standardized NFSv4 on RDMA • SDP provides SOCK_STREAM API • Over reliable transport – TCP, SCTP

  17. Open Issues • Security • TCP order processing, framing • Atomic ops • Ordering constraints – performance vs. predictability • Other transports, SCTP, TCP, unreliable • Impact on network & protocol behaviors • Next performance bottleneck? • What new applications? • Eliminates the need for large MTU (jumbos)?

More Related