1 / 28

Socket Direct Protocol over PCIe Interconnect Design

"Explore the design, implementation, and evaluation of Socket Direct Protocol (SDP) over PCI Express interconnect by Ahmed Bukhamsin. Learn about PCI Express overview, proposed system, PCIe interconnect, implementation, and evaluation. Discover how SDP enables running TCP socket applications without modifications."

bhobbs
Download Presentation

Socket Direct Protocol over PCIe Interconnect Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation By Ahmed Bukhamsin Oct 2012 1

  2. Outline • PCI Express (PCIe) Overview • PCIe Interconnect • Proposed System • Implementation • Evaluation • Conclusion School of Computing Science / Ahmed Bukhamsin 2

  3. PCI Express Overview 1 • PCI = Peripheral Component Interconnect School of Computing Science / Ahmed Bukhamsin 3

  4. PCI Express Overview 2 School of Computing Science / Ahmed Bukhamsin 4

  5. PCI Express Overview 3 • PCIe Evolution • High throughput • Low latency • Low power consumption • PCIe over cable * Aggregate bandwidth implies simultaneous traffic in both directions. School of Computing Science / Ahmed Bukhamsin 5

  6. PCIe Interconnect 1 • Issues with connecting two root complexes • Bus ownership • Endpoints resource allocation • Solution: Non-transparent bridge (NTB) School of Computing Science / Ahmed Bukhamsin 6

  7. PCIe Interconnect 2 • NTB isolates the connected PCIe domains • NTB provides a communication mechanism between the connected PCIe domains School of Computing Science / Ahmed Bukhamsin 7

  8. PCIe Interconnect 3 • NTB communication mechanism • Acquires memory in both systems • Provides address translation to allow data exchange • Provides doorbell and scratchpad registers accessible from both sides School of Computing Science / Ahmed Bukhamsin 8

  9. Proposed System 1 • We propose to develop the Socket Direct Protocol (SDP) • SDP makes it possible to run TCP based socket applications without modifications School of Computing Science / Ahmed Bukhamsin 9

  10. DMA Driver NTB Driver SDP Socket Preloaded Library IP TCP NIC Driver SDP Proposed System 2 User Space Applications Kernel Space Protocol Switch Communication API PCIe NTB Card Ethernet NIC School of Computing Science / Ahmed Bukhamsin 10

  11. DMA Driver NTB Driver IP TCP NIC Driver SDP Proposed System 2 User Space Applications • SDP Socket Preloaded Library • Rule based protocol switch SDP Socket Preloaded Library Kernel Space Protocol Switch Communication API PCIe NTB Card Ethernet NIC School of Computing Science / Ahmed Bukhamsin 11

  12. DMA Driver NTB Driver SDP Socket Preloaded Library IP TCP NIC Driver Proposed System 2 User Space Applications • SDP Socket Preloaded Library • Rule based protocol switch • SDP Kernel Module • The main component of the solution • Register itself as new address family • Implements all TCP functions Kernel Space Protocol Switch SDP Communication API PCIe NTB Card Ethernet NIC School of Computing Science / Ahmed Bukhamsin 12

  13. NTB Driver DMA Driver SDP Socket Preloaded Library IP TCP NIC Driver SDP Proposed System 2 User Space Applications • SDP Socket Preloaded Library • Rule based protocol switch • SDP Kernel Module • The main component of the solution • Register itself as new address family • Implements all TCP functions • Communication API • Group of functions exported by the driver for SDP to access the driver Kernel Space Protocol Switch Communication API PCIe NTB Card Ethernet NIC School of Computing Science / Ahmed Bukhamsin 13

  14. SDP Socket Preloaded Library IP TCP NIC Driver SDP Proposed System 2 User Space Applications • SDP Socket Preloaded Library • Rule based protocol switch • SDP Kernel Module • The main component of the solution • Register itself as new address family • Implements all TCP functions • Communication API • Group of functions exported by the driver for SDP to access the driver • DMA and NTB Drivers • Cards manufacturer drivers Kernel Space Protocol Switch Communication API DMA Driver NTB Driver PCIe NTB Card Ethernet NIC School of Computing Science / Ahmed Bukhamsin 14

  15. Create server Buffer Server Buffer Create client Buffer Interrupt Interrupt Interrupt Implementation 1 • SDP Connection Client Server Scratchpad registers - Listen() - accept() Server port Ready Conniction ID - Connect() client buf addr -Read Reg. Client Buffer IP and Port Server buff addr - Read registers Connection established Connection established School of Computing Science / Ahmed Bukhamsin 15

  16. send copy n copy 3 copy 2 send copy 1 send send recv n recv 3 recv 2 recv 1 Data Ready Signal DMA Data Copy msg 1 msg 2 msg n msg 3 Waiting ... Implementation 2 • Send and Receive Data Source Time Data Sink SDP Buf Application Application SDP Buf ... ... ... ... ... School of Computing Science / Ahmed Bukhamsin 16

  17. Evaluation 1 • Benchmarks • Throughput • Latency • CPU Utilization • Application Testing • Apache Web Server School of Computing Science / Ahmed Bukhamsin 17

  18. Evaluation 2 • Throughput School of Computing Science / Ahmed Bukhamsin 18

  19. Evaluation 3 • Throughput School of Computing Science / Ahmed Bukhamsin 19

  20. Evaluation 3 • Throughput School of Computing Science / Ahmed Bukhamsin 20

  21. Evaluation 3 • Throughput School of Computing Science / Ahmed Bukhamsin 21

  22. Evaluation 3 • Throughput School of Computing Science / Ahmed Bukhamsin 22

  23. Evaluation 4 • Latency School of Computing Science / Ahmed Bukhamsin 23

  24. Evaluation 5 • Application School of Computing Science / Ahmed Bukhamsin 24

  25. Evaluation 5 • Application School of Computing Science / Ahmed Bukhamsin 25

  26. Evaluation 5 • Application School of Computing Science / Ahmed Bukhamsin 26

  27. Evaluation 5 • Application School of Computing Science / Ahmed Bukhamsin 27

  28. Conclusion • We design and implement SDP • We evaluate it and compare it to SDP over Infiniband • Our SDP offers comparable performance to Infiniband for concurrent transfers of large message sizes • Application testing shows more u 10% improvement in performance over Infiniband • PCIe can serve as an interconnect for communication-intensive applications School of Computing Science / Ahmed Bukhamsin 28

More Related