1 / 30

InfiniBand Cable Panel: FDR and EDR Speeds and Exascale Computing

Join the panel discussion on the roadmap for InfiniBand cables, connectors, and design considerations for FDR (14G) and EDR (25G) speeds. Explore cable requirements, compliance testing, and deployment issues in the move towards exascale computing.

aart
Download Presentation

InfiniBand Cable Panel: FDR and EDR Speeds and Exascale Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2010 Sonoma Cable Panel Moving to FDR and EDR Speeds and Exascale Computing Moderator: Rupert Dance Date: 3/15/2010 www.openfabrics.org

  2. Panelists • Cinch Cables • Merrick Moeller: mmoeller@email.cinch.com • FCI • Jim David: jim.david@fci.com • Mellanox • John Monson: jmonson@mellanox.com • Luxtera • Marek Tlalka: mtlalka@luxtera.com • Molex • Jay Neer: jneer@molex.com • Volex • Toney Chew: Toney.Chew@volex.com www.openfabrics.org

  3. Agenda • InfiniBand Road Map • Current Market conditions • Connectors • Cables • Specification & Design Considerations • Deployment Issues • Compliance and Interoperability Testing www.openfabrics.org

  4. InfiniBand Roadmap www.openfabrics.org

  5. Current Market conditions • Connectors • MicroGiGaCN (aka CX4) – 4X SDR and DDR • QSFP 4X SDR, DDR and QDR, 8X QDR and 12X QDR • CXP – 12X QDR • Cables using QSFP connectors • Passive Copper – up to 7 meters in length • Active Copper – up to 15 meters in length • Active Fiber – up to 300 meters in length • Multi-Mode Fiber cables up to 100 meters • Single-Mode Fiber cables up to 4 kilometers • Gauge – copper cables • 24 AWG for max length • 32 AWG minimum gauge – used in 3 meters and less www.openfabrics.org

  6. Connectors - FDR (14G) & EDR (25G) • QSFP Connectors • Can QSFP support FDR • Can QSFP support any higher speeds • CXP Connectors • Can CXP support FDR • Can CXP support EDR • 4x Connectors for EDR • What is required for 4X connectors to support 25G speeds www.openfabrics.org

  7. Cable requirements for EDR • Cable design goals • Length supported • Gauge supported • Loss Budget • Cross talk • Insertion Loss • Return Loss • Manufacturing issues • Product Mix • Copper versus Fiber • Active versus passive www.openfabrics.org

  8. Specification & Design Considerations • Convergence of IB and Ethernet specification • Leverage volumes across markets for reduced cost • IB specification for optical connectors for EDR • Current IB spec does not cover optical spec • Currently working with OIF • Compatibility and legacy support • 8b/10b versus 64b/66b • What laser technology will be used for EDR • VCSEL vs. EML lasers • Link Integrity Issues • DDR and QDR - lessons learned • Allocating overall loss budget • Attenuation based tuning • Importance of EEPROM programming www.openfabrics.org

  9. Deployment Issues • Power requirements • Cooling requirements • Exascale Computing with 10,000+ nodes • Cable management • Distance limitation • Other Considerations • Cost • 2011 EDR availability www.openfabrics.org

  10. Compliance and Interoperability • Compliance test methodology • TDR versus VNA • WDP – pros and cons • IEEE 802.3ba • System Interoperability testing • End to end testing • Switch to Switch testing • Short trace and long trace testing • Heterogeneous testing • MPI based Interop testing www.openfabrics.org

  11. CXP Compliance Test Boards - Module Compliance Board (MCB)- (1) Req’d if testing with HCB- (2) Req’d if testing with Cables - Host Compliance Board (HCB) www.openfabrics.org 11

  12. Cable Sponsors of the OFA Workshop www.openfabrics.org

  13. Mellanox Vendor Presentation www.openfabrics.org

  14. Addressing High Bandwidth Cable Challenges Author: John Monson Date: March 15, 2010 www.openfabrics.org

  15. Challenge – Cabling at 14Gb/s • Today’s QSFP suitable for FDR • Target lengths • Passive copper up to 5 meters • Active copper up to 10 meters • Optical up to 100 meters • Active cable power similar to QDR • Cable plants will look similar to today’s QDR clusters www.openfabrics.org

  16. Challenge – Cabling at 26Gb/s • Enhanced QSFP required for EDR • Need to improve insertion loss • Optimize impedance matching • Target lengths • Passive copper up to 3 meters • Active copper up to 7-10 meters • Optical up to 100 meters • Power targets not significantly higher than QDR • More clusters will now require active cables www.openfabrics.org

  17. Total Solution Required • I/O drivers must consider media type and length • Transmitter must compensate for signal degradation over copper • Receiver must compensate with increased sensitivity • EDR will require better PCB material • Switch traces up to 17 inches

  18. Molex Vendor Presentation www.openfabrics.org

  19. InfiniBand Interconnect Infrastructure Scaling the Current Ecosystem to 25G Jay Neer Strategic Product Manager – Industry Standards March 14, 2010 www.openfabrics.org

  20. QSFP+ and CXP The Highest Density I/O Solutions RJ-45 48 Channels 480 Gb/s SFP+ 48 Channels 480 Gb/s SFP+ 48 Channels 480 Gb/s MicroGiga 112 Ch. 1120Gb/s iPass 160 Channels 1600 Gb/s QSFP+ 144 Channels 1440 Gb/s MicroGiga – 192 Ch. 960 Gb/s CXP – 384 Channels 3840 Gb/s www.openfabrics.org 20

  21. High Density InfiniBand SwitchQlogic – SC09 288 QSFP+ Ports www.openfabrics.org 21

  22. 4x QSFP+ Family of Products • Single SMT Connector • Single Cage • Ganged Cages • Stacked, Ganged Cages • Light Pipe Options • Thermal Options • EMI Solution Options • Direct Attach Copper Cables • Copper SFP+ Breakout Cables • Copper Loopbacks • 12 Fiber MTP Optical Cables • Optical LC Breakout Cables • Optical Loopbacks www.openfabrics.org

  23. 4x QSFP+ Family of Products - The QSFP+ optical cable interface is defined as the industry standard 12 fiber MTP connector - Round bulk cable provides improved cablemanagement within a rack - 3mm diameter cable for short runs - 4.5mm diameter for runs up to 25 meters - The cable assembly utilizes the exterior 4 fibers on each side of a 12 fiber ribbon cable; bulk cable with the center 4 fibers missing is also acceptable - Breakout cables to LC cables are used to connect to SFP+ applications www.openfabrics.org 23

  24. Sun Constellation System World’s Largest InfiniBand Switch 3,456 InfiniBand Channels per System with CXP 12x I/O’s www.openfabrics.org 24

  25. 12x CXP Family of Products • Ecosystem Shipping: • Single Integrated Connector • Stacked Integrated Connectors • Light Pipe Options • Thermal Options • EMI Solution Options • Direct Attach Copper Cables • Copper QSFP+ Breakout Cables • Copper Loop-backs • 24 Fiber MTP Optical Cables • Optical QSFP+ Breakout Cables www.openfabrics.org 25

  26. 12x CXP Family of Products CXP / CXP CXP / MicroGiga 12x CXP / QSFP+ CXP / MicroGiga 4x www.openfabrics.org 26

  27. 12x CXP Family of Products CXP / QSFP CXP / CXP Pluggable Optical Module w/ Passive Optical Cable Assembly www.openfabrics.org 27

  28. Single Port w/out Top Cover Access area to Top of Plug 12x CXP Family of Products Thermal Solutions Heat Sinks Optional & Application Dependent- Can be extended to the rear or to the sides- Can be square or round pins or fins- Heights based on application Clip-on Cover w/vent holes StandardCapable of cooling 1-1.5W Devices www.openfabrics.org 28

  29. CXP Development / Compliance / Test Boards - Module Compliance Board (MCB)- (1) Req’d if testing with HCB- (2) Req’d if testing with Cables - Host Compliance Board (HCB) www.openfabrics.org 29

  30. InfiniBand Roadmap Plenty of time to get to 25G www.openfabrics.org 30

More Related