It’s the Interface, Stupid!

1. It’s the Interface, Stupid! 2002 CAC Panel Shubu Mukherjee VSSAD, Intel Corporation Disclaimer: All opinions expressed in this presentation are mine and only mine. Intel or any other corporation is not liable for any of the material presented in these slides.

2. Memory Memory CPU CPU Memory Bus Memory Bus ? I/O Bus Network Interface Network Interface MPP Interconnect Cluster Interconnect • Interface = Software on CPU + Network Interface • Interface is key performance bottleneck

3. Interconnect not a bottleneck • Interconnect not a bottleneck MPP & Cluster interconnects have similar properties • Link bandwidth ~= 10s of gigabits/second • Link latency ~= nano- to micro-seconds

4. Pentium4 System Bus Pentium System Bus Cray T3D 8085 Bus TMC CM-2 MPP Interconnect Bandwidth not a bottleneck for 1-to-1 communication Alpha 21364

5. 133 MHz / 64 bit PCI-X Mellanox Infiniband Quadrics QsNet 32 MHz / 20 bit Sun Sbus Myricom Myrinet IBM PC Cluster Interconnect Bandwidthnot a bottleneck for 1-to-1 communication

6. Interconnect Latency not a bottleneck • Sample breakdown + Interconnect latency ~= 10s of nanoseconds + Network Interface ~= few microseconds + Software ~= 10s of microseconds to milliseconds • Example + Ed Felten’s Thesis, 1993 + On Intel Delta (MPP, but clusters would be similar) + Hardware = 1 microsecond + Software = 67 microsecond

7. Where are the bottlenecks? • Software Interface to interconnect • operating system intervention • protocol stacks • reliable delivery • congestion control • Hardware Interface to interconnect • Extra hop via I/O bus (MPPs usu. don’t have this problem) • Side-effect prone hardware (e.g., uncached loads) • network interface hardware designed accordingly

8. Winner’s Properties?The Right Thing To Do • User-level access to network interface + Myricom Myrinet or Quadrics QsNet (from Meiko CS2) + Infiniband or cLAN’s (with VIA) • Streamlined Network Interface + Integrated I/O bus and Cluster Interconnect + Direct Memory Access + Treat Network Interface like Cacheable Memory + most I/O bridges already do this - most network interfaces don’t support this yet

9. Tug-of-War: inertia vs. performance • Inertia to use existing software + Gigabit Ethernet with TCP/IP • Performance from Cluster Interconnects + User-level access and streamlined network interface • IDC Forecast, May 2001 (from D.K.Panda’s Infiniband Tutorial) % of Infiniband-enabled servers + 2003 : 20% of all servers + 2004 : 60% of all servers + 2005 : 80% of all servers • And, the winner is …. ULTINET (NOT) + User-Level Thin Interface NETwork

10. Don’t Discount Inertia • Software exists and works for gigabit ethernet & TCP/IP • Hardware is cheap and widely available • It is a price/performance/inertia issue • not performance alone • Infiniband will probably be a temporary I/O bus/switch/backplane • 3GIO coming up (backward compatible with PCI) • Mellanox, Quadrics, Myricom • in a niche market, which can be dangerous because the volume may not be high enough • generalizing and adapting other interfaces (e.g., ethernet) may help their business model