InfiniBand

1. InfiniBand André Larocque (1811828) Francis Lacroix (1826581) Thursday March 14th, 2002 This Report was prepared for Professor L. Orozco-Barbosa in partial fulfillment of the requirements for the course ELG/CEG 4183 This is the title page, not much to put as far as speaker’s notes.This is the title page, not much to put as far as speaker’s notes.

2. Agenda Introduction Current I/O Technologies Limitations of Current I/O Technologies Previous Solutions InfiniBand’s Role InfiniBand Architecture InfiniBand Benefits Current Status of InfiniBand The Future of InfiniBand Conclusion Again, not much value of speakers notes since this is only the agenda.Again, not much value of speakers notes since this is only the agenda.

3. Introduction “Without I/O, computing just can’t happen. But over the years traditional PCI bus, SCSI bus and ATA I/O connectors have become almost too slow for today’s computing need.”1 New I/O technology developed by the InfiniBand Trade Association which was originally formed by Compaq, Dell, Hewlett-Packard, IBM, Intel, Microsoft and Sun Microsystems “Without I/O, computing just can’t happen. But over the years traditional PCI bus, SCSI bus and ATA I/O connectors have become almost too slow for today’s computing need.”1 In order to address this need for greater performance I/O technology, the InfiniBand Trade Association was formed in October 1999. Made up of Compaq, Dell, Hewlett-Packard, IBM, Intel, Microsoft and Sun Microsystems, the aim of this association was to create an alternative to traditional I/O connectors whose technologies were now becoming obsolete. The InfiniBand Trade Association now counts over 160 members. 1Christine M. Campbell, The InifiniBand Alternative, searchEnterpriseServer, June 18 2001, http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci754589,00.html“Without I/O, computing just can’t happen. But over the years traditional PCI bus, SCSI bus and ATA I/O connectors have become almost too slow for today’s computing need.”1 In order to address this need for greater performance I/O technology, the InfiniBand Trade Association was formed in October 1999. Made up of Compaq, Dell, Hewlett-Packard, IBM, Intel, Microsoft and Sun Microsystems, the aim of this association was to create an alternative to traditional I/O connectors whose technologies were now becoming obsolete. The InfiniBand Trade Association now counts over 160 members. 1Christine M. Campbell, The InifiniBand Alternative, searchEnterpriseServer, June 18 2001, http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci754589,00.html

4. Introduction New I/O technology developed is named InfiniBand. Original design is the result of merging Future I/O and Next Generation I/O. Successor to the Peripheral Component Interconnect (PCI) technologies. Currently concentrates on server I/O. The technology since developed is called InfiniBand (previously called System I/O). The original design was the result of merging the Future I/O design developed by Compaq, IBM and Hewlett-Packard and the Next Generation I/O design developed by Intel, Microsoft and Sun Microsystems. Seen as a successor to the Peripheral Component Interconnect (PCI) technologies in both personal computers and servers, InfiniBand is an architecture and specification for data flow between processors and I/O devices. Expecting to replace PCI in both home computers and servers within a few years, InfiniBand promises greater bandwidth as well as greater expandability. However, for the time being, the InfiniBand Trade Association is concentrating its effort in developing the architecture and specifications for the server market (server-to-server, server-to-storage area and server-to-network) where the limitations of current I/O technologies are most felt and a replacement most awaited. InfiniBand hopes to solve the I/O problems as well as offer new network layout and architecture possibilities. The technology since developed is called InfiniBand (previously called System I/O). The original design was the result of merging the Future I/O design developed by Compaq, IBM and Hewlett-Packard and the Next Generation I/O design developed by Intel, Microsoft and Sun Microsystems. Seen as a successor to the Peripheral Component Interconnect (PCI) technologies in both personal computers and servers, InfiniBand is an architecture and specification for data flow between processors and I/O devices. Expecting to replace PCI in both home computers and servers within a few years, InfiniBand promises greater bandwidth as well as greater expandability. However, for the time being, the InfiniBand Trade Association is concentrating its effort in developing the architecture and specifications for the server market (server-to-server, server-to-storage area and server-to-network) where the limitations of current I/O technologies are most felt and a replacement most awaited. InfiniBand hopes to solve the I/O problems as well as offer new network layout and architecture possibilities.

5. Current I/O Technologies The figure depicted on the slide shows the architecture of a typical server, whose limit is denoted by the dotted line (and grey area). The typical server will contain a Memory Controller which will tie one or more CPU together with the system’s Memory and PCI bus bridge. In order to allow the system to perform I/O operations and interact with the outside world, the PCI bus will connect to one or more adapter cards. In turn, these adapter cards may be connected to Local Area Networks (LANs), Storage Area Networks (SANs) or other devices to give the system access to media outside it’s chassis.The figure depicted on the slide shows the architecture of a typical server, whose limit is denoted by the dotted line (and grey area). The typical server will contain a Memory Controller which will tie one or more CPU together with the system’s Memory and PCI bus bridge. In order to allow the system to perform I/O operations and interact with the outside world, the PCI bus will connect to one or more adapter cards. In turn, these adapter cards may be connected to Local Area Networks (LANs), Storage Area Networks (SANs) or other devices to give the system access to media outside it’s chassis.

6. Limitations of Current I/O Technologies “Existing PCs and server systems are hampered with an inflexible and relatively slow internal data flow system, including today’s 64-bit, 66 MHz PCI bus”2 Lack of development for the PCI technology has caused several problems to surface, specially in the server domain. “Existing PCs and server systems are hampered with an inflexible and relatively slow internal data flow system, including today’s 64-bit, 66 MHz PCI bus”2 The PCI technology currently used for I/O has known very little development compared to CPUs and Ethernet technologies, making its capacities unbalanced with the rest of the system technologies. This unbalance is the source of many problems, specially in the server domain where the demands on I/O devices is greater than in home computers. In order to understand the nature of InfiniBand it is important to understand the problems which it attempts to solve. The next few slides will explain these problems in more details, which fall into 2 main categories: 1) I/O bottleneck; 2) RAS problems. It will be shown later that InfiniBand was designed to address the very issues. 25 June 2001, http://searchstorage.techtarget.com/sDefinition/0,,sid5_gci214596,00.html“Existing PCs and server systems are hampered with an inflexible and relatively slow internal data flow system, including today’s 64-bit, 66 MHz PCI bus”2 The PCI technology currently used for I/O has known very little development compared to CPUs and Ethernet technologies, making its capacities unbalanced with the rest of the system technologies. This unbalance is the source of many problems, specially in the server domain where the demands on I/O devices is greater than in home computers. In order to understand the nature of InfiniBand it is important to understand the problems which it attempts to solve. The next few slides will explain these problems in more details, which fall into 2 main categories: 1) I/O bottleneck; 2) RAS problems. It will be shown later that InfiniBand was designed to address the very issues. 25 June 2001, http://searchstorage.techtarget.com/sDefinition/0,,sid5_gci214596,00.html

7. Limitations of Current I/O Technologies The above figure represents delays in data processing. The section below the yellow dotted line represents hardware delays (measured in seconds) while the section above the yellow dotted line represent software delays and are measured in CPU cycles. It should be noted that the sections are not to scale but never the less do give a representation of the importance of each delay within their domains. As can be seen, an increase in Ethernet bandwidth can only increase efficiency up to a maximum point. Passed a certain threshold, the PCI bus cannot handle the traffic coming from the Ethernet adapter. Therefore, as the amount of data and information coming into the system and flowing between components increases, the existing PCI bus cannot handle the increased load and becomes a performance bottleneck in the system. It should also be noted that the PCI bus delay indicated is for one device. If two or more devices are connected to the PCI bus, the bandwidth of the PCI bus will be shared and therefore create an even greater bottleneck. This problem is compounded by the fact that the distribution of the bandwidth of the PCI bus depends on the cooperation of adapter cards instead of a central authority. If one of the adapters misbehaves, the available bandwidth to the adapters might be greatly reduced. The final bottleneck problem from which PCI suffers is the relation between the amount of I/O devices which it may control and the bus speed. Due to the use of a shared parallel bus, there are electrical limitations on the number of PCI adapters that can be used at each bus speed, with fewer cards being used at higher speeds. Certain versions of PCI technologies only allow one adapter card to on the bus at the highest speed. In order to use more adapters, a lower bus speed has to be used which further increases the bottleneck problem (specially since the bandwidth is divided at the lower speed) The above figure represents delays in data processing. The section below the yellow dotted line represents hardware delays (measured in seconds) while the section above the yellow dotted line represent software delays and are measured in CPU cycles. It should be noted that the sections are not to scale but never the less do give a representation of the importance of each delay within their domains. As can be seen, an increase in Ethernet bandwidth can only increase efficiency up to a maximum point. Passed a certain threshold, the PCI bus cannot handle the traffic coming from the Ethernet adapter. Therefore, as the amount of data and information coming into the system and flowing between components increases, the existing PCI bus cannot handle the increased load and becomes a performance bottleneck in the system. It should also be noted that the PCI bus delay indicated is for one device. If two or more devices are connected to the PCI bus, the bandwidth of the PCI bus will be shared and therefore create an even greater bottleneck. This problem is compounded by the fact that the distribution of the bandwidth of the PCI bus depends on the cooperation of adapter cards instead of a central authority. If one of the adapters misbehaves, the available bandwidth to the adapters might be greatly reduced. The final bottleneck problem from which PCI suffers is the relation between the amount of I/O devices which it may control and the bus speed. Due to the use of a shared parallel bus, there are electrical limitations on the number of PCI adapters that can be used at each bus speed, with fewer cards being used at higher speeds. Certain versions of PCI technologies only allow one adapter card to on the bus at the highest speed. In order to use more adapters, a lower bus speed has to be used which further increases the bottleneck problem (specially since the bandwidth is divided at the lower speed)

8. Limitations of Current I/O Technologies In servers, RAS (Reliability, Availability & Serviceability) is extremely important. Reliability Ability of a system (in our case server) to perform its required functions under stated conditions for a certain amount of time. Shared-memory architecture which prevents separation of different I/O controller’s address space. Parallel nature of the PCI bus prevents the system from detecting and isolating a malfunctioning adapter. RAS (Reliability, Availability & Serviceability) represents the stability and performance of a system. For server systems, RAS is a crucial aspect of the system. Unfortunately, in recent years several shortcomings of the PCI technology have surfaced which prevent PCI from fulfilling the RAS requirements. Reliability: Represents the ability of a system (in our case server) to perform its required functions under stated conditions for a certain amount of time. PCI fails to meet this requirement due to many problems. First, PCI uses a shared-memory architecture which prevents separation of different I/O controller’s address space. If one adapter card encounters problems, it may accidentally interfere with the memory space of another adapter. This will lead this I/O instability and may even lead to system failure. As well, the parallel nature of the PCI bus prevents the system from detecting a malfunctioning adapter. The system cannot therefore isolate the malfunctioning card and continue operating without it.RAS (Reliability, Availability & Serviceability) represents the stability and performance of a system. For server systems, RAS is a crucial aspect of the system. Unfortunately, in recent years several shortcomings of the PCI technology have surfaced which prevent PCI from fulfilling the RAS requirements. Reliability: Represents the ability of a system (in our case server) to perform its required functions under stated conditions for a certain amount of time. PCI fails to meet this requirement due to many problems. First, PCI uses a shared-memory architecture which prevents separation of different I/O controller’s address space. If one adapter card encounters problems, it may accidentally interfere with the memory space of another adapter. This will lead this I/O instability and may even lead to system failure. As well, the parallel nature of the PCI bus prevents the system from detecting a malfunctioning adapter. The system cannot therefore isolate the malfunctioning card and continue operating without it.

9. Limitations of Current I/O Technologies Availability The degree to which a system is operational when required for use. Usually achieved through redundancy. Shared parallel nature of the PCI bus cannot support these redundant connections. Serviceability The degree at which components can be added, removed or exchanged in the system. There are no stable and reliable hot-pluggable mechanisms for PCI Cards/Adapters. Availability: The degree to which a system is operational when required for use. Availability is most commonly insured through redundancy. For I/O, this would mean having multiple connection to outside media through redundant adapters. The shared parallel nature of the PCI bus, however, cannot support these redundant connections as certain control parameters and controller functions would conflict over the bus. Serviceability: The degree at which components can be added, removed or exchanged in the system. While there exists certain system with mechanisms for hot-pluggable PCI Cards (PCI cards can be added or removed without powering down the PCI bus), these mechanisms do not posses the required stability and reliability needed for enterprise and large-scale servers. Therefore, because the cards are internal to the server, the PCI bus must be powered down in order to add, remove or exchange adapters. Unfortunately, powering down the PCI bus requires shutting down the server, which is unacceptable for many systems.Availability: The degree to which a system is operational when required for use. Availability is most commonly insured through redundancy. For I/O, this would mean having multiple connection to outside media through redundant adapters. The shared parallel nature of the PCI bus, however, cannot support these redundant connections as certain control parameters and controller functions would conflict over the bus. Serviceability: The degree at which components can be added, removed or exchanged in the system. While there exists certain system with mechanisms for hot-pluggable PCI Cards (PCI cards can be added or removed without powering down the PCI bus), these mechanisms do not posses the required stability and reliability needed for enterprise and large-scale servers. Therefore, because the cards are internal to the server, the PCI bus must be powered down in order to add, remove or exchange adapters. Unfortunately, powering down the PCI bus requires shutting down the server, which is unacceptable for many systems.

10. Previous Solutions Two main solutions: Future I/O Next Generation I/O Remedies Specialized Web Appliances New PCI Technologies Integrating Gb Ethernet into the Chipset Hardware acceleration for TCP Before the arrival of the InfiniBand architecture, many other solutions and concepts were proposed to solve the I/O problem. Many of the ideas behind these solutions have been integrated in the InfiniBand solution. The bulk of the proposed solutions consisted of the previously mentioned Future I/O and Next Generation I/O. These have now been merged to create what is now InfiniBand and will not be discussed here. In additional, certain remedies were also suggested. These remedies are more short term solution but are reflected in InfiniBand. The first is the addition of specialized Web Appliances (servers, etc…). While these may be build for higher performance for specialised tasks and facilitate the distribution of load (reducing the need for I/O on a particular machine), they unfortunately add additional Network Stack and OS penalties for each additional hop, negating any benefits. The second remedy is to develop new PCI technologies. While they offer slightly increased I/O speeds, this increase is still nowhere near the needed speed. These new PCI technologies also suffer from the same RAS problems as standard PCI. The third remedy consists of integrating the Gb Ethernet I/O technology directly into the Chipset. This remedy solves the problem for Gb Ethernet, but not for any other type of I/O. The final remedy consists of integrating TCP into the hardware. This will free a tremendous amount of CPU cycles, since a lot a time is kept maintaining the TCP stack. However, this will not solve the PCI bottleneck problem as well as create some very strong OS dependance. While none of these remedy solves the problem on its own, many aspects of these have been integrated into InfiniBand as will be seen.Before the arrival of the InfiniBand architecture, many other solutions and concepts were proposed to solve the I/O problem. Many of the ideas behind these solutions have been integrated in the InfiniBand solution. The bulk of the proposed solutions consisted of the previously mentioned Future I/O and Next Generation I/O. These have now been merged to create what is now InfiniBand and will not be discussed here. In additional, certain remedies were also suggested. These remedies are more short term solution but are reflected in InfiniBand. The first is the addition of specialized Web Appliances (servers, etc…). While these may be build for higher performance for specialised tasks and facilitate the distribution of load (reducing the need for I/O on a particular machine), they unfortunately add additional Network Stack and OS penalties for each additional hop, negating any benefits. The second remedy is to develop new PCI technologies. While they offer slightly increased I/O speeds, this increase is still nowhere near the needed speed. These new PCI technologies also suffer from the same RAS problems as standard PCI. The third remedy consists of integrating the Gb Ethernet I/O technology directly into the Chipset. This remedy solves the problem for Gb Ethernet, but not for any other type of I/O. The final remedy consists of integrating TCP into the hardware. This will free a tremendous amount of CPU cycles, since a lot a time is kept maintaining the TCP stack. However, this will not solve the PCI bottleneck problem as well as create some very strong OS dependance. While none of these remedy solves the problem on its own, many aspects of these have been integrated into InfiniBand as will be seen.

11. InfiniBand’s Role Same purpose as PCI: connect servers (CPU and memory) to external storage and LANs. Will coexist with, and then completely replace the PCI architecture. The purpose of the InfiniBand Architecture (IBA) is the same as PCI: to connect servers (CPU and memory) to external storage and LANs. In this respect, IBA simply replaces the PCI architecture. The IBA simply replaces the PCI architecture but as with any new technology, there will be migration to IBA as the system I/O interconnect, with PCI-based I/O and IBA-based I/O coexisting for a period of time. The purpose of the InfiniBand Architecture (IBA) is the same as PCI: to connect servers (CPU and memory) to external storage and LANs. In this respect, IBA simply replaces the PCI architecture. The IBA simply replaces the PCI architecture but as with any new technology, there will be migration to IBA as the system I/O interconnect, with PCI-based I/O and IBA-based I/O coexisting for a period of time.

12. InfiniBand Architecture