MarketEye: INFINIBAND: Finding Its Niche

When InfiniBand was first proposed, it was touted as the next generation I/O for servers, with many believing its application would reach down to the desktop PC.  Multiple connectors were specified and the connector market was forecast at hundreds of millions dollars.  The companies backing the standard dedicated some of their most talented designers to the effort and made significant investments to develop the technology.  The original steering committee consisted of seven stellar members: Intel, IBM, Dell, Hewlett-Packard, Compaq, Microsoft, and Sun Microsystems: virtually the who’s who of the server market.  The committee now consists of all the original members except Microsoft (and Compaq is now a part of HP) plus InfiniSwitch, Mellanox, Agilent, Infinicon Systems, and Voltaire. But what started with great fanfare soon faded when development delays, migration problems, chip complexities and cost began to take its toll.

InfiniBand is a low latency, point to point, serial network interconnect aimed at the server market.  It defines the hardware and software elements that enable an I/O message from the processor to be sent out over a point to point serial connection to a peripheral device.  As shown in the diagram, the hardware involved in this action consists of the Host Channel Adapter (HCA), which takes the parallel data from the CPU and properly formats it for transmission across a serial link.  The serial link connects to the InfiniBand switch fabric which analyzes the address data in the message to connect it to the proper destination, where a Target Channel Adapter (TCA) takes the serial channel input and changes it to the format of the peripheral device to complete the I/O process.

The InfiniBand software defines all of the formats and software controls surrounding this process.  InfiniBand offered the promise of a unified datacenter interconnect technology capable of replacing PCI, Fibre Channel and even some Ethernet interconnects as well as any peculiar buses.

Meanwhile, InfiniBand also offered performance and maintenance improvement over the venerable PCI bus.  For more than 10 years, the task of interfacing the processor with the input-output (I/O) devices has been the job of the PCI bus.  PCI is a shared bus where the peak bandwidth is divided between the devices to which it is connected.  A weak point of this technology is that a failure of any connected device will cause the entire bus to be disabled. That, along with the fact that it is limited in both bandwidth and the number of I/O devices it can handle, made InfiniBand the hope for a solution to this bottleneck in server performance.

The first sign of trouble came when development began to lag.  Then, came the loss of key support.  Intel, which was more focused on the entry level server, was pursuing a single channel (1X) solution which Microsoft was unwilling to support with native drivers in its operating system software.  It also appeared that the PC side of Intel was independently developing a simpler version of a serial I/O for the PC (ultimately named PCI Express).  These developments caused Intel to basically exit their support for InfiniBand and end their efforts at developing the enabling chipsets for the technology.  Intel has remained as a steering committee member but has disbanded its extensive InfiniBand dedicated facilities and high powered development engineering group.  Intel cited the progress of some of the start-ups as a reason for abandoning its efforts.  Microsoft has relinquished its position on the steering committee but says it is facilitating the use of third party software efforts to provide InfiniBand support.  This probably means that Intel and Microsoft realized that the smaller InfiniBand Market was not going to return a reasonable ROI to justify the capital necessary to develop their products for InfiniBand.  These actions left many wondering if InfiniBand was indeed doomed.

The conservative server vendors were reluctant to leave the proven technologies and move to InfiniBand, which required all new hardware and software solutions to be deployed.  Meanwhile the makers of incumbent PCI technology made significant improvements in speed with PCI-X.  This technology was an evolutionary step from existing PCI technology and did not require all new software.  There were also improvements to eliminate some of the maintenance concerns of the shared bus structure.  This approach was more attractive to many in the server user community.

InfiniBand faded fast and little was heard from the technology.  A couple of years passed before High Performance Computing (HPC) solutions appeared being implemented around InfiniBand technology.

It began with an announcement at the Supercomputer Symposium (SC2002) in Baltimore in November 2002 that Los Alamos National Laboratory has installed a major supercomputer made up of 128 computers interconnected by InfiniBand.  In a lab environment, the mission critical business concerns are not there.  The lure of the capability of InfiniBand was enough of an incentive to work through some of the inevitable problems that come with a new technology. In September of 2003, Mellanox announced that they had been chosen by Virginia Tech to supply InfiniBand switch technology to build a supercomputer with 44Tbits of total bandwidth.  The same month, InfiniCon announced they had been selected as the clustering fabric supplier to Fujitsu who will “build an InfiniBand cluster that will contain 2,048 Intel Xeon™ processors and deliver a peak performance of 12.4 trillion FLOPs (floating point operations per second), vaulting it to the third position in the Top 500 rankings of the world's most powerful supercomputers.”  The Fujitsu cluster will be utilized by Japan's Institute of Physical and Chemical Research (RIKEN) for biotechnology research.  Completion of these efforts represents a major step toward debugging all the new hardware and software in the InfiniBand architecture.  More importantly, the InfiniBand platform moved to a position of “verified” performance and away from its unproven status.

The good news continued with the Jan.14, 2004 announcement of the signing of a deal by IBM with InfiniBand switch supplier “Topspin” to be a reseller and design partner.  IBM will use InfiniBand to provide cost effective supercomputer products to the industry.  Dell’s “Power Edge” blade computers offer Topspin technology pass-through module to enable InfiniBand connectivity.  The value of InfiniBand in server clusters for increased performance is now being recognized.
    
Though it is not the ubiquitous replacement for PCI, InfiniBand is not a total failure.  InfiniBand remains a powerful tool for the enterprise.  Its value in the clustering environment is significant.  It has been said that InfiniBand provides “Bandwidth out of the box.”  Until now the high speed front side buses ultimately communicated outside the box through much slower interfaces.  InfiniBand was designed to accommodate server interactions with other servers, storage, and external resources.  When it comes to clustering, InfiniBand is excellent.  Likewise, for inter-processor communications, InfiniBand provides a low latency, high bandwidth path to facilitate multiple servers working on a single application.

Connectors for InfiniBand

InfiniBand selected three connector series to meet the requirements of the envisioned InfiniBand system.  These included a backplane connector (Tyco’s “SpeedPak”), a single channel (1X) I/O connector (Tyco’s “HSSDC2”) and a 4 or 12 channel I/O connector (Fujitsu’s “MicroGigaCN”).  The MicroGigaCN is second sourced by FCI (a product they call “EyeMax” and also produced by Molex (“LaneLink”) and others.  Mainly, the 4 channel (4X) version of the MicroGigaCN finds use in the industry.  In HPC the 12X version of the MicroGigaCN has found application as well.  Contrary to the early predictions of dominance, the IX connector quantities have proven to be very small and the use of the backplane connector has not been significant.

InfiniBand’s resurgence into the spotlight with its deployment in supercomputers has significance only because of the fact that most of these use clusters of low cost processors.  The high end computer market has sales of just a little over a hundred thousand systems a year.  So, if market penetration was 100% there would not be a huge market. InfiniBand’s potential for volume comes if it can move down the server market to the entry level “Blade” servers where it shows signs of life.  Sun has said that their blade servers will facilitate the use of InfiniBand and that their support is expected to grow along with industry acceptance.  Dell and IBM have already been mentioned as supporting InfiniBand.    

There is little chance that InfiniBand will ever approach the lofty heights to which it was once thought to be headed.   PCI Express has already been anointed as the high speed serial PCI replacement.  Intel avoided the potential for Microsoft to influence the fielding of PCI Express (as they did when they decided to not support a 1X InfiniBand implementation) by designing PCI Express to use the current PCI software drivers.  This ensures the operating system will supply native support.  Intel has proposed PCI Express for its servers, as well, but understands the importance of InfiniBand in connecting processors for HPC systems.  Intel presents PCI Express as a complimentary technology to InfiniBand and has participated in demonstrations such as the 10TOTS (Ten Teraflops Off The Shelf).  This was a demonstration using commercially available Intel Xeon processor based servers and third party channel adapters and switches interconnected with IB cables to configure a world class supercomputer. 

The performance of such systems will ensure a place for InfiniBand which should grow as software progresses to take advantage of such computing capability. 

In spite of all the predictions of doom, InfiniBand is still a viable and useful technology.  Though it probably never will replace Fibre Channel as the interconnect to storage or Ethernet in the LAN, it has now established itself as a proven technology.  With maturity, recognition of its potential to be the “universal” solution, as was once envisioned, could be recognized.  With working hardware and software, there is always the potential to re-emerge as the universal server I/O technology.