IEEE 1394 (FireWire) compliant cards and cables make for good, cheap cluster High Speed Interconnects (HSI) within some strict confines. Since much of the traditional network gear (switches, routers, etc.) is not available under IP-over-FireWire, this standard lends itself most to directly connected hypercubes and similar constructs. By virtue of the fact that you are restricted to this arrangement, though, you can make some network configuration refinements that ameliorate some of the shortcomings of this implementation. Many operating systems tuned for clustering already support an implementation of IP-over-FireWire, and most of these are coming into compliance with the applicable standards specifications, including RFC 2734, IPv4 over IEEE 1394.
What is FireWire?
FireWire (also called iLink) is a peer-to-peer high speed serial bus standard. It was developed jointly by Sony and Apple and has gained acceptance among PC manufacturers since standardization. It is a specification like USB but capable of higher speeds (up to 400Mbps in the original design). Like the SCSI standards, one can daisy-chain peripherals without regard to sequence. But unlike SCSI, you can hot-swap and add up to 63 devices. The dream of FireWire was to interconnect various peripherals (that could not be interconnected with earlier protocols), with less worry over electrical shorts or other concerns, using the same cables. Today, you can buy (external) FireWire drives and other peripherals including scanners. Since the original 1394 standard developed in 1995, newer versions have envisioned faster speed (using fiber cables) and a number of different printer adaptations like the p1394 series, the Imaging Protocol Standards.
Various manufacturers have implemented 1394 in hardware. Texas Instruments has their PCILynx family of chipsets and many developers subscribe to the OHCI (Open Host Controller Interface) specifications.
Why is FireWire appropriate for clustering?
We looked at a couple of trends in PC clustering technologies before initiating experiments with FireWire; the movement towards using multiple NICs and therefore hypercubes (or at least Flat Networks), the trend towards integrated NICs, the proliferation of PCI slots on current motherboards, and the increasing speeds (often achieved through the overclocking options in contemporary BIOS settings) on 32-bit PCI buses. The bottleneck in hypercube clustering performance has long been thought to be the network. While you can get good High Speed Interconnects between nodes, or cheap ones, you could rarely combine the 2 in the same network topology. FireWire promises to deliver at least one 400Mbps network connection per card. This would represent a good bang for the buck, considering A) the low cost of PCI-based FireWire cards and B) the gap between inexpensive network topologies and fast (or at least low-latency) topologies. Proprietary interconnects have filled in this gap between Fast Ethernet (802.3) and Gigabit Ethernet (802.3z) traditionally. Considering that there is about a Gigabit per second of theoretical bandwidth available from the PCI bus (most ATX motherboards have 33Mhz / 32 bit PCI buses and all OHCI cards are 32 bit), it would seem that there would be some way to get more throughput for not much more money. Enter FireWire cards… cheap, ubiquitous, and with plenty of promise. Compared to earlier attempts at using IP over other serial standards, FireWire benefits not only from faster throughput but also from a greater number of devices supported on a given bus, a larger array of devices, ease of interconnect, and reuse of multipurpose equipment.
FireWire network infrastructure: theory.
Many modern ATX motherboards have 5 or more PCI slots. Assuming you use all of them for FireWire cards, as you are likely to do with a hypercube, and you used 2 ports per card, you would realize all of your theoretical bandwidth even with today’s drivers. In fact, it is more likely you would run out of IRQs before you maximized your bandwidth usage. There are FireWire hubs, but not much more networking gear. As of this writing, there are no FireWire switches, routers or other traditional packet switched equipment. This was part of the foundation for the case for building a FireWire hypercube.
FireWire network infrastructure: practice.
Most of the operating systems associated with PC clustering offer some level of FireWire and IP-over-FireWire support. Microsoft, for instance, has had IP-over-FireWire implemented for some time. Unibrain (www.unibrain.com), in particular, has supplied users of the modern Microsoft operating systems (from Windows 98 on) with drivers for most IEEE1394-compatible cards. With the release of Win ME, Microsoft has embraced networking over FireWire and will ship it stock with subsequent operating systems.
Unibrain also offers a port of their FireNet Station driver to support the current generation of Apple computers which have FireWire connections built in. The Unibrain driver does not follow the RFC 2374 standard and will therefore not fit in seamlessly to a hybrid or heterogeneous network, but as long as you are connecting units into a Unibrain FireNet, you shouldn’t have intercommunication problems. Unibrain also offers a FireNet server product for server editions of the popular operating systems.
FireWire and IP-over-1394 under Linux.
The Linux implementation, while not as seasoned as the Windows variety, does subscribe more fully to the RFC and will probably catch up in performance measures soon. Currently, you can get 100Mbps solid using the eth1394 driver and 120Mbps to 130Mbps using ip1394 for certain packet sizes. Configuration is fairly predictable after you have the module installed. Here again, the OS would support multiple cards and multiple ports on cards. Below is our test setup and methodology.
Test configuration (hardware)
We performed throughput and latency tests on the following motherboards, usually in multiples of 2. Our tests show that the number and arrangement of boards, and therefore chipset, did not impact the results noticeably.
A) Abit BP6, Dual Celeron 500 Mhz
B) Abit KT7, Duron 700 Mhz
C) Asus A7A266, Athlon 1.33 Ghz
As you can see, we tested a number of different configurations to see if FSB speed (or the other variables associated with the various motherboards) had an impact upon performance. They did not. We used at least 128Megs of RAM in these configurations (of either PC100, PC133, or PC2100, respectively). Also, it did not appear that performance either suffered or improved on the SMP boxes. For FireWire cards, we used generic OHCI-compliant cards.
Test configuration (software)
The software to use FireWire with Linux is available via the Linux1394 Project (linux1394.sourceforge.net) and associated links. Internode communication tests were performed on machines that were running the Linux 2.4.6 kernel with the ip1394 and eth1394 device drivers (available from the aforementioned site).
We used Guido Fiala’s softnet ports of the eth1394 and ip1394 modules from earlier this year and recreated his test scenario (www.s.netic.de/gfiala) with better machines so we wouldn’t hit the CPU roadblocks he hit. While we tested many different configurations, we took his advice and set the MTU to 2030 in our maximum performance tests. We have a proprietary tool in-house that opens a TCP/IP connection between cards and then calls write() 4096 times with the same 4096-byte buffer. This ran stably and returned similar results to Netpipe. Netpipe (which ran unstably with the ip1394 module) is what we will use to present our data. Netpipe (www.scl.ameslab.gov/netpipe) does have an NT port but the FireNet drivers currently require Unibrain cards so we were not able to perform this test.
Under Linux we had 100Mbps running under eth1394 consistently and stably. We could get up to 130Mbps in certain configurations under ip1394 but these would not hold for all scenarios (large packet sizes) and would eventually crash the kernel. We could use ping and telnet with ip1394 but ftp and most other TCP/IP applications would not work. Netpipe results for the range of packet sizes from 1000 bits to 10000 bits using ip1394 are shown logarithmically. Here are the 2 throughput vs. blocksize graphs for the 2 modules.
As you can see from the ip1394 graph, we hit a maximum throughput (135.5 Mbps) at a packet size just under 25000 bytes.
You will notice that the Netpipe test for the eth1394 module went through to completion but it leveled off at its cap bandwidth (100Mbps) at 10000 bytes.
Both graphs show an irregularity at the 2000 – 3000 byte point suggesting that the packet fragmenting algorithm works best with packets larger than the MTU size.
IP-over-1394 is ready for primetime within some restrictions. If you are running a Microsoft operating system, like Windows 2000 Advanced or Datacenter Server, then a hypercube may satisfy your intercommunication needs. The bandwidth will certainly be greater than a similarly constructed Fast Ethernet cube. A popular driver for this architecture, FireNet, will be RFC 2734 compliant in version 3.0.
If you are using Linux as your OS, then you can also use your 1394 cards as NICs. Driver development is continuing and promises to make IP-over-FireWire a viable alternative network choice for directly-connected clusters. It is unlikely we will ever see a switch for FireWire, since this would only be useful for this application.
The future of this use of the IEEE 1394 standard may depend upon the price and performance advances of competing technologies and upon commodity hardware market trends but research continues at a breakneck pace. There are also some new standards that have come into being recently and are being awarded close scrutiny by the FireWire development community. These are RFC 2855, DCHP for IEEE 1394, and investigation of IPv6 over IEEE 1394. We will post updates to the Linux drivers as they come available to extreme-linux.com.