provided by: EDN High-performance external storage is a godsend for consumers averse to cracking open their gear's enclosure to upgrade its storage capacity or for manufacturers and service providers averse to having their customers tackle such surgery. Looking back over the past few years' worth of interface history, the following chronologically ordered achievements stand out as particularly notable:
- In April 2000, the USB Version 2.0 specification, which added support for 480-Mbps High-Speed mode, a substantial performance boost over its 12-Mbps predecessor, debuted.
- In May 2001, the IEEE 1394 standards committee introduced the Version B specification amendment, adding support for 800-Mbps interface speeds, which were twice as fast as the first-generation IEEE 1394a-that is, FireWire 400. In April 2002, the committee subsequently approved IEEE 1394b, also known as FireWire 800.
- On Aug 23, 2005, the SATA (Serial Advanced Technology Attachment) International Organization announced Version 2.5 of the interface specification after roughly one year of public development. The specification documents, among other things, the eSATA (external-SATA) interface. This standardized eSATA implementation supersedes numerous proprietary external-SATA approaches that predated it (Reference 1).
Silicon Image and its partners and competitors in the SATA industry, such as Addonics Technologies, have been talking up eSATA connections since at least mid-2003 (Reference 2). And, over the past four years, I've repeatedly wondered why. Granted, transferring small, occasional bursts of data into and out of a hard drive's integrated RAM cache might derive a benefit from SATA's 1.5- to 3-Gbps speeds, as might a cluster of parallel-accessed hard drives, such as a large RAID (redundant-array-of-inexpensive-disks) 0 striped array. But it has always seemed to me that conventional hard-disk drives' 40- to 60-Mbyte/sec sustained-transfer rates would constrain the performance of most applications and most external-storage configurations with just one or a few hard drives. FireWire and USB2, at least on paper, should be equally adept at supporting this per-drive sustained-speed metric, and you might already be planning on using one of these more general-purpose interfaces for other peripheral-tethering purposes (Table A at www.edn.com/070510cs).
Assuming that your design already has USB2, FireWire, or both, what, if any, justification is there for spending incremental IC, hardware, and software budget to also include one or several eSATA ports on the system back panel? Analogously, if you're planning on using FireWire 400, should you upgrade your design to support more costly but backward-compatible-in conjunction with connector-translation dongles-FireWire 800? Those two fundamental questions prompted me to tackle this hands-on project, whose initial results you'll find in this print article and that will live on in cyberspace (see sidebar "Online addenda exceed print's storage capacity").
A top-notch test bed
Benchmarking projects are tricky, because up-front assumptions can heavily influence the outcome. If I select a combination of equipment, software, and usage-model variables that are too specific, my results would be meaningful to only a narrow set of readers. Choose a too-broad set of options, on the other hand, and I end up with a bewildering plethora of outcome data. Peeking ahead at the tables of results in this article, you might accurately conclude that I've erred on the side of breadth. Nonetheless, I still see plenty of potential external-storage-application scenarios that this project doesn't encompass. It's important, therefore, that you understand my assumptions up-front before proceeding to the conclusions. In that way, you can judge the results from an informed perspective and, if necessary, repeat my testing with altered assumptions that better match your application characteristics.
Note that I conducted my testing on a high-end desktop PC running an up-to-date version of Microsoft's Windows XP Professional (Table B at www.edn.com/070510cs and Figure 1). I selected Windows XP, as I have in past hands-on projects, because I'm intimately familiar with it and because a diversity of potential benchmarking utilities are available for it. However, if you plan to run an embedded-Linux distribution in your design, for example, you might want to minimally educate yourself on the differences between the two operating systems with respect to storage performance. Consider, as well, taking the next step and using my project as a template for your own Linux-based analysis.
Similarly, consider my choice of an AMD Quad FX system in light of your target-design parameters. I selected this platform for two reasons. First, it encompasses the PCI and PCIe (PCI Express) expansion-interface flavors necessary to minimize any system-bus-induced bottlenecks in testing. Second, I hoped that using dual 3-GHz FX-74 CPUs, each containing two discrete CPU cores, would eliminate any system-processor limits on external-storage performance. However, it's likely that your target design's microprocessor plans are more judicious; if so, keep in mind that software-crunching bottlenecks elsewhere in the system may suppress any speed disparity between external-storage alternatives that this study reveals.
Benchmark abundance
In preparing for this project, I surveyed a number of published storage-benchmarking studies from a variety of online and print publications to assess what Windows-based testing suites they used. I came up with a list of approximately 20 synthetic-benchmarking packages (with each package often comprising multiple subutilities) in addition to the wide variety of real-life application-based benchmark sets that commonly find use. Clearly, I'd need to whittle down the options list to a reasonable subset if I ever wanted to finish my work! After much investigation and gnashing of teeth, I settled on two benchmarking utilities-File Systems and Physical Disks-within a testing suite I've used several times in the past, SiSoftware's Sandra.
SiSoftware's documentation states: "File Systems benchmark[s] mounted file systems volumes. [It] shows how your file systems connected to storage adapters and storage hosts compare to other devices in a typical computer. This is not the raw disk performance that other benchmarks test-but the speed of the volume itself that depends on many more factors like file system, operating system cache, position on disk. ... Thus this is the performance you get at the file system level. ... Drive Index: is a composite figure representing an overall performance rating based on the average of the read, write, and seek tests, and file and cache size. The Drive Index is intended to represent drive performance under typical use in a PC. A larger number means better performance. The weighting of the results is not equal; it represents the distribution of different files sizes as used on these devices (obtained through field research).
"Physical Disks benchmark[s] hard disks ... the disk itself, not the file system). [It] shows how your physical disks connected to the storage adapters or hosts compare to other disks in a typical computer. As the test measures raw performance it is independent [of] the file system the disk uses and any volumes mounted off the disk. Read Test: Sequential across disk. Write Test: Sequential across disk. Seek Test: random, full stroke. ... Drive Index is a composite figure representing an overall performance rating based on the highest read or write speed across the whole disk ... the higher the better. Access Time is the average time to read a random sector on the disk, analogous to latency response time ... the lower the better."
I configured both utilities to use write-through mode-that is, to bypass Windows' write cache with the intent of more accurately measuring the speed of the external-storage interfaces and their connected storage peripherals. Also, analyzing the benchmarking-results reports provides other useful information: The File Systems test employs a 2-Gbyte test-file size, and both it and the Physical Disks test use overlapped I/O commands with a four-request I/O-queue depth. I left enabled the default-benchmark support for multithreading, commensurate with the four CPU cores in the system.
Interface options
Working outward, after considering the CPU and its associated software as potential external-storage-performance bottlenecks, the various system interfaces are next to face scrutiny. The Asus L1N64-SLI WS motherboard within the AMD Quad FX reference system contains Nvidia's nForce 680a SLI (Scalable Link Interface) core-logic chip set, which comprises two MCP (media-and-communications-processor) north-bridge chips, each supporting 16- and eight-lane PCIe buses. XFX GeForce 7900 GTX graphics cards running in an SLI configuration populate the motherboard's two 16-lane PCIe connectors; my efforts focused on the "upper" eight-lane PCIe bus along with the motherboard's 32-bit, 33-MHz PCI port (Table C at www.edn.com/070510cs). Note that the test system supported Version 1 PCIe speeds.
I employed the motherboard's built-in USB2, FireWire 400, and eSATA capabilities, along with several add-in boards. Asus implemented FireWire 400 and eSATA, respectively, with Via Technologies 6308P and Silicon Image single-lane PCIe Sil3531 transceivers. I tested two FireWire 800 cards-Belkin's 32-bit, 33-MHz F5U23-APL PCI product and NitroAV's one-lane PCIe board. Both PCBs (printed-circuit boards) employ Texas Instruments' TSB82AA2 FireWire 800 IC, revealing that the NitroAV board also contains a PCI-to-PCIe bridge chip.
By press time and despite multiple requests, I was unable to obtain a FireWire 800 reference board from any of the vendors that are supposedly shipping native PCIe ICs. However, because the TSB82AA2 supports both 32- and 64-bit, 33-MHz PCI operation, I anticipated that NitroAV's card might run the PCIe bridge chip in 64-bit PCI mode, thereby boosting performance beyond the capability of the motherboard's 32-bit PCI bus and explaining why I decided to test both FireWire 800 boards.
I also attempted to supplement my testing of the motherboard's native 3-Gbps eSATA capabilities with a Silicon Image eight-lane PCIe add-in board. Its foundation silicon, the 3-Gbps SiI3124ACBHU, contains a PCI-X interface, so the reference board also includes Intel's PCI-X-to-PCIe bridge. According to Alex Chervet, a marketing manager at Silicon Image, "In a PCIe [eight-lane]-inclusive [Apple] G5 Power Mac, we've clocked over 800 Mbytes/sec sustained performance when [we connected it] to four SV2000s comprising a total of 20 drives." That assertion whetted my appetite to testdrive both available eSATA options.
Unfortunately, I figured out after I'd completed my testing that there was a mismatch between the firmware image on the card and the corresponding driver suite I had installed for it and the SATARaid5Manager software. As a result, to date, I've been able to benchmark the Silicon Image SteelVine SV2000 only in its lowest potential performance hard-disk-drive configuration when I tethered it to the SiI3124. For more, visit my blog at www.edn.com/briansbrain, where I hope to post a full suite of testing results on the SiI3124 by the time you read this. I also aspire to post test results for AMCC's 3ware 9650SE-4LPME native PCIe board, which tethers to the Sidecar external-storage peripheral over a four-lane proprietary eSATA bus that AMCC refers to as xSATA.
Storage devices
And what storage peripherals will I mate with the external-storage interfaces (Figure 2)? (Also see Table D at www.edn.com/070510cs.) First up is Seagate's ST3500601XS-RK, which supports 3-Gbps eSATA speeds and contains a single 500-Gbyte hard drive that operates at 7200 rpm and has a 16-Mbyte cache. Maxtor's (now part of Seagate) OneTouch III Turbo Edition includes two 500-Gbyte hard drives operating at 7200 rpm with 16-Mbyte caches. You can arrange these drives in RAID 0 and RAID 1 configurations using the company's OneTouch Manager software. The OneTouch III Turbo Edition offers USB2-, FireWire 400-, and FireWire 800-interface options. Finally, I tested Silicon Image's 3-Gbps SteelVine SV2000, which has five 160-Gbyte hard drives operating at 7200 rpm and with 8-Mbyte caches. The SV2000 can operate in five-partition ("contiguous," in Silicon Image parlance), JBOD (just a bunch of disks, or "concatenated"), RAID 0 ("striped"), RAID 1 ("mirrored"), RAID 1+0-that is, RAID 10 ("mirrored striped"), and, when in a software-based RAID arrangement, RAID 5 ("parity-RAID") configurations.
At first glance, these three products may appear to differ so vastly as to be benchmark-incomparable. Reality, however, isn't nearly so bleak. Note that the two-drive Maxtor unit, which "stripes" its reads and writes across both drives in parallel in its highest performance RAID 0 mode, can also operate in a single-drive-equivalent RAID 1 mode. Similarly, the five-drive SV2000 can also operate as a single-drive (discrete, JBOD, and RAID 1) and dual-drive (RAID 1+0) equivalent. Keep these common features in mind as you peruse the results.
Keep in mind, too, that RAID configurations incur processing overhead beyond their single-drive counterparts. For mirrored modes, the system needs to write common data to both drives, and it needs to ensure that the parallel data it reads from both drives results in a match. RAID 0 writes a subset of the total information to each drive, and it combines partial-data fetches from each drive during a read operation. RAID 1+0 combines the RAID 0 and RAID 1 requirements, and RAID 5 is even more complex, requiring the calculation of parity information during writes and validation by means of this same parity data during reads.
Silicon Image's SV2000 includes a SiI4726 storage processor whose embedded RISC engine optionally offloads all hard-drive-management tasks, including RAID control, from the host computer's CPUs. A RAID-software stack running on the host computer could-and, with simpler multidrive external-storage peripherals, does-accomplish this same function. Which approach produces a higher performance result depends on, among other things, the relative amount of available processing muscle on each side of the eSATA link. To date, I've collated benchmark results for the SV2000 with the SiI4726 in JBOD mode and with all RAID functions consequently implemented in system software through the SATARaid5Manager. The SV2000's configuration capability, regardless of whether software or the SiI4726 handles the RAID functions, is impressive; users can configure the five hard drives' raw capacity in an infinite variety of ways, encompassing multiple partitions of multiple types and spanning one drive to many drives as well as to a subset of any drive.
The two-drive internal RAID 0 system array, whose stats I include for comparison, comprises two 150-Gbyte Western Digital Raptor hard-disk drives operating at 10,000 rpm and with 16-Mbyte caches. Nvidia's RAID-management utility configures these drives with a 64-kbyte block setting and divides them into two partitions: NTFS (New Technology File System)-formatted drives C and D. The stripe setting for the SV2000 in its striped, mirrored-striped, and parity-RAID configurations was an 8-kbyte chunk. I couldn't determine the stripe size that the Maxtor OneTouch III Turbo Edition employs in RAID 0 mode. I used the default Windows XP cluster size for drives that I formatted in NTFS: Maxtor's OneTouch III Turbo Edition, which was initially formatted as HFS (Hierarchical File System) for use on Apple's OS X; Seagate's ST3500601XS-RK, which was in FAT (file-allocation-table) 32 format; and the SV2000, which was unpartitioned and, therefore, unformatted when I received it.
File Systems
I uncovered a few issues during the course of this project; for that reason, I was unable to test some configurations (Table 1). Among those problems, as previously mentioned, was the fact that I could not access the SteelVine SV2000 through the SATARaid5Manager configuration software when it connects to the SiI3124 reference card. For this reason, I could test the SiI3124-and-SV2000 combination with the storage peripheral only in a contiguous configuration. Maxtor's OneTouch Manager also immediately terminated with an error message whenever I tried running it with the PCIe FireWire 800 card plugged into the system, regardless of whether the OneTouch III Turbo Edition was connected to the card. I didn't observe this issue with either the native FireWire 400 ports or the PCI FireWire 800 card, and it precluded me from reconfiguring the OneTouch III Turbo Edition from RAID 0 to RAID 1 mode. Because my primary motivation was to test the storage peripheral in its fastest possible configuration (RAID 0), thereby potentially maximizing the achievable FireWire 800 bus speed, I didn't focus much effort on resolving or working around the OneTouch Manager issue with the NitroAV card.
When comparing one storage-expansion bus with another, begin by matching up data sets for common-drive-count configurations, such as the RAID 1-configured Maxtor OneTouch III Turbo Edition versus the Seagate ST3500601XS-RK and versus the Silicon Image SV2000 in its concatenated, contiguous, or mirrored modes. Then, see whether reading and writing multiple drives in parallel-the OneTouch III Turbo Edition in RAID 0 mode or the SV2000 in mirrored-striped, parity-RAID, or striped modes, for example-results in a performance increase. These guidelines will help you discern when a given interface, rather than the drive or drives connecting to it, is the performance bottleneck.
Also, note the significant performance impact of running the SV2000 in its parity-RAID (RAID 5) mode. Keep in mind that, in this case, the SiI4726 storage processor handles the parity calculation and confirmation. In contrast, RAID utilities from companies such as Intel with its Matrix Storage Manager and Nvidia with its MediaShield handle this task on the host CPU. Because the SiI4726 storage processor doesn't support RAID 5, the software-based approach I've tested is your only option with this RAID mode. Silicon Image's SteelVine product manager, Jason Green, comments, "I would expect SATARaid5 to run quicker when using the 3124 host-bus adapter. It will be slow on the 3531 link due to the limit of single-lane PCIe." On the other hand, though, note how speedy the SV2000 was in its five-drive RAID 0 striped configuration, approaching and, in some cases, even exceeding the performance of the internal two-drive RAID 0 array. This internal array comprises hard-disk drives with nearly 40% higher rotational speeds and each with twice the RAM cache of its SV2000 equivalent.
USB2's consistently lower read performance than that of FireWire 400 is evident from the results, despite the fact that USB2 has a 20% faster signaling rate. I suspect that higher USB-protocol overhead and USB's need for the host to provide the arbitration and scheduling of transactions are both partially to blame for this discrepancy. The need for the host's actions couples with the relatively simple point-to-point FireWire connections that this benchmarking project uses. These connections negate any peer-arbitration overhead that might be present in a more complex FireWire configuration.
Physical Disks
In an attempt to better discern the performance potential of various interface-and-drive combinations by operating them in an operating-system-generic fashion, I also ran them through Sandra's Physical Disks benchmark. This portion of the project produced another set of intriguing data, although it brought up at least as many new questions as it solved!
The issues with the NitroAV/OneTouch Manager and SiI3124/SV2000 combinations explain most of the "not-tested" sections of Table 2. Two additional omissions this time around both relate to the internal-drive RAID 0 array. I couldn't do a partition-free read test on the array because it had the operating system and applications suite on it. Similarly, Sandra's Physical Disk write test refused to run because it found a valid partition present on the internal-drive set. "For security, so that users don't destroy their data on disks by mistake, the disk to be write-tested must not have any partitions," says SiSoftware spokesman Catalin Adrian Silasi. "The benchmark does not test if the partitions are empty, since each format is different, but whether there are any. Thus, delete any partitions from the disk before benchmarking it."
In many cases, you'll notice a decrease in read and write performance as the drive's head moves across the platter, from the 0 to the 100% position (Figure 3). This well-known phenomenon reflects the different linear velocities at a constant rotational speed at various points across the platter's radius, along with different data-storage densities at those same points. If you inspect the output plot of the SV2000 in its concatenated mode, you can clearly see the repeated access-time-degradation pattern as the benchmark cycles from one hard drive to another across the five-drive array.
The Physical Disk data is interesting, but it's dubious enough that I resist drawing any definitive conclusions from it. Look, for example, at the USB2 numbers. How can a bus capable of only 480-Mbps signaling rates generate 83-Mbyte/sec-that is, 664-Mbps-transfer speeds? SiSoftware's Silasi notes, "The test does raw sector reads and writes using I/O Windows functions. Instead of opening a file as the other benchmarks-file system [and] removable storage-do, it opens the disk. The test specifies no buffering, and it reads and writes a multiple of the disk and controller sector size. Although the test specifies no buffering, it is up to the driver to actually honor these requests. As you probably know, when Microsoft introduced Windows XP and Windows 2003, SCSI drives had "worse" write performance-not just in Sandra, but also in other benchmarks. The reason for this [poor performance was that SCSI drives] honored these flags, whereas IDE drivers would buffer regardless of the flag setting. It is possible that the driver for your disk caches and reads ahead for better performance, even when told not to," he says.
Here's another oddity: I would frequently get vastly different read-performance results depending on whether the storage peripheral was partitioned, even though the benchmark was supposedly accessing the device in a non-OS-specific fashion. "The benchmark uses raw-sector I/O with no buffering," says Silasi. "It makes no difference whether the drive is formatted or not. ... [The varying performance depending on the presence or absence of a partition] should not happen." Again, he adds, the driver could cache the data, using system memory it allocates for this purpose, in certain cases, thereby possibly explaining the discrepancy.
The benchmark data is also questionable when it's not repeatable to within a reasonably small percentage, accounting for slight run-to-run variability. I tested the SV2000's read performance in its contiguous mode in the following sequence of steps, each of which I conducted in a drive-by-drive fashion:
- With no partition and with one-lane PCIe-based eSATA,
- With no partition and with eight-lane PCIe-based eSATA,
- With a partition and one-lane PCIe-based eSATA, and
- With a partition and eight-lane PCIe-based eSATA.
Through Step 1, the five contiguous drives each delivered 55-Mbps read speeds at the 0% head position (Figure 4). However, part of the way through Step 2, the 0%-position speed leaped to 93 Mbps, and it stayed there through steps 3 and 4. On a hunch, I went back and retested all five contiguous drives in the Step 1 and Step 2 configurations, and, this time, they all exhibited the 93-Mbps initial-access uptick.
Finally, note that the severe performance penalty that occurred when I ran the SV2000 in its parity-RAID (RAID 5) mode on the File Systems test was not in evidence with the Physical Disks benchmarks.
Next steps
By the time you read this article, I hope to have made continued progress in a number of areas. Please visit my blog at www.edn.com/briansbrain on EDN's Web site for additional results reflecting this subsequent analysis.
Shortly before wrapping up this article for print, I recalled that Windows XP SP2 at least initially ran 1394a and 1394b FireWire devices at only 100 Mbps (Reference 3). I recall that a subsequent OS patch from Windows Update fixed this error, and my benchmark-study results reveal a performance gain for FireWire 400 over USB2 and for FireWire 800 over FireWire 400. However, Web research gives inconclusive information on whether I still need to install an obscure Windows "hot fix" to unlock all of FireWire's potential. After I get a definitive ruling on the matter from Microsoft, I'll rerun my FireWire 400 and FireWire 800 tests on the OneTouch III Turbo Edition if necessary.
I plan to rerun my SV2000 tests with the SiI3531, this time with the SteelVine storage appliance's SiI4726 storage processor rather than the AMD FX-74 CPUs handling hard-disk management. I also hope to test the SV2000, in both software- and hardware-based RAID configurations, with the SiI3124 card. The SV2000's performance, especially in striped mode, is impressive, even when you tether it to a one-lane PCIe transceiver. I'd like to see how much faster it could be with an eight-lane PCIe connection.
Speaking of higher eSATA speeds, the AMCC 3ware Sidecar, in conjunction with the 9650SE-4LPME native PCIe board and its quadruple-lane xSATA capabilities, is intriguing. I plan to focus my attention here after tackling the preceding challenges.
SiSoftware's Silasi reports that my erratic test results have spurred him to work on an updated Physical Disks benchmark methodology. "We're working on improving the benchmark to add an offset in subsequent runs," he says. "Basically, instead of reading sectors A, B, C, and D, we're going to read A+block, B+block, and so forth on the second run, thus slightly out of sequence compared to the first run, and similarly alter the pattern again on the third run." I promised him I'd run the benchmark again after he finalizes the update, and I'll report back to both him and you any improvements I encounter.
Finally, I'd like to replace the 7200-rpm hard drives in all four storage peripherals with 10,000-rpm Raptor-drive equivalents and rerun my tests to assess how much faster the results can be in conjunction with Western Digital's SAS (serial-attached SCSI)-threatening speed demons.
Online addenda exceed print's storage capacity
As usual, page-count limitations precluded me from passing along in this print article some of the interesting information that I uncovered during my research. And, again as usual, the EDN Web site provides an alternative publishing venue for this material. Visit the Brian's Brain blog at www.edn.com/briansbrain. There, the posting "AMD's Quad FX: system and strategy" provides additional details on the system I used in my article and analyzes AMD's first plunge into quad-core systems for consumers versus Intel's dual-die, single-package alternative approach. "Analysis downloads and additional results" is the nexus for downloading the multitude of report files and screenshots that my testing generated, as well as the output from the additional planned testing I mention in the article. Speaking of multitude, "Benchmark alternatives" will provide links and critiques of other storage-testing suites you might consider for your follow-on studies. "Education: next steps" provides suggestions for further cultivating your external-storage knowledge. And "Share your thoughts," as its name implies, provides a forum for you to comment on data from my study that you found interesting and to peruse and discuss the observations of your peers. The blog also offers other relevant external-storage postings.
For more information
You can reach Senior Technical Editor Brian Dipert at 1-916-760-0159, bdipert@edn.com, and www.bdipert.com.
References:REFERENCES
- Dipert, Brian, "Speedy simplicity: serial-storage interfaces," EDN, Jan 22, 2004, pg 33, www.edn.com/article/CA373882.
- www.addonics.com/news/product_announcements/2003/External%20Serial%20ATA.pdf.
- http://support.microsoft.com/kb/885222.
author: by Brian Dipert, Senior Technical Editor
EDN. Copyright © 2007 Reed Business Information, a division of Reed Elsevier Inc. All rights reserved.