Performance testing an external USB drive array

This is a somewhat more technical post than is my usual fare, so my apologies to my non-geek friends.

Last night, I started doing some performance testing of my new external drive array. The storage array is an Addonics Storage Tower USB/JBOD, which provides a USB to IDE adapter that supports four drives. The storage array shows up to the system as a single USB device with four targets. USB 2.0 operates at a nominal speed of 60 MB/s, although due to command overhead, performance usually maxes out around 50-55 MB/s. Most typical USB drives usually max out between 15 and 20 MB/s.

I wrote a small benchmarking tool to test sustained sequential write performance. I chose write performance as my benchmark standard for a couple of reasons. First, it is easier to eliminate the effects of caching. An fsync() call at the end prior to closing will ensure that all data has been committed to disk. Second, it can be used to highlight the performance drop (if any) that is associated with calculating and writing parity on RAID volumes. The following table gives the performance of each configuration, and transfer rates at different transfer sizes. I included an internal SCSI drive in the tests as a comparison between a server-class drive and the external USB drives.

		MB/s
Configuration	8K	64K	128K
UFS single drive SCSI	36.57	35.07	35.80
UFS single drive USB	12.80	12.59	12.00
ZFS single drive USB	10.94	12.58	13.03
ZFS dual master-master stripe USB	10.34	12.25	12.86
ZFS dual master-slave stripe USB	10.34	12.49	12.90
ZFS RAID-Z four drives USB	7.12	7.83	7.93

One interesting result of the tests was that striping two drives together did not improve performance at all. This indicates that the performance is limited somewhere between the USB controller card and the USB/IDE adapter. Given that a single USB/IDE adapter is used to connect all four drives, it is likely that the adapter is the point of contention. If there were multiple adapters, the host system might be able to queue writes to multiple drives simultaneously.

The ZFS performance is based on default settings. I have started testing a variety of ZFS block sizes. Setting the block size to 8K improved the performance of the small writes to be similar to the UFS configuration.

Overall, the performance of the storage array in a RAID-Z configuration is not great for writes, but it is tolerable for a USB setup. Given the fact that most of my drive writes will be across the network, the array speed is fast enough to keep up with the network. I may experiment with adding a second USB/IDE adapter inside the enclosure, to see if I can take advantage of parallel writes.

Geek night

I recently ordered a new drive enclosure and drives for my server, and the enclosure arrived today. I am doing some experimenting with Sun's new filesystem, ZFS, and the new drives will help me with the research. I now have 1.2 Terrabytes of drive space configured in a 960G RAID-Z configuration. Needless to say, I think I will be all set for disk space for the next several years. And while ZFS has some nifty data protection features, such as online data checksums, built-in snapshots, and RAID, I'm still trying to figure out how I will do offsite backups of all that data.

A quick visit home

I flew home to Michigan this weekend in order to get some work done on my house. A friend of mine is becoming my roommate, so I had to get things ready for him to move in. In the process, I achieved a significant milestone. Three years after I bought my house, I finally moved in to the master bedroom. It was a little weird sleeping in a completely different room, but it felt good to be sleeping in my bedroom. I still have some minor work to finish on the room, such a terminating the phone and network, but the room is quite liveable. I'm hoping that I can put the finishing touches on it when I'm home again later in September.

In other news, my brother is moving from DC to Denver next week. He has been desperately trying to move out of the DC area, and he finally got a reasonable offer. It's going to be really weird for me to not have him in the same state, let alone the same time zone. Still, I totally understand his desire to get out of DC. DC is a fun place to hang out and visit, but it's not for everyone. What's interesting is that I had looked at Denver as a possible place to live about two years ago. The tech industry has a significant presence out there, and I could probably find some decent contracts out there if I wanted to.

A Tri-County Ride

Today was an absolutely beautiful day, so I headed out to the western edge of Loudon County for a bike ride. I'm not quite sure what fit of insanity gripped me today, but I wanted to bike over a mountain. I started in Purcellville, and proceeded to Snickersville Pike. Snickersville Pike runs through "hunt country", and is filled with large estates, horse farms, and vineyards. At the far edge of Bluemont, I headed south on VA-601. The road immediately starts climbing the mountain, and keeps going up. It eventually peaks at 1825 feet, about 1300 feet higher than when I started. By the time I reached the top, I was pretty worn out - and I still had another thirty miles to go.


Paris, VA

Down near Paris, VA, I left Clarke County and crossed into Fauquier County. I had to follow US-50 for a few miles, and thankfully there were reasonable shoulders to ride on. I headed southeast past Upperville, and then headed back north on winding gravel roads. By the time I finally made it back to Purcellville, I had clocked about forty-seven miles on the trip.

A Northern Virginia Ramble

I headed out yesterday for another bike ride, even though the temperature was around ninety-five degrees. As I started on the W&OD trail, the waves of heat rose from the asphalt, and it felt like I was riding into a hair dryer. Thankfully, the trail west of Leesburg is reasonably shaded, and the ride got a little easier. I decided to leave the trail and go exploring rural Virginia. At Hamilton, I headed north towards the little town of Waterford. Waterford is a quaint little town that has been preserved for its historical buildings. The town is extremely small, and seems like something from the early 1900's. From there, I looped over to Wheatland on John Wolford Road, which is a gravel road. The gravel and washboard surface was rather challenging to navigate on my road tires. From Wheatland, it was back south to the W&OD trail, and off to Purcellville and back.


The W&OD Trail

While I was riding on the W&OD Trail near Hamilton, I came across an unusual sight. A rooster and a hen were walking along the side of the trail. The rooster just looked at me while I rode past, like "hey, don't interrupt my game!"


Yesterday's Ramble

An Ann Arbor Weekend

Last weekend, I went home to Ann Arbor for a long weekend. My house is part of Ann Arbor's Footing Drain Disconnection Program, so I needed to be home to get some estimates for the work. The visits by the city program manager and the contractors did not require much time, so I had plenty of time to enjoy being home in Ann Arbor. I got to visit friends, enjoy Ann Arbor attractions like Ashley's and ABC, spend some time running, and even get some workout time at the NCRB. Overall, it was a very low-stress weekend.

While I was home, I stayed up extremely late one night finishing up Harry Potter and the Deathly Hallows. The book was fun to read, and wrapped up a lot of the unanswered questions from the earlier books. There were a few parts that seemed a bit contrived, but it still was an enjoyable book.