Raising private storage to the limits

A few months ago, I was looking for a new storage system, since my existing file server with it's 2.5TB of data were running full and I didn't want to put another drive in it. The existing file server was pretty loud and it made some trouble with the network connection since some time. I first looked into existing NAS systems, but with a good number of bays (between six and eight) were pretty expensive, while still suffering from the network bottleneck. They also were always just build with MDADM, a Linux software-RAID. My next idea was building a new, more silent file server, but with the right components I would also have gotten to a pretty high price, while still having the problem with the network bottlenecks.

Finally I made the decision that I, as I live alone, do not need a secondary machine anymore, which is running all the time. So I looked around for DAS systems, in the hope they would be cheaper. I came to the conclusion, that this was not the case, and thus looked for some enterprise models. Eventually I found my new storage, which should be a Sonnet Fusion DX800RAID with a hardware RAID controller and eight bays. It was pretty expensive with a price of 2200€, but that shouldn't be a show-stopper. The next thing would be to find the right drives for my new toy, and after getting some information from several friends, I decided to get eight Western Digital RE4-GP with 2 TB each. At that time, all decisions were made and it was time to order all the stuff for about 4400€.

After this month-long looking around, it should only take a week until everything arrived. When I came home at 18 o'clock I directly unpacked the new RAID system and started studying the installation manual. I was a bit shocked when I read that I first have to disassemble the trays from their transport bridges, but after an hour I had it done. It took me another 30 minutes then to prepare the drives with the threadlocker and mount them into their trays, but then I were ready to go on with the RAID itself.

The next part was inserting the RAID controller card into a PCI-E slot, which only took a couple of minutes. Time to tell you, I already bought a new mainboard a few weeks ago as I knew that I would need another PCI-E 8x slot. Anyway, after closing my PC again, I connected the DX800RAID with two mini-SAS cables to the PC and with a power cable to the power supply. Before I should boot the system, I surely had to insert all eight drives, which was done in a couple of minutes as well. Finally I should boot the system.

At this point, I didn't need any manual anymore, since it was straight forward. After the computer BIOS loaded, the BIOS of the RAID controller showed up and I pressed the hot-key to get into the setup for it. There I quickly setted up the RAID6 over all eight drives and rebootet the system. After Ubuntu loaded and the drivers and configuration tool were installed I had to see that the RAID had started rebuilding as it was degraded. As I didn't wanted to do anything wrong, I planned to wait until the rebuild was done. After playing QuakeLive for some hours, I returned to the system at 1 o'clock in the night.

The rebuild was done and I could start setting up an empty partition with DM-Crypt on it with an EXT4 partition, but it shouldn't be that easy as I thought. I were surely already telling some friends about my new toy, and one of them was going to tell me some things I had to consider when setting up everything. After talking a bit about the technical details of the RAID, we figured out that the chunk size was 128kb and the sector size was 521b, thus the empty partition had to start at 1536. This took a while, as GParted wasn't able to setup partition starts at sectors, and parted wasn't so easy to use. The nxt step should be to setup DM-Crypt correctly, and after a while we figured out that you can define the payload offset for encrypted data, which would be 1536 as well. Finally I could set up the EXT4 partition, and with the help of my friend we choosed the parameters for it wisely. At the time we were done, it was already 4 o'clock in the night, so I decided to get to bed after running a quick write-performance benchmark (which showed me nice 200 MB/s) and telling my friend that I owe him a beer.

When I came home the next day, I started copying all data from my old file server to the new RAID system with rsync, which should take a while. One day later, it wasn't even done with the first 1 TB disks (which were running in RAID1), so I decided to move it from the file server to my desktop PC, as I was going to use them as system disk for my next Ubuntu 10.04 installation anyway. This was a pretty hard task, as my huge graphics card blocked two HDD bays and I had to rearrange the disks a bit. I restarted the copying then from the local disk, which only took another 16 hours. The next day I started copying all other data parallely from my file server, which was done just 1.5 days later. Finally I could shut down my old file server forever.

My next step was to setup PAM_mount, as I didn't want to enter a password for DM-Crypt at boot time so I could boot the system remotely. Thanks to good tutorials on the internet, this was a pretty easy task. Eventually everything was done and I could get to bed with no loud machines running in my room anymore. Now it is saturday, one day after this multiple-day experience, and I have to leave now for a barbecue, so here is my conclusion:

It was a joy to setup everything, but less joy to copy all the data, but I learned a lot in this short amount of time. The performance of the DX800RAID is very good, and with the 9 TB of free space (12 TB in total plus 4 TB parity information) I should have a future-proof storage system for the next years. If anyone of you is looking for a good solution, I can recommend you the Sonnet Fusion products.