Part II

Well...after all this groping around, let's see what Mr. X came up with!

Storage Scalability requirement: 13 TB
User Base: 25 simultaneous clients.
Mr. X decided to go with a standard, mid-tower setup with a standard ATX form factor. The first and the most crucial component is the selection of the chipset and a server motherboard that utilizes the full features of the chipset. There are quite a few server motherboards available in market. What do we have to look for w.r.t building a fileserver? Here:

Chipset and Motherboard: The Front Side Bus connects the processor and the chipset, which in turn interfaces to other devices of the system through various interfaces like PCI, AGP, etc. The speed of the FSB is a crucial component, which if selected improperly, can lead to bottlenecks in the system. A slow FSB will cause the CPU to spend significant amounts of time waiting for data to arrive from system memory. In today’s market, FSB is overtaken by other technologies, like the Intel Shared Bus Architecture, or the more industry standard HyperTransport technology adopted by many vendors. We shall choose the HyperTransport technology and AMD processors for its various advantages in both compatibility and performance. The Hypertransport speeds extend from 400MHz to 2.6GHz. The most common speed available in today’s chipsets are in the range of 1 GHz. There are various vendors making AMD chipsets in the market. nVidia nForce Professional 3600 chipset for our purpose. Considering the processor and chipset we chose, and comparing various server motherboards in the market, we’ve reached a conclusion on the Tyan Thunder n3600S S2927 dual proc motherboard with the nVidia nForce Professional 3600 chipset. A detailed explanation of the board’s features can be found at http://www.tyan.com/product_board_detail.aspx?pid=175.

I really wish it could be this one, I really really do:
http://www.tyan.com/product_board_detail.aspx?pid=554
But as you see, we’ve got a budget of 2000$ to catch up, and this buddy’s a big stretch. However, if your disk server really has a huge load to serve, go for it (along with scaling the other components).
The most notable features are:

FSB: 1 GHz HyperTransport link
Processor Support: 2 AMD Socket 1207 L1 Sockets supporting AMD Opteron™ 2000 rev F processors.
Chipset: nVidia nForce Professiona™l 3600 chipset
Memory Support: 32 GB DDR2 ECC 667/533/400 DIMMs
Expansion Slots: 1 PCI-E 16x + 1 PCI-E 8x + PCI

Processor: We have limited the choice of the processor between vendors Intel® and AMD, as they are the most commonly available in the market today. As both the processors provide comparable performances on a single processor motherboard, what is left to check is the architecture and scalability to more than one processor. Intel has been following the Shared Bus Architecture, which utilizes a single bus to fetch and write to the memory and devices. When we use more than one processor, it becomes difficult to scale. Intel has created many workarounds for this, mainly various caching and prefetching techniques, but the architectural constrain still remains. The AMD architecture on the other side, has adopted the industry standard point to point Hypertransport technology, eliminating the need for a shared FSB, and simplifying the architecture by eliminating many interfacing chipsets, and using standard tunnel chips for interfacing. (Wow, I didn’t know I could create such a big and complex sentence!). In multiprocessors AMD uses the Direct Connect Architecture where the microprocessor is directly connected to other CPUs through a proprietary extension running on top of two additional natively implemented HyperTransport interfaces; whereas Intel multiprocessors have to hit the shared bus each time it has to access the memory, resulting in latency. In Xeon processors, they try to circumvent this limitation by providing a good amount of cache (4MB). The argument between Intel and AMD architecture can continue, with both sides having its pros and cons. Anyways, I feel its silly to fight over the powers of processors, and if you wanna, check out the various discussion forums where they do that. However, in light of the points mentioned here, and considering the lower costs of the AMD chip, we chose to go with socket 1207 AMD 2000 series Rev F processors, specifically, the AMD Opteron 2212 (dual proc) 2.0 GHz. Besides, we chose an AMD motherboard, remember?

Memory: The speed of the memory is directly related to it’s interconnect with the processor. For a data-centric application we require the system to be of high stability, thus eliminate any possibility of over-clocking and choose ECC technology. Fully buffered memory has been seen to be used in some systems, especially HP and MAC systems for more stability, but considering the cost and performance degradation of using Fully Buffered memory, w e choose against using it. ECC proper cooling, and proper clocking should suffice our requirement. We can limit the speed on 667MHz, as a higher speed would increase the possibility of more memory errors. As memory in our setup is in a NUMA (local to each processor) configuration, the speed does not have to scale as more processors are added. The memory supported by the chipset is 32GB, 16GB per processor, thus we have plenty of room to scale.

Disk Array: The nVidia 3600 chipset has support for RAID 0, 1, 0+1, 5, and JBOD. However, it is better to offload the RAID handling from the chipset to a separate controller, more specifically built to handle RAID and has more manageability and backup features. The main features we look for this are the interconnect speeds, manageability and functionality is provides. Manageability includes local and remote monitoring, support for standards like SNMP, firmware updates, etc. Interconnect for most modern systems are PCIe, under various lane configurations (32, 16x, 8x, 4x and 1x). Considering the speed of the SATA disks available in market today, which support 3.0 Gbps transfer rates, and the number of disks handled by the controller, we probably would go for an 3ware 9650SE-12ML, which has superb RAID levels support, including RAID 6. It also provides a PCIe interface of 8 lanes, which theoretically should give a maximum transfer speed of 20Gbps under PICe 1 specification of 2.5 Gbps per lane. The 3ware card does automatic assignment of LUN’s for volumes greater than 2TB for OSes which doesn’t support it. It also provides 64 bit LBA addressing. If you calculate on a very vague level, whether there might be any bottlenecks, by taking the average disk speed as 50-54 MBps, for a 12 disk configuration; we find that it leaves plenty of space on the PCIe 8x bus.

Disks: The cost per GB of hard disks has reduced to as low as 0.014$ in 2007 for SATA disks, and still going down steeply. Capacity and speed are main metrics to measure hard disk performance, but there are many other server class features in today’s hard disks, like reliability (measured in MTT, mean time between failures), noise level, heat generation, NCQ, RAID optimization, etc. The traditional fileserver storage component is SCSI disks, offering high reliability and speed. However, today the SATA and SAS technologies have become so mature that major storage giants have entire product lines released on these technologies. We choose SATA disks for our storage server, due to the increased reliability at a relatively low price. Specifically, we choose the Western Digital RE2 RAID Edition 500GB SATA disks. They have special optimizations made in for performing better in RAID environments and have server class storage metrics like high MTT, 16MB Cache, NCQ support, etc. They are not as fast as the 10k RPM disks, but overall, comparing the RAID performance, reliability and other metrics, we can say that these disks are one of the best choices for building a storage server at this price.
Network Interface Cards:
No matter how much the performance of the disk server is, what comes out of a NAS box is what the user gets, and that’s what the really matters. It is essential that whatever throughput the server can give under a specified load should be what the user base should get. For this, we have to carefully choose a NIC, which can handle such throughputs. Comparing various products in the market, we have chosen the Intel PRO/1000 PT Quad Port Server Adapter. The Intel PRO/1000 PT Quad Port Server Adapter has a Quad port configuration (4x 1Gbps) and a PCIe 4 lane interface to the mainboard. It has dual controllers handle the network traffic of the 4 ports, and the ports can be bonded in failover aggregate mode, providing a virtual 4Gbps link under a single IP address. The PCIe interface ensures that data can be supplied to the controllers under the rate it wants, eliminating any bottleneck in the interface part(4x2.5 = 10 Gbps). Remember that these being standard PCIe devices, can be replaced by an FC interface card to convert the box to SAN storage, if needed, in future (provided you have the software to support it).

Chassis: The choice of the chassis is one of the often overlooked aspects while building a server, not just in the issue of cooling, but also of various features that can be integrated into it. Many of the important features are redundant power supply, hot swappable disks, cooling, expansion, etc. The stability of the system depends a lot on the running temperature, the power input, proper shielding from interference, etc. Due to these reasons, even though a bit costlier option, I’m gonna go with a chassis which is specifically built to handle file server type loads and environments. The Supermicro SuperChassis 745TQ-800 is a high availability chassis, with lot of useful features needed in server type architectures. Some of the key features are redundant 800W High-efficiency Power Supply, 8 x 3.5" SAS/SATA Hot-swappable drive bays, 7x full-height, full-length expansion slot, 3x Hot-Swappable Cooling Fans and 2x 5000 RPM Hot-Swappable Rear-Exhaust Fans.

Proceed to next part>