I myself have built a NAS server on OpenSolaris using ZFS on 8 hard drives in a raidz2 configuration (tantamount to hw RAID6) and so far I'm happy with it. ZFS is currently the safest alternative available and its free of charge. I chose OpenSolaris because it has the most up to date ZFS implementation but there are other alternatives as well. FreeNAS (freenas.org) is another popular solution that uses ZFS.
The hardware I would recommend is any hardware that is compatible with OpenSolaris and/or FreeBSD. What is important is that the SATA controller and the network adapter work well on the system.
For SAS/SATA I would recommend any of LSI's controllers or any LSI chipset based controller. Examples are LSI SAS3081E-R, Intel UC8I, Dell Perc 5i (I think, check that is has the LSI SAS1068E chip), Fujitsu-Siemens Megaraid controller, ... Many motherboards come with JMicron based controllers which are really bad. The motherboards could be good but I would stay away from the JMicron controllers. I would also be a little careful with Marvell based SATA controllers since they have poor compatibility with OpenSolaris. I would also recommend staying away from the RocketRAID controllers. I don't think they are poor controllers but they have poor driver support (the manufacturer are slow on the updates and so on). They work on FreeBSD but not OpenSolaris. What is important if you choose to use ZFS is that you choose a card that has no hardware raid at all and only presents the drives as is with no hardware based "error correction" at all. This is important because ZFS needs to "see" the drives for what they really are and not get obscured by the controller in any way. In short, the controller should operate in IT mode (Initiator-Target). The good news about it is that this is found on the cheapest SAS/SATA controllers.
As for ethernet, most motherboards I have seen comes with a Realtek gigabit network controllers. The latest builds of opensolaris works fine with these. However, if you want good performance I would recommend buying a good Intel card that is used for fileservers. But I'm happy with my Realtec NIC and have so far no plans to upgrade to an Intel server card.
I cannot speak about freenas since I haven't tried it but I have heard good things about it and it seems to be a popular alternative.
I would stay away from these HW RAID based NAS boxes out there but that's me. I just don't trust them and believe that if you have some time and patience it is worth it to build your own both from an economical perspective and from a reliability perspective.
Opensolaris worked for me out of the box and it was extremely simple to set up a ZFS pool with file systems. Setting up CIFS/Samba was a little "quirky" but the intstructions on the web made it a cakewalk. Updating my system to the latest build was a little tricky but you can download the latest build and install it all from scratch without problems (you get it from pkg.opensolaris.org/dev) and it should be no problems. OpenSolaris is quite simple when it comes to administration and operation of NAS, ftp, http, and so on but doing other less conventional things can be tricky. I have problems with making it read NTFS formatted volumes. I have so far not managed to compile fuse and ntfs-3 to operate without making the system crash. Other people may have had better luck with it.