Sign in with
Sign up | Sign in
Your question

REPOST - help building new Ubuntu file+media server

Last response: in Linux/Free BSD
Share
July 4, 2010 4:50:39 AM

NOTE: sorry to repost this from the CPU-server forums, but I am hoping to find some seasoned Ubuntu advice for parts that you KNOW works together for the below... and I suspect I will be coming back a lot to this forum once I have assembled the server and do my configuration...


I have had several different types of servers running in my house for 10+ years, but they have always been some flavor of Microsoft server-based systems, and mostly used for pure business type reasons (file storage, email, database, etc.)... The time has come for a change, and I want to jump into Linux / open source AND I have changed my focus for the server somewhat.

IMPORTANT: would like to stay in the $3-4K range for new hardware

Functionality Considerations

Functionality needed on server:
  • Mass storage is definitely still in, but will now also include very large media files (movies etc.)
  • Automated backup of clients to the server is definitely in
  • Remote backup of server is definitely in (looking for tips on this by the way)
  • Media streaming and POSSIBLY on-the-fly encoding of the same (been looking at MediaTomb)
  • Remote connectivity + FTP (beyond the LAN)
  • Print server (easy to do yes?)

    Functionality no longer needed on server:
  • Email + database is gone... using external providers for next to nothing

    Technical Considerations

  • Minimum 10TB of storage (with expandability to 20TB+)
  • Processing power to handle key functionality above
  • Reliability is key, with some redundancy (must have hot-swap HDs, thinking Raid 5 as well)
  • Green is a plus, quiet is a big plus
  • OS: thinking Ubuntu initial/basic install (using CLI), but doing most configuration using eBox - yeah, needless to say I need the visual interface (can't be helped), but didn't want to expose the server to a full XWindow server
  • All the various components should be proven to work together (hopefully) and my target OS


    Need input on:
  • Mobo + CPU (would two be needed??... seems like "not needed" with the bloat of MS Server removed)
  • RAM (and how much)
  • Case (would like one nicely setup for hotswap, and also with at least one optical drive slot... rack or freestanding doesn't matter)
  • HD + HotSwap casing options
  • Controller, assuming RAID 5 is a good idea
  • PSU
  • Cooling options (fans, etc.)

    ASSUMPTION: a GPU provides no value on the server (never has in the past, assuming that is still the case given above functionality) + NIC on mobo should suffice... also assuming after initial / basic install of Ubuntu, it can go headless (eBox admin will be done from a client)?


    I know this is a request for pretty much everything, but I have never built a server, and would much prefer some sage / experienced advice - note that I have built plenty of PCs and am very comfortable with assembly of parts. Furthermore, I am actually excited to shed my Windows Server anchor and move to Linux (should be a fun / learning experience).

    On a somewhat unrelated note, but more for those that might be interested... My current server (Windows Server 2003) has been acting as the local DNS + DHCP server as well; I am thinking for this new server (and to hopefully minimize administration), I will simply use the ISP router's DHCP and completely ditch the DNS service - the server will of course get a static IP. There are roughly 10 clients that will use this network + various media devices.
    July 4, 2010 6:03:02 AM

    1.)get ubuntu server and install openbox or lxde. forget installing gnome or kde, just bloat you dont need. just start it when you need it, and forget it when you dont.
    2.)Forget on-the-fly encoding, its really not worth the effort. Just encode to a format that most of your clients will be able to playback. H.264+AAC in mp4 container should be playable by most every device.
    3.) save some $$$ and pick consumer parts over server parts!
    4.) if all your doing is just file serving, you dont need much cpu or ram power... so pick some low power parts and save on electricty bill too! 2gb of ram and a low power dual core amd cpu will be more than enough.

    You also might be interested in ZFS filesystem, on BSD and opensolaris natively and linux via fuse. It really is an awesome filesystem!
    http://en.wikipedia.org/wiki/ZFS
    http://zfs-fuse.net/
    also btrfs, which is kind of still experimental, but similar to ZFS and native under linux.
    http://en.wikipedia.org/wiki/Btrfs
    m
    0
    l
    July 4, 2010 4:46:53 PM

    The ZFS filesystem looks interesting... but what impact does it have on the rest of the server installation? Does it restrict what you can and cannot load etc. especially given how fundamental filesystem selection tends to be? Doesn't the normal Ubuntu filesystem have a lot of the same stuff (swear I saw software raid and stuff there too)?

    I like what I see, but would hate to have it be like (forgive the Windows lingo) installing FAT realizing later it needed to be NTFS...

    On a related note, can software raid support hot-swappable drives?


    Also, there will surely be a ton of questions related to Linux from yours truly soon... but I really need proven hardware advice from someone who has built their own, fully-functional Ubuntu server please please! NOTE: I have pretty much resigned myself to some type of server case in order to support the total number of drives + ability to hot swap drives.
    m
    0
    l
    Related resources
    July 4, 2010 5:20:49 PM

    Yes there is software RAID, however it adds an extra layer over the top of existing filesystems. ZFS has this (and alot of other great features that others dont have) built in to the filesystem, and yes it supports hotswap!

    http://www.enterprisestorageforum.com/technology/featur...

    ZFS is here to stay, however BTRFS is also on the horizon, and its already usable, but its not at the level that ZFS is (yet). In particular it doesnt (yet) support RAID5 like support. Please do more reading (there is way too much for me to write here) on these two new filesystems, as other linux file systems (ext4) are on their way out.

    btw, really there isnt a whole lot of difference between linux desktop install, and server install... just software! My advice to you is pick low power, good quality consumer parts. As far as a case goes, just pick one that fits your needs and budget.
    m
    0
    l
    July 5, 2010 5:08:51 AM

    Did some reading around various sites, and am getting a little freaked out about filesystem issues - people are complaining about it, even though I don't know enough to really understand what the root of the problem is.

    ZFS looks pretty awesome, but using it through FUSE seems to be like rolling the dice a bit (on several boards, there seem to be a lot of people unable to get it to work)? And then the software raid built into Ubuntu is apparently buggy for release 10.04 - was a LONG thread about it... not sure what to take away from all of this, other than I am getting a little bit nervous - I do NOT want to be a tester for my server environment.

    So, assuming I want to be safe & simple (i.e. load up the server OS and have it work without a lot of workarounds), seems I should stick with the hardware raid controller - then put the OS maybe on a raid 1, and data on raid 5... I think some of the stuff I have read looks promising & interesting, but I am extremely nervous about things that require a lot of tweaking to get it working (and outright reluctant to do anything requiring work-arounds) on a server environment, especially since I am completely new to Linux / Ubuntu.
    m
    0
    l
    July 5, 2010 5:45:55 AM

    why limit your self to linux? there is always opensolaris of freebsd!

    Also i have heard good things about ZFS under fuse, its considered stable now, and is very very simple to setup. Where have you heard otherwise? and remember RAID is just one of the features... snapshots and copy on write are other great features that will make your life much easier.
    m
    0
    l
    July 5, 2010 9:07:16 AM

    Why don't ya start with ext3 or 4 with lvm and really good hardware raid from areca or 3ware?

    Keep it simple and straightforward :) 

    Start messing with ubuntu and fedora and get comfortable with linux then in a couple of years you'll know enough that you'll be able to mess with opensolaris of freebsd with relative ease. [ fixed typo ]

    Besides, in a couple of years BTRFS should be much more mature and ZFS will either be nicer or dead if oracle has their way.

    Good luck :) 
    m
    0
    l
    July 5, 2010 1:21:10 PM

    yes, liscense is an issue with ZFS that i would be concerned about too.
    m
    0
    l
    July 5, 2010 7:47:11 PM

    First let me say thank you all who are trying to help this n00b get to the right server solution!

    Now, after having spent a good portion of the evening doing more research (and definitely NOT getting enough sleep - pseudo disclaimer if what follows doesn't make sense), it seems to me that FreeNAS might be the ticket for me.

    FreeNAS:
    Has native ZFS & would allow me to do its powerful version of software raid
    Is very easy to install & administer, with some nice media serving apps
    NOTE: I need to be careful about HD selection (something to do with sector size that can cause performance issues unless you do a geli or gnop work around to explicitly define the HD sector size)
    NOTE2: I still haven't found a clear answer about hot-swap procedures and associated hardware constraints - could be that I am just tired

    So assuming I am on the right track, it seems the hardware considerations for me are:
  • Case - seems like I should definitely look for something server-size with lots of drive bays + hot swap ability
  • RAM - using ZFS for raidz and many TBs of storage recommends lots of RAM... at least 4G
  • Processor - again, ZFS does seem to benefit, and since a raid controller is no longer needed, can funnel the money here and on RAM
  • HDs - gotta get some good ones here, preferably 1.5T or 2T size, and that will play nice with the FreeNAS
  • Mobo - something with good features, likely server mobo?
  • Power supply - need a good quality one that will power storage + expansion

    Question: given that I will eventually have quite a few HDs on this server, do I need to get a SATA controller (not necessarily a RAID controller) for maximum expansion? I am a little concerned that the mobo will have enough ports to handle the volume of drives... any suggestions for where to start?

    Question2: power mgmt is definitely a concern, what is the best way to handle it given my stated build objective? It seems with ZFS that there is usually a fairly continuous access of drives - does this pretty much negate power efficiency on the system?

    Question3: what is the purpose of having a separate log pool?? This has me concerned I need yet another mirror (not just for the OS itself)...

    Question4: backup... given the amount of data, I know I need to seriously develop an overall backup strategy; however, what is the most common way for handling large backup of data here? Is it some type of external USB drive? Heck, I probably need to start another thread for this alone...


    Additional ramblings:
    Various ZFS best practices that I seemingly need to consider: put OS on its own mirrored array (raid 1) - guess a couple of smaller HDs should be fine, but will take up 2 drive slots; put data on raidz or raidz2 with 1 or 2 spares; adding to the storage array will require adding new array sets (as appropriate) - cannot grow storage 1 drive at a time; try to define 1 storage pool per whole disk...
    m
    0
    l
    July 5, 2010 8:06:49 PM

    Doing my own rough draft of tech requirements:

    4G of RAM
    2x250G HDs for OS (mirrored) - some are using SSD HDs here, although I don't know why given how "light" FreeNAS is
    7x2T HDs for data (raidz) - one for parity, 5 storage (10T), 1 spare

    Still need help on selecting mobo, processor, PSU, case and actual parts for the above

    Also, to hit the possible future 20T storage limit, it would seem there is no mobo that could support that many drives - thus needing an actual SATA controller card? With my prelim draft above, I already need 9 SATA connections... which is then brought up to 16 total using the same configuration for 20T of total data storage.
    m
    0
    l
    July 5, 2010 9:19:30 PM

    You'll need to get a RAID/Storage controller of some sort; most Motherboards don't have anything more than 6 SATA ports. 3Ware and Areca make some nice *Nix/BSD friendly controllers.
    m
    0
    l
    July 6, 2010 2:40:58 AM

    Here is what I have specced out at newegg.... will the controller be able to handle all the HDs? Also, if anyone sees anything that might not work with FreeNAS, please let me know. I am bumping up against the top of what I would like to spend, but it is a pretty powerful server, and at least I am not shelling out $2k+ for the OS.

    1 BYTECC 18" Sata and Slim Sata Power 7+6pin Cable, for Sata Slim OD Model SATA-XP118
    Item #: N82E16812270183
    Return Policy: Standard Return Policy
    $6.99
    1 SONY 8X DVD Multi Writer Black Slim SATA Model AD-7700S - OEM
    Item #: N82E16827118034
    Return Policy: Standard Return Policy
    $44.99
    2 Western Digital Caviar Black WD6401AALS 640GB 7200 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
    Item #: N82E16822136319
    Return Policy: Standard Return Policy
    $149.98
    ($74.99 each)
    7 Western Digital Caviar Black WD2001FASS 2TB 7200 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
    Item #: N82E16822136456
    Return Policy: Standard Return Policy
    $1,329.93
    ($189.99 each)
    1 CORSAIR HX Series CMPSU-850HX 850W ATX12V 2.3 / EPS12V 2.91 80 PLUS SILVER Certified Modular Active PFC Power Supply
    Item #: N82E16817139011
    Return Policy: Standard Return Policy -$20.00 Instant

    $10.00 Mail-in Rebate Card17-139-011
    $189.99
    $169.99
    1 ASUS Z8PE-D12(ASMB4-IKVM) Dual LGA 1366 Intel 5520 Tylersburg SSI EEB 3.61 Dual Intel Xeon 5500 and 5600 Series w/ Remote ...
    Item #: N82E16813131373
    Return Policy: Standard Return Policy
    $359.99
    1 3ware 9650SE-4LPML KIT PCI Express Lanes: 4 SATA II Controller Card
    Item #: N82E16816116042
    Return Policy: Standard Return Policy
    $329.99
    1 NORCO RPC-4020 4U Rackmount Server Case - OEM
    Item #: N82E16811219021
    Return Policy: Standard Return Policy
    $319.99
    2 Kingston 4GB 240-Pin DDR3 SDRAM DDR3 1066 (PC3 8500) ECC Registered Server Memory Model KVR1066D3Q8R7S/4G
    Item #: N82E16820139024
    Return Policy: Memory Standard Return Policy
    $269.98
    ($134.99 each)
    2 ARCTIC COOLING Freezer 7 Pro Rev.2 92mm Fluid Dynamic CPU Cooler
    Item #: N82E16835186134
    Return Policy: Standard Return Policy -$5.00 Instant

    $79.96
    $69.96
    ($34.98 each)
    2 Intel Xeon E5520 Nehalem 2.26GHz LGA 1366 80W Quad-Core Server Processor BX80602E5520
    Item #: N82E16819117185
    Return Policy: CPU Replacement Only Return Policy
    $759.98
    ($379.99 each)
    Subtotal: $3,811.77
    m
    0
    l
    July 6, 2010 8:55:46 PM

    Why the hell are you buying an expensive server motherboard, and 2 quadcore server CPUs when all your doing is file serving? You could get away with an ATOM based system with what your doing... Its one way to blow $1100 i suppose, but again WHY? Spend extra money on more harddrives or something...

    everything else looks peachy besides that.

    btw just a random thought i had: Your hardrives are likely to become obsolete a long long time before the filesystem does ;) 
    m
    0
    l
    July 6, 2010 9:21:57 PM

    Skittle - are you suggesting a single processor with a standard mobo? If so, can you point me towards some solid options? Savings could definitely help as it turns out I will need a better controller card... Also, will a standard mobo be mountable inside the case I selected? Finally, have you run ZFS on any of your servers/boxes?
    m
    0
    l
    July 6, 2010 9:50:59 PM

    At most a modern low power dual processor and 2gb of ram will be more than you will ever need for file serving. Also yes a standard ATX motherboard will be mountable in that case.

    I have a small NAS - 3 1.5tb drives with ZFS running on a old celeron - in my crawlspace.

    One of these:
    http://www.newegg.com/Product/Product.aspx?Item=N82E168...
    paired with a regular ATX motherboard will be more than enough.
    I like this motherboard, lots of features and lots of PCIe slots for SATA controllers, also if you ever want to upgrade your CPU, that option is of course available too.
    http://www.newegg.com/Product/Product.aspx?Item=N82E168...
    m
    0
    l
    July 6, 2010 10:49:38 PM

    I will explore this some more for sure then... that is a huge price difference. My worry is about using PC-grade gear in a server box - could be just my own mindset that needs to change.

    Also, quick FreeNAS/ZFS question: are your windows clients able to map network drives??
    m
    0
    l
    July 6, 2010 11:36:56 PM

    http://www.newegg.com/Product/Product.aspx?Item=N82E168...

    Dual-core atom motherboard/CPU, meant for server applications, compatible with Windows, *nix, BSD, you name it. That would be plenty powerful with 2 to 4GB of RAM. No PCIe-x16 slot though, so you wouldn't be able to add a video card for streaming. I'm sure other models include that feature, I was just showing you these products exist.
    m
    0
    l
    July 7, 2010 3:09:26 PM

    skittle and Pyroflea have made some really good suggestions :) 

    The supermicro's nice and low power but it doesn't have the IO, memory bandwidth and IO slots that an AMD does.

    I'd suggest a nice midrange to high end AMD ATX board and an areca or 3ware with 8 ports and a battery.

    An NAS doesn't need a big expensive CPU unless you're planing to use it for more than just serving files across the LAN.

    http://www.newegg.com/Product/Product.aspx?Item=N82E168...

    A 250's only gonna run you 62 bucks out the door and you've got the option to upgrade to x3, x4 or x6 down the road and ya don't have to change your board if you've picked a really nice one right off the bat :) 

    The onboard memory controller on AMD CPUs also supports ECC, desktop CPUs take non-registered ECC, server CPUs take registered ECC.

    You'll save a boatload of money going with AMD, just make sure you get top notch parts all around for a nice reliable build. The corsair power supply you picked seems real nice :)  Watch out tho, there's a lot of cheaply made parts out there that you'll need to avoid, a lot of systems are failing due to faulty parts and very bad quality control, desktop and server alike.

    Good luck :) 
    m
    0
    l
    July 7, 2010 5:41:32 PM

    regarding windows sharesL Yes, either through SMB/CIFS (windows native share) or throurgh NFS shares
    m
    0
    l
    July 7, 2010 6:17:39 PM

    Pyroflea said:
    Dual-core atom motherboard/CPU, meant for server applications, compatible with Windows, *nix, BSD, you name it. That would be plenty powerful with 2 to 4GB of RAM.


    Personally I'm finding my Atom server a bit slow, though I do use it for MythTv recording and transcoding as well as file serving. I'm leaning towards replacing it with an i3 or i5, which should give similar idle power consumption and better performance when it's required.
    m
    0
    l
    July 7, 2010 6:24:22 PM

    i3 is nice, but expensive... new dualcore sempron or athlon II will be comparable at much smaller price.
    m
    0
    l
    July 7, 2010 6:34:35 PM

    skittle said:
    i3 is nice, but expensive... new dualcore sempron or athlon II will be comparable at much smaller price.


    Isn't the Athlon's idle power much higher than the i3 or i5? My server spends most of the time idle and just doesn't have enough CPU power when I need to run it at 100%.
    m
    0
    l
    July 7, 2010 6:45:34 PM

    dont know about athlon, but i3 is about the same as i5
    m
    0
    l
    July 7, 2010 11:06:29 PM

    MarkG said:
    Personally I'm finding my Atom server a bit slow, though I do use it for MythTv recording and transcoding as well as file serving. I'm leaning towards replacing it with an i3 or i5, which should give similar idle power consumption and better performance when it's required.


    Yeah, I realize that for what some people want they just don't quite cut it. I was just throwing it out there, trying to show you don't need to spend a lot to get reasonable results.

    I'd go AMD if you're trying to save money. Intel makes great stuff, but they just can't compete with AMD's prices. My server I'm working on is running an AMD Regor :p 
    m
    0
    l
    !