Should be interesting, eh? There's no dedicated video card because it'll primarily be a HPC rig. This motherboard is absolutely great: it would support up to 64GB of memory without a problem. It's basically what the Seaburg chipset, the one in the skulltrail platform, was designed for: this really is a server, and isn't a server chip pretending to be a cool gaming-oriented platform. Having said that, it's the same chipset, so this is basically another take on the Skulltrail platform.
I unpacked everything and proceeded to the build. I have a few pictures here that I'm about to upload and will post the links very shortly. It's already up and running. I'll also install an OS and try and run some benchmarks. Here are two:
The two OEM processors and one of the Thermalright HR-01-Xs. Apparently, the X5472 is more widespread as OEM version for system integrators. The guy I bought this from said he couldn't find boxed top-range Xeons anywhere. So no stock coolers, but... who cares? Honestly, against the HR-01-X?...
The motherboard, still inside protective bag. Quite a big motherboard: I've never seen one of this size. And certainly not with 16 memory slots!
So that's what we're talking about... system up and running. Wires are still a little messy: I did the cabling properly only for the HDD cables. The fan wires in particular are all over the place. I'll take care of that later. There are 7 active noctua fans in this system, plus another for the main intake, but the noctuas are not noisy. Cooling is great, because there's kind of a vertical wind tunnel going on here. Also, for those who were wondering, the Zalman retention bracket was my way to keep the memory cool. I'm still considering using the supplied rear fan (a black nine-bladed Silverstone FM121) for the retention bracket, just for the looks. The noctuas look great but have very strange colors. What do you think?
Here's another one. I haven't figured out what "93" (or is it "E6"?) means, but it certainly is cool. The system was running memtest while this picture was taken. Plus, here you can see that I haven't even zip tied the front panel cables (reset, power and so on).
Huh, I wonder if that's enough to run vista? These are the 16 DDR2-800 FB-DIMM modules with dedicated cooling by a Noctua NF-P12-1300 fan. Just imagine if they were 2GB modules or even 4GB modules! Really, all you need is money for that: the motherboard and technology for it is there.
The assembled system, closed. OK, so I still haven't bothered to hide the fan cables and still haven't zip tied the front panel cables, but come on, this already looks quite good! System is off here.
Two more from other angles. In this last one, you can see that I've done the cabling for the hard drives with zip ties in a more respectable fashion. I'll do the rest later.
This is what I ran first: a memtest86+. Doesn't recognize processor or chipset, but did you see that memory speed? (remember, the theoretical maximum bandwidth for quad-channel DDR2-800 is 25.6GB/s) I'll run something a little more beefy later on.
So, Mephistopheles walks into Office Depot carrying this monster
with his 2 arms, and he asks the Computer Department
salesman if this computer can do word processing.
The Computer Dept. salesman answers:
"I don't know; let me ask my boss."
(Kinda of a true story, in fact:
the motherboard had 4 CPU sockets
instead of 2, however.)
Sincerely yours,
/s/ Paul Andrew Mitchell, Inventor and
Webmaster, Supreme Law Library
http://www.supremelaw.org/
You might want to use a dedicated graphic card since onboard graphics uses the system RAM (even though it's only 32MB) the onboard will have a negative effect on RAM bandwidth. Just grab a cheap PCIe card like a 8400/8600.
Holy Sh** that is one f**king nice system, (swearing increases as jelousy does.)
obviously not gaming, but what is the main purpose of this sytem? caluculations? rendering? Server?
Shadow if it used as a server or as a workstation of calculation then there is really very little need for a GPU.
none of the servers i have come into contact with have dedicated graphics (although thats te wrong word, as on board graphics still has a GPU so doesnt strain the CPU) because there is just no need for them unles it has some graphical processing to do or if the MB doesnt haev built in graphics. there just more to go wrong.
very nice build and the total cost???
Message edited by dobby on 05-03-2008 at 12:13:27 AM
OK now: switch your thinking caps
into the realm of 64GB of FB-DIMM RAM.
Heck, relax all current IT restraints and
consider RAMs ranging from 64GB to 1TB
(they'll be here sooner than you can blink an eye!)
One of our recent U.S. patent applications
designates a user-defined region of RAM
(e.g. uppermost 10-12GB of RAM, enough for XP)
to host a Windows C: system partition,
and permits Windows Setup or GHOST restore
to load directly into this RAMDISK:
Now, your routine file system I/O e.g. swap file,
browser caches etc. all run at ~20GB/second (NOT MB/second).
See his Memtest86 results.
We got the idea from the RamDisk Plus
product sold by www.superspeed.com .
More details will be announced on the Internet
as soon as they are available.
Sincerely yours,
/s/ Paul Andrew Mitchell, Inventor and
Webmaster, Supreme Law Library
http://www.supremelaw.org/
Actually, shadow, I looked it up, and according to Tyan's website, the i5400PW uses a ZGI Z9S integrated PCI graphics system with 32MB DDR2 dedicated memory, so the video system doesn't access the main system memory.
While I was speccing this system, I found that it is common practice for server motherboards to include a few memory chips on their boards just for graphics. Only in the desktop arena do integrated graphics share memory, apparently!
Well that is pretty sweet . Awesome in fact...
On a side note, sorry to go slightly off topic - I know Xeon's and Opteron's are designed for server platforms but if they were put with a decent graphics card how do they compare to desktop gaming platforms? With 16gb it can't be that bad at gaming right?
Actually, shadow, I looked it up, and according to Tyan's website, the i5400PW uses a ZGI Z9S integrated PCI graphics system with 32MB DDR2 dedicated memory, so the video system doesn't access the main system memory.
While I was speccing this system, I found that it is common practice for server motherboards to include a few memory chips on their boards just for graphics. Only in the desktop arena do integrated graphics share memory, apparently!
Cool. I should have checked out the board first lol. Thanks for the correction.
> if they were put with a decent graphics card > how do they compare to desktop gaming platforms? > With 16GB it can't be that bad at gaming right?
Aside from the obvious importance of the graphics subsystem,
don't forget that file I/O can be moved to a RAMDISK using a product like RamDisk Plus from www.superspeed.com .
Not only have you thereby reduced the wear and tear on your rotating disk drives; by virtue of that transition you've also
accelerated routine file I/O to an astounding 20 Gigabytes per second,
as long as the file(s) in question are relocated to the RAMDISK!
NOT TOO SHABBY!
The very fastest hard drives are currently doing ~125 MB per second
sustained, e.g. 15,000 rpm 2.5" Seagate enterprise HDDs.
That's an average file I/O speed improvement of 20,000/125 = 160X !!!
Sincerely yours,
/s/ Paul Andrew Mitchell, Inventor and
Webmaster, Supreme Law Library
http://www.supremelaw.org/
supremelaw, funny story about the guy who asked if a 4-socket rig could do word processing. The only sad thing is that people who don't understand this will probably frown away from the system's price without acknowledging that this is, in fact, a ridiculously powerful number cruncher that is not like those cheap desktops you pick up at your local marketplace...
dobby: it'll primarily be used for processing custom physics code for an important professor at a local university. He and a bunch of students have parallel code up and running at it will really make a difference for them!
Also, as for price, we paid in brazilian reais, and we probably paid quite a respectable amount of import taxes, so it's not easy to compare with prices you'd pay in other countries. If you think about prices in the US, you're looking at ~US$2500-2600 for mobo+CPUs, ~US$1000 of memory, US$600 case and PSU, ~US$200 for hard drives, US$140 for 7 high-end noctua fans, ~US$110 for the HR-01-Xs, and 5 or 10 dollars for the Zalman retention bracket, which, come to think of it, was really a good idea for the FB-DIMMs. This adds up to a total cost of US$4550-4650, which, actually, isn't that expensive for such a ridiculous system.
For the sake of comparison, the top alienware system (the ALX), without monitor, costs US$5149 in its default configuration. It's a X38-based desktop with a quad-core penryn extreme-edition chip overclocked to 4Ghz equipped with a petty 2GB of DDR3-1333 and 320GB (2x160GB) hard drives, but it has a dedicated video card. OK, it uses liquid cooling and clocks higher, but it doesn't feature a 1.6Ghz FSB (or two) and is no match for the throughput of this workstation for HPC. Plus, it's more expensive! But the extra US$500-600 could be put to use in a dedicated video card. But, anyway, for HPC this is appropriately priced against ultra-high-end desktops.
Out of curiosity, we could have upgraded the memory to 32GB total system memory for an extra US$1000, and if you were really hell-bent on having your own private memory nirvana, you could get 64GB for an extra ~US$5200 (ouch)! We didn't have the money to do so, but the guy from the store even gave me some prices in brazilian reais for these higher-density modules (2GB and 4GB DDR2-800 FB-DIMM): he could have obtained them if we had the need/money!!!
HP's blackbird high-end desktop also doesn't match this and costs north of US$5K. Plus, hey, I got to build the damned thing and it was fun!!!
Too bad this computer won't also be used as a development workstation - there won't even be a monitor involved. If that was the case, you could plug in, say, a rather cost-effective 8800GT or something and a 24" widescreen LCD. Now that would be a perfect place to write, debug and run code.
Over the next few weeks I will try and run some benchmarks and stability tests. I have some n-threaded code that I'm eager to test on this server. I'm also interested in comparing the performance this system achieves to a few other reference systems. What I have available consists of one Q6600-based system, one C2D E6700 and one 16-CPU numalink-connected Itanium server (CPUs @ 1.5Ghz) to test. Personally, I think the Itaniums will get trounced, with this system besting a 16-way Itanium, but I'll tell you in a few days.
Sorry for not posting any more pics. The "über-rig" is not here with me at home, it's at the university. I'll post more pictures later. Also, I need to do the final wiring. But not today. Now I'm off to bed!
Message edited by Mephistopheles on 05-03-2008 at 04:09:55 AM
what OS are you gonna be running. most of the research PC i have come into contaact with have run Win Server (03 mainly) so there program can make use of the .net libary make coding easy as (i love MS codes, C# FTW)]
or use Solaris for real serius stuff.
i assume it gonna have to be an x64 OS to make use of that ram, but is your program gonna be x64?
what OS are you gonna be running. most of the research PC i have come into contaact with have run Win Server (03 mainly) so there program can make use of the .net libary make coding easy as (i love MS codes, C# FTW)]
or use Solaris for real serius stuff.
i assume it gonna have to be an x64 OS to make use of that ram, but is your program gonna be x64?
thank for posting BTW. appreichate it
I wonder what kind of number crunching you do if you do it mostly on Windows machines. From what I've seen, my guess is you're running fairly short-run (less than a day's runtime) jobs that have small (<2 GB) working sets on 32-bit Windows. I also will guess that you are running some pretty high-dollar proprietary software on some of the machines (e.g. MATLAB) and you're running them on machines that are basically desktops with all of the RAM slots filled with maybe a few dual-socket workstations thrown in there. That basically describes the only Windows number crunchers I've ever seen, and it wasn't all that many. Almost all of the number-crunchers I've come across have been big headless machines running a Unixy OS such as Linux, BSD, or Solaris and you used ssh to log into them remotely. The Windows machines ran the high-dollar proprietary software programs that worked on problems that took usually at most an hour to crunch whereas the custom-written stuff that required a lot of memory and runtime ran on the Unixy hardware. I am sure a few guys do code HPC stuff in C# but everything I've seen has been FORTRAN, C, or C++ and little of it has any sort of a GUI- the most you would do is have the number cruncher spit out a table or other results file that you'd open and look at on your desktop or laptop and that would be the GUI.
Also can't you use CUDA for MATLAB? ( know you might have to learn some new things, but the raw crunching power of a 8800GTX or the like are awesome). I remember there was a thread that some guy was trying to use CUDA and 2*9800GTX2s for number crunching.
.
