Home-Built Supercomputer for Scientific Computing

Hello all!

I am an Aerospace Engineer and looking to build a homemade "super computer", for lack of a better name. The computer's sole purpose will be to perform computational fluid dynamics (CFD). As you may know, CFD is very demanding when it comes to cpu performance. I would like the system to run a linux os. It will primarily running FORTRAN 90, C, and C++. It will also be running MATLAB, but in a less demanding manor. Along with fortran and C++, GAMBIT will be used for CFD grid generation and Fluent will be used for its cfd codes.
Complex cfd programs can produce data files in the terabytes range, although I dont plan on getting that in depth. Speed is primary concern. What can I build for under $10k USD?

Thanks Guys

how about a VX50? I know it's E-bay....but this guy has some real high-end stuff with warranty. Check out the double wide chassis S4985 he's got listed if you need alot of drive space too. About as close as you'll get to a "personal supercomputer".

http://shop.ebay.com/merchant/mars [...] omZQQ_mdoZ

If you have your heart set on building one yourself, you can always pick-up a barebones VX50 or the "cheaper" FT48. Don't know if this is the type of thing you have in mind, or if you're looking for something a little more traditional.

I actually don't know anything about that type of software I've used tyan stuff for years for CAD. I'm actually in the process of building a FT48 (16-core) right now as a virtualization/CAD rig. I do know that the advertised primary use for these type of boxes are CFD and they definately will run Linux. From what I do understand, this type of work usually requires a good number of cores, a large amount of RAM, and some fast HDD's or some sort of clustering set-up using infiniband networking. I did some quick googling, and apparently results can drastically differ with slightly different types of calculation depending on how you set it up to run. Most of the info I've found is waaayyyy over my head. Sorry I can't be of more help....

I would start with a careful inventory of all production software
you intend to run on a regular basis.

Let me give you a good example from our daily workload:
after we installed COPERNIC desktop search software
on a quad-core Q6600 workstation, we compared it
head-to-head with a dual-core D 945 in a similar workstation.

The Q6600 finishes just about twice as fast as the D 945
updating the very same 5GB database.

That can only be the result of parallel programming in COPERNIC.

If your software is not coded to exploit multiple cores,
you're wasting your time bulking up on multi-socket systems.
Those are intended chiefly for busy servers with lots of
multi-tasking to process.

After you've completed your analysis of your production software,
the real choice for you is between a single-socket Core i7 machine,
or a twin-socket Xeon with Core i7 architecture.

A single-socket Core i7 machine has 4 CPU cores with hyperthreading:
each socket is THUS capable of running 8 threads simultaneously.

That should be enough for any software you want to throw at it,
particularly if you are only running one copy of your fluid dynamics model
at any given point in time.

Next decision is the clock speed of the Core i7: go for the fastest
because you've got plenty of budget for it, and it was just reported
on the Internet that all Intel Core i7 CPUs are "unlocked" --
meaning they can all be overclocked.

So, buy a motherboard that makes overclocking easy, e.g. ASUS P6T.

As for RAM, X58 chipsets now support either 12 or 24GB of DDR3 RAM:
look into Corsair's high-end Dominator series, with the memory module cooler.
Kingston make engineering samples of 4GB DIMMs x 6 = 24GB total e.g.:

http://www.hexus.net/content/item.php?item=17187


The Corsair power supplies w/ 850 Watts and up should be enough:
they are very highly regarded for high-end workstations like the
one you want to build:

http://www.newegg.com/Product/Prod [...] 6817139009

I'll leave your graphics hardware choices up to your own good research.

If you are really serious about writing very large data files,
then be sure to give serious consideration to motherboards
and/or RAID controllers that can pump data quickly to and from
SAS (Serial Attached SCSI) hard drives: these come in both
10,000 and 15,000 rpm e.g. Seagate.

Despite what lots of amateurs claim, in ignorance,
a RAID 0 with multiple (4 x or 8 x) HDDs can really move a lot of raw data
very fast: 250-300MB/second is quite easy, and 500MB/second
is within reach, if you know what to buy and how to configure
your storage subsystem. The key is choosing a RAID controller
that does parity computations in hardware, e.g. Areca, 3Ware
or Highpoint's "enterprise" class controllers (there are others).

Out on the "bleeding" edge you will find Fusion-io's ioDrive Duo
(Google that one), or OCZ's new Z-drive (actually just a
Highpoint RocketRAID with 4 x OCZ MLC SSDs inside the
plastic shell).

Lastly, we've had enormous success with ramdisks
by installing RamDisk Plus from SuperSpeed, LLC
in Sudbury, Massachusetts: www.superspeed.com

A little bit of intelligent memory management, tailored
to your application software, can pay enormous dividends:
Core i7 memory bandwidth BEGINS at 25,000 MB/second
(25GB/second), and goes up from there when triple-channel
DDR3 is overclocked.


This is THE hottest computer hardware available
at the present time for workstation-class machines.
AMD still can't even come close.


hope this helps


MRFS

Six Core i7 platforms compared here at Tom's Hardware:

http://www.tomshardware.com/review [...] ,2164.html


(sorry for the repeat posting)


MRFS

I first began working with computers during the Summer
before my first year of grad school at U.C. Irvine in 1971.

I've been using and developing advanced computer systems
ever since then, now 38+ years.

I build about one new workstation every year or so,
with the "trickle-downs" going to friends and neighbors.

I have also submitted 3 patent applications to the U.S.
Patent Office, with a fourth ready to submit, in the area
of high-speed solid-state data storage subsystems.

(I've also been banned by certain websites merely
for asking what "VISTA" stands for: nevertheless,
witness how fast MS is now pushing Windows 7.)


MRFS (= Memory Resident File Systems)

http://www.tomshardware.com/news/T [...] ,5672.html

[begin quote]

Besides performance improvements, the T10P also delivers 64-bit or double-precision capability, which is required for most fluid dynamics and financial stream processing applications. Double precision is substantially more intensive than single precision calculations and with decrease the performance of the card dramatically. Nvidia told us that double-precision calculations will result in a 90% speed penalty and deliver only 100 GFlops per T10P processor.

[end quote]


Also:

http://www.tomshardware.com/review [...] ,1954.html


MRFS

http://news.softpedia.com/news/Int [...] 8425.shtml

http://www.techradar.com/news/comp [...] ked-487131


Just to confirm that our eyes did not deceive us, we also gave Nehalem EP a quick going over with the Stars Euler3D benchmark. It's a computational fluid dynamics simulation that majors on floating point performance.

Sure enough, Nehalem EP roasts all comers in this benchmark, too – it's twice as quick as a pair of 2.7GHz Shanghai processors (14.34 seconds to complete five instances versus 30.32 seconds).

[end quote]


MRFS

http://www.crn.com/white-box/215900275


Intel To Launch Nehalem Server Chips March 30

By Damon Poeter, ChannelWeb
8:08 PM EDT Fri. Mar. 13, 2009

Intel (NSDQ:INTC) is prepping its channel for the wide release of its Nehalem-class Xeon server microprocessors and platforms to whitebox partners on March 30, Channelweb.com has learned.

Code-named Gainestown, the first quad-core Xeon parts featuring the company's next-generation Nehalem microarchitecture hit the market in early March with the launch of Apple's new Mac Pro workstations.

[end quote]


MRFS

"Nehalem EP" is the code phrase for multiple Core i7 CPUs
running in the same motherboard e.g. dual-socket
(e.g. see photos above of the ASUS MP version).

"MP" means multi-processor (i.e. multiple CPUs).

As far as I know, the instruction sets are the same,
with extensions.

You may need to re-compile, but the languages you
mentioned fully support the x86 instruction set by now.

AMD's 64-bit implementation was designed from the outset
natively to support the 32-bit instructions of that x86 set.


What machine(s) are your fluid dynamics codes
running on presently?


I used to do a LOT of FORTRAN conversions (long time ago):
the main problems resulted from system-specific
SUBROUTINE calls, and different compiler defaults
e.g. INTEGER defaulted to *2 on some machines
and *4 on other machines.

In your case, you should look into the
defaults for DOUBLE PRECISION:
is the default REAL*8 or REAL*16?

You may need to hard-code REAL*8 to achieve maximum performance,
because REAL*16 may cause a severe performance penalty.


Do you have any colleagues currently running Core i7 machines?


From what I've gleaned after a lot of reading,
ASUS and Gigabyte are the only games in town
for a serious single-socket Core i7 machine.

"Core i7" is synonymous with "Nehalem".

With your budget, I would strongly suggest that
you start with a modest "test" machine, e.g.:

http://enthusiast.hardocp.com/news [...] FzdCwsLDE=

http://www.asus.com/Product.aspx?P_ID=6i86Hj0lGriFHfY9

Then, when your codes are running, you can either demote
this mATX motherboard to the role of a backup server, via a Gigabit switch;
or, cannibalize the parts for your dream workstation e.g. dual-socket.
The single-socket Core i7 CPU will need to stay in the mATX motherboard, however.

SDRAM vendors are competing fiercely right now for this Core i7 market,
and you wouldn't need to buy THE fastest DDR3 for such a "test machine".
Start with 6GB of triple-channel RAM, 1 x VelociRaptor for your OS,
and 1 x 1TB SATA/3G hard drive for data storage e.g. Western Digital.


Dual-socket Core i7 machines are now or will very soon
be available from ASUS, Tyan, Supermicro and Intel.
I don't know enough about these MP motherboards
to make a recommendation to you, however.


If you like, I can put you in touch with the ASUS
Marketing Manager for North America. Let me know.


MRFS

The Intel Core i7 CPU aka "Nehalem" is a quad-core chip
with hyperthreading, that installs into a single LGA1366 socket.

Dual-socket motherboards are being called Nehalem EP.

I think you should start out with a single socket motherboard
like the ones built by ASUS or Gigabyte (see above).


MRFS

SuperCruise, I am also a CUDA enthusiast (bought a 8-series video-card only to see how can CUDA practically work), but keep in mind that what you are talking about is theoretical computational power. Not only his software needs to be ported on CUDA, but also the algorithms have to be re-designed in order to use effectively the large amount of cores. Also you need to understand the CUDA constrains about synchronization, memory access, etc. The incentive of this sustained effort is the tremendous speed improvement to benefit of.
bgood44, what you have right now is a very mobile computer. Its performance is measured in pounds and battery life when idling, it was clearly not designed for extensive computational work. If you purchase a decent price i7 processor, with 4-8 GB of RAM and a regular 1T B hard drive, your will be on a budget less than $800 and you will observe a significant improvement in performance, I would say x5 in computations and x2 in data storage. Later on, when you have very clear in mind what algorithm do you have to implement, how many independent session you have to support, how long is a session supposed to last, you can decide between a large computer with 2 or even 4 processors, a CUDA approach based on NVidia hardware, a cluster of inexpensive one processor nodes or a mixture of these 3.

NVidiA Tesla HPC cards do 1 Teraflop of single precision floating point operations per second and 280 GFLOPS of double precision operations per second. They are completely scalable for many Teraflops just buy using additional cards. The cards are about $1399 at Tiger Direct and fit in any PCIe 16 X slot. They do however, require lots of system RAM. They use CUDA and are well suited to fluid dynamics. They are not a video card, strictly a massive number crunching machine.

like others earlier I strongly recommend looking at CUDA. CUDA uses

Start here: http://www.nvidia.com/object/compu [...] amics.html

Then check your software to see what is supported.

If you care about your results, consider waiting for the next gen from nVidia codenamed “Fermi”. It actually has error checking so you won't need to re-run your code a few times to see if you got the right results or if a sunspot flipped a bit on you. (RAS of current accelerators is mostly missing because they are built from graphics cards that don't care about data integrity)

Suggest: Append to CUDA forums describing your application needs in CFD. Ask for config help, verify software works as advertised, check benchmarks.

Suggest: Start with a 'gamer' PC of your choice. Specify a high end nVidia graphic card. Cost will be $1K to $2K total from Dell or some specialty house. Get the software you need working with CUDA. If the performance is not there then you can consider higher end solutions. Don't throw $20K at this problem until you understand the ups and downs of the $1-2K system.

REMEMBER I can build a 10TFLOP machine with the new supermicro board. Screamers here...

LEAN GREEN COMPUTING POWER... mark

for a real personal super computer under 10k USD try nvidia tesla super computer its really amazing

Get your own supercomputer. Experience cluster level computing performance-up to 250 times faster than standard PCs and workstations- right on your desk. The NVIDIA® Tesla™ Personal Supercomputer is powered by up to 960 parallel processing cores and 16 GB of dedicated compute memory to solve large datasets. Based on the revolutionary NVIDIA® CUDA™ parallel computing architecture, the Tesla Personal Supercomputer puts nearly 4 teraflops of computing muscle at your fingertips.
Up to 960 Core Parallel Supercomputer on Your Desk, 250 Times Faster than a PC.
Up to 4 teraflops of compute capability for under $10,000.
Computation comparable to a large server cluster that fills a room.
250 times faster than a multi-CPU core PC or workstation using traditional serial computing.
GPU parallelism designed to execute tens of thousands of concurrent threads.
Up to16 GB of dedicated compute memory meets the computing requirements of large datasets.
Single- and double-precision floating point.
Scalable architecture meets computational demands of complex scientific applications.
Based on Revolutionary NVIDIA® CUDA® Parallel Computing Architecture.
Unlocks the power of GPU parallel computing.
Over 25,000 application developers worldwide using CUDA.
Over 100,000,000 CUDA-enabled processors shipped.
Accessible to Everyone
Supercomputing available to anyone, anytime.
Iterate faster; discover more.
Computing whenever you need it. No need to ask permission to an administrator.
Buy it anywhere, put it anywhere.
Plugs into standard power strip.

http://www.connoiseur.com/NVIDIA_T [...] -list.aspx

My honest and sincere advice is that if you are a recent graduate and want a computer to do CFD at home, just buy a good PC, don't worry about clusters, GPUs and servers.... I suppose you are not intending to model the airflow in a commercial plane with a detailed geometry and under project time pressures, right? Everybody waits for CFD results.