Best configuration for this high performance task

Toggle sidebar Toggle sidebar
R

river251

Distinguished
Sep 13, 2011
25
0
18,530
Hi, my first post....first, I'm looking for a place appropriate for this, so if this is not it, please pardon me. Any pointers to an online forum for high performance computing would be more appreciated.

I have a Dell Tower with 64 gigs of ram, 12 processors, a 250gb SST, and 2 2tb hard drives in RAID 0.

I run matlab, to analyze EEG (brainwave) data sets, about 4gb each. When I run an independent component analysis (ICA), all the cores are engaged, and the ram is about half full, taking about a day to complete. That is for one subject. I also do other analyses which are not multi-threaded, so don't use all 12 processors, but operate on 15 4gb datasets (all 15 subjects in an experiment) at once. The toolbox I use is designed to pull data from the hard drive as needed for this, since few machines will have RAM for 15 4gb datasets. Also, as any of these analyses are done, equally large output files are created.


I have windows 7 and Matlab installed on the SST. I am thinking of reconfiguring the system with Windows 7 and Matlab on the RAID to leave the SST free for the EEG data. My thinking is that the ICA and the 15-subject analyses involve a lot of back and forth between the drive and the CPUs, and to optimize performance here, I need to move the OS and applications off of the SST. At the moment it's pretty full with the OS and applications, leaving not much room for the SST to be used in analyses. I hope having the data on the SST will cause the SST to be the main place where data is read from and written to during the analyses.

Thanks much,

Jim Kroger

NMSU


 
Solution
hang-the-9
I have worked with EEG devices before, although specifically the system I worked with was over the network real time data.

If you have anti-virus program setup on the computer, put in an exception for it to not scan any directories the data files are in or get created in. This actually caused data issues on the system I worked on.

You can check on the drive usage also using device manager. Did you use anything like performance manager to gather long term drive, ram, cpu usage or just ram and cpu?

Since you have such a high end system, you should get a second large solid state drive for it fully, not use the other drives in RAID.

I'm sure the few $100 spent on the hard drive would be easily approved.
Sort by date Sort by votes
hang-the-9

hang-the-9

Titan
Mar 25, 2010
60,387
1,081
152,940
I have worked with EEG devices before, although specifically the system I worked with was over the network real time data.

If you have anti-virus program setup on the computer, put in an exception for it to not scan any directories the data files are in or get created in. This actually caused data issues on the system I worked on.

You can check on the drive usage also using device manager. Did you use anything like performance manager to gather long term drive, ram, cpu usage or just ram and cpu?

Since you have such a high end system, you should get a second large solid state drive for it fully, not use the other drives in RAID.

I'm sure the few $100 spent on the hard drive would be easily approved.
 
Upvote 0 Downvote
Solution
R

river251

Distinguished
Sep 13, 2011
25
0
18,530


Thanks much. Can you elaborate on a couple points:
How would I use device manager to check drive usage? What would I be looking for?
How would I use performance manager? Is that in windows?
Also, I know another SST would be faster than the RAID, but the RAID is 4tb, it's my strorage for everything. My thought was to put the data I am analyzing on the SST and not involve the RAID in the analysis. Not sure if that's possible though.
Thank you very much.

 
Upvote 0 Downvote
R

river251

Distinguished
Sep 13, 2011
25
0
18,530


Hi, thanks. Is the idea that if the computations are happening in RAM for ICA (or anything else) and there is not much read/write, then the SST vs RAID doesn't matter much, but if there is a lot, than the SST will be much faster execution-wise?

Thanks
 
Upvote 0 Downvote
R

river251

Distinguished
Sep 13, 2011
25
0
18,530


Thanks. I agree that switching to all SSTs will eliminate the concerns. And I do have another slot for another SST. But forgive me if I am not understanding, but it doesn't seem to change the question. I can move the OS to the RAID array, and hope to find a way to make the computer use the SST for computation read/writes (maybe by putting the data on the SST).

Unless I misunderstand, having the OS on the RAID won't slow things down (not sure though), so why not just put the OS on the RAID, freeing up the SST for the computations? Then why would I need a second SST?

Thank you!
Jim
 
Upvote 0 Downvote
USAFRet

USAFRet

Titan
Moderator
Mar 16, 2013
171,455
17,911
184,590


The SSD are faster (by a lot) than the RAID 0 HDD.
Leave the OS and applications on the current SSD drive
Add another SSD of whatever size fits in the budget.
Use this for your working files.
When done, then move that file off to HDD for long term storage.

 
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom