For work i need to do matrix operations on really really huge data sets. 30 Gb and more.
We have some budged, but not that much what would be really required.
If we put all our ram in one single basket, we can have a 64gb machine (667 MHz ddr2 fb ram). And even that is still not enough, (there is additinal memory required for the intermediate results and opperations) so our programs still have to scratch on the HD. which slows things down of course.
Now the idea was to use some SSD's in raid or a revo drive as a huge scratch disk (like putting the windows page file on the complete drive). In order to minimize the HD bottleneck.
Now my question is what would be the best putting 2 120gb drives in raid 0 or buying a revodrive 3 or 3x2 if lucky on ebay. (limited to sata2 and pcie 2.0)
My googling comes up that it all depends on the use. But how can i monitor my use?
I mean if i had both disk i could run my problem as a benchmark. but i dont have the disks and i still like to know to wich benchmarks my problem would compare the best.
How do i know how the programs are writting to the disk? are they doing that i small 4k blocks or huge 500 mb blocks? is the program regulating that, or is windows/linux regulating that? can i change such settins some where? does there exist a kind of monitor software such that i can run it together with my problem, and then that it tells some statistics about how the hardisk has been used?
Any suggestions are welcome! thanks in advance.
We have some budged, but not that much what would be really required.
If we put all our ram in one single basket, we can have a 64gb machine (667 MHz ddr2 fb ram). And even that is still not enough, (there is additinal memory required for the intermediate results and opperations) so our programs still have to scratch on the HD. which slows things down of course.
Now the idea was to use some SSD's in raid or a revo drive as a huge scratch disk (like putting the windows page file on the complete drive). In order to minimize the HD bottleneck.
Now my question is what would be the best putting 2 120gb drives in raid 0 or buying a revodrive 3 or 3x2 if lucky on ebay. (limited to sata2 and pcie 2.0)
My googling comes up that it all depends on the use. But how can i monitor my use?
I mean if i had both disk i could run my problem as a benchmark. but i dont have the disks and i still like to know to wich benchmarks my problem would compare the best.
How do i know how the programs are writting to the disk? are they doing that i small 4k blocks or huge 500 mb blocks? is the program regulating that, or is windows/linux regulating that? can i change such settins some where? does there exist a kind of monitor software such that i can run it together with my problem, and then that it tells some statistics about how the hardisk has been used?
Any suggestions are welcome! thanks in advance.