SATA disk chokes performance. Looking for improvements.

Lentebriesje · May 28, 2017

Hi everyone,

I use a tool called World Machine. It's rendering software to generate terrains and it requires massive amounts of memory.

This is my system:

After a minute of running the RAM memory will be filled up after which the program will use "disk swapping" to save all the files. So it will temporarily store data it doesn't need at that moment to an assigned disk and only keep files it uses to generate in the RAM memory. It will just recall files it needs later on and store other stuff.

After my RAM memory is filled up the CPU will generally work on 1~6%, the RAM memory will be 90~99% full and disk write averages probably at 98%.

This leads me to believe that the SATA disk i write temporary files to chokes the performance of my system. (One of the Seagate ST2000DM's)

is there anything i could try to improve disk saving speed, or would you suggest buying additional hardware? Or maybe there's not much efficiency to be gained?

Thanks in advance for any help!

ko888 · May 28, 2017

How much of the Samsung 840 EVO's capacity is used or is still available?

Which drive are you reading from (i.e. source drive)?

Which drive are you writing to (i.e. destination drive)?

Which drive are you using as the temporary swap drive?

Preferably, the temporary swap drive should be on an SSD.

Lentebriesje · May 28, 2017

@ko888
-How much of the Samsung 840 EVO's capacity is used or is still available?
Not enough space for what i generally create in terms of temporary swap files

- Which drive are you reading from (i.e. source drive)?
D drive (Seagate ST2000DM)
Only swap reading. There are no source files. It's generated from thin air.

- Which drive are you writing to (i.e. destination drive)?
E drive (Seagate ST2000DM (the other))
Permanent saving only happens at the very end.

- Which drive are you using as the temporary swap drive?
D drive (Seagate ST2000DM)

- Preferably, the temporary swap drive should be on an SSD.
The HDD saves 90~100MB/s. How much could i get realistically with a SSD do you think?

USAFRet · May 28, 2017

Question:

What size worlds and complexity are you creating?
I just downloaded the free Basic version, just to play with it.

I have 32GB RAM, and all SSD drives.
I could run a test and see what happens with SSD vs your HDD.

dudeman509 · May 28, 2017

Sounds like you'd want to move the swap file to a SSD. You could even pick up a cheap 120G drive and dedicate it to that and temporary files from that program alone. Platter drives are terrible at swap files and operations that require random I/O.

USAFRet · May 28, 2017

dudeman509 :

That is precisely what my old 120GB Kingston drive is doing.
Scratch/temp space for video and image editing applications.

Lentebriesje · May 28, 2017

@USAFRet
I'm not sure that it's worth going into the nitty gritty of the software. But for example earlier today i generated a world export measuring 6144*6144 pixels with about 30 outputs. Which isn't much in terms of size, but i generated 1650 heighfields which is nearly all stored to swap because I don't use macro's. Macro's are more efficient when saving but the used compression is unacceptable for my usage.

What you could do is create device chains, as many chains as you have threads to render. use combiners to all link them up in the end so they will be generated simultaneously. If the chains are long enough to fill your RAM and spill into swap memory you should be able to see how the swap saving performs. I think the basic free version is limited to 1024 pixel renders, so you need long chains.

Edit: i gather i should just get a SSD. If i'm not mistaken there are SSDs that are more wear resistant than others? or do you think i shouldnt concern myself with that? I think i'll need 256GB SSD for the swap disk. Suggestions?

USAFRet · May 28, 2017

Yes, an SSD would work well here.
120 or 250GB.

ko888 · May 28, 2017

Stephen Schmitt, the creator of World Machine, says in answer to a user's question "SSD setup - does it help?":

Funny you ask this..

Short version: absolutely. Just make sure your temp folder is on the SSD path and you're set.

Long version:

World Machine does its own paging to keep memory use under the level that causes the "thrashing of death" that typically occurs once you hit memory exhaustion. One of the things I"m working on right now in the next WM 2.3 beta is fixing a couple paging bugs that lead to instability. In the process of doing some testing, I've setup my temp folder to a solid state drive I have here, and it is a remarkable thing -- it nicely expands the highest practical resolution that you can work with. The only limitation is that you still need to have enough memory to let a device load in all its inputs at once, which for some devices is the main limiting factor.

Have you monitored what size the largest swap file has reached? SLC NAND flash SSDs are most suited for this use. You can also use larger TLC and MLC NAND flash SSDs where some of its capacity is set aside (a.k.a. over-provisioning) for wear leveling.

Lentebriesje · May 28, 2017

ko888 :

I havent monitored todays packet file sizes. Based on a few lingering packets they seem to be between 3GB and 65MB.

Search

SATA disk chokes performance. Looking for improvements.

Lentebriesje

Reputable

ko888

Titan

Lentebriesje

Reputable

USAFRet

Titan

dudeman509

Dignified

USAFRet

Titan

Lentebriesje

Reputable

USAFRet

Titan

ko888

Titan

Lentebriesje

Reputable

TRENDING THREADS

Latest posts

Moderators online

Share this page