How do I write large (250 GB) files continuously to an SSD without slowing down?

loganboy427 · Mar 25, 2014

I am doing scientific imaging and need to write very large, consecutive images to disk.

The camera I have is a 16 bit camera. Images from this camera are large (11,000Kb per frame). I want to record 60,000 images(frames) at a rate of 50 frames per second which equals 550 Mb/s of data writing to the disk. This needs to be done without interruption for the entire 60,000 frames which should take 20 minutes to capture all these images at the required frame rate.

I have upgraded to an SSD and am able to accomplish my goal but only for the first 3,000 frames. After 3,000 frames have been taken the PC slows down and the rate at which images are taken decreases to about 4 Frames/sec for roughly 1,000 images then it speeds back up for a while. This varying frame rate continues for the duration of the 60,000 images.

Unfortunately, this lag produces non-equivalent time between all my images.

I have plenty of storage space on my SSD, SATA III connection to the Motherboard.

Is there a way to continuously write these images to the Drive without having this lag?

Will purchasing a PCIe SSD solve this issue?

laviniuc · Mar 25, 2014

it slows down because the ssd garbage collector kicks in when it runs out of free nand cells. what's the ssd model? you can probably get away with a good quality 500gb-ish ssd which is way cheaper compared to the pci ex ssds.

try to run a ssd optimization before starting it should help. also install the os to another hdd and leave the ssd only for this. at the very least set the windows page file to another drive.

edit: not many ssd-s can sustain 550mb/s, certainly not for the whole capacity, any chance you could get to somewhere around 300-400mb/s? your best bet is a samsung 840 pro or sandink extreme II or intel 730 but at least 500gb in size and these can cost a bit...

Gee Bee · Mar 25, 2014

whats the ssd?

Someone Somewhere · Mar 25, 2014

Firstly, do an ATA Secure Erase on the drive each time before you write the file. This will ensure that garbage collection doesn't get in the way.

You might want to investigate getting a second SSD and doing RAID 0 or having it write the images to alternating SSDs.

loganboy427 · Mar 25, 2014

Thanks. The SSD is a Samsung 840 Pro.

I can cut down the frame rate but my experiment is critical to running at the full 50 FPS.

I will add a HDD and install the OS on that if you think it will help.

I guess I need to read up on the "Garbage Collector" as I am not familiar with it. I assume there are setting I can change this?

laviniuc · Mar 25, 2014

no, no setting, it's internal to the ssd. what's the size of the ssd? 256gb?
you can do a quick optimization also on the samsung magician

edit: the secure erase suggested above is better. it also deletes everything on the drive so be weary of that

Gee Bee · Mar 25, 2014

What size is the 840 pro, this can make a huge difference in write times. Especially since they are dropping off quite quickly after 3000 frames.

loganboy427 · Mar 25, 2014

500GB

loganboy427 · Mar 25, 2014

I do notice the write times one this SSD are only 520Mb/s which should still allow for 40FPS.

I found this: http://www.maketecheasier.com/12-things-you-must-do-when-running-a-solid-state-drive-in-windows-7/

Useful or rubbish for what I want to do?

Someone Somewhere · Mar 25, 2014

Number one only, if your OS is on another drive.

laviniuc · Mar 25, 2014

ok, for the 500gb version you should be able to make it work.

1.first try:
-free up as much space as possible on the ssd (75%+)
-move the windows page file to another drive (or disable it completely if you have enough ram)
-disable rapid mode
-use the ssd performance optimization quick cleanup on samsung magician
-try again

2.second try:
-install windows completely to another drive
-boot from there, do a full secure erase on the ssd
-try again

let us know how it works on both cases if the first one doesnt cut it...

menetlaus · Mar 25, 2014

http://www.anandtech.com/show/2829/9

that article is a few years old (thus most of the issues presented have solutions) - but it has some of the earliest explanations about the issue you are seeing. TRIM is the general low-level command that the garbage collection utilities use to "clean" an SSD.

loganboy427 · Mar 27, 2014

Thanks for all the assistance. I ended up getting the write speeds that I need by turning off write cache buffer dump in windows device manager for that drive. I'm able to get the max write speed from the ssd.
Since I am taking long segments of scientific data that represents a "repeatable experiment" I am not worried about a sudden loss of power. Plus it writes directly to disk at such a quick rate there is no real need that I see for the cache to be useful.

Gee Bee · Mar 27, 2014

So simple, duh...nice one, happy 'repeatable experiments'

Search

How do I write large (250 GB) files continuously to an SSD without slowing down?

loganboy427

Honorable

loganboy427

laviniuc

Judicious

Gee Bee

Honorable

Someone Somewhere

Titan

loganboy427

Honorable

laviniuc

Judicious

Gee Bee

Honorable

loganboy427

Honorable

loganboy427

Honorable

Someone Somewhere

Titan

laviniuc

Judicious

menetlaus

Distinguished

loganboy427

Honorable

Gee Bee

Honorable

TRENDING THREADS

Latest posts

Moderators online

Share this page