Sign in with
Sign up | Sign in
Your question
Solved

How do I write large (250 GB) files continuously to an SSD without slowing down?

Last response: in Storage
Share
March 25, 2014 7:38:44 AM

I am doing scientific imaging and need to write very large, consecutive images to disk.

The camera I have is a 16 bit camera. Images from this camera are large (11,000Kb per frame). I want to record 60,000 images(frames) at a rate of 50 frames per second which equals 550 Mb/s of data writing to the disk. This needs to be done without interruption for the entire 60,000 frames which should take 20 minutes to capture all these images at the required frame rate.

I have upgraded to an SSD and am able to accomplish my goal but only for the first 3,000 frames. After 3,000 frames have been taken the PC slows down and the rate at which images are taken decreases to about 4 Frames/sec for roughly 1,000 images then it speeds back up for a while. This varying frame rate continues for the duration of the 60,000 images.

Unfortunately, this lag produces non-equivalent time between all my images.

I have plenty of storage space on my SSD, SATA III connection to the Motherboard.

Is there a way to continuously write these images to the Drive without having this lag?

Will purchasing a PCIe SSD solve this issue?
a b G Storage
March 25, 2014 7:44:40 AM

it slows down because the ssd garbage collector kicks in when it runs out of free nand cells. what's the ssd model? you can probably get away with a good quality 500gb-ish ssd which is way cheaper compared to the pci ex ssds.

try to run a ssd optimization before starting it should help. also install the os to another hdd and leave the ssd only for this. at the very least set the windows page file to another drive.

edit: not many ssd-s can sustain 550mb/s, certainly not for the whole capacity, any chance you could get to somewhere around 300-400mb/s? your best bet is a samsung 840 pro or sandink extreme II or intel 730 but at least 500gb in size and these can cost a bit...
m
1
l
March 25, 2014 7:45:49 AM

whats the ssd?

m
0
l
Related resources
a c 115 G Storage
March 25, 2014 8:01:21 AM

Firstly, do an ATA Secure Erase on the drive each time before you write the file. This will ensure that garbage collection doesn't get in the way.

You might want to investigate getting a second SSD and doing RAID 0 or having it write the images to alternating SSDs.
m
0
l
March 25, 2014 8:14:58 AM

Thanks. The SSD is a Samsung 840 Pro.

I can cut down the frame rate but my experiment is critical to running at the full 50 FPS.

I will add a HDD and install the OS on that if you think it will help.

I guess I need to read up on the "Garbage Collector" as I am not familiar with it. I assume there are setting I can change this?
m
0
l
a b G Storage
March 25, 2014 8:17:56 AM

no, no setting, it's internal to the ssd. what's the size of the ssd? 256gb?
you can do a quick optimization also on the samsung magician

edit: the secure erase suggested above is better. it also deletes everything on the drive so be weary of that
m
0
l
March 25, 2014 8:20:30 AM

What size is the 840 pro, this can make a huge difference in write times. Especially since they are dropping off quite quickly after 3000 frames.
m
0
l
March 25, 2014 8:32:43 AM

500GB
m
0
l
a c 115 G Storage
March 25, 2014 8:38:31 AM

Number one only, if your OS is on another drive.
m
0
l
a b G Storage
March 25, 2014 8:39:09 AM

ok, for the 500gb version you should be able to make it work.

1.first try:
-free up as much space as possible on the ssd (75%+)
-move the windows page file to another drive (or disable it completely if you have enough ram)
-disable rapid mode
-use the ssd performance optimization quick cleanup on samsung magician
-try again

2.second try:
-install windows completely to another drive
-boot from there, do a full secure erase on the ssd
-try again

let us know how it works on both cases if the first one doesnt cut it...
m
0
l
March 25, 2014 8:41:45 AM

http://www.anandtech.com/show/2829/9

that article is a few years old (thus most of the issues presented have solutions) - but it has some of the earliest explanations about the issue you are seeing. TRIM is the general low-level command that the garbage collection utilities use to "clean" an SSD.
m
0
l

Best solution

March 27, 2014 5:53:16 AM

Thanks for all the assistance. I ended up getting the write speeds that I need by turning off write cache buffer dump in windows device manager for that drive. I'm able to get the max write speed from the ssd.
Since I am taking long segments of scientific data that represents a "repeatable experiment" I am not worried about a sudden loss of power. Plus it writes directly to disk at such a quick rate there is no real need that I see for the cache to be useful.
Share
March 27, 2014 7:04:32 AM

So simple, duh...nice one, happy 'repeatable experiments'
m
0
l
!