limit on number of files copied at once?

[dirtbagjack]

I'm not sure if this is a windows problem or a hardware problem so we'll start in the windows forum and see....

I've got a few million small files in folders and sub-folders and even more sub-folders that I want to copy to a external HD. Somewhere around 500,000 to 600,000 files, windows won't copy anymore and gives me errors. Restarting the computer solves the problem and lets me continue copying (once I find out where things left off). Currently I'm processing an image which results in about 150 smaller files which I save directly to an external HD but after 500,000 or so, things just stop and don't work anymore. I haven't tried saving to the internal HD as I don't want 3.5 million files on the drive. I've also tried copying from external to internal drive (or vice-versa) and the same problem occurs.

Details:

The PC is a junky Gateway running some sort of nVidia chipset (all this thing does is batch process images and copy files to an external HD so I didn't need anything fancy). The internal hard drive is a 750gb SATA. The boot drive is an IDE so the SATA where the images are coming from is a secondary drive. Originally the SATA drive was connnected to the SATA port on the motherboard. I got a SATA PCI card (-X or express, I can't remember) and tried running with that but it didn't make a difference. The external hard drive is a Lacie Porsche USB only.

I have no clue if this is hardware (computer chipset), software (windows) or the chipset in the external drive. Does anyone know what's going on here or how to fix it? I can't seem to find any info.

Thanks

 

[g-paw]

Could you clarify a bit? Millions of files is an enormous number if you're talking about data, what kind of files are they? How many GB of files do you have? You said you have folders, is there a reason you can't just copy a given number of folders and then copy some more? Why do you need to copy them all at the same time?

 

[dirtbagjack]

These files are tiny little jpegs. I have 29,000+ tiffs (between 15-20mb each) and I need to turn them into zoomified versions (www.zoomify.com) which results in 120-170 little jpegs per tiff, which are stored in a folder along with a little text file. In total I think all those little jpegs add up to about 60gb. Copying 4000 of those folders is about 8gb of data I think.

I've copied the 16,000+ tiff files at once without any issues and that's 285gb worth of data. Right after that was done, I've done 6000 more files which was another 110gb. No restarts, no problem. Based on that it would seem my problem is the total number of files, not the amount of gigs.

Right now I've got a batch file calling the conversion program for each image, one after the other. I can break it up into smaller parts for processing but I'd prefer not to because it slows me down. I figure it's going to take 29-32 hours to process everything but now that I have to go in smaller batches, it's taking 3 days and I have to keep an eye on the computer to start the next batch. Get distracted doing something else and I could lose 2-3 hours of processing time.

When I've had to copy the files from drive to drive, I prefer to use the command prompt. Windows wastes time thinking before copying whereas the command prompt starts right away. Not sure how to copy only a range of folders using that method though.

I don't have to copy/process everything at once but I would much prefer it as I can let things go overnight and don't have to keep an eye on the computer (and I get the drive off via fedex on time instead of several days late and then have to spend more money to send it overnight). I've asked someone else who has done this a while ago and they don't recall having this copying problem. (Not sure if they copies with Linux/Unix or Windows though). I'm thinking it's something with the nForce chipset on the motherboard but I'm not sure. Don't want to buy a new machine and end up with the same issue if it's a windows thing.

Thanks

 

[g-paw]

What percentage of your hdds are free? My understanding is that it should be around 20% or so to avoid problems. Have you tried using the external as an internal? The transfer speed for the external is likely slower than when it's internal unless it's an eSATA connection, which I believe is about the same speed as an internal. It sounds like you're sending the external drive to customer and it would be a pain to use it as an internal but would be worth a try to see if it helps. This would be a lot easy with a case where the drives face out like the Antec Sonata II. Finally, it's possible you're over stressing machine and need something more powerful. Using what you can from the Gateway, which would be the drives, you could put a machine together for $500 or so, maybe a bit more depending on how CPU intensive this is, especially using on board video. If this is for a business, might be worthwhile in the long run and likely tax deductible.

 

[dirtbagjack]

hard drives have plenty of free space, more than 20% so that's not an issue.

The external drive is going to the customer. I don't know if the HD's case can be opened (i'm assume it's possible but I don't know if I could get it open and then closed again). Even if I can open it, I don't know if getting the drive out is possible or not.

This is for business and I did get the OK to buy a new machine. Problem is I don't know if buying a new computer will solve the problem so I'm sill trying to track down info before I do anything. I'm thinking of buying a small HP ProLiant server instead of building my own machine. I know it costs more but it's easier for me this way and in theory, whatever they use should be able to handle the load (I've emailed HP so we'll see what they say). I can check the specs and see how much buying the components myself would cost as well.

Thanks

 

[g-paw]

If these are off the shelf external drives, opening them would probably invalidate any warranty. Do you need a server, i.e., is there more than one person working on these files? What are the specs on the Gateway? Right Clicking on My Compute, Properties will give you the CPU and RAM. Device Manager will tell you the video card. As for buying or building, the advantage to building is that you can insure the best part for the money and likely save a little, at around $900 should see 10% to 15% and the savings go up from there. Plus you get a better warranty, hdd are now 3 to 5 years, CPU are 3 years, and RAM lifetime. Of course you become support. What kind of budget have then given you? Will the machine be yours or the company?

 

[dirtbagjack]

Technically I don't need a server. I'm only doing simple file sharing to transfer files to and from the machine. Once on there, they are processed and stored there, waiting to be copied to an external HD for the client. A decent workstation would probably be OK but I'm thinking server as buying a server will give me more internal room and better i/o.

It's the weekend so I'm not at work and can't check the computer but if the spec sheet I found hanging around at home is correct...

Athlon 64 X2 dual-core 3800+ process
2GB ram (PC3200)
250gb HD (no images are stored or copied to/from this HD)

Computer is running win XP pro. I installed a netgear gigabit ethernet card and an SATA card (can't remember the make and it didn't make a difference over the onboard sata ports)

I thought with those specs I would be ok. I just wanted something to do the copying and powerful enough to do lots of batch processing. It's got integrated graphics (GeForce 6100) but I don't really care about that.

The internal 750gb HD (where images are copied from) is an SATA seagate (with the jumper set for 1.5 operation... it was at 3 originally and I tried 1.5 to see if that would help but it didn't and I didn't notice any difference in speed)

I've never built my own machine before so I'm shying away from it. I'm not worried about saving money, I just want it to work, have plenty of internal storage space and plenty of cooling. I've installed ram, HD's, swapped HDs and CD/DVD drives, power supplies, a processor once but never anything with a motherboard. I'm sure I could build the thing myself if I really wanted to... but unless I can be guaranteed I won't run into the same issue, I'm better of spending more and going back to HP if they gave me bad info.

I told the boss we're looking at $1000-2000 for a new machine and he was ok with that.

I don't mind being support, I hate calling support lines as those guys are generally useless anyway. I just want a machine that can handle the load.

The machine will be the companies, not mine.

If you have any recommendations on which parts to buy, please share. Maybe I should post in one of the other forums (new system build perhaps?)

Thanks

 

[g-paw]

Check out this CPU chart. I compared the X2 3800 using Photo Shop, seemed to be the closest to what you're doing. Looks like the Intel 6750 would be you're best bet at $212.
http://www23.tomshardware.com/cpu_2007.html?modelx=33&model1=927& model2=874&chart=437
http://www.newegg.com/Product/Product.aspx?Item=N82E16 819115029

While onboard video would work the advantage of a lower cost video card is it would use less of the computers resources and probably every little bit would help. You could easily put this machine together for around $1000 or less. As far as building, looks like you've done everything, just not all at the same time. Something like this mobo would be good. Check out the manual and see if you're comfortable with the assembly instructions.
http://www.newegg.com/Product/Product.aspx?Item=N82E16813128057 They're pretty much the same regardless of the board.

If you can disconnect from the Internet while doing this, I'd disable the antivirus and any other security programs other than the firewall. I'd also go with XP , Pro would probably be easier on a business network, rather than Vista. I know you said you don't want to work with fewer files but if a more powerful machine doesn't solve the problem, you may have no choice. It seems like the basic issue as you pointed out is that you're working with a very large amount of data that is compounded by the fact you are transferring it to an external drive that is likely in most instances uses a USB connection. While the 6750 should definitely help with the processing not sure what you can do about the transfer rate. One thing you could also do is check out "Industrial" level machines used for heavy duty processing and transfers, which are likely very expensive. Check out the specs and then put a similar machine together for considerably less, e.g., semi professional machines designed for video, audio, or photo processing. There are companies that specialize in these machines.

 

[dirtbagjack]

thanks for your help g-paw. I'll keep looking into all this and see what I can do with building my own machine. Just one question... would I need to attach a heatsink to a processor with that thermal paste stuff? Never done that before so not sure how much/how little to use. Please say the heatsinks are already attached or I don't need that thermal compound

 

[g-paw]

You'll have to attach the HSF to the CPU and the HSF will have the thermal paste already. The Retail CPU will come with the the HSF. This is probably the easiest part of the build. Appreciate your concern because installing the CPU/HSF was my biggest fear for my first build was really surprised how easy it was. Instructions come with both the mobo and CPU. Again, check out the mobo manual on line. This is a starting point and I included everything, e.g., XP Pro, small hdd for OS/Programs and 750GB for storage. Suggest you set up a couple of Wish Lists at newegg to price different configurations.

Here would be a place to start:
Case $50 http://www.newegg.com/product/product.asp?item=N82E16811119106
Mobo $100 http://www.newegg.com/product/product.asp?item=N82E16813128057
PSU $75 http://www.newegg.com/product/product.asp?item=N82E16817194011
CPU $212 http://www.newegg.com/product/product.asp?item=N82E16819115029
2GB RAM $110 http://www.newegg.com/product/product.asp?item=N82E16820146565
Floppy $7 $210 http://www.newegg.com/product/product.asp?item=N82E16821103203
HDD 750GB http://www.newegg.com/product/product.asp?item=N82E16822148134
HDD 80GB $43 http://www.newegg.com/product/product.asp?item=N82E16822148231
DVD Burner $30 http://www.newegg.com/product/product.asp?item=N82E16827151133
XP Pro OEM $140 http://www.newegg.com/product/product.asp?item=N82E16832116059
GPU $65 w Rebate http://www.newegg.com/product/product.asp?item=N82E16814130029
Total without shipping $1062

 

[dirtbagjack]

thanks for all the links g-paw. Much appreciated.

 

[g-paw]

No problem. Let us know what you decide