more then 105000 folders in one folder

[Guest]

Archived from groups: microsoft.public.win2000.setup (More info?)

Hello,

we have an performance issue, sometimes our server has a lack of
performance.
Operating System Windows 2000 and Windows 2003 (both give the performance
problemens)
When opening the Explorer and only the drive which contains the files (not
the folder) the explorer is using 50 - 90 % of the CPU and the memory is
fluctuating between 60 and 110 kb, it fluctuates continouisly.

I think that the problem is caused by this :
We've an application which uses in ONE single folder more then 105.000
folders, each folder contains ONE single file.

I told it to our programmers but they don't see any problems with it. I've
some questions :
** 1 ** Has Microsoft a recommendation about the maximum number of files in
one folder?
** 2 ** Which performance counters are the best to use to monitor if this is
a problem?

thank you in advance,

Marco J.L.
luyendyk@yahoo.newscom

 

[Guest]

Archived from groups: microsoft.public.win2000.setup (More info?)

Marco,

The maximum number of objects in an NTFS partition is 4,294,967,295. I
strongly doubt that you are anyway near that boundary. What you are
probably seeing is a performance bottleneck associated with accessing these
many files -- I would start looking at caching events for drive writes.

While we don't know what kind of an application this is, it does seem as
though it is poorly designed or, at best, really inefficient, but I am sure
they had a reason.

Your quickest (though sloppiest) solution would probably be to move this to
a hardware-cached RAID array. This should have a positive effect on your
disk access times.

--
Ryan Hanisco
MCSE, MCDBA
Flagship Integration Services

"Marco J.L." <luyendyk@yahoo.com.news> wrote in message
news:ubpHTYv1EHA.1192@tk2msftngp13.phx.gbl...
> Hello,
>
> we have an performance issue, sometimes our server has a lack of
> performance.
> Operating System Windows 2000 and Windows 2003 (both give the performance
> problemens)
> When opening the Explorer and only the drive which contains the files (not
> the folder) the explorer is using 50 - 90 % of the CPU and the memory is
> fluctuating between 60 and 110 kb, it fluctuates continouisly.
>
> I think that the problem is caused by this :
> We've an application which uses in ONE single folder more then 105.000
> folders, each folder contains ONE single file.
>
> I told it to our programmers but they don't see any problems with it. I've
> some questions :
> ** 1 ** Has Microsoft a recommendation about the maximum number of files
in
> one folder?
> ** 2 ** Which performance counters are the best to use to monitor if this
is
> a problem?
>
> thank you in advance,
>
> Marco J.L.
> luyendyk@yahoo.newscom
>
>
>

 

[Guest]

Archived from groups: microsoft.public.win2000.setup (More info?)

Marco J.L. wrote:
> Hello,
>
> we have an performance issue, sometimes our server has a lack of
> performance.
> Operating System Windows 2000 and Windows 2003 (both give the performance
> problemens)
> When opening the Explorer and only the drive which contains the files (not
> the folder) the explorer is using 50 - 90 % of the CPU and the memory is
> fluctuating between 60 and 110 kb, it fluctuates continouisly.
>
> I think that the problem is caused by this :
> We've an application which uses in ONE single folder more then 105.000
> folders, each folder contains ONE single file.
>

This is an *old* problem. It was an NT 4.0 problem that
was not fixed with W2K. I did some simple experiments on
a wide variety of systems and found that the time T it
takes for W2K or NT to process a folder with n items in it
is like this:
T = a*n + b*n^2

The constant "a" is much larger than "b", so for only
a few items in a folder you can ignore the quadratic
term and approximate it as
T = a*n

However, when n gets above 1000 to 2500 - which seems
to depend mostly on your cpu - the quadratic term begins
to dominate and you can approximate the time as
T = b*n^2

What this suggests to me is that
1.) The linear term a*n is determined by the drives,
SCSI controllers, RAID card, etc. It is the time it
takes the OS to *find* the n items in the folder.
2.) The quadratic term b*n^2 is the time it takes the
OS to do whatever the heck it does with the n items it
finds. The fact that it determined by the square of n
suggests that it is being done in a *very* stupid fashion.
The most obvious thing is the use of inefficient sorting
algorithms like a bubble sort or an insertion sort - nice
simple code that works well for small lists where the
hardware is the limiting factor, but extremely bad when
dealing with large lists.

Tell your programmers to grow up and start organizing
things into subfolders that contain smaller numbers of
items. 105 folders each with 1000 items will process
a *lot* faster - and hence hog the CPU for less time -
than one folder with 105,000 items in it.


> I told it to our programmers but they don't see any problems with it. I've
> some questions :

Their practice of putting 1 file in each subfolder is not
too bright, but there is not much of a performance penalty
for doing so. What is giving you such sh*tty performance
is having the 105,000 subfolders all in a single folder.

> ** 1 ** Has Microsoft a recommendation about the maximum number of files in
> one folder?
> ** 2 ** Which performance counters are the best to use to monitor if this is
> a problem?
>

A stop-watch works just fine

> thank you in advance,
>
> Marco J.L.
> luyendyk@yahoo.newscom
>
>
>

 

[Guest]

Archived from groups: microsoft.public.win2000.setup (More info?)

Just a thought fwiw ... do you have the Indexing Service turned on for
this drive?




Marco J.L. wrote:
> Hello,
>
> we have an performance issue, sometimes our server has a lack of
> performance.
> Operating System Windows 2000 and Windows 2003 (both give the performance
> problemens)
> When opening the Explorer and only the drive which contains the files (not
> the folder) the explorer is using 50 - 90 % of the CPU and the memory is
> fluctuating between 60 and 110 kb, it fluctuates continouisly.
>
> I think that the problem is caused by this :
> We've an application which uses in ONE single folder more then 105.000
> folders, each folder contains ONE single file.
>
> I told it to our programmers but they don't see any problems with it. I've
> some questions :
> ** 1 ** Has Microsoft a recommendation about the maximum number of files in
> one folder?
> ** 2 ** Which performance counters are the best to use to monitor if this is
> a problem?
>
> thank you in advance,
>
> Marco J.L.
> luyendyk@yahoo.newscom

 

[Guest]

Archived from groups: microsoft.public.win2000.setup (More info?)

Just an idea, did you or could you disable DOS short file names? This
might improve performance a little bit until you cure the root cause of
the problem. By default Windows sets NTFS to create short file names
for every file and folder created with long file names to make them
compatible with DOS applications. If you don't use DOS apps turning off
this behaviour might (?) increase performance on the file system as NTFS
will not have to keep track of these extra names. It can be set at:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

Value Name: NtfsDisable8dot3NameCreation
Data Type: REG_DWORD
Value: 0

0 -NTFS generates short names, 1 -NTFS doesn’t generate short names. If
you don’t run any DOS apps you don’t need short file names, set value to 1.

If the folders and files within are already all short names it wont make
any difference, only if there are long filenames.

John

Marco J.L. wrote:
> Hello,
>
> we have an performance issue, sometimes our server has a lack of
> performance.
> Operating System Windows 2000 and Windows 2003 (both give the performance
> problemens)
> When opening the Explorer and only the drive which contains the files (not
> the folder) the explorer is using 50 - 90 % of the CPU and the memory is
> fluctuating between 60 and 110 kb, it fluctuates continouisly.
>
> I think that the problem is caused by this :
> We've an application which uses in ONE single folder more then 105.000
> folders, each folder contains ONE single file.
>
> I told it to our programmers but they don't see any problems with it. I've
> some questions :
> ** 1 ** Has Microsoft a recommendation about the maximum number of files in
> one folder?
> ** 2 ** Which performance counters are the best to use to monitor if this is
> a problem?
>
> thank you in advance,
>
> Marco J.L.
> luyendyk@yahoo.newscom
>
>
>