File size/Number of file limits

Typo correction in the before last paragraph. LFN's require at least
*2* extra entries, (requires 3 or more entries).

John John wrote:

98 Guy wrote:

The claimed max number of files per directory is 65535.


You seem to think (or you are implying) that this is incorrect, by one
of your yet unknown methods are we to assume that you can also overcome
this limitation? In the Directory Table the object's first cluster is
held in a 16 bits wide feild so, being that objects cannot share
clusters, the maximum number of objects that can be contained in the
Directory Table is 65536.


The max number of files possible using FAT-32: 268,435,437

The number 268,435,437 (max number of files possible under FAT32) is
equal to 16k x 16k (ie, 16k directories, each with 16k files) or 4k x
64k (ie - 4k directories, each with 64k files).


Sigh... Where did you get this mathematical explanation for the
268,435,437 file or cluster limit on FAT32? FAT32 is 32 bits wide but 4
bits are reserved or unused so it is really 28 bits wide and 2^28 less a
couple of bits (for something or other, that I am not sure why) =
268,435,437 clusters.


The limiting factor for the number of files on a win-98 system (under
FAT-32) will be the number of allocation units available on the
volume. You can't have more files than you have allocation units (aka
clusters).


Well duh! And you can't puts 3 pints in a quart!


As far as this being a long file name issue, I don't think so. The
size of the directory tables are variable under FAT-32 and can grow as
needed.


Well think again! Long filenames require at least 2 extra directory
object entries, so unless you figured out how to "widen" the 16 bit
field, long file names will reduce the number of available directory
objects.

I expect that you will soon tell us that you have blown FAT32's 4gb file
size limit!

John

John John wrote:

98 Guy wrote:

The claimed max number of files per directory is 65535.

You seem to think (or you are implying) that this is incorrect, by one
of your yet unknown methods are we to assume that you can also overcome
this limitation? In the Directory Table the object's first cluster is
held in a 16 bits wide feild so, being that objects cannot share
clusters, the maximum number of objects that can be contained in the
Directory Table is 65536.

Actually, a FAT32 directory could hold many more entries than this.
The specification says that FAT32 drivers must have this limit. That's
because a FAT32 directory is always searched sequentially, so the
bigger it gets, the more time you spend searching it. NTFS
directories, by contrast, are stored as BTrees (balanced trees), which
makes searching *much* faster. (It does slow down the process of
retrieving all file names, but not hugely.)

The spec says this:

quote
* DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
file system driver must not allow a cluster chain to be created that
is longer than 0x100000000 bytes, and the last byte of the last
cluster in a chain that long cannot be allocated to the file. This
must be done so that no file has a file size 0xFFFFFFFF bytes. This
is a fundamental limit of all FAT file systems. The maximum allowed
file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.
/quote

and since each entry is 32 bytes long, that means no more than 65,536
entries per directory.

--
Tim Slattery
MS MVP(DTS)

http://members.cox.net/slatteryt

Tim Slattery wrote:

John John wrote:


98 Guy wrote:


The claimed max number of files per directory is 65535.

You seem to think (or you are implying) that this is incorrect, by one
of your yet unknown methods are we to assume that you can also overcome
this limitation? In the Directory Table the object's first cluster is
held in a 16 bits wide field so, being that objects cannot share
clusters, the maximum number of objects that can be contained in the
Directory Table is 65536.


Actually, a FAT32 directory could hold many more entries than this.
The specification says that FAT32 drivers must have this limit. That's
because a FAT32 directory is always searched sequentially, so the
bigger it gets, the more time you spend searching it. NTFS
directories, by contrast, are stored as BTrees (balanced trees), which
makes searching *much* faster. (It does slow down the process of
retrieving all file names, but not hugely.)

The spec says this:

quote
* DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
file system driver must not allow a cluster chain to be created that
is longer than 0x100000000 bytes, and the last byte of the last
cluster in a chain that long cannot be allocated to the file. This
must be done so that no file has a file size 0xFFFFFFFF bytes. This
is a fundamental limit of all FAT file systems. The maximum allowed
file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.
/quote

and since each entry is 32 bytes long, that means no more than 65,536
entries per directory.

Isn't the DIR_FileSize field used to record the size of the file? Hence
the 4,294,967,295 bytes maximum?

Shouldn't the maximum number of objects in the directory be limited by
DIR_FstClusLO or DIR_FstClusHI fields? Isn't the file location just
kept by the first cluster in the Directory Table, and isn't the rest of
the cluster map outside of the Directory Table, just sort of "daisy
chained" within the actual and successive clusters?

Much of this is new territory to me ;-) so I'm just being inquisitive, I
always appreciate getting good solid information on some of these
scantly documented (and complicated!) subjects.

John

John John wrote:


The spec says this:

quote
* DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
file system driver must not allow a cluster chain to be created that
is longer than 0x100000000 bytes, and the last byte of the last
cluster in a chain that long cannot be allocated to the file. This
must be done so that no file has a file size 0xFFFFFFFF bytes. This
is a fundamental limit of all FAT file systems. The maximum allowed
file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.
/quote

and since each entry is 32 bytes long, that means no more than 65,536
entries per directory.

Isn't the DIR_FileSize field used to record the size of the file? Hence
the 4,294,967,295 bytes maximum?

Exactly

Shouldn't the maximum number of objects in the directory be limited by
DIR_FstClusLO or DIR_FstClusHI fields?

As the quote shows, the 65,536 entries limit is purely arbitrary. The
spec says that any conforming driver must impose this limit. It's not
the result of anything in the structure/

Isn't the file location just
kept by the first cluster in the Directory Table, and isn't the rest of
the cluster map outside of the Directory Table, just sort of "daisy
chained" within the actual and successive clusters?

Yes, the directory entry points to the first cluster (or allocation
unit). To find where the rest of the file is, you have to look in the
File Allocation Table itself, where the entry for a given cluster will
point to the next cluster in the file. And another chain keeps track
of free space.

Much of this is new territory to me ;-) so I'm just being inquisitive, I
always appreciate getting good solid information on some of these
scantly documented (and complicated!) subjects.

There is solid documentation of FAT32 (unlike NTFS). It's he
http://www.microsoft.com/whdc/system...re/fatgen.mspx

MS does not (AFAIK) make that kind of documentation of NTFS available
- at least without paying for it. There is an open source program
that's trying to make NTFS available for Linux. They have published
documentation at http://www.linux-ntfs.org/doku.php

--
Tim Slattery
MS MVP(Shell/User)

http://members.cox.net/slatteryt

Tim Slattery wrote:

As the quote shows, the 65,536 entries limit is purely arbitrary. The
spec says that any conforming driver must impose this limit. It's not
the result of anything in the structure/

I see, the limit is imposed be the file system driver. Thanks for
steering me straight on this.

John

I performed a series of serial file generations to see how many files
could be created in a subdirectory while changing the length of the
file name.

The following table shows the file-name length and the corresponding
number of files that could be created before this eror was generated:

The directory or file cannot be created

6.0 - 65,533
6.3 - 32,767
12.3 - 21,845
15.3 - 21,845
17.3 - 21,845
23.3 - 16,384
47.3 - 13,107
63.3 - 9,362
96.3 - 7,282

So in the first case, with a filename composed of 6 characters (and no
suffix) I was able to create 65,533 files. In the second case, the
filename was composed of 6-characters.3-characters, and the directory
would only hold 32,767 of those files. So as the filename length
increases, there is a decrease in the number of possible files.

According to the OP:

The target 'contracts' directory is 599MB and contains 16135
objects. We are appending a few hundred files. We get the
following error:
'The directory or file cannot be created.

For the above to happen, the existing 16,135 files would have to have
long file names (about 20 characters, more likely 23 characters,
possibly slightly more).

On Thu, 25 Oct 2007 12:44:07 -0400, Tim Slattery
put finger to keyboard and composed:

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.
/quote

and since each entry is 32 bytes long, that means no more than 65,536
entries per directory.

I'm thinking that this restriction could in extreme circumstances
result in file system corruption. For example, let's assume that the
directory is full (65,536 entries) and that the machine has been
booted in Win9x real DOS mode. What happens if you try to create an
additional file? AFAICS, DOS sees the existing LFNs as deleted files
(each entry has a leading 0xE5 flag byte), so wouldn't DOS then
overwrite one of these LFNs? Or does Win9x DOS understand that these
entries are LFNs even though it doesn't support them?

- Franc Zabkar
--
Please remove one 'i' from my address when replying by email.

I performed a series of serial file generations to see how many files
could be created in a subdirectory while changing the length of the
file name.

6.0 - 65,533
6.3 - 32,767

strange that it starts to use LFN-names when you add extension to
the filenames? are you sure you didn't got lowercase in the
fileextensions somehow?

12.3 - 21,845
15.3 - 21,845
17.3 - 21,845
23.3 - 16,384
47.3 - 13,107
63.3 - 9,362
96.3 - 7,282

Actually, a FAT32 directory could hold many more entries than this.
The specification says that FAT32 drivers must have this limit. That's

nice to hear that it is just a rule with a number they chosed to be
"enough"
and not something forced by the format. I guess then you could relatively
easily
modify your filesystem drivers to a number you like more.
(but some of your file-utilities may not see all your files then, and
verry sloppily coded ones could perhaps crash)

That said, I must say that I think 65536 is way more than enough that it
is,
if you want more than 10000 files in the same directory without hierachy,
your program should really use some database instead.

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.

good to know that I never need more than 2MB to hold a directory list in
memory :-)

I'm thinking that this restriction could in extreme circumstances
result in file system corruption. For example, let's assume that the
directory is full (65,536 entries) and that the machine has been
booted in Win9x real DOS mode. What happens if you try to create an
additional file? AFAICS, DOS sees the existing LFNs as deleted files

nope! the LFNs are not marked as deleted files.
The first char in the filename-part in the LFN-posts are a letter
that describes number or LFN-posts the LFN-filename have. A is one LFN-post
B is two, C is three etc.

the special marking of the LFN-posts is instead by using fileattribute
DiskLabel-System-Hidden-ReadOnly

(each entry has a leading 0xE5 flag byte), so wouldn't DOS then
overwrite one of these LFNs? Or does Win9x DOS understand that these
entries are LFNs even though it doesn't support them?

dos without LFN-support understands them as "bad" and don't touch them.

(And if you run old scandisk type of diskutilities, like norton disk
doctor etc,
and the find these "errorus lines" and removes them, nothing more than the
long filenames is destroyed. the files can allways be used with the short
names)