Large files on linux

Fernan Aguero <fernan@iib.unsam.edu.ar> writes:

I am having trouble with large files on a Linux box (RH 6.2). I know there
is a limit of 2 GB on the file size, but do not know if this is kernel
related, filesystem related or both.

I am researchiing the issue (having some trouble to find info) and found
that there are several alternatives. Can you give me advice for/against the
ones you think are best/worse. Also, if you happen to have links to other
places which might have information on this issue, please pass them along.

Here are my options:

i) patch the kernel.
I didn't find much information about this one. I don't know yet whether
this is a kernel related limitation or filesystem-related or both. I have
also read that programs had to be patched to use this 'large_file' patch.
Will probably avoid this one.

ii) try a new kernel, 2.4-test
have to do some research into this yet, but apparently the new linux kernel
will support large files. My understanding right now is that ext2fs do not
support files this large,

It does.

iii) try a new filesystem.
several filesystems appear to support large files (but not ext2fs, and
apparently not ext3fs)

On 2.2, it's VFS related. ext3/ext2 doesn't have such problems.

Red Hat Linux 7.0 includes a kernel (the "enterprise" one) with the
LFS patched included and test. It's also ready for 2.4

--
Trond Eivind Glomsr�d
Red Hat, Inc.

Fernan Aguero writes:

I am having trouble with large files on a Linux box (RH 6.2). I know there
is a limit of 2 GB on the file size,

...but that doesn't affect table size, database size, or whatever you're
thinking of.

--
Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/

* Fernan Aguero <fernan@iib.unsam.edu.ar> [001211 12:45] wrote:

Dear all:

I am having trouble with large files on a Linux box (RH 6.2). I know there
is a limit of 2 GB on the file size, but do not know if this is kernel
related, filesystem related or both.

Afaik it's both.

Honestly Postgresql should be able to deal with this limitation by
using more than one file per table.

But if you really want to support large files on a free UNIX,
I'd try FreeBSD.

best of luck,
--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Alfred Perlstein schrieb:

* Fernan Aguero <fernan@iib.unsam.edu.ar> [001211 12:45] wrote:

Dear all:

I am having trouble with large files on a Linux box (RH 6.2). I know there
is a limit of 2 GB on the file size, but do not know if this is kernel
related, filesystem related or both.

Afaik it's both.

Honestly Postgresql should be able to deal with this limitation by
using more than one file per table.

But if you really want to support large files on a free UNIX,
I'd try FreeBSD.

best of luck,
--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."

Kernel >= 2.4 can manage large files. As far as I have tried this it works
perfectly well.

Hans

Peter Eisentraut wrote:

Fernan Aguero writes:

I am having trouble with large files on a Linux box (RH 6.2). I know there
is a limit of 2 GB on the file size,

...but that doesn't affect table size, database size, or whatever you're
thinking of.

Nope, PostgreSQL segments nicely for tables.

But, unless you do chunking, it _does_ affect dumpfile size. Someone
posted awhile back a script that did dumpchunking. Should be in the
archives.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Quoting Fernan Aguero <fernan@iib.unsam.edu.ar>:

Dear all:

I am having trouble with large files on a Linux box (RH 6.2). I know
there
is a limit of 2 GB on the file size, but do not know if this is kernel
related, filesystem related or both.

IIRC Partly Kernel, Partly FileSystem.

For large tables we split the tables at the 1Gb point. The Storage Manager then
deals with which file it needs to open when it knows that the file position is
beyond any one 1Gb point.

NB: The source was doing this at the 2Gb point but (6.3.x or 6.4.x I think) I
did some tests for another project and found that Linux didn't like files
exactly 2Gb in size. After discussion here, 1Gb was chosen as a nice round
figure.

--
Peter Mount peter@retep.org.uk
PostgreSQL JDBC Driver: http://www.retep.org.uk/postgres/
RetepPDF PDF library for Java: http://www.retep.org.uk/pdf/

Bruce Momjian wrote:

This has nothing to do with PostgreSQL. We don't need large files to go
over 4gig tables. We split them on our own.

Peter Eisentraut wrote:

But, unless you do chunking, it _does_ affect dumpfile size. Someone
posted awhile back a script that did dumpchunking. Should be in the
archives.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Sorry for not being clear in my first post. The problems I am having
are when dumping data, as Peter rightly guessed.
I remembered reading something about this on the list, that's why i
asked my questions here.
Although I considered writing a script to divide the dumped data into
smaller chunks, i (for other reasons) preferred to go for a cleaner
solution, and that is having a system able to cope with large files.

Thanks to all for your replies.

Here's a brief summary of things:

Linux kernel 2.4 supports large files.
As mentioned, SuSE and RedHat, both at version 7.0 come with patched
2.2.x kernels that support large files. SuSE uses the ReiserFS, while
RedHat uses ext2fs.

Since I'm using RH 6.2, the best thing to do is to update to 7.0 and
see if that helps. BTW, RH 7.0 is ready for 2.4 kernels (Thanks Trond)
--

Lic. Fernan Aguero Tel:
(54-11) 4752-0021
Instituto de Investigaciones Biotecnologicas Fax:
(54-11) 4752-9639
Universidad Nacional de General San Martin