Linux Largefile Support In Postgresql RPMS

Note, I'm not sure this belongs in -hackers so I've added -general but left
-hackers in so that list can at least see that it's going to -general.

On Thu, 8 Aug 2002, mark Kirkwood wrote:

Hi all,

I just spent some of the morning helping a customer build Pg 7.2.1 from
source in order to get Linux largefile support in pg_dump etc. They
possibly would have kept using the binary RPMs if they had this feature.

This got me to wondering why the Redhat/Mandrake...etc binary RPMS are
built without it.

Would including default largefile support in Linux RPMs be a good idea ?

(I am presuming that such RPMs are built by the Pg community and
"supplied" to the various distros... apologies if I have this all wrong...)

I must admit that I am fairly new to PostgreSQL but I have used it and read
stuff about it and I'm not sure what you mean. Could you explain what you
did?

A quick scan of the source shows that there may be an issue in
storage/file/buffile.c:BufFileSeek() is that the sort of thing you are talking
about? Or maybe I've got it completely wrong and you're talking about adding
code to pg_dump although I thought that could already handle large
objects. Actually, I'm going to shut up now before I really do show my
ignorance and let you answer.

--
Nigel J. Andrews
Director

---
Logictree Systems Limited
Computer Consultants

Hi,

just my two cents worth: I like having the files sized in a way I can
handle them easily with any UNIX tool on nearly any system. No matter
wether I want to cp, tar, dump, dd, cat or gzip the file: Just keep it at
a maximum size below any limits, handy for handling.

For example, Oracle suggests it somewhere in their documentation, to keep
datafiles at a reasonable size, e.g. 1 GB. Seems right to me, never had
any problems with it.

Kind regards
... Ralph ...

On Thursday 08 August 2002 05:36 pm, Nigel J. Andrews wrote:

Matt Kirkwood wrote:

I just spent some of the morning helping a customer build Pg 7.2.1 from
source in order to get Linux largefile support in pg_dump etc. They
possibly would have kept using the binary RPMs if they had this feature.

And you added this by doing what, exactly? I'm not familiar with pg_dump
largefile support as a standalone feature.

(I am presuming that such RPMs are built by the Pg community and
"supplied" to the various distros... apologies if I have this all
wrong...)

You have this wrong. The distributions do periodically sync up with my
revision, and I with theirs, but they do their own packaging.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

On Fri, 2002-08-09 at 06:07, Lamar Owen wrote:

On Thursday 08 August 2002 05:36 pm, Nigel J. Andrews wrote:

Matt Kirkwood wrote:

I just spent some of the morning helping a customer build Pg 7.2.1 from
source in order to get Linux largefile support in pg_dump etc. They
possibly would have kept using the binary RPMs if they had this feature.

And you added this by doing what, exactly? I'm not familiar with pg_dump
largefile support as a standalone feature.

As far as I can make out from the libc docs, largefile support is
automatic if the macro _GNU_SOURCE is defined and the kernel supports
large files.

Is that a correct understanding? or do I actually need to do something
special to ensure that pg_dump supports large files?

--
Oliver Elphick Oliver.Elphick@lfix.co.uk
Isle of Wight, UK
http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C
========================================
"...ask, and ye shall receive, that your joy may be
full." John 16:24

As far as I can make out from the libc docs, largefile support is
automatic if the macro _GNU_SOURCE is defined and the kernel supports
large files.

Is that a correct understanding? or do I actually need to do something
special to ensure that pg_dump supports large files?

in this case you still have to use large file functions in the code
explicitly

the easiest way to get large file support is to pass
-D_FILE_OFFSET_BITS=64 to the preprocessor, and I think I remember doing
this once for pg_dump

see /usr/include/features.h

Best regards
Helge

On Fri, 9 Aug 2002, Helge Bahmann wrote:

As far as I can make out from the libc docs, largefile support is
automatic if the macro _GNU_SOURCE is defined and the kernel supports
large files.

Is that a correct understanding? or do I actually need to do something
special to ensure that pg_dump supports large files?

in this case you still have to use large file functions in the code
explicitly

the easiest way to get large file support is to pass
-D_FILE_OFFSET_BITS=64 to the preprocessor, and I think I remember doing
this once for pg_dump

see /usr/include/features.h

There is some commentary on this in my /usr/doc/libc6/NOTES.gz, which I presume
Oliver has already found since I found it after reading his posting. It gives a
bit more detail that the header file for those who want to check this out. I
for one was completely unaware of those 64 bit functions.

--
Nigel J. Andrews
Director

---
Logictree Systems Limited
Computer Consultants

Lamar Owen wrote:

And you added this by doing what, exactly? I'm not familiar with pg_dump
largefile support as a standalone feature.

Enabling largefile support for the utilities was accomplished by :

CFLAGS="-O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" ./configure ...

It seemed to me that the ability to dump databases >2G without gzip,
split etc was a "good thing". What do you think ?

You have this wrong. The distributions do periodically sync up with my
revision, and I with theirs, but they do their own packaging.

I see.... so if you enabled such support, they they would probably sync
that too ?

Ralph Graulich wrote:

Hi,

just my two cents worth: I like having the files sized in a way I can
handle them easily with any UNIX tool on nearly any system. No matter
wether I want to cp, tar, dump, dd, cat or gzip the file: Just keep it at
a maximum size below any limits, handy for handling.

Good point... however I was thinking that being able to dump the entire
database without resporting to "gzips and splits" was handy...

For example, Oracle suggests it somewhere in their documentation, to keep
datafiles at a reasonable size, e.g. 1 GB. Seems right to me, never had
any problems with it.

Yep, fixed or controlled sizes for data files is great... I was thinking
about databases rather than data files (altho I may not have made that
clear in my mail)

best wishes

Mark

Oliver Elphick wrote:

As far as I can make out from the libc docs, largefile support is
automatic if the macro _GNU_SOURCE is defined and the kernel supports
large files.

Is that a correct understanding? or do I actually need to do something
special to ensure that pg_dump supports large files?

I defined

_LARGEFILE_SOURCE and
_FILE_OFFSET_BITS=64

however _GNU_SOURCE may well be a cleaner way of getting the same effect
(guess I should browse the .h files...)

regards

Mark

On Sat, 2002-08-10 at 00:25, Mark Kirkwood wrote:

Ralph Graulich wrote:

Hi,

just my two cents worth: I like having the files sized in a way I can
handle them easily with any UNIX tool on nearly any system. No matter
wether I want to cp, tar, dump, dd, cat or gzip the file: Just keep it at
a maximum size below any limits, handy for handling.

Good point... however I was thinking that being able to dump the entire
database without resporting to "gzips and splits" was handy...

For example, Oracle suggests it somewhere in their documentation, to keep
datafiles at a reasonable size, e.g. 1 GB. Seems right to me, never had
any problems with it.

Yep, fixed or controlled sizes for data files is great... I was thinking
about databases rather than data files (altho I may not have made that
clear in my mail)

I'm actually amazed that postgres isn't already using large file
support. Especially for tools like dump. I do recognize the need to
keep files manageable in size but my file sizes for my needs may differ
from your sizing needs.

Seems like it would be a good thing to enable and simply make it a
function for the DBA to handle. After all, even if I'm trying to keep
my dumps at around 1GB, I probably would be okay with a dump of 1.1GB
too. To me, that just seems more flexible.

Greg

On Sat, 2002-08-10 at 06:32, Mark Kirkwood wrote:

Oliver Elphick wrote:

As far as I can make out from the libc docs, largefile support is
automatic if the macro _GNU_SOURCE is defined and the kernel supports
large files.

Is that a correct understanding? or do I actually need to do something
special to ensure that pg_dump supports large files?

I defined

_LARGEFILE_SOURCE and
_FILE_OFFSET_BITS=64

however _GNU_SOURCE may well be a cleaner way of getting the same effect
(guess I should browse the .h files...)

It seems that it has to be defined explicitly.

--
Oliver Elphick Oliver.Elphick@lfix.co.uk
Isle of Wight, UK
http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C
========================================
"If ye abide in me, and my words abide in you, ye shall
ask what ye will, and it shall be done unto you."
John 15:7

On Sat, Aug 10, 2002 at 09:21:07AM -0500, Greg Copeland wrote:

I'm actually amazed that postgres isn't already using large file
support. Especially for tools like dump.

Except it would only cause confusion if you ran such a program on a
system that didn't itself have largefile support. Better to make the
admin turn all these things on on purpose, until everyone is running
64 bit systems everywhere.

A
--
----
Andrew Sullivan 87 Mowat Avenue
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M6K 3E3
+1 416 646 3304 x110

On Mon, 2002-08-12 at 09:39, Andrew Sullivan wrote:

On Sat, Aug 10, 2002 at 09:21:07AM -0500, Greg Copeland wrote:

I'm actually amazed that postgres isn't already using large file
support. Especially for tools like dump.

Except it would only cause confusion if you ran such a program on a
system that didn't itself have largefile support. Better to make the
admin turn all these things on on purpose, until everyone is running
64 bit systems everywhere.

If by "turn...on", you mean recompile, that's a horrible idea IMO.
Besides, you're expecting that an admin is going to know that he even
needs to recompile to obtain this feature let alone that he'd interested
in compiling his own installation. Whereas, more then likely he'll know
off hand (or can easily find out) if his FS/system supports large files
(>32 bit sizes).

Seems like, systems which can natively support this feature should have
it enabled by default. It's a different issue if an admin attempts to
create files larger than what his system and/or FS can support.

I guess what I'm trying to say here is, it's moving the problem from
being a postgres specific issue (not compiled in -- having to recompile
and install and not knowing if it's (dis)enabled) to a general body of
knowledge (does my system support such-n-such capabilities).

If a recompile time is still much preferred by the core developers,
perhaps a log entry can be created which at least denotes the current
status of such a feature when a compile time option is required. Simply
having an entry of, "LOG: LARGE FILE SUPPORT (DIS)ENABLED (64-bit file
sizes)", etc...things along those lines. Of course, having a
"--enable-large-files" would be nice too.

This would seemingly make sense in other contexts too. Imagine a
back-end compiled with large file support and someone else using fe
tools which does not support it. How are they going to know if their
fe/be supports this feature unless we let them know?

Greg

On Mon, Aug 12, 2002 at 10:15:46AM -0500, Greg Copeland wrote:

If by "turn...on", you mean recompile, that's a horrible idea IMO.

Ah. Well, that is what I meant. Why is it horrible? PostgreSQL
doesn't take very long to compile.

I guess what I'm trying to say here is, it's moving the problem from
being a postgres specific issue (not compiled in -- having to recompile
and install and not knowing if it's (dis)enabled) to a general body of
knowledge (does my system support such-n-such capabilities).

The problem is not just a system-level one, but a filesystem-level
one. Enabling 64 bits by default might be dangerous, because a DBA
might think "oh, it supports largefiles by default" and therefore not
notice that the filesystem itself is not mounted with largefile
support. But I suspect that the developers would welcome autoconfig
patches if someone offered them.

A

-- 
----
Andrew Sullivan                               87 Mowat Avenue 
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110

On Monday 12 August 2002 11:30 am, Andrew Sullivan wrote:

The problem is not just a system-level one, but a filesystem-level
one. Enabling 64 bits by default might be dangerous, because a DBA
might think "oh, it supports largefiles by default" and therefore not
notice that the filesystem itself is not mounted with largefile
support. But I suspect that the developers would welcome autoconfig
patches if someone offered them.

Interesting point. Before I could deploy RPMs with largefile support by
default, I would have to make sure it wouldn't silently break anything. So
keep discussing the issues involved, and I'll see what comes of it. I don't
have an direct experience with the largefile support, and am learning as I go
with this.

Given that I have to make the source RPM's buildable on distributions that
might not have the largefile support available, so on those distributions the
support will have to be unavailable -- and the decision to build it or not to
build it must be automatable.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

On Mon, Aug 12, 2002 at 11:44:24AM -0400, Lamar Owen wrote:

The problem is not just a system-level one, but a filesystem-level
one. Enabling 64 bits by default might be dangerous, because a DBA
might think "oh, it supports largefiles by default" and therefore not
notice that the filesystem itself is not mounted with largefile
support.

keep discussing the issues involved, and I'll see what comes of it. I don't
have an direct experience with the largefile support, and am learning as I go
with this.

I do have experience with both of these cases. We're hosted in a
managed-hosting environment, and one day one of the sysadmins there
must've remounted a filesystem without largefile support. Poof! I
started getting all sorts of strange pg_dump problems. It wasn't
hard to track down, except that I was initially surprised by the
errors, since I'd just _enabled_ large file support.

This is an area that is not encountered terribly often, actually,
because postgres itself breaks its files at 1G. Most people's dump
files either don't reach the 2G limit, or they use split (a
reasonable plan).

There are, in any case, _lots_ of problems with these large files.
You not only need to make sure that pg_dump and friends can support
files bigger than 2G. You need to make sure that you can move the
files around (your file transfer commands), that you can compress the
files (how is gzip compiled? bzip2?), and even that you r backup
software takes the large file. In a few years, when all
installations are ready for this, it seems like it'd be a good idea
to turn this on by default. Right now, I think the risks are at
least as great as those incurred by telling people they need to
recompile.

A

-- 
----
Andrew Sullivan                               87 Mowat Avenue 
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110

On Mon, 2002-08-12 at 10:30, Andrew Sullivan wrote:

On Mon, Aug 12, 2002 at 10:15:46AM -0500, Greg Copeland wrote:

If by "turn...on", you mean recompile, that's a horrible idea IMO.

Ah. Well, that is what I meant. Why is it horrible? PostgreSQL
doesn't take very long to compile.

Many reasons. A DBA is not always the same thing as a developer (which
means it's doubtful he's even going to know about needed options to pass
-- if any). Using a self compiled installation may not install the same
sense of reliability (I know that sounds odd) as using a distribution's
package. DBA may not be a SA, which means he should probably not be
compiling and installing software on a system. Furthermore, he may not
even have access to do so.

Means upgrading in the future may be problematic. Someone compiled with
large file support. He leaves. New release comes out. Someone else
upgrades and now finds things are broken. Why? If it supported it out
of the box, issue is avoided.

Lastly, and perhaps the most obvious, SA and DBA bodies of knowledge are
fairly distinct. You should not expect a DBA to function as a SA.
Furthermore, SA and developer bodies of knowledge are also fairly
distinct. You shouldn't expect a SA to know what compiler options he
needs to use to compile software on his system. Especially for
something as obscure as large file support.

I guess what I'm trying to say here is, it's moving the problem from
being a postgres specific issue (not compiled in -- having to recompile
and install and not knowing if it's (dis)enabled) to a general body of
knowledge (does my system support such-n-such capabilities).

The problem is not just a system-level one, but a filesystem-level
one. Enabling 64 bits by default might be dangerous, because a DBA
might think "oh, it supports largefiles by default" and therefore not
notice that the filesystem itself is not mounted with largefile
support. But I suspect that the developers would welcome autoconfig
patches if someone offered them.

The distinction you make there is minor. A SA, should know and
understand the capabilities of the systems he maintains (this is true
even if the SA and DBA are one). This includes filesystem
capabilities. A DBA, should only care about the system requirements and
trust that the SA can deliver those capabilities. If a SA says, my
filesystems can support very large files, installs postgres, the DBA
should expect that match support in the database is already available.
Woe is his surprise when he finds out that his postgres installation
can't handle it?!

As for the concern for danger. Hmm...my understanding is that the
result is pretty much the same thing as exceeding max file size. That
is, if you attempt to read/write beyond what the filesystem can provide,
you're still going to get an error. Is this really more dangerous than
simply reading/writing to a file which exceeds max system capabilities?
Either way, this issue exists and having large file support, seemingly,
does not effect it one way or another.

I guess I'm tying to say, the risk of seeing filesystem corruption or
even database corruption should not be effected by the use of large file
support. Please correct me if I'm wrong.

Greg

On Mon, 2002-08-12 at 11:04, Andrew Sullivan wrote:

On Mon, Aug 12, 2002 at 11:44:24AM -0400, Lamar Owen wrote:

keep discussing the issues involved, and I'll see what comes of it. I don't
have an direct experience with the largefile support, and am learning as I go
with this.

I do have experience with both of these cases. We're hosted in a
managed-hosting environment, and one day one of the sysadmins there
must've remounted a filesystem without largefile support. Poof! I
started getting all sorts of strange pg_dump problems. It wasn't
hard to track down, except that I was initially surprised by the
errors, since I'd just _enabled_ large file support.

And, what if he just remounted it read only. Mistakes will happen.
That doesn't come across as being a strong argument to me. Besides,
it's doubtful that a filesystem is going to be remounted while it's in
use. Which means, these issues are going to be secondary to actual
product use of the database. That is, either the system is working
correctly or it's not. If it's not, guess it's not ready for production
use.

Furthermore, since fs mounting, if being done properly, is almost always
a matter of automation, this particular class of error should be few and
very far between.

Wouldn't you rather answer people with, "remount your file system",
rather than, recompile with such-n-such option enabled, reinstall. Oh
ya, since you're re-installing a modified version of your database,
probably a good paranoid option would be to back up and dump, just to be
safe. Personally, I'd rather say, "remount".

There are, in any case, _lots_ of problems with these large files.
You not only need to make sure that pg_dump and friends can support
files bigger than 2G. You need to make sure that you can move the
files around (your file transfer commands), that you can compress the
files (how is gzip compiled? bzip2?), and even that you r backup
software takes the large file. In a few years, when all
installations are ready for this, it seems like it'd be a good idea
to turn this on by default. Right now, I think the risks are at
least as great as those incurred by telling people they need to
recompile.

All of those are SA issues. Shouldn't we leave that domain of issues
for a SA to deal with rather than try to force a single view down
someone's throat? Which, btw, results is creating more work for those
that desire this feature.

Greg

On Mon, 2002-08-12 at 16:44, Lamar Owen wrote:

Interesting point. Before I could deploy RPMs with largefile support by
default, I would have to make sure it wouldn't silently break anything. So
keep discussing the issues involved, and I'll see what comes of it. I don't
have an direct experience with the largefile support, and am learning as I go
with this.

Given that I have to make the source RPM's buildable on distributions that
might not have the largefile support available, so on those distributions the
support will have to be unavailable -- and the decision to build it or not to
build it must be automatable.

I raised the question on the Debian developers' list. As far as I can
see, the general feeling is that it won't break anything but will only
work with kernel 2.4. It may break with 2.0, but 2.0 is no longer
provided with Debian stable, so I don't mind that.

The thread starts at
http://lists.debian.org/debian-devel/2002/debian-devel-200208/msg00597.html

I intend to enable it in the next version of the Debian packages (which
will go into the unstable archive if this works for me) by adding
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 to CFLAGS for the entire build.

One person said:
However compiling with largefile support will change the size
of off_t from 32 bits to 64 bits - if postgres uses off_t or
anything else related to file offsets in a binary struct in one
of the database files you will break stuff pretty heavily. I
would not compile postgres with largefile support until it
is officially supported by the postgres developers.

but I cannot see that off_t is used in such a way.
--
Oliver Elphick Oliver.Elphick@lfix.co.uk
Isle of Wight, UK
http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C
========================================
"And he spake a parable unto them to this end, that men
ought always to pray, and not to faint."
Luke 18:1