New Linux xfs/reiser file systems

* Bruce Momjian <pgman@candle.pha.pa.us> [010502 14:01] wrote:

I was talking to a Linux user yesterday, and he said that performance
using the xfs file system is pretty bad. He believes it has to do with
the fact that fsync() on log-based file systems requires more writes.

With a standard BSD/ext2 file system, WAL writes can stay on the same
cylinder to perform fsync. Is that true of log-based file systems?

I know xfs and reiser are both log based. Do we need to be concerned
about PostgreSQL performance on these file systems? I use BSD FFS with
soft updates here, so it doesn't affect me.

The "problem" with log based filesystems is that they most likely
do not know the consequences of a write so an fsync on a file may
require double writing to both the log and the "real" portion of
the disk. They can also exhibit the problem that an fsync may
cause all pending writes to require scheduling unless the log is
constructed on the fly rather than incrementally.

There was also the problem that was brought up recently that
certain versions (maybe all?) of Linux perform fsync() in a very
non-optimal manner, if the user is able to use the O_FSYNC option
rather than fsync he may see a performance increase.

But his guess is probably nearly as good as mine. :)

--
-Alfred Perlstein - [alfred@freebsd.org]
http://www.egr.unlv.edu/~slumos/on-netbsd.html

The "problem" with log based filesystems is that they most likely
do not know the consequences of a write so an fsync on a file may
require double writing to both the log and the "real" portion of
the disk. They can also exhibit the problem that an fsync may
cause all pending writes to require scheduling unless the log is
constructed on the fly rather than incrementally.

Yes, this double-writing is a problem. Suppose you have your WAL on a
separate drive. You can fsync() WAL with zero head movement. With a
log based file system, you need two head movements, so you have gone
from zero movements to two.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

* Bruce Momjian <pgman@candle.pha.pa.us> [010502 15:20] wrote:

The "problem" with log based filesystems is that they most likely
do not know the consequences of a write so an fsync on a file may
require double writing to both the log and the "real" portion of
the disk. They can also exhibit the problem that an fsync may
cause all pending writes to require scheduling unless the log is
constructed on the fly rather than incrementally.

Yes, this double-writing is a problem. Suppose you have your WAL on a
separate drive. You can fsync() WAL with zero head movement. With a
log based file system, you need two head movements, so you have gone
from zero movements to two.

It may be worse depending on how the filesystem actually does
journalling. I wonder if an fsync() may cause ALL pending
meta-data to be updated (even metadata not related to the
postgresql files).

Do you know if reiser or xfs have this problem?

--
-Alfred Perlstein - [alfred@freebsd.org]
Daemon News Magazine in your snail-mail! http://magazine.daemonnews.org/

Yes, this double-writing is a problem. Suppose you have your WAL on a
separate drive. You can fsync() WAL with zero head movement. With a
log based file system, you need two head movements, so you have gone
from zero movements to two.

It may be worse depending on how the filesystem actually does
journalling. I wonder if an fsync() may cause ALL pending
meta-data to be updated (even metadata not related to the
postgresql files).

Do you know if reiser or xfs have this problem?

I don't know, but the Linux user reported xfs was really slow.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:

I was talking to a Linux user yesterday, and he said that performance
using the xfs file system is pretty bad. He believes it has to do with
the fact that fsync() on log-based file systems requires more writes.

With a standard BSD/ext2 file system, WAL writes can stay on the same
cylinder to perform fsync. Is that true of log-based file systems?

I know xfs and reiser are both log based. Do we need to be concerned
about PostgreSQL performance on these file systems? I use BSD FFS with
soft updates here, so it doesn't affect me.

I did see poor performance on reiserfs, I have not as yet ventured into using
xfs.

I occurs to me that journalizing file systems will almost always be slower on
an application such as postgres. The journalizing file system is trying to
maintain data integrity for an application which is also trying to maintain
data integrity. There will always be extra work involved.

This behavior raises the question about file system usage in Postgres. Many
databases, such as Oracle, create table space files and operate directly on the
raw blocks, bypassing the file system altogether.

On one hand, Postgres is easy to use and maintain because it cooperates with
the native file system, on the other hand it incurs the overhead of whatever
silliness the file system wants to do.

I would bet it is a huge amount of work to use a "table space" system and no
one wants that. lol. However, it should be noted that a bit more control over
database layout would make some great performance improvements.

The ability to put indexes on a separate volume from data.
The ability to put different tables on different volumes.
And so on.

In the short term, I think poor performance on a journalizing file system is to
be expected, unless there is an IOCTL to tell the FS to leave the files alone
(and postgres calls it). A Linux HOWTO which informs people that certain file
systems will have performance issues and why should handle the problem.

Perhaps we can convince the Linux community to create a "dbfs" which is a
stripped down simple no nonsense file system designed for applications like
databases?

--
I'm not offering myself as an example; every life evolves by its own laws.
------------------------
http://www.mohawksoft.com

On Thu, 3 May 2001, mlw wrote:

I would bet it is a huge amount of work to use a "table space" system
and no one wants that.

From some stracing of 7.1, the most common syscall issued by
postgres is an lseek() to the end of the file, presumably to
find its length, which seems to happen up to about a dozen
times per (pgbench) transaction.

Tablespaces would solve this (not that lseek is a particularly
expensive operation, of course).

Perhaps we can convince the Linux community to create a "dbfs" which
is a stripped down simple no nonsense file system designed for
applications like databases?

Sync-metadata ext2 should be fine. Filesystems fsck pretty
quick when they contain only a few large files.

Otherwise, something like "smugfs" (now obsolete) might do.

Matthew.

Matthew Kirkwood <matthew@hairy.beasts.org> writes:

From some stracing of 7.1, the most common syscall issued by
postgres is an lseek() to the end of the file, presumably to
find its length, which seems to happen up to about a dozen
times per (pgbench) transaction.

Tablespaces would solve this (not that lseek is a particularly
expensive operation, of course).

No, they wouldn't; or at least they'd just create a different problem.
The reason for the lseek is that the file length may have changed since
the current backend last checked it. To avoid lseek we'd need some
shared data structure that maintains the current length of every active
table, which would be a nuisance to maintain and probably a source of
contention delays.

(Of course, such a data structure would just be the tip of the iceberg
of what we'd have to maintain for ourselves if we couldn't depend on the
kernel to do it for us. Reimplementing a filesystem doesn't strike me
as a profitable use of our time.)

regards, tom lane

I know xfs and reiser are both log based. Do we need to be concerned
about PostgreSQL performance on these file systems? I use BSD FFS with
soft updates here, so it doesn't affect me.

I did see poor performance on reiserfs, I have not as yet ventured into using
xfs.

I occurs to me that journalizing file systems will almost always be slower on
an application such as postgres. The journalizing file system is trying to
maintain data integrity for an application which is also trying to maintain
data integrity. There will always be extra work involved.

Yes, the problem is that extra work is required on PostgreSQL's part.
Log-based file systems make sure all the changes get onto the disk in an
orderly way, but I believe it can delay what gets written to the drive.
PostgreSQL wants to be sure all the data is on the disk, period.
Unfortunately, the _orderly_ part makes the _fsync_ part do more work.
By going from ext2 to a log-based file system, we are getting _farther_
from a raw device that if we just sayed with ext2.

ext2 has serious problems with corrupt file systems after a crash, so I
understand the need to move to another file system type. I have been
waitin for Linux to get a more modern file system. Unfortunately, the
new ones seem to be worse for PostgreSQL.

This behavior raises the question about file system usage in Postgres. Many
databases, such as Oracle, create table space files and operate directly on the
raw blocks, bypassing the file system altogether.

OK, we have considered this, but frankly, the new, modern file systems
like FFS/softupdates have i/o rates near raw speed, with all the
advantages a file system gives us. I believe most commercial dbs are
moving away from raw devices and toward file systems. In the old days
the SysV file system was pretty bad at i/o & fragmentation, so they used
raw devices.

The ability to put indexes on a separate volume from data.
The ability to put different tables on different volumes.
And so on.

We certainly need that, but raw devices would not make this any easier,
I think.

In the short term, I think poor performance on a journalizing file system is to
be expected, unless there is an IOCTL to tell the FS to leave the files alone
(and postgres calls it). A Linux HOWTO which informs people that certain file
systems will have performance issues and why should handle the problem.

Perhaps we can convince the Linux community to create a "dbfs" which is a
stripped down simple no nonsense file system designed for applications like
databases?

It could become a serious problem as people start using reiser/xfs for
their file systems and don't understand the performance problems. Even
more likely is that they will turn off fsync, thinking reiser doesn't
need it, when in fact, I think it does.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Matthew Kirkwood <matthew@hairy.beasts.org> writes:

From some stracing of 7.1, the most common syscall issued by
postgres is an lseek() to the end of the file, presumably to
find its length, which seems to happen up to about a dozen
times per (pgbench) transaction.

Tablespaces would solve this (not that lseek is a particularly
expensive operation, of course).

No, they wouldn't; or at least they'd just create a different problem.
The reason for the lseek is that the file length may have changed since
the current backend last checked it. To avoid lseek we'd need some
shared data structure that maintains the current length of every active
table, which would be a nuisance to maintain and probably a source of
contention delays.

Seems we should cache the file lengths somehow. Not sure how to do it
because our file system cache is local to each backend.

(Of course, such a data structure would just be the tip of the iceberg
of what we'd have to maintain for ourselves if we couldn't depend on the
kernel to do it for us. Reimplementing a filesystem doesn't strike me
as a profitable use of our time.)

Ditto. The database is complicated enough.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

This behavior raises the question about file system usage in Postgres. Many
databases, such as Oracle, create table space files and operate directly on the
raw blocks, bypassing the file system altogether.

OK, we have considered this, but frankly, the new, modern file systems
like FFS/softupdates have i/o rates near raw speed, with all the
advantages a file system gives us. I believe most commercial dbs are
moving away from raw devices and toward file systems. In the old days
the SysV file system was pretty bad at i/o & fragmentation, so they used
raw devices.

I'm starting to like the idea of raw FS for a few reasons:

1) Considering that postgresql now does WAL, the need for a logging FS
for the database doesn't seem as needed (is it needed at all?).

2) Given the fact that postgresql is trying to support many OSs,
depending on, for example, XFS on a linux system will cause many
problems. What about solaris? How about BSD? Etc.. Using raw db MAY be
easier than dealing with the problems that will arise from supporting
multiple filesystems.

That said, the ability to use the system's FS does have it's advantages
(backup, moving files, etc).

Just some thoughts..

- Brandon

b. palmer, bpalmer@crimelabs.net
pgp: www.crimelabs.net/bpalmer.pgp5

kernel to do it for us. Reimplementing a filesystem doesn't strike me
as a profitable use of our time.)

Ditto. The database is complicated enough.

Maybe some kind of recommendation would be a good thing. That is, if the
PostgreSQL community has enough knowledge.

A section in the docs that discusses various file systems, so people can make
an intelligent choice.

--
Kaare Rasmussen --Linux, spil,-- Tlf: 3816 2582
Kaki Data tshirts, merchandize Fax: 3816 2501
Howitzvej 75 �ben 14.00-18.00 Web: www.suse.dk
2000 Frederiksberg L�rdag 11.00-17.00 Email: kar@webline.dk

On Thu, 3 May 2001, mlw wrote:

This behavior raises the question about file system usage in Postgres. Many
databases, such as Oracle, create table space files and operate directly on the
raw blocks, bypassing the file system altogether.

On one hand, Postgres is easy to use and maintain because it cooperates with
the native file system, on the other hand it incurs the overhead of whatever
silliness the file system wants to do.

It is not *that* hard to write a 'postgresfs' but you have to look at
the problems it creates. One of the biggest problems facing sys admins of
large sites is that the Oracle/DB2/etc DBA, having created the
purpose-build database filesystem, has not allowed enough room for
growth. Like I said, a basic file system is not difficult, but volume
management tools and the maintenance of the whole thing is. Currently,
postgres administrators are not faced with such a problem.

There is, of course, the argument that pgfs need not been enforced. The
problem is that many people would probably use it so as to have a
'superior' installation. This then entails the problems above, creating
more work for core developers.

Gavin

Just put a note in the installation docs that the place where the database
is initialised to should be on a non-Reiser, non-XFS mount...

Chris

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org]On Behalf Of mlw
Sent: Thursday, 3 May 2001 8:09 PM
To: Bruce Momjian; Hackers List
Subject: [HACKERS] Re: New Linux xfs/reiser file systems

Bruce Momjian wrote:

I was talking to a Linux user yesterday, and he said that performance
using the xfs file system is pretty bad. He believes it has to do with
the fact that fsync() on log-based file systems requires more writes.

With a standard BSD/ext2 file system, WAL writes can stay on the same
cylinder to perform fsync. Is that true of log-based file systems?

I know xfs and reiser are both log based. Do we need to be concerned
about PostgreSQL performance on these file systems? I use BSD FFS with
soft updates here, so it doesn't affect me.

I did see poor performance on reiserfs, I have not as yet ventured into
using
xfs.

I occurs to me that journalizing file systems will almost always be slower
on
an application such as postgres. The journalizing file system is trying to
maintain data integrity for an application which is also trying to maintain
data integrity. There will always be extra work involved.

This behavior raises the question about file system usage in Postgres. Many
databases, such as Oracle, create table space files and operate directly on
the
raw blocks, bypassing the file system altogether.

On one hand, Postgres is easy to use and maintain because it cooperates with
the native file system, on the other hand it incurs the overhead of whatever
silliness the file system wants to do.

I would bet it is a huge amount of work to use a "table space" system and no
one wants that. lol. However, it should be noted that a bit more control
over
database layout would make some great performance improvements.

The ability to put indexes on a separate volume from data.
The ability to put different tables on different volumes.
And so on.

In the short term, I think poor performance on a journalizing file system is
to
be expected, unless there is an IOCTL to tell the FS to leave the files
alone
(and postgres calls it). A Linux HOWTO which informs people that certain
file
systems will have performance issues and why should handle the problem.

Perhaps we can convince the Linux community to create a "dbfs" which is a
stripped down simple no nonsense file system designed for applications like
databases?

--
I'm not offering myself as an example; every life evolves by its own laws.
------------------------
http://www.mohawksoft.com

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

There might be a problem, but if no one mentions it to the maintainers of
those
fs's, it will not get fixed...

Regards
John

Just put a note in the installation docs that the place where the database
is initialised to should be on a non-Reiser, non-XFS mount...

Sure, we can do that now. What do we do when these are the default file
systems for Linux? We can tell them to create other types of file
systems, but that is a pretty big hurdle. I wonder if it would be
easier to get reiser/xfs to make some modifications.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Well, arguably if you're setting up a database server then a reasonable DBA
should think about such things...

(My 2c)

Chris

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Friday, 4 May 2001 9:42 AM
To: Christopher Kings-Lynne
Cc: mlw; Hackers List
Subject: Re: [HACKERS] Re: New Linux xfs/reiser file systems

Just put a note in the installation docs that the place where the database
is initialised to should be on a non-Reiser, non-XFS mount...

Sure, we can do that now. What do we do when these are the default file
systems for Linux? We can tell them to create other types of file
systems, but that is a pretty big hurdle. I wonder if it would be
easier to get reiser/xfs to make some modifications.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Well, arguably if you're setting up a database server then a reasonable DBA
should think about such things...

Yes, but people have trouble installing PostgreSQL. I can't imagine
walking them through a newfs.

(My 2c)

Chris

-----Original Message-----
From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
Sent: Friday, 4 May 2001 9:42 AM
To: Christopher Kings-Lynne
Cc: mlw; Hackers List
Subject: Re: [HACKERS] Re: New Linux xfs/reiser file systems

Just put a note in the installation docs that the place where the database
is initialised to should be on a non-Reiser, non-XFS mount...

Sure, we can do that now. What do we do when these are the default file
systems for Linux? We can tell them to create other types of file
systems, but that is a pretty big hurdle. I wonder if it would be
easier to get reiser/xfs to make some modifications.

--
Bruce Momjian                        |  http://candle.pha.pa.us
pgman@candle.pha.pa.us               |  (610) 853-3000
+  If your life is a hard drive,     |  830 Blythe Avenue
+  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:

Just put a note in the installation docs that the place where the database
is initialised to should be on a non-Reiser, non-XFS mount...

Sure, we can do that now. What do we do when these are the default file
systems for Linux? We can tell them to create other types of file
systems, but that is a pretty big hurdle. I wonder if it would be
easier to get reiser/xfs to make some modifications.

I have looked at Reiser, and I don't think it is a file system suited for very
large files, or applications such as postgres. The Linux crowd should lobby
against any such trend. It is ok for many moderately small files. ReiserFS
would be great for a cddb server, but poor for a database box.

XFS is a real big file system project, I'd bet that there are file properties
or management tools to tell it to leave directories and files alone. They
should have addressed that years ago.

One last mention..

Having better control over WHERE various files in a database are located can
make it easier to deal with these things.

Just a thought. ;-)

--
I'm not offering myself as an example; every life evolves by its own laws.
------------------------
http://www.mohawksoft.com

Just put a note in the installation docs that the place where the

database

is initialised to should be on a non-Reiser, non-XFS mount...

Sure, we can do that now.

I still think this is not necessarily the right approach either. One
major purpose of using a journaling fs is for fast boot up time after
crash. If you have a 100 GB database you may wish to have the data
on XFS. I do think that the WAL log should be on a separate disk and
on a non-journaling fs for performance.

Best Regards,
Carl Garland

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com