PostgreSQL benchmarked on XFS vs ZFS vs btrfs vs ext4

On 09/13/2011 08:15 PM, Toby Corkindale wrote:

Hi,
Some months ago, I ran some (probably naive) benchmarks looking at how pgbench performed on an identical system with differing filesystems. (on Linux).

Since then the kernel-level version of ZFS became usable, and there have been improvements to btrfs, and no doubt various updates in the Linux kernel and PostgreSQL that should help performance.

I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again.

The latter combination showed a considerable performance improvement overall - although I didn't investigate to find out whether this was due to kernel improvements, postgres improvements, or virtio improvements.

The results are measured in transactions-per-second, with higher numbers being better.

Results:

ext4 (data=writeback,relatime):
natty: 248
oneiric: 297

ext4 (data=writeback,relatime,nobarrier):
natty: didn't test
oneiric: 1409

XFS (relatime):
natty: didn't test
oneiric: 171

btrfs (relatime):
natty: 61.5
oneiric: 91

btrfs (relatime,nodatacow):
natty: didn't test
oneiric: 128

ZFS (defaults):
natty: 171
oneiric: 996

Conclusion:
Last time I ran these tests, xfs and ext4 pulled very similar results, and both were miles ahead of btrfs. This time around, ext4 has managed to get a significantly faster result than xfs.

However we have a new contender - ZFS performed *extremely* well on the latest Ubuntu setup - achieving triple the performance of regular ext4!
I'm not sure how it achieved this, and whether we're losing some kind of data protection (eg. like the "barrier" options in XFS and ext4).
If ext4 has barriers disabled, it surpasses even ZFSs high score.

Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I can't explain this. Any ideas?

Cheers,
Toby

Did you test unplugging the power cable in the middle of a test to see which would come back up?

-Andy

On 14/09/11 12:56, Andy Colson wrote:

On 09/13/2011 08:15 PM, Toby Corkindale wrote:

Hi,
Some months ago, I ran some (probably naive) benchmarks looking at how
pgbench performed on an identical system with differing filesystems.
(on Linux).

[snip]

Did you test unplugging the power cable in the middle of a test to see
which would come back up?

Heh, no, but it'd be interesting to see..

I wonder if turning a virtual machine off has the same effect?

On 14/09/2011 09:30, Toby Corkindale wrote:

On 14/09/11 12:56, Andy Colson wrote:

On 09/13/2011 08:15 PM, Toby Corkindale wrote:

Hi,
Some months ago, I ran some (probably naive) benchmarks looking at how
pgbench performed on an identical system with differing filesystems.
(on Linux).

[snip]

Did you test unplugging the power cable in the middle of a test to see
which would come back up?

Heh, no, but it'd be interesting to see..

I wonder if turning a virtual machine off has the same effect?

No, never the exact same effect.

There are two things to consider: if/when/how the OS flushes the data to
the hardware and if/when/how the hardware flushes the data to physical
storage. You can simulate only the failure of the first part with a
virtual machine, but not the second (unless you bring down the VM host...).

Can you go into some more detail on how you set up ZFS on these systems?

Robert Treat
conjecture: xzilla.net
consulting: omniti.com

Dne 14.9.2011 03:15, Toby Corkindale napsal(a):

Hi,
Some months ago, I ran some (probably naive) benchmarks looking at how
pgbench performed on an identical system with differing filesystems. (on
Linux).

Since then the kernel-level version of ZFS became usable, and there have
been improvements to btrfs, and no doubt various updates in the Linux
kernel and PostgreSQL that should help performance.

I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the
system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again.

The latter combination showed a considerable performance improvement
overall - although I didn't investigate to find out whether this was due
to kernel improvements, postgres improvements, or virtio improvements.

The results are measured in transactions-per-second, with higher numbers
being better.

Results:

ext4 (data=writeback,relatime):
natty: 248
oneiric: 297

ext4 (data=writeback,relatime,nobarrier):
natty: didn't test
oneiric: 1409

XFS (relatime):
natty: didn't test
oneiric: 171

btrfs (relatime):
natty: 61.5
oneiric: 91

btrfs (relatime,nodatacow):
natty: didn't test
oneiric: 128

ZFS (defaults):
natty: 171
oneiric: 996

Conclusion:
Last time I ran these tests, xfs and ext4 pulled very similar results,
and both were miles ahead of btrfs. This time around, ext4 has managed
to get a significantly faster result than xfs.

However we have a new contender - ZFS performed *extremely* well on the
latest Ubuntu setup - achieving triple the performance of regular ext4!
I'm not sure how it achieved this, and whether we're losing some kind of
data protection (eg. like the "barrier" options in XFS and ext4).
If ext4 has barriers disabled, it surpasses even ZFSs high score.

Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I
can't explain this. Any ideas?

Those results look very suspisicous I guess. Unless we know all the
details about the fs options, pgbench runs and system (drives, cache,
controller) it's very difficult to say anything.

For example the ZFS results seem really strange - I've done a quite
thorough benchmark of file systems last month (http://bit.ly/oEdkSH) and
I really don't think the differences can be that huge under the same
conditions - increase from 171 to 996 is ridiculous.

The only explanation I can come up with is that oneiric uses
significantly weaker ZFS options (e.g. disabled write barriers) and
there's a controller with write cache.

But that also means you should not compare ZFS to plain ext4 (297 tps)
but to ext4 with write barriers disabled (1409 tps). That changes the
results a bit, because instead of being 3x faster, ZFS is about 30% slower.

Tomas

On 15/09/11 02:46, Robert Treat wrote:

Can you go into some more detail on how you set up ZFS on these systems?

I'm afraid my knowledge of ZFS is rather weak compared to the other
filesystems - all I really did was zpool create followed by zfs create,
using all the defaults.

The zpool was created against an LVM logical volume (which was the same
one used for all the filesystems measured in the tests). That LV was
itself part of a volume group that was striped over three disks (Western
Digital WD1003FBYX).

I'm happy to re-run the benchmark with different ZFS options if you can
suggest some. Unfortunately I can't easily bypass the LVM layer here, as
the disks involved are fully committed to the volume group and I don't
want to rebuild the machine.

Cheers,
Toby

On Tue, Sep 13, 2011 at 9:15 PM, Toby Corkindale
<toby.corkindale@strategicdata.com.au> wrote:

However we have a new contender - ZFS performed *extremely* well on the
latest Ubuntu setup - achieving triple the performance of regular ext4!

Did you do any tuning to ZFS? There are many tweaks to it, like
putting a cache disk in front of it, or moving the logs to SSD and
such. I haven't run any produciton DBs on ZFS yet, but it sure is
tempting. The speed penalty for the features it gives you (snapshots,
robust against power fails, etc.) is worth the tradeoff.

On Wed, Sep 14, 2011 at 11:18 PM, Toby Corkindale
<toby.corkindale@strategicdata.com.au> wrote:

The zpool was created against an LVM logical volume (which was the same one
used for all the filesystems measured in the tests). That LV was itself part
of a volume group that was striped over three disks (Western Digital
WD1003FBYX).

So you're throwing in the complication of a logical volume manager
interfering with zfs. If you put ZFS on three separate drives, it has
a better chance of optimizing its operations.

On 17/09/11 00:09, Vick Khera wrote:

On Tue, Sep 13, 2011 at 9:15 PM, Toby Corkindale
<toby.corkindale@strategicdata.com.au> wrote:

However we have a new contender - ZFS performed *extremely* well on the
latest Ubuntu setup - achieving triple the performance of regular ext4!

Did you do any tuning to ZFS? There are many tweaks to it, like
putting a cache disk in front of it, or moving the logs to SSD and
such. I haven't run any produciton DBs on ZFS yet, but it sure is
tempting. The speed penalty for the features it gives you (snapshots,
robust against power fails, etc.) is worth the tradeoff.

No, I didn't do that kind of tuning - agreed, it'd improve the
performance. But then putting an SSD in the mix and storing journals on
it would have improved the performance of XFS and ext4 as well..

I'll re-run the tests again in the future, no doubt, and hopefully I'll
have a spare SSD by then. Also maybe I'll have learnt more about ZFS;
I'm a bit of a noob at the moment.

I agree that ZFS does seem to offer some rather nice features though!
I'm tempted to start using it on my personal server now; although I'll
be leaving it for some time before considering using it in production at
work.

Cheers,
Toby