Server Performance

On Tue, Mar 31, 2009 at 3:37 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Hi

Been having interesting times with an IBM x3650 with 8 15k RPM 73GB drives
in RAID 10 and a ServRAID 8K controller with Write-Back cache enabled
(battery installed and working).  Currently getting a pgbench score of 4.7
transactions per second!  After playing with the postgresql configuration
file, I'm certain that this is not a postgresql problem.  I have tried two
different Linux distro's upon the server both with the same problems.  I'm
fairly certain that this is a problem with the hardware configuration /
setup, however I'm still waiting for IBM to contact me!

Initially I started with the OS on a RAID 1 array and a 6 drive RAID 10
array for postgresql.  With this setup I got 3tps, altering the RAID
configuration to a single 8 drive array, running both the OS and postgresql.
 I was able to reach 700tps, however after upgrading to the latest RAID
controller firmware this has now fallen back to 4tps.

Benchmarking another server I have access to, 4 15k 73GB RPM disks with a
Dell Perc 5/i controller. I consistently get a pgbench score of 1400tps.
 Therefore taking a linear extrapolation I expect the IBM x3650 to manage
~3000tps.  Additionally my Laptop with a 5400 RPM sata disk was able to
score ~200tps.

SNIP

What SAS RAID controllers are people using?

What RAID configurations are people using?

What SAS RAID controllers would anyone recommend purchasing?

I am using an Areca 1680 series controller. 16 SAS 15k5 disks. 2
RAID-1, 12 RAID-10, 2 hot spares. 512Meg bbu. RHEL 5.2 I can sustain
around 3000 tps with pgbench and 30 minute runs.

I'd call IBM and ask them to come pick up their boat anchors.

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:

On Tue, Mar 31, 2009 at 3:37 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Hi

Been having interesting times with an IBM x3650 with 8 15k RPM 73GB

drives

in RAID 10 and a ServRAID 8K controller with Write-Back cache enabled
(battery installed and working). Currently getting a pgbench score of

4.7

transactions per second! After playing with the postgresql

configuration

file, I'm certain that this is not a postgresql problem. I have tried

two

different Linux distro's upon the server both with the same problems.

I'm

fairly certain that this is a problem with the hardware configuration

/

setup, however I'm still waiting for IBM to contact me!

Initially I started with the OS on a RAID 1 array and a 6 drive RAID

10

array for postgresql. With this setup I got 3tps, altering the RAID
configuration to a single 8 drive array, running both the OS and

postgresql.

I was able to reach 700tps, however after upgrading to the latest

RAID

controller firmware this has now fallen back to 4tps.

Benchmarking another server I have access to, 4 15k 73GB RPM disks

with a

Dell Perc 5/i controller. I consistently get a pgbench score of

1400tps.

Therefore taking a linear extrapolation I expect the IBM x3650 to

manage

~3000tps. Additionally my Laptop with a 5400 RPM sata disk was able

to

score ~200tps.

SNIP

What SAS RAID controllers are people using?

What RAID configurations are people using?

What SAS RAID controllers would anyone recommend purchasing?

I am using an Areca 1680 series controller. 16 SAS 15k5 disks. 2
RAID-1, 12 RAID-10, 2 hot spares. 512Meg bbu. RHEL 5.2 I can sustain
around 3000 tps with pgbench and 30 minute runs.

Thanks for the Info.

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Chris Ellis
******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************

On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:

On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Yes I will be, I never really did trust IBM and I certainly don't now!

I just need to choose the correct RAID card now, good performance at the
right price.

Chris Ellis

******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************

On Tue, Mar 31, 2009 at 8:54 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:

On Tue, Mar 31, 2009 at 8:21 AM,  <Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Yes I will be, I never really did trust IBM and I certainly don't now!

I just need to choose the correct RAID card now, good performance at the
right price.

The older Areca and 3ware cards are pretty reasonably priced (Areca
12xx series, 3ware 95xx series) and the newer ones aren't too badly
priced for the performance you get. What's your budget for the RAID
card?

Note that you might also be able to get away with the crappy RAID card
they sold you if you put it into jbod mode and use software RAID
instead. Just guessing.

On Tue, 31 Mar 2009, Chris.Ellis@shropshire.gov.uk wrote:

Been having interesting times with an IBM x3650 with 8 15k RPM 73GB
drives in RAID 10 and a ServRAID 8K controller with Write-Back cache
enabled (battery installed and working). ï¿œCurrently getting a pgbench
score of 4.7 transactions per second!

pgbench is a very high level test of your system. It can tell you when a
system is doing well, but it's almost useless for figuring out what's
wrong if there's a problem.

You shouldn't run pgbench until you've first done a lower-level benchmark
such as bonnie++ on the hardware. That should give you a better idea
what's going on here, and if the badness shows up there it will be much
easier to get someone at IBM to pay attention too.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

From pgsql-general-owner@postgresql.org Tue Mar 31 18:21:52 2009

Received: from localhost (unknown [200.46.208.211])
by mail.postgresql.org (Postfix) with ESMTP id 16744635157
for <pgsql-general-postgresql.org@mail.postgresql.org>; Tue, 31 Mar 2009 17:10:53 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024)
with ESMTP id 02444-09
for <pgsql-general-postgresql.org@mail.postgresql.org>;
Tue, 31 Mar 2009 17:10:47 -0300 (ADT)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from mail-gx0-f176.google.com (mail-gx0-f176.google.com [209.85.217.176])
by mail.postgresql.org (Postfix) with ESMTP id 935D9635126
for <pgsql-general@postgresql.org>; Tue, 31 Mar 2009 17:10:51 -0300 (ADT)
Received: by gxk24 with SMTP id 24so7650339gxk.19
for <pgsql-general@postgresql.org>; Tue, 31 Mar 2009 13:10:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=gamma;
h=domainkey-signature:mime-version:date:received:message-id:subject
:from:to:content-type;
bh=xKXngZmZXW2Ri47mTMnc0AEUTsdNmYzG6hMO0q5bhmk=;
b=xJgxe1KM/0NKTnCnkcxhVARvsMXFxYeVx6caQLcIhi/xyt6xGaKwLOql2dOQvvMjky
89PJ42P48MRibHjmGDuXYD7EpDGHoz3UmGwCMQkJtYtki4fOYgdpYHJ0xkyOvmbje1xD
QeDOzpYZfKQ/LbRbbnAs5IjEhawtW5thQ06B4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
d=gmail.com; s=gamma;
h=mime-version:date:message-id:subject:from:to:content-type;
b=TdaycuYT5oVAbLVI/ONdT1p/pdFQ34mM0Zs13bi3wAvHlW5k4/0vxELvz7uz7HoJyO
dzVck7i03QCVzEbckPBw8g+kU1sBchlbRD33cPWdqHB3vX/3NoQh27c8nfT47UQB8QaW
2MGdFCVxy/o1BxJp42MK1hJu6PlUP+wNQmcRY=
MIME-Version: 1.0
Date: Tue, 31 Mar 2009 16:10:35 -0400
Received: by 10.150.156.9 with SMTP id d9mr13286152ybe.182.1238530250515; Tue,
31 Mar 2009 13:10:50 -0700 (PDT)
Message-ID: <aaff70a90903311310u429e8722y420540da40d36312@mail.gmail.com>
Subject: =?windows-1252?Q?ERROR=3A_XX001=3A_could_not_read_block_2354_of_relation?=
=?windows-1252?Q?=85?=
From: Patrick Desjardins <mrdesjardins@gmail.com>
To: pgsql-general@postgresql.org
Content-Type: multipart/alternative; boundary=000e0cd6b21aa02e3104666fcbe9
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=0.001 tagged_above=0 required=5
tests=HTML_MESSAGE=0.001
X-Spam-Level:
X-Archive-Number: 200903/1362
X-Sequence-Number: 145948

--000e0cd6b21aa02e3104666fcbe9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

I have the error ERROR: XX001: could not read block 2354 of relation
1663/17633/17925: read only 0 of 8192 bytes, but only sometime, when trying
to Insert data into a table. I would say that 99% of Insert works and 100%
of read works. This is only happenning since few weeks. I have done Vaccum
Analyze without any success (the vaccum take 65 minutes but nothing is
fixed, still have the ERROR XXX001). I have this error in the backend
application and when I do some Insert Query in the PgAdmin tool.

What can I do to fix that problem?

Thank you,

--000e0cd6b21aa02e3104666fcbe9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<br><br>I have the error ERROR: XX001: could not read block 2354 of rela=
tion 1663/17633/17925: read only 0 of 8192 bytes, but only sometime, when t=
rying to Insert data into a table. I would say that 99% of Insert works and=
100% of read works. This is only happenning since few weeks. I have done V=
accum Analyze without any success (the vaccum take 65 minutes but nothing i=
s fixed, still have the ERROR XXX001). I have this error in the backend app=
lication and when I do some Insert Query in the PgAdmin tool.<br>

<br>What can I do to fix that problem?<br><br>Thank you,<br>

--000e0cd6b21aa02e3104666fcbe9--

Chris.Ellis@shropshire.gov.uk wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:

On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Yes I will be, I never really did trust IBM and I certainly don't now!

I just need to choose the correct RAID card now, good performance at the
right price.

you are jumping to conclusions too quickly - while the 8k is not the
worlds fastest raid card available it is really not (that) bad at all.
we have plenty of x3650 in production and last time I tested I was
easily able to get >>2000tps even on an untuned postgresql install and
with fwer disks.

So I really think you are looking at another problem here (be it
defective hardware or a driver/OS level issue).

is your SLES10 install updated to the latest patch levels available and
are you running the recommended driver version for that version of SLES?

Stefan

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote on 01/04/2009
06:53:07:

Chris.Ellis@shropshire.gov.uk wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:

On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk>

wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009

15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Yes I will be, I never really did trust IBM and I certainly don't now!

I just need to choose the correct RAID card now, good performance at

the

right price.

you are jumping to conclusions too quickly - while the 8k is not the
worlds fastest raid card available it is really not (that) bad at all.
we have plenty of x3650 in production and last time I tested I was
easily able to get >>2000tps even on an untuned postgresql install and
with fwer disks.

Could you provide any more information upon your configurations if
possible, please?

So I really think you are looking at another problem here (be it
defective hardware or a driver/OS level issue).

Hardware is always a possiblity, finally managed to get hold of IBM too.
I have tried two different Linux distro's, with different kernels, My
current Mandriva test using a fairly upto date kernel.
I may try a custom kernel.

is your SLES10 install updated to the latest patch levels available and
are you running the recommended driver version for that version of SLES?

Yes

Stefan

******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************

Chris.Ellis@shropshire.gov.uk wrote:

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote on 01/04/2009 06:53:07:

Chris.Ellis@shropshire.gov.uk wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:

On Tue, Mar 31, 2009 at 8:21 AM,

<Chris.Ellis@shropshire.gov.uk> wrote:

Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009

15:16:01:

I'd call IBM and ask them to come pick up their boat anchors.

My sentiments exactly, unfortunately, I seem stuck with them :(

Can you at least source your own RAID controllers?

Yes I will be, I never really did trust IBM and I certainly don't now!

I just need to choose the correct RAID card now, good performance

at the

right price.

you are jumping to conclusions too quickly - while the 8k is not the
worlds fastest raid card available it is really not (that) bad at all.
we have plenty of x3650 in production and last time I tested I was
easily able to get >>2000tps even on an untuned postgresql install and
with fwer disks.

Could you provide any more information upon your configurations if
possible, please?

x3650, dual quadcore Xeon 5430. Servraid 8k with 256MB-BBWC and likely
RAID6 during that testing. OS was/is debian etch/amd64. Don't have the
exact (pgbench) test parameters handy anymore though...

So I really think you are looking at another problem here (be it
defective hardware or a driver/OS level issue).

Hardware is always a possiblity, finally managed to get hold of IBM too.
I have tried two different Linux distro's, with different kernels, My
current Mandriva test using a fairly upto date kernel.
I may try a custom kernel.

also test with different IO schedulers(especially deadline and noop).

Stefan

On Wed, Apr 1, 2009 at 2:01 PM, Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:

also test with different IO schedulers(especially deadline and noop).

But wasn't the OP getting something like 6 tps? I mean, something is
so horrifically wrong a simple change like the io scheduler can't hope
to fix things.