Disk I/O Question

On 11/10/2015 06:34 AM, teeeebro wrote:

Hello - I am new to this forum and know nothing about PostgreSQL. I have
managed Microsoft SQL for 15 years (I know ....)

We have a third party application that uses PostgreSQL for the database
engine. Performance writing to our Dell Compellent SAN is lacking.

What version of Postgres?

What is the writing doing?

What are the performance numbers?

What I've read about PostgreSQL is that it's a single-thread disk I/O system
where only one read/write operation executes at a time.

Can you tell me if this is correct?

Thanks!!
Terry

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

2015-11-10 15:34 GMT+01:00 teeeebro <terryb@esi.net>:

Hello - I am new to this forum and know nothing about PostgreSQL. I have
managed Microsoft SQL for 15 years (I know ....)

We have a third party application that uses PostgreSQL for the database
engine. Performance writing to our Dell Compellent SAN is lacking.

What I've read about PostgreSQL is that it's a single-thread disk I/O
system
where only one read/write operation executes at a time.

it is 100% isn't true - any client can read data, writer/wal-writer,
autovacum can write

Regards

Pavel

On 10 November 2015 at 14:34, teeeebro <terryb@esi.net> wrote:

We have a third party application that uses PostgreSQL for the database
engine. Performance writing to our Dell Compellent SAN is lacking.

What I've read about PostgreSQL is that it's a single-thread disk I/O
system
where only one read/write operation executes at a time.

Can you tell me if this is correct?

​
There's a similar thread here that might be worth a read:

http://postgresql.nabble.com/concurrent-IO-in-postgres-td3316768.html​

​Geoff​

On 11/10/15 8:34 AM, teeeebro wrote:

What I've read about PostgreSQL is that it's a single-thread disk I/O system
where only one read/write operation executes at a time.

Definitely false. However...

Other databases support asynchronous IO, where the database tries to
issue IO requests before they're needed and let the kernel notify when
the IO is complete. Postgres does some other things in this area, but
it's not the same. The end result is that if there's much latency in
your IO subsystem at all then performance is really going to take a hit.
I've specifically seen this on iSCSI. The good news is that if you have
enough concurrent queries you can still get pretty close to maximum
theoretical throughput, so for common OLTP workloads you should still be
able to scale pretty well.

I'd suggest setting effective_io_concurrency to something > 1. That will
allow some operations to attempt pre-fetching data.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

The third party application is locked down and we do not have root access to
the server. We have reached out to them for the version number of
PostgreSQL.

The application server is a virtual machine (RedHat 6) running on VMware
5.5. This is a stand-alone host server running only the third party
application. This application is used to monitor just over 5000 network
devices (routers, switches, firewalls, servers, etc.). It receives a LOT of
information and writes it to disk.

The host server has 2 local 15k drives mirrored. The host server is also
attached to the SAN via 8GB fiber channel.

For testing we did a base installation of Centos 6 and ran a command line
test provided by the application vendor.
*rm -f /tmp/foo && sync && dd if=/dev/zero of=/tmp/foo bs=4k count=12k
oflag=dsync*

They said we should be see between 7 and 10 MB/Sec.

Writing to the SAN we get around 1.7 MB/Sec.
Writing to the local drives we get around 11.3 MB/Sec.

Just a note: We have a 4 host server cluster of VMware servers running off
the SAN. There are multiple MSSQL servers running in this cluster as well
as web servers, etc. etc. About 45 servers in all. We have zero
performance issues with any of these.

Thoughts?

Thanks again!!
Terry

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873398.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 11/10/2015 08:23 AM, teeeebro wrote:

The third party application is locked down and we do not have root access to
the server. We have reached out to them for the version number of
PostgreSQL.

The application server is a virtual machine (RedHat 6) running on VMware
5.5. This is a stand-alone host server running only the third party
application. This application is used to monitor just over 5000 network
devices (routers, switches, firewalls, servers, etc.). It receives a LOT of
information and writes it to disk.

The host server has 2 local 15k drives mirrored. The host server is also
attached to the SAN via 8GB fiber channel.

For testing we did a base installation of Centos 6 and ran a command line
test provided by the application vendor.
*rm -f /tmp/foo && sync && dd if=/dev/zero of=/tmp/foo bs=4k count=12k
oflag=dsync*

They said we should be see between 7 and 10 MB/Sec.

Writing to the SAN we get around 1.7 MB/Sec.
Writing to the local drives we get around 11.3 MB/Sec.

Not sure what the command line test has to do with Postgres?

The numbers you report, are they from the script, Postgres or some
combination?

More to the point where does Postgres enter into this?

Just a note: We have a 4 host server cluster of VMware servers running off
the SAN. There are multiple MSSQL servers running in this cluster as well
as web servers, etc. etc. About 45 servers in all. We have zero
performance issues with any of these.

Thoughts?

Thanks again!!
Terry

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873398.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

På tirsdag 10. november 2015 kl. 17:23:15, skrev teeeebro <terryb@esi.net
<mailto:terryb@esi.net>>:
The third party application is locked down and we do not have root access to
the server.  We have reached out to them for the version number of
PostgreSQL.

The application server is a virtual machine (RedHat 6) running on VMware
5.5.  This is a stand-alone host server running only the third party
application.  This application is used to monitor just over 5000 network
devices (routers, switches, firewalls, servers, etc.).  It receives a LOT of
information and writes it to disk.

The host server has 2 local 15k drives mirrored.  The host server is also
attached to the SAN via 8GB fiber channel.

For testing we did a base installation of Centos 6 and ran a command line
test provided by the application vendor.
 *rm -f /tmp/foo && sync && dd if=/dev/zero of=/tmp/foo bs=4k count=12k
oflag=dsync*

They said we should be see between 7 and 10 MB/Sec.

Writing to the SAN we get around 1.7 MB/Sec. 
Writing to the local drives we get around 11.3 MB/Sec.

Just a note: We have a 4 host server cluster of VMware servers running off
the SAN.  There are multiple MSSQL servers running in this cluster as well
as web servers, etc. etc.  About 45 servers in all.  We have zero
performance issues with any of these.

Thoughts?

Thanks again!!
Terry
 
 
We used to host our solution on VMWare. We started to experience severe
performance-issues which turned out to be when we hit the SAN's 8GB disk-cache.
We got ~1|MB/Sec, which is bad.
 
The command we used to test was:
 
iozone  -r  8  -i  0  -i  2  -s  10000000
 
PG reads/writes using 8K-pages and it's important to test on >8GB, hence "-s
10000000"
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com&gt;
<https://www.visena.com&gt;

Fair question

Let me elaborate. The command was simply to test the speed at which data
can be written to disk.

Postgre comes into play with the monitoring application itself. Information
pertaining to the 5000+ devices being monitored is sent to the monitoring
application and written to the Postgre database.

The issue we're having is that around 10% (or so) of the data received is
being discarded because according to the vendor (SolarWinds) it cannot be
written to disk fast enough. Simply too much data coming in and nowhere to
go so it gets dropped.

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873425.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 11/10/2015 10:42 AM, teeeebro wrote:

Fair question

Let me elaborate. The command was simply to test the speed at which data
can be written to disk.

Postgre comes into play with the monitoring application itself. Information
pertaining to the 5000+ devices being monitored is sent to the monitoring
application and written to the Postgre database.

The issue we're having is that around 10% (or so) of the data received is
being discarded because according to the vendor (SolarWinds) it cannot be
written to disk fast enough. Simply too much data coming in and nowhere to
go so it gets dropped.

So SolarWinds is the third party/monitoring application?

If not what application is using Postgres?

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873425.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

The application is called Nable and is owned by Solarwinds

http://www.n-able.com

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873428.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 11/10/2015 10:58 AM, teeeebro wrote:

The application is called Nable and is owned by Solarwinds

http://www.n-able.com

It is not uncommon for DAS to be much faster than NAS/SAN.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564
PostgreSQL Centered full stack support, consulting and development.
New rule for social situations: "If you think to yourself not even
JD would say this..." Stop and shut your mouth. It's going to be bad.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 11/10/2015 10:58 AM, teeeebro wrote:

The application is called Nable and is owned by Solarwinds

http://www.n-able.com

Is it the application that is using Postgres?

What would really help is some sort of schematic of how data is flowing
through the system and exactly where the hangup is.

--
View this message in context: http://postgresql.nabble.com/Disk-I-O-Question-tp5873364p5873428.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general