Re: open items for 9.4

On 2014-09-29 11:50:19 -0400, Robert Haas wrote:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

I vote for keeping it.

- TAP tests still have some pretty severe problems

Some of these issues have been fixed; but maybe not all.

If there are, they don't seem blockers to me personally.

- autovacuum scheduling starvation and frenzy

This doesn't seem to be a new problem, so I don't think we should
consider it a stop-ship issue.

+1.

- pg_dump fails with --if-exists and blobs

This looks like a 9.4 regression.

Alvaro, IIRC you were looking at this one?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Sep 29, 2014 at 5:53 PM, Andres Freund <andres@2ndquadrant.com> wrote:

On 2014-09-29 11:50:19 -0400, Robert Haas wrote:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

I vote for keeping it.

No real preference. But ISTM that if we're uncertain, it's probably a
good idea to keep them.

- TAP tests still have some pretty severe problems

Some of these issues have been fixed; but maybe not all.

If there are, they don't seem blockers to me personally.

+1. Good to have, but not release blockers.

- autovacuum scheduling starvation and frenzy

This doesn't seem to be a new problem, so I don't think we should
consider it a stop-ship issue.

+1.

+1.

- pg_dump fails with --if-exists and blobs

This looks like a 9.4 regression.

Alvaro, IIRC you were looking at this one?

This does seem to be the "major" release stopper, other than the json stuff.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund wrote:

On 2014-09-29 11:50:19 -0400, Robert Haas wrote:

- pg_dump fails with --if-exists and blobs

This looks like a 9.4 regression.

Alvaro, IIRC you were looking at this one?

I am.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Dne 29.9.2014 18:00 "Magnus Hagander" <magnus@hagander.net> napsal(a):

On Mon, Sep 29, 2014 at 5:53 PM, Andres Freund <andres@2ndquadrant.com>

wrote:

On 2014-09-29 11:50:19 -0400, Robert Haas wrote:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

I vote for keeping it.

No real preference. But ISTM that if we're uncertain, it's probably a
good idea to keep them.

- TAP tests still have some pretty severe problems

Some of these issues have been fixed; but maybe not all.

If there are, they don't seem blockers to me personally.

+1. Good to have, but not release blockers.

- autovacuum scheduling starvation and frenzy

This doesn't seem to be a new problem, so I don't think we should
consider it a stop-ship issue.

+1.

+1.

- pg_dump fails with --if-exists and blobs

This looks like a 9.4 regression.

Alvaro, IIRC you were looking at this one?

This does seem to be the "major" release stopper, other than the json

stuff.

I sent two variants of fix month ago. It should be somewhere in mailing
list archive

On 09/29/2014 08:53 AM, Andres Freund wrote:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

I vote for keeping it.

Time for me to put on my "we have too many darned GUCs" curmudgeon hat.

1. What does it do?
2. Are there reasons why users would want to change it from the default?
3. Can we explain those reasons in the form of documentation?

IMHO, if the answers to 2 or 3 are "no", then we shouldn't have a GUC.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-09-29 11:28:07 -0700, Josh Berkus wrote:

On 09/29/2014 08:53 AM, Andres Freund wrote:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

I vote for keeping it.

Time for me to put on my "we have too many darned GUCs" curmudgeon hat.

1. What does it do?

Change scalability in write heavy loads.

2. Are there reasons why users would want to change it from the
default?

Yes, to improve scalability.

3. Can we explain those reasons in the form of documentation?

Yes. Try and benchmark it. It'll be hardware and workload dependant.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

The lack of any documentation for the GUC (neither in config.sgml or
postgresql.conf.sample) suggests very very strongly that it was not
meant to be shipped. If we don't remove it I will certainly insist
that it be documented adequately.

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, Sep 29, 2014 at 2:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

The lack of any documentation for the GUC (neither in config.sgml or
postgresql.conf.sample) suggests very very strongly that it was not
meant to be shipped. If we don't remove it I will certainly insist
that it be documented adequately.

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

+1.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-09-29 14:44:42 -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

The lack of any documentation for the GUC (neither in config.sgml or
postgresql.conf.sample) suggests very very strongly that it was not
meant to be shipped.

That's definitely the case.

If we don't remove it I will certainly insist that it be documented
adequately.

Definitely.

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

That right now it can make quite noticeable differences in
scalability. And we have not much data to justify 8 as the default. And
that's surely not going to change if we require recompiles.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund <andres@2ndquadrant.com> writes:

On 2014-09-29 14:44:42 -0400, Tom Lane wrote:

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

That right now it can make quite noticeable differences in
scalability. And we have not much data to justify 8 as the default. And
that's surely not going to change if we require recompiles.

I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-09-29 16:16:38 -0400, Tom Lane wrote:

Andres Freund <andres@2ndquadrant.com> writes:

On 2014-09-29 14:44:42 -0400, Tom Lane wrote:

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

That right now it can make quite noticeable differences in
scalability. And we have not much data to justify 8 as the default. And
that's surely not going to change if we require recompiles.

I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".

Because that'd be horrible performancewise on a system with many
wal_buffers. There's several operations where all locks are checked in
sequence (to see whether there's any stragglers that need to finish
inserting) and even some where they're acquired concurrently (e.g. for
xlog switch, checkpoint and such).

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund <andres@2ndquadrant.com> writes:

On 2014-09-29 16:16:38 -0400, Tom Lane wrote:

I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".

Because that'd be horrible performancewise on a system with many
wal_buffers. There's several operations where all locks are checked in
sequence (to see whether there's any stragglers that need to finish
inserting) and even some where they're acquired concurrently (e.g. for
xlog switch, checkpoint and such).

Hm. Well, if there are countervailing considerations as to how large is a
good value, that makes it even less likely that it's sensible to expose
it as a user tunable. A relevant analogy is that we don't expose a way
to adjust the number of lock table partitions at runtime.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-09-29 16:35:12 -0400, Tom Lane wrote:

Andres Freund <andres@2ndquadrant.com> writes:

On 2014-09-29 16:16:38 -0400, Tom Lane wrote:

I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".

Because that'd be horrible performancewise on a system with many
wal_buffers. There's several operations where all locks are checked in
sequence (to see whether there's any stragglers that need to finish
inserting) and even some where they're acquired concurrently (e.g. for
xlog switch, checkpoint and such).

Hm. Well, if there are countervailing considerations as to how large is a
good value, that makes it even less likely that it's sensible to expose
it as a user tunable.

Aren't there such considerations for most of the performance critical
gucs?

A relevant analogy is that we don't expose a way
to adjust the number of lock table partitions at runtime.

Which has worked out badly for e.g. the number of buffer partitions...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Sep 30, 2014 at 3:44 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Robert Haas <robertmhaas@gmail.com> writes:

The items I see are:

- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!

The text seems to indicate that there's some disagreement on this
point. I don't have a strong opinion on whether or not to keep the
GUC, but if we're going to remove it it should probably happen before
beta3. It's going to be impossible to remove once we've released with
it, I suspect.

The lack of any documentation for the GUC (neither in config.sgml or
postgresql.conf.sample) suggests very very strongly that it was not
meant to be shipped. If we don't remove it I will certainly insist
that it be documented adequately.

Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?

I tend to go in this direction too. It is unclear how it is really able to
improve scalability, or at least some documentation should be here to help
users to set it. An additional thought as well: set it with a configure
switch at compilation instead of a GUC.
--
Michael

On 09/29/2014 11:41 PM, Andres Freund wrote:

On 2014-09-29 16:35:12 -0400, Tom Lane wrote:

Andres Freund <andres@2ndquadrant.com> writes:

On 2014-09-29 16:16:38 -0400, Tom Lane wrote:

I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".

Because that'd be horrible performancewise on a system with many
wal_buffers. There's several operations where all locks are checked in
sequence (to see whether there's any stragglers that need to finish
inserting) and even some where they're acquired concurrently (e.g. for
xlog switch, checkpoint and such).

Hm. Well, if there are countervailing considerations as to how large is a
good value, that makes it even less likely that it's sensible to expose
it as a user tunable.

Aren't there such considerations for most of the performance critical
gucs?

A relevant analogy is that we don't expose a way
to adjust the number of lock table partitions at runtime.

Which has worked out badly for e.g. the number of buffer partitions...

Number of buffer partitions is the analogy I've also had in mind. It has
worked out pretty well, IMHO. Sure, with a system with a lot of CPUs you
sometimes want to increase the hardwired default, but most
configurations are not too sensitive to it.

No-one has stepped up to do any testing on the effects if the GUC during
the beta period, while the GUC has been there. Somehow I don't think
anyone's going to do any rigorous tuning of it before commissioning a
system into production, either.

This might come up again in 9.5 cycle, once we get the improvements in
LWLock and buffer cache scalability. That can make scalability of the
WAL insertion more visible again.

There seems to be no decisive consensus here. I'm going to put my foot
on the ground and go remove it, as I'm leaning towards that option, and
we need to get the release out. But if someone objects loudly enough to
actually write the documentation and commit it that way, I'm happy with
that too.

- Heikki

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 09/30/2014 04:56 AM, Heikki Linnakangas wrote:

There seems to be no decisive consensus here. I'm going to put my foot
on the ground and go remove it, as I'm leaning towards that option, and
we need to get the release out. But if someone objects loudly enough to
actually write the documentation and commit it that way, I'm happy with
that too.

I'm also in favor of removing it. We really don't need more GUCs which
nobody has any clear idea how to change or why.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 9/29/14, 2:30 PM, Andres Freund wrote:

Can we explain those reasons in the form of documentation?

Yes. Try and benchmark it. It'll be hardware and workload dependant.

I missed this whole thing, and eventually I have to circle back to it.
I could do it this week. Could you (or someone else familiar with the
useful benchmarks) give me more detail on how to replicate one case
where things should improve? I can do that, and I have a lab full of
hardware if it's easier to spot on one type of server. That exercise
will probably lead to a useful opinion on the feature in its final form,
any associated GUC, tunables, and necessary level of associated
documentation in a day or two.

This is a pretty low bar, right? Examples and documentation sufficient
for just me to catch up is not asking for very much. If it isn't
possible to do that in a very short period of time, it would not bode
well for broader consumption.

--
Greg Smith greg.smith@crunchydatasolutions.com
Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 09/30/2014 09:10 PM, Gregory Smith wrote:

On 9/29/14, 2:30 PM, Andres Freund wrote:

Can we explain those reasons in the form of documentation?

Yes. Try and benchmark it. It'll be hardware and workload dependant.

I missed this whole thing, and eventually I have to circle back to it.
I could do it this week.

Ah, that would be great!

Could you (or someone else familiar with the
useful benchmarks) give me more detail on how to replicate one case
where things should improve? I can do that, and I have a lab full of
hardware if it's easier to spot on one type of server. That exercise
will probably lead to a useful opinion on the feature in its final form,
any associated GUC, tunables, and necessary level of associated
documentation in a day or two.

To see the most improvement from the patch, try a workload that's
otherwise bottlenecked on XLogInsert. For example, with pgbench:

psql postgres -c "create table foo (id int4)"

pgbench postgres -n -f fooinsert.sql -c 4 -T10

and in the test script:

insert into foo select g from generate_series(1, 10000) g;

- Heikki

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers