pg_control is missing a field for LOBLKSIZE

On 06/04/2014 10:03 AM, Tom Lane wrote:

I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.

I think there ought to be a guard for that, for exactly the same reasons
that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk
data requires that this value match the original database configuration.

Obviously it's too late to do anything about this in existing branches,
but I propose to add a field to pg_control after we branch off 9.4.

If we did an initdb-requiring change for 9.4 could we piggy-back this
onto it?

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 2014-06-04 10:25:07 -0400, Andrew Dunstan wrote:

If we did an initdb-requiring change for 9.4 could we piggy-back this onto
it?

Do you know of a problem requiring that?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Andrew Dunstan (andrew@dunslane.net) wrote:

On 06/04/2014 10:03 AM, Tom Lane wrote:

I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.

I think there ought to be a guard for that, for exactly the same reasons
that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk
data requires that this value match the original database configuration.

Obviously it's too late to do anything about this in existing branches,
but I propose to add a field to pg_control after we branch off 9.4.

If we did an initdb-requiring change for 9.4 could we piggy-back
this onto it?

I was thinking more-or-less the same thing...

Then again, I've never heard of a field complaint regarding this, so
pehraps it's not worth it.

Thanks,

Stephen

On 06/04/2014 10:27 AM, Andres Freund wrote:

On 2014-06-04 10:25:07 -0400, Andrew Dunstan wrote:

If we did an initdb-requiring change for 9.4 could we piggy-back this onto
it?

Do you know of a problem requiring that?

No, just thinking ahead.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Stephen Frost <sfrost@snowman.net> writes:

* Andrew Dunstan (andrew@dunslane.net) wrote:

On 06/04/2014 10:03 AM, Tom Lane wrote:

I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.

Then again, I've never heard of a field complaint regarding this, so
pehraps it's not worth it.

I've not heard one either, but there was just somebody asking in
pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
That's not a big enough sample size to make me panic about getting a
hasty fix into 9.4, but I do think we should fix this going forward.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

Stephen Frost <sfrost@snowman.net> writes:

* Andrew Dunstan (andrew@dunslane.net) wrote:

On 06/04/2014 10:03 AM, Tom Lane wrote:

I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.

Then again, I've never heard of a field complaint regarding this, so
pehraps it's not worth it.

I've not heard one either, but there was just somebody asking in
pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
That's not a big enough sample size to make me panic about getting a
hasty fix into 9.4, but I do think we should fix this going forward.

Agreed.

Thanks,

Stephen

On 2014-06-04 10:03:09 -0400, Tom Lane wrote:

I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.

I think there ought to be a guard for that, for exactly the same reasons
that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk
data requires that this value match the original database configuration.

Obviously it's too late to do anything about this in existing branches,
but I propose to add a field to pg_control after we branch off 9.4.

Btw, I had wondered before if we shouldn't also add sizeof(long) to
pg_control to catch cases where a database is copied between a LLP64
(64bit windows) and an LP64 (nearly every other 64bit system) system. I
have my doubts that we're completely clean about the size
difference. Not to speak of extension datatypes.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Andres Freund <andres@2ndquadrant.com> writes:

Btw, I had wondered before if we shouldn't also add sizeof(long) to
pg_control to catch cases where a database is copied between a LLP64
(64bit windows) and an LP64 (nearly every other 64bit system) system. I
have my doubts that we're completely clean about the size
difference. Not to speak of extension datatypes.

I don't believe that this is necessary. It's certainly true that some
in-memory structures will be laid out differently, but not on-disk.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Stephen Frost <sfrost@snowman.net> writes:

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

I've not heard one either, but there was just somebody asking in
pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
That's not a big enough sample size to make me panic about getting a
hasty fix into 9.4, but I do think we should fix this going forward.

Agreed.

BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE,
I noticed that the tuptoaster.c functions are reasonably paranoid about
checking that toast chunks are the expected size, but the large object
functions are not: the latter have either no check at all, or just an
Assert that the size is not more than expected. So we could provide at
least a partial guard against a wrong LOBLKSIZE configuration by making
all the large-object functions throw elog(ERROR) if the length of a LO
chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE
is an expected case, so this would only help in one direction. Still,
it'd be an easy and back-patchable change that would provide at least some
defense, so I'm thinking of doing it.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wed, Jun 4, 2014 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Stephen Frost <sfrost@snowman.net> writes:

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

I've not heard one either, but there was just somebody asking in
pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
That's not a big enough sample size to make me panic about getting a
hasty fix into 9.4, but I do think we should fix this going forward.

Agreed.

BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE,
I noticed that the tuptoaster.c functions are reasonably paranoid about
checking that toast chunks are the expected size, but the large object
functions are not: the latter have either no check at all, or just an
Assert that the size is not more than expected. So we could provide at
least a partial guard against a wrong LOBLKSIZE configuration by making
all the large-object functions throw elog(ERROR) if the length of a LO
chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE
is an expected case, so this would only help in one direction. Still,
it'd be an easy and back-patchable change that would provide at least some
defense, so I'm thinking of doing it.

This seems like a pretty weak argument for adding run-time overhead.
Maybe it can be justified on the grounds that it would catch corrupted
TOAST data, but I've never heard of anyone changing LOBLKSIZE and see
no reason to get agitated about it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Robert Haas <robertmhaas@gmail.com> writes:

On Wed, Jun 4, 2014 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE,
I noticed that the tuptoaster.c functions are reasonably paranoid about
checking that toast chunks are the expected size, but the large object
functions are not: the latter have either no check at all, or just an
Assert that the size is not more than expected. So we could provide at
least a partial guard against a wrong LOBLKSIZE configuration by making
all the large-object functions throw elog(ERROR) if the length of a LO
chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE
is an expected case, so this would only help in one direction. Still,
it'd be an easy and back-patchable change that would provide at least some
defense, so I'm thinking of doing it.

This seems like a pretty weak argument for adding run-time overhead.
Maybe it can be justified on the grounds that it would catch corrupted
TOAST data, but I've never heard of anyone changing LOBLKSIZE and see
no reason to get agitated about it.

One if-test per fetched tuple hardly seems likely to add measurable
overhead. As for "never heard of", see today's thread in pgsql-general:
/messages/by-id/CAGou9Mg=9qPYTdh18NDO3LTJtwQN8uRdTwABfkcyMRUt6D_fJw@mail.gmail.com
There was a similar gripe a few months ago:
/messages/by-id/CACg6vWXy_84ShCCXzNCsz9xLfWnx5sZvQRU6aNcrR0c3XW1Bxg@mail.gmail.com

There are at least two places in inv_api.c where we have
"Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
into a local variable of size LOBLKSIZE, so that the only thing standing
between us and a stack-smash security issue that's trivially exploitable
in production builds is that on-disk data conforms to our expectation
about LOBLKSIZE. I think it's definitely worth promoting these checks
to regular runtime-if-test-and-elog.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

There are at least two places in inv_api.c where we have
"Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
into a local variable of size LOBLKSIZE, so that the only thing standing
between us and a stack-smash security issue that's trivially exploitable
in production builds is that on-disk data conforms to our expectation
about LOBLKSIZE. I think it's definitely worth promoting these checks
to regular runtime-if-test-and-elog.

Agreed. Promoting that to a run-time check seems well worth it to me.

Thanks,

Stephen

Stephen Frost <sfrost@snowman.net> writes:

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

There are at least two places in inv_api.c where we have
"Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
into a local variable of size LOBLKSIZE, so that the only thing standing
between us and a stack-smash security issue that's trivially exploitable
in production builds is that on-disk data conforms to our expectation
about LOBLKSIZE. I think it's definitely worth promoting these checks
to regular runtime-if-test-and-elog.

Agreed. Promoting that to a run-time check seems well worth it to me.

Here's a draft patch for this. Barring objections I'll commit the whole
thing to HEAD, and the inv_api.c changes to the back branches as well.

regards, tom lane

On Wed, Jun 4, 2014 at 06:57:31PM -0400, Tom Lane wrote:

Stephen Frost <sfrost@snowman.net> writes:

* Tom Lane (tgl@sss.pgh.pa.us) wrote:

There are at least two places in inv_api.c where we have
"Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
into a local variable of size LOBLKSIZE, so that the only thing standing
between us and a stack-smash security issue that's trivially exploitable
in production builds is that on-disk data conforms to our expectation
about LOBLKSIZE. I think it's definitely worth promoting these checks
to regular runtime-if-test-and-elog.

Agreed. Promoting that to a run-time check seems well worth it to me.

Here's a draft patch for this. Barring objections I'll commit the whole
thing to HEAD, and the inv_api.c changes to the back branches as well.

Uh, I think pg_upgrade needs to check that they match too.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian <bruce@momjian.us> writes:

Uh, I think pg_upgrade needs to check that they match too.

Possibly. What do you think it should do when examining a pg_control
version that lacks the field?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Uh, I think pg_upgrade needs to check that they match too.

Possibly. What do you think it should do when examining a pg_control
version that lacks the field?

Good question. I have existing cases where fields were removed, but not
ones that were added. As we have no way to query the old cluster's
value for LOBLKSIZE, I think I will just add code to compare them if
they _both_ exist.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian wrote:

On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Uh, I think pg_upgrade needs to check that they match too.

Possibly. What do you think it should do when examining a pg_control
version that lacks the field?

Good question. I have existing cases where fields were removed, but not
ones that were added. As we have no way to query the old cluster's
value for LOBLKSIZE, I think I will just add code to compare them if
they _both_ exist.

Can't you compare it to the historic default value? I mean, add an
assumption that people thus far has never tweaked it.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote:

Bruce Momjian wrote:

On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Uh, I think pg_upgrade needs to check that they match too.

Possibly. What do you think it should do when examining a pg_control
version that lacks the field?

Good question. I have existing cases where fields were removed, but not
ones that were added. As we have no way to query the old cluster's
value for LOBLKSIZE, I think I will just add code to compare them if
they _both_ exist.

Can't you compare it to the historic default value? I mean, add an
assumption that people thus far has never tweaked it.

Well, if they did tweak it, then they would be unable to use pg_upgrade
because it would complain about a mismatch if they actually matched the
old and new servers.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian <bruce@momjian.us> writes:

On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote:

Can't you compare it to the historic default value? I mean, add an
assumption that people thus far has never tweaked it.

Well, if they did tweak it, then they would be unable to use pg_upgrade
because it would complain about a mismatch if they actually matched the
old and new servers.

What about comparing to the symbolic value LOBLKSIZE? This would make
pg_upgrade assume that the old installation had been tweaked the same
as in its own build. This ends up being the same as what you said,
ie, effectively no comparison ... but it might be less complicated to
code/understand.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers