8.2 beta blockers

A long time ago, in a galaxy far, far away, tgl@sss.pgh.pa.us (Tom Lane) wrote:

I see the following items standing between us and putting out 8.2beta1:
* AIX linking issues

This isn't necessarily a beta-stopper, but it'd be nice to get it done
so we can be sure that any beta testing done on AIX will test the
revisions.

This has to do with the discussion about shared vs static libs?

If a patch is put out, it'll get build-farm results on AIX within a
couple hours. If there's something worth trying, then try away...
--
select 'cbbrowne' || '@' || 'gmail.com';
http://cbbrowne.com/info/wp.html
When I met th'POPE back in '58, I scrubbed him with a MILD SOAP or
DETERGENT for 15 minutes. He seemed to enjoy it ...

On Sun, Sep 17, 2006 at 07:38:38PM -0400, Tom Lane wrote:

We have three possible choices for this: do nothing, install a
bug-compatible, allegedly-clean-room implementation in contrib:
http://archives.postgresql.org/pgsql-patches/2006-09/msg00077.php
or put a hopefully-cleaner design into core, eg per my suggestions here:
http://archives.postgresql.org/pgsql-hackers/2006-09/msg00467.php
I favor the third alternative, mainly because by changing the API
we remove all doubt as to whether any "intellectual property"
remains from the original GPL'd code. However, we've got to make up
our minds and get on with it.

FWIW, I'm +1 on the cleaner design you suggested. While I understand the
concerns of adding features/API this late; as a user, I'd rather not wait
another year to have these available in core(yes, I know alternative measures
would exist if it did not make it into core, but the convenience of having it
there would certainly be nice). That is, I really like the waiting variant.
It is something that I would use. The lack thereof(IIRC) in the current contrib
implementation is something that I have recently lamented about.

I understand that "want" is not a reason to compromise the feature freeze, so I
hope the legal concerns Tom mentions will be enough. =)

On 2006-09-18, James William Pye <pgsql@jwp.name> wrote:

FWIW, I'm +1 on the cleaner design you suggested. While I understand the
concerns of adding features/API this late;

Adding features is one thing, breaking existing users of the code is another.

--
Andrew, Supernews
http://www.supernews.com - individual and corporate NNTP services

Christopher Browne <cbbrowne@acm.org> writes:

A long time ago, in a galaxy far, far away, tgl@sss.pgh.pa.us (Tom Lane) wrote:

I see the following items standing between us and putting out 8.2beta1:
* AIX linking issues

This has to do with the discussion about shared vs static libs?

If a patch is put out, it'll get build-farm results on AIX within a
couple hours.

Yeah, I know, which is why I don't find it absolutely critical that
this make it to beta1. But one of the concerns mentioned in the thread
is that the changes might break things for older AIX versions. If we
get it into beta1, we have a better chance of finding out before release
whether there are any issues with AIX versions that aren't represented
in buildfarm.

regards, tom lane

Andrew - Supernews <andrew+nonews@supernews.com> writes:

On 2006-09-18, James William Pye <pgsql@jwp.name> wrote:

FWIW, I'm +1 on the cleaner design you suggested. While I understand the
concerns of adding features/API this late;

Adding features is one thing, breaking existing users of the code is another.

We already have an established solution for people who need backwards
compatibility: download the old userlock code from pgfoundry. I'm
unclear why we should duplicate that project within core.

regards, tom lane

On Sun, Sep 17, 2006 at 07:38:38PM -0400, Tom Lane wrote:

* The contrib/userlock replacement issue

We have three possible choices for this: do nothing, install a
bug-compatible, allegedly-clean-room implementation in contrib:
http://archives.postgresql.org/pgsql-patches/2006-09/msg00077.php
or put a hopefully-cleaner design into core, eg per my suggestions here:
http://archives.postgresql.org/pgsql-hackers/2006-09/msg00467.php
I favor the third alternative, mainly because by changing the API
we remove all doubt as to whether any "intellectual property"
remains from the original GPL'd code. However, we've got to make up
our minds and get on with it.

One problem I see with userlock is that you're asking for lock ID
conflicts unless you control everything on the system that's using
userlocks. Right now, doing so probably isn't that difficult, but my
hope is that as PostgreSQL becomes more acceptable we'll see more
software written to use it... software that might need to play well with
other software within the same database.

I think the best way to fix this is to set aside some range of the lock
ID space for a naming scheme that is based on schemas, or some other
kind of text identifier. I think this really just amounts to putting
some recommended practices into the docs; perhaps not using any
int4,int4 IDs where the first int is larger than 16k or whatever the
maximum system OID is. That would allow for segmenting locks based on
schema OIDs. Down the road we might want a nicer, schema-based
interface, but just setting aside the space should be fine for now.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Tom Lane wrote:

I see the following items standing between us and putting out 8.2beta1:

* Set client encoding based on OS environment - Peter E.

[snip]

Personally I'm willing to commit to making the VALUES-list docs and
userlock replacement code happen tomorrow. Bruce seems to be close
on the release notes, and if the other two items aren't ready, well,
beta1 can ship without 'em.

If you talk about the "client encoding thing" here...

Since the client encoding change is a feature addition and a behavioral
change, I think this should really be done before going to beta, IMHO. Btw.
was there any mailing list discussion on this item? Other than this:
http://archives.postgresql.org/pgsql-hackers/2003-05/msg00737.php.

I really hope that this change will only affect psql, not pg_dump, as Peter
wrote in 2003. I would strongly object to such a change (as much as my
voice counts). The current behavior of dumping with the database encoding
is exactly the right thing to do. I have a database in UTF-8 here. Using
any LATIN based encoding for dumping the database will simply break,
because there are always characters that don't map into the encoding.

Even with psql there could be issues with existing scripts, but I see a
benefit at least.

Best Regards
Michael Paesold

Tom Lane wrote:

I see the following items standing between us and putting out 8.2beta1:

* Set client encoding based on OS environment - Peter E.

I'm not sure whether Peter is intending to complete this item for 8.2
or not, but if it's to be done it ought to be done before we start beta.

This just reminds me, are there plans to take into account multibyte
server encodings inside the client quote function?

regards,
Lukas

Quoth tgl@sss.pgh.pa.us (Tom Lane):

Christopher Browne <cbbrowne@acm.org> writes:

A long time ago, in a galaxy far, far away, tgl@sss.pgh.pa.us (Tom Lane) wrote:

I see the following items standing between us and putting out 8.2beta1:
* AIX linking issues

This has to do with the discussion about shared vs static libs?

If a patch is put out, it'll get build-farm results on AIX within a
couple hours.

Yeah, I know, which is why I don't find it absolutely critical that
this make it to beta1. But one of the concerns mentioned in the thread
is that the changes might break things for older AIX versions. If we
get it into beta1, we have a better chance of finding out before release
whether there are any issues with AIX versions that aren't represented
in buildfarm.

Well, systems running elder versions of AIX are doing so without any
IBM support. AIX 5.1 is no longer supported...

I'd have a problem with running any of those elder versions due to the
absence of support, myself...
--
(format nil "~S@~S" "cbbrowne" "gmail.com")
http://linuxdatabases.info/info/nonrdbms.html
"Microsoft is a cross between the Borg and the Ferengi. Unfortunately,
they use Borg to do their marketing and Ferengi to do their
programming." -- Simon Slavin in asr

Am Montag, 18. September 2006 09:20 schrieb Lukas Kahwe Smith:

This just reminds me, are there plans to take into account multibyte
server encodings inside the client quote function?

Huh?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Am Montag, 18. September 2006 01:38 schrieb Tom Lane:

* Set client encoding based on OS environment - Peter E.

This is not an item for 8.2 in my mind.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Am Montag, 18. September 2006 09:20 schrieb Lukas Kahwe Smith:

This just reminds me, are there plans to take into account multibyte
server encodings inside the client quote function?

Huh?

Ah, I just checked the libpq docs and there seems to be a
PQescapeStringConn. Not sure when this was added, I think PHP does not
yet use it. I will investigate this and will make sure its used in favor
of the deprecated old PQescapeString function.

regards,
Lukas

Am Montag, 18. September 2006 14:25 schrieb Lukas Kahwe Smith:

Peter Eisentraut wrote:

Am Montag, 18. September 2006 09:20 schrieb Lukas Kahwe Smith:

This just reminds me, are there plans to take into account multibyte
server encodings inside the client quote function?

Huh?

Ah, I just checked the libpq docs and there seems to be a
PQescapeStringConn. Not sure when this was added, I think PHP does not
yet use it. I will investigate this and will make sure its used in favor
of the deprecated old PQescapeString function.

That was part of the big security update we had a while back. If PHP missed
that, it's their problem.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Michael Paesold <mpaesold@gmx.at> writes:

* Set client encoding based on OS environment - Peter E.

I really hope that this change will only affect psql, not pg_dump, as Peter
wrote in 2003. I would strongly object to such a change (as much as my
voice counts). The current behavior of dumping with the database encoding
is exactly the right thing to do.

Actually, I realize after a quick look at the pg_dump code that its
current behavior is to dump in
1. Specified encoding if a -E switch is given.
2. PGCLIENTENCODING, if that environment var exists.
3. Else, server encoding.
So there's already an environment dependency, although it's for
something much less likely to be set than LANG. I tend to agree
that we'd better avoid having dumps depend on LANG ... wonder if
we should remove the dependency on PGCLIENTENCODING too.

regards, tom lane

"Jim C. Nasby" <jimn@enterprisedb.com> writes:

One problem I see with userlock is that you're asking for lock ID
conflicts unless you control everything on the system that's using
userlocks.

Well, the lock IDs already include the database OID under the hood,
so you only need to control stuff within your database. Beyond that,
yeah, it's your responsibility...

regards, tom lane

Am Montag, 18. September 2006 15:48 schrieb Tom Lane:

So there's already an environment dependency, although it's for
something much less likely to be set than LANG. I tend to agree
that we'd better avoid having dumps depend on LANG ... wonder if
we should remove the dependency on PGCLIENTENCODING too.

I'd say that in principle, any console application should respect the
console's encoding settings by default. pg_dump is not only a backup tool
but also useful for quasi-interactive use. We could check if stdout goes to
a terminal or something like that.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Am Montag, 18. September 2006 01:38 schrieb Tom Lane:

* Set client encoding based on OS environment - Peter E.

This is not an item for 8.2 in my mind.

OK, moved to TODO:

* Set client encoding based on the client operating system encoding

Currently client_encoding is set in postgresql.conf, which
defaults to the server encoding.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, Sep 18, 2006 at 10:10:32AM -0400, Tom Lane wrote:

"Jim C. Nasby" <jimn@enterprisedb.com> writes:

One problem I see with userlock is that you're asking for lock ID
conflicts unless you control everything on the system that's using
userlocks.

Well, the lock IDs already include the database OID under the hood,
so you only need to control stuff within your database. Beyond that,
yeah, it's your responsibility...

Which is what I'm worried about for the future... that's why I'd like to
recommend to users some kind of best practice that allows for a system
in the future that doesn't require managing every single lock in the
system. I believe recommending that you not use locks with the first
int4 above 16k (and whatever the equivalent would be for int8) would be
a good way to do that, as it would allow for segregating locks by schema
OID.

It's important to recommend this now so that people don't write software
that will clobber other locks in the future.
--
Jim Nasby jimn@enterprisedb.com
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

"Jim C. Nasby" <jimn@enterprisedb.com> writes:

I believe recommending that you not use locks with the first
int4 above 16k (and whatever the equivalent would be for int8) would be
a good way to do that, as it would allow for segregating locks by schema
OID.

That seems pretty content-free to me, if not counter-productive. To the
extent that we can foresee the usage for the two-int4s lock style, the
first one is likely to be an OID (eg, a user table's OID) which the user
is not going to be able to control the range of. Also, if you are
locking on the basis of object OIDs, there's no need to worry which
schema they are in.

regards, tom lane

Lukas,

Ah, I just checked the libpq docs and there seems to be a
PQescapeStringConn. Not sure when this was added, I think PHP does not
yet use it. I will investigate this and will make sure its used in favor
of the deprecated old PQescapeString function.

PHP driver authors and major PHP projects (PHPBB, SQL-Ledger, GForge, etc.)
were notified about this *before* it happened. I beleive that PDO will
support it soon, pg_connect already does, and PEAR::DB2 does.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

All,

Is UserLocks a cool enough feature to be worth mentioning in the 8.2 PR?
If so, can someone explain it to me off-list? I still don't get what it
does ...

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org
Yeah, I know, which is why I don't find it absolutely critical that
this make it to beta1. But one of the concerns mentioned in
the thread
is that the changes might break things for older AIX versions. If we
get it into beta1, we have a better chance of finding out
before release
whether there are any issues with AIX versions that aren't represented
in buildfarm.

As a backdoor, we can leave in the code to compile static for the old
versions, but I am not sure its worth it.

As a side note, I tried to build CVS on an AIX 4.1 box, and it already
has issues. The TOC is too large for ld, amongst others.

I had tried to get our older box onto the buildfarm, but the version of
Perl on that box was too ancient to run the buildfarm code, and I didn't
think people would care too much about it missing, since its such an old
version anyway. Then I discovered the issues with trying to build by
hand, and gave up at that point.

I think we should make our cut-off version as 4.3. This is the first to
support run time linking, so the changes we are talking about for today
will work on that version. It came out in October 97 and was EOL'd Dec
2001.

If someone really wants to run on something older than a 9 year old OS,
that was EOL'd almost 5 years ago. I think they should do so at their
peril, and its not up to us to go out of our way to support them.

I don't think we will be making an AIX uesrs sad by adding support for
dynamic linking, but breaking old versions of the OS.

Thanks,
-rocco

Josh Berkus wrote:

All,

Is UserLocks a cool enough feature to be worth mentioning in the 8.2 PR?
If so, can someone explain it to me off-list? I still don't get what it
does ...

Probably not.

--
Bruce Momjian bruce@momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

Josh Berkus wrote:

Is UserLocks a cool enough feature to be worth mentioning in the 8.2 PR?

Probably not.

Especially not since the capability has been there right along.

regards, tom lane

On 9/18/06, Josh Berkus <josh@agliodbs.com> wrote:

All,

Is UserLocks a cool enough feature to be worth mentioning in the 8.2 PR?
If so, can someone explain it to me off-list? I still don't get what it
does ...

yes, i can explain it in detail, and am willing to kick in some
documentation. it is very cool, and relatively undiscovered. email
me for details.

the short answer is you can do long term locking (not scoped to
transaction) using the postgresql high speed internal lock engine.
this allows you to bypass various locking hacks such as keeping
transactions (too) long term, or maintaining a lock flag in a table,
which has cleanup issues.

they are great if you need to emulate pessimistic locking and
transactions is an awkward fit. its up to you, the user to determine
when and what to lock.

i really beat on the user (advisory locks) when doing a cobol/isam
conversion project. they work fabulous.

merlin

On 9/18/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Bruce Momjian <bruce@momjian.us> writes:

Josh Berkus wrote:

Is UserLocks a cool enough feature to be worth mentioning in the 8.2 PR?

Probably not.

Especially not since the capability has been there right along.

I disagree, almost nobody knows about them, due to lousy name and lack
of documentation. I think the pr hook should be merited to their
general usefullness, previous implementation notwithstanding.
techinally, we added a wait version too, right? 8.1 table
partitioning was dressed up inheritance right? :) (not quite the
same, i know but...)

merlin

"Merlin Moncure" <mmoncure@gmail.com> writes:

yes, i can explain it in detail, and am willing to kick in some
documentation.

Ah-hah, you're on the hook for docs then ;-).

I'm going to go ahead with implementing it in-core per my last proposal:

void pg_advisory_lock(int8) wait
void pg_advisory_lock_shared(int8) wait
bool pg_try_advisory_lock(int8) no wait
bool pg_try_advisory_lock_shared(int8) no wait
bool pg_advisory_unlock(int8) returns T if successful
bool pg_advisory_unlock_shared(int8) returns T if successful

plus all the above taking 2 int4's, plus

void pg_advisory_unlock_all()

but am happy to let someone else do the docs.

As far as the PR material goes, something like "advisory locks
incorporated into core" would be OK, but don't make it sound like
there was nothing there before ...

regards, tom lane

On 9/18/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

"Merlin Moncure" <mmoncure@gmail.com> writes:

yes, i can explain it in detail, and am willing to kick in some
documentation.

Ah-hah, you're on the hook for docs then ;-).

sure no problem. the prototypes you suggested are imo the way to go,
with two small considerations:

is it worth considering using the oid type instead of int4 since the
'locktag' fields are unsigned?
also, the userlocks raised a warning if you tried to release a
non-existing lock. should that stay or go?

As far as the PR material goes, something like "advisory locks
incorporated into core" would be OK, but don't make it sound like
there was nothing there before ...

ok, thats a good compromise.

merlin

Tom, Merlin,

As far as the PR material goes, something like "advisory locks
incorporated into core" would be OK, but don't make it sound like
there was nothing there before ...

ok, thats a good compromise.

Yes, although if I'm doing this for PR, I need to use language which is
standard in the industry or I won't be understood. What about
"high-concurrency pessimistic locking mechanism"?

I don't have a problem covering this in PR even though it's not really "new":
a) it's never been in our PR before;
b) while it's been available for 4+ versions, who knew about it?
c) we have a new refactored code and cleaned up API

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

"Merlin Moncure" <mmoncure@gmail.com> writes:

sure no problem. the prototypes you suggested are imo the way to go,
with two small considerations:

is it worth considering using the oid type instead of int4 since the
'locktag' fields are unsigned?

Hmm ... I was thinking it didn't matter, but on closer look, the
int4->oid cast is implicit while the oid->int4 one is only assignment.
So you'd need to write a cast to pass an OID if we declare the functions
as taking int4. But you'll need a cast anyway if you want to pass a
single OID to the int8-taking version (that's an assignment cast too).

The downside of declaring the functions to take OID is that people might
think they could *only* use OIDs, which isn't so, they can use any
int4-sized key they feel like.

Not seeing a strong reason one way or the other ... what do you think?

also, the userlocks raised a warning if you tried to release a
non-existing lock. should that stay or go?

That's in the core code I think, so it won't change.

regards, tom lane

Josh Berkus <josh@agliodbs.com> writes:

As far as the PR material goes, something like "advisory locks
incorporated into core" would be OK, but don't make it sound like
there was nothing there before ...

Yes, although if I'm doing this for PR, I need to use language which is
standard in the industry or I won't be understood. What about
"high-concurrency pessimistic locking mechanism"?

"Advisory lock" *is* a standard term. The other sounds, well, mighty PR-ish.

regards, tom lane

On 9/18/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Hmm ... I was thinking it didn't matter, but on closer look, the
int4->oid cast is implicit while the oid->int4 one is only assignment.
So you'd need to write a cast to pass an OID if we declare the functions
as taking int4. But you'll need a cast anyway if you want to pass a
single OID to the int8-taking version (that's an assignment cast too).

The downside of declaring the functions to take OID is that people might
think they could *only* use OIDs, which isn't so, they can use any
int4-sized key they feel like.

hm. this is really a byproduct of oid being the catchall unsigned int4
type since it has the most built in casts. i agree 100% though on the
oid perception however, i don't like userland oids at all, until such
time as an 8 bit one comes out. i would say leave as int4 unless you
were willing to sql typedef the oid to some other name.

merlin

On Mon, Sep 18, 2006 at 05:06:09PM -0400, Merlin Moncure wrote:

On 9/18/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Hmm ... I was thinking it didn't matter, but on closer look, the
int4->oid cast is implicit while the oid->int4 one is only assignment.
So you'd need to write a cast to pass an OID if we declare the functions
as taking int4. But you'll need a cast anyway if you want to pass a
single OID to the int8-taking version (that's an assignment cast too).

The downside of declaring the functions to take OID is that people might
think they could *only* use OIDs, which isn't so, they can use any
int4-sized key they feel like.

hm. this is really a byproduct of oid being the catchall unsigned int4
type since it has the most built in casts. i agree 100% though on the
oid perception however, i don't like userland oids at all, until such
time as an 8 bit one comes out. i would say leave as int4 unless you
were willing to sql typedef the oid to some other name.

Would adding OID versions of the functions (so there'd be int8, (int4,
int4) and (oid,oid)) be overkill?
--
Jim Nasby jimn@enterprisedb.com
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

"Jim C. Nasby" <jimn@enterprisedb.com> writes:

Would adding OID versions of the functions (so there'd be int8, (int4,
int4) and (oid,oid)) be overkill?

I think what it'd mostly accomplish is to create "couldn't resolve
ambiguous function" errors :-(

regards, tom lane

Lukas Kahwe Smith wrote:

Peter Eisentraut wrote:

Am Montag, 18. September 2006 09:20 schrieb Lukas Kahwe Smith:

This just reminds me, are there plans to take into account multibyte
server encodings inside the client quote function?

Huh?

Ah, I just checked the libpq docs and there seems to be a
PQescapeStringConn. Not sure when this was added, I think PHP does not
yet use it. I will investigate this and will make sure its used in favor
of the deprecated old PQescapeString function.

This will be fixed as of PHP 5.2.0

regards,
Lukas