Roadmap for FE/BE protocol redesign
This is an attempt to lay out a road map for updating the frontend/backend
protocol in 7.4. I don't at this point want to get into details on any
one of the TODO items, just get consensus that this is the set of tasks
to be tackled. Are there any areas that I've missed (that require
protocol changes)?
* Extend ERROR and NOTICE messages to carry multiple fields, including
(as appropriate) a spec-compliant error code, a textual position in the
original query string, information about the source-code location where
the error was reported, etc.
* Consider extending NOTIFY messages to allow a parameter to be carried.
* Fix COPY protocol to allow graceful error recovery (more graceful than
aborting the connection, at least) and to support COPY BINARY to/from
frontend.
* Redesign fastpath function call protocol to eliminate the problems cited
in the source code comments (src/backend/tcop/fastpath.c), and to
eliminate the security hole of accepting unchecked internal representation
from frontend. Also consider a fastpath for execution of PREPAREd queries.
* Re-institute type-specific send/receive conversion routines to allow
some modest amount of architecture independence for binary data. This'd
provide a place to check for bogus internal representation during fastpath
input and COPY BINARY IN, too, thus alleviating security concerns.
* Get rid of hardwired field sizes in StartupPacket --- use
variable-length null-terminated strings. Fixes problem with usernames
being limited to 32 characters, gets around unreasonable limitation on
PGOPTIONS length. Also can remove unused fields.
* Backend should pass its version number, database encoding, default
client encoding, and possibly other data (any ideas?) to frontend during
startup, to avoid need for explicit queries to get this info. We could
also consider eliminating SET commands sent by libpq in favor of adding
variable settings to startup packet's PGOPTIONS field. Ideally we could
get back to the point where a standard connection startup takes only one
packet in each direction.
* Backend's ReadyForQuery message (Z message) should carry an indication
of current transaction status (idle/in transaction/in aborted transaction)
so that frontend need not guess at state. Perhaps also indicate
autocommit status. (Is there anything else that frontends would Really
Like To Know?)
* XML support? If we do anything, I'd want some extensible solution to
allowing multiple query-result output formats from the backend, not an
XML-specific hack. For one thing, that would allow the actual appearance
of any XML support to happen later.
One of the $64 questions that has to be answered is how much work we're
willing to expend on backwards compatibility. The path of least
resistance would be to handle it the same way we've done protocol
revisions in the past: the backend will be able to handle both old and new
protocols (so it can talk to old clients) but libpq would be revised to
speak only the new protocol (so new/recompiled clients couldn't talk to
old backends). We've gotten away with this approach in the past, but the
last time was release 6.4. I fully expect to hear more complaints now.
One way to tamp down expectations of client backwards compatibility
would be to call the release 8.0 instead of 7.4 ;-)
Comments?
regards, tom lane
Tom Lane wrote:
<snip>
One way to tamp down expectations of client backwards compatibility
would be to call the release 8.0 instead of 7.4 ;-)Comments?
Actually, I've been thinking about the numbering of the next PostgreSQL
version for a few days now.
The scenario that's appealing to me the most is this for the next release:
PostgreSQL 8.0
**************
+ Includes PITR and the Win32 port
+ Not sure where Satoshi is up to with his 2 phase commit proposal, but
that might make sense to incorporate into a wire protocol revision.
From memory he received funding to work on it, so it might be coming
along nicely.
+ Other things optional of course.
Personally, I'd rather we go for PostgreSQL 8.0, waiting a while extra
for PITR and Win32 if needed, and also properly co-ordinate all of the
release process information (website updates, package builds, Announce
to the mailing lists and news sources).
Regards and best wishes,
Justin Clift
--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
+ Not sure where Satoshi is up to with his 2 phase commit proposal, but
that might make sense to incorporate into a wire protocol revision.
From memory he received funding to work on it, so it might be coming
along nicely.
One should note that his protocol changes had absolutely nothing to do
with 2 phase commits -- but were used as a marker to direct replication.
We may want to consider leaving some space for a server / server style
communication (Cluster ID, etc.)
--
Rod Taylor <rbt@rbt.ca>
PGP Key: http://www.rbt.ca/rbtpub.asc
Tom Lane wrote:
* Backend should pass its version number, database encoding, default
client encoding, and possibly other data (any ideas?) to frontend during
startup, to avoid need for explicit queries to get this info. We could
also consider eliminating SET commands sent by libpq in favor of adding
variable settings to startup packet's PGOPTIONS field. Ideally we could
get back to the point where a standard connection startup takes only one
packet in each direction.
Should we pass this in a way where we can add stuff later, like passing
it as a simple NULL-terminated string that can get split up on the
client end.
One of the $64 questions that has to be answered is how much work we're
willing to expend on backwards compatibility. The path of least
resistance would be to handle it the same way we've done protocol
revisions in the past: the backend will be able to handle both old and new
protocols (so it can talk to old clients) but libpq would be revised to
speak only the new protocol (so new/recompiled clients couldn't talk to
old backends). We've gotten away with this approach in the past, but the
last time was release 6.4. I fully expect to hear more complaints now.
I think such compatibility is sufficient. We can mention in the
releases notes that they should upgrade there servers before their
clients.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073Justin Clift <justin@postgresql.org> writes:
The scenario that's appealing to me the most is this for the next release:
PostgreSQL 8.0
+ Includes PITR and the Win32 port
If the folks doing those things can get done in time, great. I'm even
willing to push out the release schedule (now, not later) to make it
more likely they can get done. What I'm not willing to do is define
the release in terms of "it happens when these things are done". We
learned the folly of that approach in 7.1 and 7.2. Setting a target
date and sticking to it works *much* better.
+ Not sure where Satoshi is up to with his 2 phase commit proposal, but
that might make sense to incorporate into a wire protocol revision.
I can't see any need for protocol-level support for such a thing.
Why wouldn't it just be some more SQL commands?
(Not that I believe in 2PC as a real-world solution anyway, but that's
a different argument...)
regards, tom lane
* Backend's ReadyForQuery message (Z message) should carry an indication
of current transaction status (idle/in transaction/in aborted transaction)
so that frontend need not guess at state. Perhaps also indicate
autocommit status. (Is there anything else that frontends would Really
Like To Know?)
Could it include transaction depth with the assumption nested
transactions will arrive at some point?
* XML support? If we do anything, I'd want some extensible solution to
allowing multiple query-result output formats from the backend, not an
XML-specific hack. For one thing, that would allow the actual appearance
of any XML support to happen later.
One of the $64 questions that has to be answered is how much work we're
willing to expend on backwards compatibility. The path of least
resistance would be to handle it the same way we've done protocol
revisions in the past: the backend will be able to handle both old and new
protocols (so it can talk to old clients) but libpq would be revised to
speak only the new protocol (so new/recompiled clients couldn't talk to
old backends). We've gotten away with this approach in the past, but the
last time was release 6.4. I fully expect to hear more complaints now.
I wouldn't worry about backward compatibility complaints too much BUT
I'd be tempted to make a startup packet that will allow libpq to revert
back to old protocols easily enough for the future so that we can do
incremental changes to the protocol.
--
Rod Taylor <rbt@rbt.ca>
PGP Key: http://www.rbt.ca/rbtpub.asc
Bruce Momjian <pgman@candle.pha.pa.us> writes:
Tom Lane wrote:
* Backend should pass its version number, database encoding, default
client encoding, and possibly other data (any ideas?) to frontend during
startup, to avoid need for explicit queries to get this info.
Should we pass this in a way where we can add stuff later, like passing
it as a simple NULL-terminated string that can get split up on the
client end.
Yeah, I was envisioning something with multiple labeled fields so that
more stuff can be added later without a protocol change (likewise for
StartupPacket and ErrorMessage). But again, I don't want this thread to
get into any details about specific tasks --- let's try to get a view of
the forest before we start hewing down individual trees...
We've gotten away with this approach in the past, but the
last time was release 6.4. I fully expect to hear more complaints now.
I think such compatibility is sufficient. We can mention in the
releases notes that they should upgrade there servers before their
clients.
I'd be really happy if we can make that stick. There's enough work to
be done here without trying to develop a multiprotocol version of
libpq.
It would be good to hear some words from the JDBC and ODBC developers
about what sort of plans they'd have for updating those interfaces.
regards, tom lane
On Mon, 2003-03-10 at 14:05, Justin Clift wrote:
Tom Lane wrote:
<snip>One way to tamp down expectations of client backwards compatibility
would be to call the release 8.0 instead of 7.4 ;-)Comments?
Actually, I've been thinking about the numbering of the next PostgreSQL
version for a few days now.The scenario that's appealing to me the most is this for the next release:
PostgreSQL 8.0
**************+ Includes PITR and the Win32 port
+ Not sure where Satoshi is up to with his 2 phase commit proposal, but
that might make sense to incorporate into a wire protocol revision.
From memory he received funding to work on it, so it might be coming
along nicely.+ Other things optional of course.
Personally, I'd rather we go for PostgreSQL 8.0, waiting a while extra
for PITR and Win32 if needed, and also properly co-ordinate all of the
release process information (website updates, package builds, Announce
to the mailing lists and news sources).
I don't think PITR or Win32 (or even replication) warrant an 8.0, since
none of those should effect client/server interaction and/or backward
compatibility. (Or at least not as much as schema support did, which
required most "adminy" apps to be worked over)
A protocol change however, would warrant a version number bump IMHO. I
would guess that by the time all of the protocol changes could be
completed, we'd have win32 or pitr, so it will hopefully be moot.
Robert Treat
Rod Taylor <rbt@rbt.ca> writes:
I'd be tempted to make a startup packet that will allow libpq to revert
back to old protocols easily enough for the future so that we can do=20
incremental changes to the protocol.
We already have that: you send a startup packet with a version less than
the latest, and the backend speaks that version to you.
One thing I want to do though is relax the protocol-level constraints
on certain message contents: for example, if ErrorMessage becomes a
collection of labeled fields, it should be possible to add new field
types without calling it a protocol revision. The protocol need only
specify "ignore any fields whose label you do not recognize".
regards, tom lane
On Mon, 2003-03-10 at 14:30, Tom Lane wrote:
Rod Taylor <rbt@rbt.ca> writes:
I'd be tempted to make a startup packet that will allow libpq to revert
back to old protocols easily enough for the future so that we can do=20
incremental changes to the protocol.We already have that: you send a startup packet with a version less than
the latest, and the backend speaks that version to you.
Yes, but that requires you know the backend is less than the latest.
If you send version A, and the backend responds don't know A, but I know
A - 2, then libpq may want to try speaking A - 2.
types without calling it a protocol revision. The protocol need only
specify "ignore any fields whose label you do not recognize".
This is probably just as good, if it's done for both sides.
--
Rod Taylor <rbt@rbt.ca>
PGP Key: http://www.rbt.ca/rbtpub.asc
Justin Clift wrote:
PostgreSQL 8.0
**************+ Includes PITR and the Win32 port
*snip*
I feel like the upcoming 7.4 is the most important release since the
introduction of toast, maybe even since the introduction of the sql
language. I wholeheartedly agree with your proposition.
Merlin
Import Notes
Resolved by subject fallback
Rod Taylor <rbt@rbt.ca> writes:
We already have that: you send a startup packet with a version less than
the latest, and the backend speaks that version to you.
Yes, but that requires you know the backend is less than the latest.
As opposed to knowing what? You send the version number you wish to speak;
either the backend can handle it, or not.
regards, tom lane
* Backend should pass its version number, database encoding,
default client encoding, and possibly other data (any ideas?) to
frontend during startup, to avoid need for explicit queries to get
this info. We could also consider eliminating SET commands sent by
libpq in favor of adding variable settings to startup packet's
PGOPTIONS field. Ideally we could get back to the point where a
standard connection startup takes only one packet in each
direction.
How about a bitmap field of "additional features" supported by
client/backend that can be negotiated? (Doesn't have to be bitmap, of
course, but the idea..) That way the server (or client - doesn't matter
who goes first, I guess) could say "I support X and Y, but not Z and Q".
If the client supports both X and Y, both are enabled. If it supports
only X, then only X is enabled.
X and Y? Well, the first thing that comes to mind is SSL support. I'm
not sure if it's still that way, but at least it used to be a pretty
ugly kludge there with the connection being dropped and re-connected in
some cases. I also seem to recall there was some talk about on-wire
compression support - that would also be ideal for something like this.
I'm sure there can be more...
It should be easy to make it version-independent - just make both client
and server reject any "protocol features" that are unknown.
These are features that can be enabled/disabled in a backend/client of
the same version. It cannot (at least not easily) be handled with
protocol versioning, since you can have for example 7.4 with or without
SSL.
//Magnus
Import Notes
Resolved by subject fallback
"Magnus Hagander" <mha@sollentuna.net> writes:
X and Y? Well, the first thing that comes to mind is SSL support. I'm
not sure if it's still that way, but at least it used to be a pretty
ugly kludge there with the connection being dropped and re-connected in
some cases.
SSL support is a bad example, since it would have to be negotiated long
before any more general-purpose negotiation could occur. (You do want
the connection authentication exchange to happen under cover of SSL, no?)
ISTM most of the other features you might want to turn on and off can be
handled as SET commands: the client tries to SET a variable, the backend
either accepts it or returns an error. No need for special protocol
support if you do it that way. Can you point to any examples that have
to have a special protocol feature instead?
regards, tom lane
Tom Lane wrote:
This is an attempt to lay out a road map for updating the frontend/backend
protocol in 7.4. I don't at this point want to get into details on any
one of the TODO items, just get consensus that this is the set of tasks
to be tackled. Are there any areas that I've missed (that require
protocol changes)?
What about binding variables ala oracle's ociparse -> ocibindbyname ->
ociexecute -> ocifetch ? I know you can do most of it via SQL
(PREPARE/EXECUTE) but you can't do 'RETURN x INTO :x' as it stands.
This would also get around the problem of getting values from newly
inserted rows (eg PKs) without resorting to OIDs. Not entirely a FE/BE
issue... but worth considering in any redesign.
Ashley Cambrell
On Tue, 2003-03-11 at 06:52, Tom Lane wrote:
Rod Taylor <rbt@rbt.ca> writes:
We already have that: you send a startup packet with a version less than
the latest, and the backend speaks that version to you.Yes, but that requires you know the backend is less than the latest.
As opposed to knowing what? You send the version number you wish to speak;
either the backend can handle it, or not.
If the backend can not handle the version I request, but can handle a
prior version, I'd like to know. I am planning on having handlers for
multiple protocol versions in the same memory space (I'm using
Smalltalk, BTW) so that one application can talk to various databases of
various vintages.
I suppose that the client can just keep retrying the connection with
different versions until it gets a match, though.
On Mon, 2003-03-10 at 14:52, Tom Lane wrote:
Rod Taylor <rbt@rbt.ca> writes:
We already have that: you send a startup packet with a version less than
the latest, and the backend speaks that version to you.Yes, but that requires you know the backend is less than the latest.
As opposed to knowing what? You send the version number you wish to speak;
either the backend can handle it, or not.
At some point PostgreSQL will have enough users that changing it will
piss them off. If the backend cannot handle whats been requested, we
may want to consider negotiating a protocol that both can handle.
Anyway, it doesn't really affect me one way or the other. So whatever
you like is probably fine.
--
Rod Taylor <rbt@rbt.ca>
PGP Key: http://www.rbt.ca/rbtpub.asc
Bruce Badger <bruce_badger@badgerse.com> writes:
I suppose that the client can just keep retrying the connection with
different versions until it gets a match, though.
Right now, the backend just barfs with
FATAL: unsupported frontend protocol
so you have to do a blind search to see what it will take.
It would probably be a good idea to improve that message to mention
exactly what range of protocol versions the backend does support.
However, this will do you little good for talking to existing backends
:-(
regards, tom lane
On Mon, 2003-03-10 at 16:37, Ashley Cambrell wrote:
This would also get around the problem of getting values from newly
inserted rows (eg PKs) without resorting to OIDs.
That's not a problem: ensure that the newly inserted row has a SERIAL
column, and use currval().
Cheers,
Neil
--
Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
Neil Conway <neilc@samurai.com> writes:
On Mon, 2003-03-10 at 16:37, Ashley Cambrell wrote:
This would also get around the problem of getting values from newly
inserted rows (eg PKs) without resorting to OIDs.
That's not a problem: ensure that the newly inserted row has a SERIAL
column, and use currval().
There was some talk awhile back of inventing INSERT ... RETURNING and
UPDATE ... RETURNING commands so that you could pass back computed
values to the frontend without an extra query. It doesn't seem to have
gotten further than a TODO item yet, though. AFAICS this does not need
a protocol extension, anyway --- it'd look just the same as a SELECT
at the protocol level.
regards, tom lane
