BUG #4053: libpq documentation should express clearly, that integers are passed in network octet order
The following bug has been logged online:
Bug reference: 4053
Logged by: Aleksej Saushev
Email address: asau@inbox.ru
PostgreSQL version: 8.3
Operating system: NetBSD, but this does no matter
Description: libpq documentation should express clearly, that
integers are passed in network octet order
Details:
The documentation on PQexecParams is silent about the precise format,
expected by server for binary parameters, it should be more explicit
and note that integers are expected in network octet order.
Aleksej Saushev wrote:
The following bug has been logged online:
Bug reference: 4053
Logged by: Aleksej Saushev
Email address: asau@inbox.ru
PostgreSQL version: 8.3
Operating system: NetBSD, but this does no matter
Description: libpq documentation should express clearly, that
integers are passed in network octet order
Details:The documentation on PQexecParams is silent about the precise format,
expected by server for binary parameters, it should be more explicit
and note that integers are expected in network octet order.
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote:
Aleksej Saushev wrote:
The following bug has been logged online:
Bug reference: 4053
Logged by: Aleksej Saushev
Email address: asau@inbox.ru
PostgreSQL version: 8.3
Operating system: NetBSD, but this does no matter
Description: libpq documentation should express clearly, that
integers are passed in network octet order
Details:The documentation on PQexecParams is silent about the precise format,
expected by server for binary parameters, it should be more explicit
and note that integers are expected in network octet order.This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?
They don't have to if they use the libpqtypes library :)
project: http://pgfoundry.org/projects/libpqtypes/
online docs: http://libpqtypes.esilo.com/
andrew
Bruce Momjian wrote:
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?
This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.
merlin
Import Notes
Resolved by subject fallback
"Merlin Moncure" <mmoncure@gmail.com> writes:
Bruce Momjian wrote:
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.
No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.
Aleksej Saushev wrote:
"Merlin Moncure" <mmoncure@gmail.com> writes:
Bruce Momjian wrote:
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.
Agreed.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote:
Aleksej Saushev wrote:
"Merlin Moncure" <mmoncure@gmail.com> writes:
Bruce Momjian wrote:
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.Agreed.
Correct, libpqtypes is not part of the core.
What we are saying is, libpqtypes solves this problem. I don't think
docs is the solution because that just supports the idea of libpq apps
directly handling binary formats; which makes changing these formats
MUCH harder in future releases. Lots of libpq apps would be dependant
on these binary formats.
By all means, documenting this is probably a good idea. I just don't
think it solves any of the interesting problems.
--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/
Andrew Chernow <ac@esilo.com> writes:
Bruce Momjian wrote:
Aleksej Saushev wrote:
"Merlin Moncure" <mmoncure@gmail.com> writes:
Bruce Momjian wrote:
This brings up a good question. Exactly how do users know what format
_binary_ is? int4 is network byte order, but what about int8, float4,
inet?This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.Agreed.
Correct, libpqtypes is not part of the core.
What we are saying is, libpqtypes solves this problem. I don't
think docs is the solution because that just supports the idea
of libpq apps directly handling binary formats; which makes
changing these formats MUCH harder in future releases. Lots of
libpq apps would be dependant on these binary formats.
Right, but that's upon developers to decide, if they're going to
support changing API. I'm pretty sure, I won't use it much anyway,
and internal documentation is good enough to fix the problem
in future.
By all means, documenting this is probably a good idea. I just
don't think it solves any of the interesting problems.
The easiest way to address the lack of documentation is adding
reference to libpqtypes at appropriate place. Shipping libpqtypes
along is even better.
Andrew Chernow wrote:
This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.Agreed.
Correct, libpqtypes is not part of the core.
What we are saying is, libpqtypes solves this problem. I don't think
docs is the solution because that just supports the idea of libpq apps
directly handling binary formats; which makes changing these formats
MUCH harder in future releases. Lots of libpq apps would be dependant
on these binary formats.By all means, documenting this is probably a good idea. I just don't
think it solves any of the interesting problems.
Agreed that libpqtypes would solve the problem, but the odd thing to me
is that we have gotten very few requests for binary format information
since we added binary prepared parameters years ago, so is no one
actually using it?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
On Thu, May 8, 2008 at 12:27 PM, Bruce Momjian <bruce@momjian.us> wrote:
Andrew Chernow wrote:
This is exactly what libpqtypes solves. Not only do we handle
formatting of binary formats, we provide a level of protection from
internal format changes for libpq users. See the example here:
http://libpqtypes.esilo.com/. So, documentation of binary formats
(including network byte ordering) are not required.No, it is still required. There's not a single reference to libpqtypes
in Postgres documentation, and libpqtypes isn't part of the distribution,
if I understand it right.Agreed.
Correct, libpqtypes is not part of the core.
What we are saying is, libpqtypes solves this problem. I don't think
docs is the solution because that just supports the idea of libpq apps
directly handling binary formats; which makes changing these formats
MUCH harder in future releases. Lots of libpq apps would be dependant
on these binary formats.By all means, documenting this is probably a good idea. I just don't
think it solves any of the interesting problems.Agreed that libpqtypes would solve the problem, but the odd thing to me
is that we have gotten very few requests for binary format information
since we added binary prepared parameters years ago, so is no one
actually using it?
libpqtypes is safe to use right now if you are willing to patch libpq
yourself back to at least 8.2. back to 7.4 is possible but some
internal formats may have changes (we didn't check back that far) so
you'd have to be a little careful.
If I searched the archives, I could probably come up with several
requests for documentation/explanations on how to use the binary
format, with at least one or two other patches to do byteswapping.
Since libpqtypes automatically parameterizes your query and handles
all the details of byteswapping, why would you ever want to do it
another way? What possible advantage would there be to ever invoking
PQexecParams manually? (PQexec has a small speed advantage for simple
queries, so that at least still has a place).
Add this to the fact that you now have a clean way to handle things
like arrays and composites without all the tedious and error prone
text parsing. I understand arguments related to code maintenance and
the like, but for libpq developers, there would never be a reason
(outside of occasional PQexec), to run a non 'params' based query
because it's faster, easier, and safer.
It's a catch-22 here...a lot of people are simply unaware of what the
postgresql binary flag is even supposed to do, since it's so poorly
documented. By the way, even with documentation, any app relying on
binary formats without the protection of libpqtypes could suddenly and
spectacularly break since there is zero protection from binary format
changes.
merlin
Bruce Momjian wrote:
Agreed that libpqtypes would solve the problem, but the odd thing to me
is that we have gotten very few requests for binary format information
since we added binary prepared parameters years ago, so is no one
actually using it?
I think part of the answer is lack of documentation and possibly how
tedious of an API call PQexecParams can be. If anything, using the
parameterized method promotes much better "safer" coding practices, but
currently its at quite a programming effort cost.
--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/
Aleksej Saushev wrote:
The following bug has been logged online:
Bug reference: 4053
Logged by: Aleksej Saushev
Email address: asau@inbox.ru
PostgreSQL version: 8.3
Operating system: NetBSD, but this does no matter
Description: libpq documentation should express clearly, that
integers are passed in network octet order
Details:The documentation on PQexecParams is silent about the precise format,
expected by server for binary parameters, it should be more explicit
and note that integers are expected in network octet order.
I have added this libpq documentation paragraph about how to determine
how to pass binary values. I also added a few comments to the libpq C
code.
Thanks for the report; please let me know if I can improve it more.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +