real/float example for testlibpq3

Hi Mark,
Fetching a "real" type field from libpq doesn't look that intuitive.
So this example is super helpful. Thanks.

I am wondering whether we should provide PQgetfloatvalue() which does
what that example shows but transparently. It will return a float
value of the given field instead of char *. That will standardize the
way to fetch real typed values in libpq. That leads to the next
question. Do we need to introduce different PQget*value() for standard
C/SQL data types.

--
Best Wishes,
Ashutosh Bapat

Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> writes:

I am wondering whether we should provide PQgetfloatvalue() which does
what that example shows but transparently. It will return a float
value of the given field instead of char *.

I'm against that, precisely because ...

That will standardize the
way to fetch real typed values in libpq. That leads to the next
question. Do we need to introduce different PQget*value() for standard
C/SQL data types.

... I do not want to go here. Where would you stop? How would you
deal with cross-machine inconsistencies in integer widths?

regards, tom lane

On 02/28/22 10:19, Tom Lane wrote:

That will standardize the
way to fetch real typed values in libpq. That leads to the next
question. Do we need to introduce different PQget*value() for standard
C/SQL data types.

... I do not want to go here. Where would you stop? How would you
deal with cross-machine inconsistencies in integer widths?

This stimulates a question for me.

Not just libpq, but all drivers implemented in whatever language, if they
wish to support binary protocol, depend on knowing what the committed
send/recv wire formats are for whichever types they mean to support.

In the current state of affairs, what's considered the ur-source of that
information?

I have often seen those formats documented in code comments, usually above
the recv function in the .c file for a given adt.

Have we got any more, well, machine-readable collection of that knowledge?

Regards,
-Chap

Chapman Flack <chap@anastigmatix.net> writes:

This stimulates a question for me.

Not just libpq, but all drivers implemented in whatever language, if they
wish to support binary protocol, depend on knowing what the committed
send/recv wire formats are for whichever types they mean to support.

In the current state of affairs, what's considered the ur-source of that
information?

The source code for the type's send/receive functions :-(. One could
wish for something better, but no one has stepped up to produce such
documentation.

regards, tom lane

On Mon, 28 Feb 2022 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Chapman Flack <chap@anastigmatix.net> writes:

In the current state of affairs, what's considered the ur-source of that
information?

The source code for the type's send/receive functions :-(. One could
wish for something better, but no one has stepped up to produce such
documentation.

Fwiw the client library I heard of attempting to have good binary mode
support was the Crystal language client
https://github.com/will/crystal-pg. I think he was aiming for full
coverage of the built-in data types. That might make a good reference
implementation to write up documentation from. He probably uncovered
some corner cases in development that one might not find from just
inspection of the server code.

--
greg

On Wed, Mar 30, 2022 at 01:16:37PM -0400, Greg Stark wrote:

On Mon, 28 Feb 2022 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Chapman Flack <chap@anastigmatix.net> writes:

In the current state of affairs, what's considered the ur-source of that
information?

The source code for the type's send/receive functions :-(. One could
wish for something better, but no one has stepped up to produce such
documentation.

Fwiw the client library I heard of attempting to have good binary mode
support was the Crystal language client
https://github.com/will/crystal-pg. I think he was aiming for full
coverage of the built-in data types. That might make a good reference
implementation to write up documentation from. He probably uncovered
some corner cases in development that one might not find from just
inspection of the server code.

Checking in for quick feedback to see if this refactor makes sense.

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.

I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

Regards,
Mark

On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:

Checking in for quick feedback to see if this refactor makes sense.

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.

I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

I think this could be a reasonable kind of thing to do, but I think
you left the "ts" output out of the example output in the comments,
and also your code's apparently not portable, because my compiler is
OK with testlibpq3 right now, but with your patch it emits lengthy
unhappy moaning, starting with:

testlibpq3.c:88:10: error: expected ')'
uint64_t ntohll(uint64_t);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_endian.h:140:25:
note: expanded from macro 'ntohll'
#define ntohll(x) __DARWIN_OSSwapInt64(x)
^

Apparently macOS defines ntohll as a macro, and it's not amused by
your attempt to include a function prototype for it.

I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program. But
doing this much seems OK to me.

--
Robert Haas
EDB: http://www.enterprisedb.com

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.
I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program. But
doing this much seems OK to me.

Yeah, "hiding impromptu documentation inside a test program" is what
this looks like, and I'm not sure that's a reasonable way to go.

(1) Who's going to think to look in src/test/examples/testlibpq3.c for
documentation of binary formats?

(2) The useful details are likely to get buried in notational and
portability concerns, as I think your build failure illustrates.

(3) I bet few if any packagers install these files, so that the new
info would be unavailable to many people.

I think some new appendix in the main SGML docs would be the appropriate
place if we want to provide real documentation.

regards, tom lane

On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.
I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program. But
doing this much seems OK to me.

Yeah, "hiding impromptu documentation inside a test program" is what
this looks like, and I'm not sure that's a reasonable way to go.

(1) Who's going to think to look in src/test/examples/testlibpq3.c for
documentation of binary formats?

(2) The useful details are likely to get buried in notational and
portability concerns, as I think your build failure illustrates.

(3) I bet few if any packagers install these files, so that the new
info would be unavailable to many people.

I think some new appendix in the main SGML docs would be the appropriate
place if we want to provide real documentation.

Ok, I'll leave the testlibpq3.c as it was then. If it's worth keeping
any of those changes, then I can remove the timestamp example because of
the ntohll() portability since that is trivial.

I'll start a new appendix and share again when I have something to show.

Regards,
Mark

--
Mark Wong
EDB: http://www.enterprisedb.com

On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote:

Robert Haas <robertmhaas@gmail.com> writes:

On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.
I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program. But
doing this much seems OK to me.

Yeah, "hiding impromptu documentation inside a test program" is what
this looks like, and I'm not sure that's a reasonable way to go.

(1) Who's going to think to look in src/test/examples/testlibpq3.c for
documentation of binary formats?

(2) The useful details are likely to get buried in notational and
portability concerns, as I think your build failure illustrates.

(3) I bet few if any packagers install these files, so that the new
info would be unavailable to many people.

I think some new appendix in the main SGML docs would be the appropriate
place if we want to provide real documentation.

Just wanted to do another quick check in before I go too far. How do
does this start look? Extending the libpq section with a code snippet
and description per data type?

Regards,
Mark

On 01.07.22 19:18, Mark Wong wrote:

Just wanted to do another quick check in before I go too far. How do
does this start look? Extending the libpq section with a code snippet
and description per data type?

If we want to start documenting the binary format for all data types, it
should go into the "Data Types" chapter. There, we already document the
text format in great detail, so putting the binary format in there as
well would make sense.

The libpq chapter could then contain some examples of applying that
information in the context of libpq. But the libpq documentation
shouldn't be the primary place where the formats are documented.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

If we want to start documenting the binary format for all data types, it
should go into the "Data Types" chapter. There, we already document the
text format in great detail, so putting the binary format in there as
well would make sense.

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.

regards, tom lane

On 01.11.22 09:15, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

If we want to start documenting the binary format for all data types, it
should go into the "Data Types" chapter. There, we already document the
text format in great detail, so putting the binary format in there as
well would make sense.

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.

Would that clutter the protocol docs? ;-)

I suppose figuring out exactly where to put it and how to mark it up,
etc., in a repeatable fashion is part of the job here.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 01.11.22 09:15, Tom Lane wrote:

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.

Would that clutter the protocol docs? ;-)

I said "near", not "in". At the time I was thinking "new appendix",
but I now recall that the wire protocol docs are not an appendix
but a chapter in the Internals division. So that doesn't seem like
quite the right place anyway.

Perhaps a new chapter under "IV. Client Interfaces" is the right
place?

If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.

I suppose figuring out exactly where to put it and how to mark it up,
etc., in a repeatable fashion is part of the job here.

Yup.

regards, tom lane

On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 01.11.22 09:15, Tom Lane wrote:

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.

Would that clutter the protocol docs? ;-)

I said "near", not "in". At the time I was thinking "new appendix",
but I now recall that the wire protocol docs are not an appendix
but a chapter in the Internals division. So that doesn't seem like
quite the right place anyway.

Perhaps a new chapter under "IV. Client Interfaces" is the right
place?

If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.

I suppose figuring out exactly where to put it and how to mark it up,
etc., in a repeatable fashion is part of the job here.

Yup.

I'll take a stab at adding a new chapter and share how that looks.

Regards,
Mark

--
Mark Wong
EDB https://enterprisedb.com

On Wed, Nov 09, 2022 at 12:47:23PM -0800, Mark Wong wrote:

I'll take a stab at adding a new chapter and share how that looks.

This thread has stalled for three weeks, so I have marked it as
returned with feedback. Once you have a new patch, please feel free
to submit it.
--
Michael

On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote:

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:

On 01.11.22 09:15, Tom Lane wrote:

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.

Would that clutter the protocol docs? ;-)

I said "near", not "in". At the time I was thinking "new appendix",
but I now recall that the wire protocol docs are not an appendix
but a chapter in the Internals division. So that doesn't seem like
quite the right place anyway.

Perhaps a new chapter under "IV. Client Interfaces" is the right
place?

If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.

I suppose figuring out exactly where to put it and how to mark it up,
etc., in a repeatable fashion is part of the job here.

Yup.

How does this look?

I've simply moved things around into a new "Binary Format" section with
the few parts that I've started for some quick feedback about whether
this is looking like the right landing place.

Regards,
Mark

Mark Wong <markwkm@gmail.com> writes:

On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote:

Perhaps a new chapter under "IV. Client Interfaces" is the right
place?

If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.

I suppose figuring out exactly where to put it and how to mark it up,
etc., in a repeatable fashion is part of the job here.

How does this look?

I've simply moved things around into a new "Binary Format" section with
the few parts that I've started for some quick feedback about whether
this is looking like the right landing place.

I took a quick look at this. The patch cannot be applied as given
because it seems to be editing binary-format.sgml from some prior
state, but that file doesn't exist at all yet. However, there's
enough visible in the patch to comment on.

Personally I'm not excited about supplying fragments of C code
like this. Those seem quite useless to non-C users, and I'm not
really sure that they'd be very helpful even for C users, because
you have to make a whole lot of assumptions about what the user
wants to do with the value. I think what *would* be helpful is
just plain prose documentation of the on-the-wire binary format.

I don't mind if you write something like

A float4 value is a 4-byte IEEE single-precision floating point
number. It is transmitted in network byte order, so you must
convert to local byte order. (C users can do this portably
using the standard ntohl() function.)

but I'm not sure an example is worth more than such a parenthetical
comment. Perhaps others disagree, though.

regards, tom lane

On Fri, Jan 20, 2023 at 12:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I don't mind if you write something like

A float4 value is a 4-byte IEEE single-precision floating point
number. It is transmitted in network byte order, so you must
convert to local byte order. (C users can do this portably
using the standard ntohl() function.)

but I'm not sure an example is worth more than such a parenthetical
comment. Perhaps others disagree, though.

I don't disagree with that.

I do think that when you suggested documenting this rather than just
adding some examples, you moved the goalposts a long way. If we're
going to add this to the documentation, it probably ought to cover
every data type we ship. Overall, I think that would be a better
result than just adding a few examples for the most common data types
to testlibpq*.c, but it's also substantially more work. I do agree
with you that if we're going to document this, as opposed to provide
examples, then a narrative style is more appropriate than a whole
bunch of small sample programs; maintaining working code in the
documentation seems like an annoying amount of maintenance and is
probably not the most efficient way to communicate useful information.

--
Robert Haas
EDB: http://www.enterprisedb.com