PRI?64 vs Visual Studio (2022)

On 31.03.25 08:28, Kyotaro Horiguchi wrote:

If you're already aware of this and have taken it into account, please
feel free to ignore this.

As described in the recent commit a0ed19e0a9e, many %ll? format
specifiers are being replaced with %<PRI?64>.

I hadn’t paid much attention to this before, but I happened to check
how this behaves on Windows, and it seems that with VS2022, PRId64
expands to "%lld". As a result, I suspect the gettext message catalog
won't match these messages correctly.

I think this is working correctly. Gettext has a built-in mechanism to
translate the %<PRI...> back to the appropriate %lld or %ld. See also
<https://www.gnu.org/software/gettext/manual/html_node/c_002dformat.html&gt;.

On Wed, Apr 2, 2025 at 2:04 AM Peter Eisentraut <peter@eisentraut.org> wrote:

On 31.03.25 08:28, Kyotaro Horiguchi wrote:

I hadn’t paid much attention to this before, but I happened to check
how this behaves on Windows, and it seems that with VS2022, PRId64
expands to "%lld". As a result, I suspect the gettext message catalog
won't match these messages correctly.

I think this is working correctly. Gettext has a built-in mechanism to
translate the %<PRI...> back to the appropriate %lld or %ld. See also
<https://www.gnu.org/software/gettext/manual/html_node/c_002dformat.html&gt;.

Interesting report though. Commit 962da900 assumed that our in-tree
printf implementation still needed to understand that %I64 stuff in
case it came to us from system headers, but it looks like it
disappeared with MSVCRT:

1. I checked with CI (VS 2019). puts(PRId64) prints out "lld".
2. MinGW's inttypes.h[1]https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/inttypes.h only uses "I64" et al if you build against MSVCRT.

So I think we should delete that stuff. Attached.

I worried that GNU gettext() might still know about %I64 somewhere,
but it just expands the macros to whatever inttypes.h defines[2]https://github.com/autotools-mirror/gettext/blob/637b208fbe13f1c306f19d4f31c21fec7e9986d2/gettext-runtime/intl/loadmsgcat.c#L473.
Good.

We don't even test -Dnls on the Windows CI task, so the fact that it
passes there doesn't mean much (if our tests would even pick up
<PRI*64> expansion failure, not sure). We should probably do
something about that and/or its absence from the build farm. We're
effectively counting on the EDB packaging team or end users to tell us
if we break localisation on this platform.

I was also curious to know if the nearby floating point formatting
kludge added by commit f1885386 was still needed today. CI passes
without it, and the standard is pretty clear: "The exponent always
contains at least two digits, and only as many more digits as
necessary to represent the exponent". I didn't look too closely at
the fine print, but that text was already present in C89 so I guess
MSVCRT just failed to conform on that point.

[1]: https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/inttypes.h
[2]: https://github.com/autotools-mirror/gettext/blob/637b208fbe13f1c306f19d4f31c21fec7e9986d2/gettext-runtime/intl/loadmsgcat.c#L473

Thomas Munro <thomas.munro@gmail.com> writes:

We don't even test -Dnls on the Windows CI task, so the fact that it
passes there doesn't mean much (if our tests would even pick up
<PRI*64> expansion failure, not sure). We should probably do
something about that and/or its absence from the build farm. We're
effectively counting on the EDB packaging team or end users to tell us
if we break localisation on this platform.

I'm pretty certain that we do not test NLS localization at all,
anywhere :-(. (There are no test cases checking enable_nls,
which would be a necessary thing to not fail on buildfarm critters
not using NLS.)

I agree that starting to rely on PRI?64 in translatable strings
is raising the bar a good deal, so maybe it's time to do something
about that.

regards, tom lane

On Wed, Nov 19, 2025 at 3:28 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I agree that starting to rely on PRI?64 in translatable strings
is raising the bar a good deal, so maybe it's time to do something
about that.

Perhaps meson/configure should do a po -> mo -> gettext() check with a
canned test message? That'd also make sure your msgfmt and libintl
are compatible, something I worried about when I wrote about musl
recently.

Thomas Munro <thomas.munro@gmail.com> writes:

Perhaps meson/configure should do a po -> mo -> gettext() check with a
canned test message? That'd also make sure your msgfmt and libintl
are compatible, something I worried about when I wrote about musl
recently.

No, I don't think that's a good approach. That is testing the library
available at configure time, not the one you are actually running
with (possibly years later and on a different machine, even without
considering cross-compilation cases). I think we should do it
honestly with a regression test. It doesn't need to be very
complicated --- I think checking one message in one translation is
sufficient, so long as it includes a PRI?64 usage.

regards, tom lane

On 19.11.25 04:15, Tom Lane wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

Perhaps meson/configure should do a po -> mo -> gettext() check with a
canned test message? That'd also make sure your msgfmt and libintl
are compatible, something I worried about when I wrote about musl
recently.

No, I don't think that's a good approach. That is testing the library
available at configure time, not the one you are actually running
with (possibly years later and on a different machine, even without
considering cross-compilation cases). I think we should do it
honestly with a regression test. It doesn't need to be very
complicated --- I think checking one message in one translation is
sufficient, so long as it includes a PRI?64 usage.

We could generate an English message catalog that translates all
messages unchanged, and run the whole test suite with that. This would
exercise the whole gettext run-time machinery.

Generating the message catalog is easy, gettext provides a tool for
that. What's a little tricky is convincing all our testing
infrastructure to *not* disable NLS-related locale settings. See
attached for a rough, incomplete demo.

On 19.11.25 03:13, Thomas Munro wrote:

Interesting report though. Commit 962da900 assumed that our in-tree
printf implementation still needed to understand that %I64 stuff in
case it came to us from system headers, but it looks like it
disappeared with MSVCRT:

1. I checked with CI (VS 2019). puts(PRId64) prints out "lld".
2. MinGW's inttypes.h[1] only uses "I64" et al if you build against MSVCRT.

So I think we should delete that stuff. Attached.

Looks good to me.

Peter Eisentraut <peter@eisentraut.org> writes:

On 19.11.25 04:15, Tom Lane wrote:

I think we should do it
honestly with a regression test. It doesn't need to be very
complicated --- I think checking one message in one translation is
sufficient, so long as it includes a PRI?64 usage.

We could generate an English message catalog that translates all
messages unchanged, and run the whole test suite with that. This would
exercise the whole gettext run-time machinery.

... except that if it were actually doing nothing whatsoever, you
could not tell. This seems particularly troublesome for gettext,
since its fallback behavior is exactly to return the given string.
I'd prefer a test that fails in a visible way.

regards, tom lane

On 2025-Nov-19, Tom Lane wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

On 19.11.25 04:15, Tom Lane wrote:

I think we should do it
honestly with a regression test. It doesn't need to be very
complicated --- I think checking one message in one translation is
sufficient, so long as it includes a PRI?64 usage.

We could generate an English message catalog that translates all
messages unchanged, and run the whole test suite with that. This would
exercise the whole gettext run-time machinery.

... except that if it were actually doing nothing whatsoever, you
could not tell.

You could feed the message catalog a translated string that differs from
the original in some simple way, say, by adding a constant prefix
"[translated]" or something like that.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/

On Wed, Nov 19, 2025 at 9:07 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:

You could feed the message catalog a translated string that differs from
the original in some simple way, say, by adding a constant prefix
"[translated]" or something like that.

`xgettext -m` can do that. (But I wish I'd known about msgen earlier...)

We could additionally use preloadable_libintl.so, in combination with
GETTEXT_LOG_UNTRANSLATED, and check if the log contains entries from
our domains. I was doing that just last week. But beware that the log
file can grow very quickly. And we'd probably have to differentiate
the "no domain" text belonging to other software from accidental
no-domain strings in our own code, like what I described in [1]/messages/by-id/CAOYmi+kQQ8vpRcoSrA5EQ98Wa3G6jFj1yRHs6mh1V7ohkTC7JA@mail.gmail.com.

--Jacob

[1]: /messages/by-id/CAOYmi+kQQ8vpRcoSrA5EQ98Wa3G6jFj1yRHs6mh1V7ohkTC7JA@mail.gmail.com

On Thu, Nov 20, 2025 at 4:44 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter@eisentraut.org> writes:

On 19.11.25 04:15, Tom Lane wrote:

I think we should do it
honestly with a regression test. It doesn't need to be very
complicated --- I think checking one message in one translation is
sufficient, so long as it includes a PRI?64 usage.

We could generate an English message catalog that translates all
messages unchanged, and run the whole test suite with that. This would
exercise the whole gettext run-time machinery.

... except that if it were actually doing nothing whatsoever, you
could not tell. This seems particularly troublesome for gettext,
since its fallback behavior is exactly to return the given string.
I'd prefer a test that fails in a visible way.

How about a test module with a test_nls() function that just raises an
error containing "hello %" PRId64 ", ..." with all the macros we care
about, and regression test that calls it, and two alternative expected
files with "hello ...", "hola ...", matching en.po and es.po (or
choose some other second language that we think is likely to be tested
by a subset of BF animals and/or a CI task)? Then if you didn't
enable -Dnls it'd still pass with English, and if you did it'd pass
for any language. Since there are no other .po files, if you had some
third language it'd fall back to English, and the .po would have a "do
not translate" comment or even prefix in the message to avoid
confusing the translation team. That assumes that modules are allowed
to supply .po files, I didn't check, if that's not true then maybe
it'd have to be in core instead. That'd test quite a lot of moving
parts at once.

The reason I thought about a contrived message with lots of macros is
that I'd stumbled across a partial implementation[1]/messages/by-id/CA+hUKG+pp==d-3LVhdNOvOAzwQN0vP4gBSxtHkmxnmfQD3NY=w@mail.gmail.com in Alpine's
alternative non-GNU msgfmt program, which appears to have PRIu64 but
not PRIx64 and others. It also has some other way of encoding this
stuff in the .mo that musl's alternative built-in libintl
implementation can understand (it looks like they have arranged to be
able to mmap the .mo and use it directly as shared read-only memory,
while GNU's implementation has to allocate memory to translate them to
%lld etc in every process, clever but (I assume) broken if
msgfmt/libintl implementations are mixed), so I figured it'd be a good
idea to make sure that we test that all the macros actually work. I
didn't try to understand the implications of Wolfgang's reply, but I
guess that to have any chance of Alpine's libintl pickle being
straightened out, we'd ideally want a test case that someone
interested in that could use to validate the whole localisation
pipeline conclusively.

[1]: /messages/by-id/CA+hUKG+pp==d-3LVhdNOvOAzwQN0vP4gBSxtHkmxnmfQD3NY=w@mail.gmail.com

On Thu, Nov 20, 2025 at 6:07 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:

You could feed the message catalog a translated string that differs from
the original in some simple way, say, by adding a constant prefix
"[translated]" or something like that.

Oh, that's probably better than my nearby en.po + es.po suggestion.
Combining the ideas, you could have just an en.po translation, but
expected files to match "hello ..." and "[translated] hello ...".
Though, hmm, I suppose that fails to fail if it didn't translate when
it should have, so maybe a TAP test or a test_nls() function that
internally checks the translation rather than using error() and
expected files...

Thomas Munro <thomas.munro@gmail.com> writes:

On Thu, Nov 20, 2025 at 6:07 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:

You could feed the message catalog a translated string that differs from
the original in some simple way, say, by adding a constant prefix
"[translated]" or something like that.

Oh, that's probably better than my nearby en.po + es.po suggestion.
Combining the ideas, you could have just an en.po translation, but
expected files to match "hello ..." and "[translated] hello ...".
Though, hmm, I suppose that fails to fail if it didn't translate when
it should have,

Yeah. I think it's critical that the test be set up so that
failure-to-translate cannot look like a success.

I agree with the idea of just using a single test message that checks
all the PRI* macros we care about. I don't think we need to invent a
whole new translation for this. I'd be inclined to just get the
desired translated string pushed into one or two .po files in HEAD,
then we can start testing with those specific languages, and we're
good. Over time the translators would presumably get translations
into other .po files, and then maybe we'd want to expand the set of
tested languages, or maybe that wouldn't really buy much. (Managing
the encoding of the expected-file might be tricky if you got too
ambitious about that.)

regards, tom lane

I wrote:

I agree with the idea of just using a single test message that checks
all the PRI* macros we care about. I don't think we need to invent a
whole new translation for this. I'd be inclined to just get the
desired translated string pushed into one or two .po files in HEAD,
then we can start testing with those specific languages, and we're
good. Over time the translators would presumably get translations
into other .po files, and then maybe we'd want to expand the set of
tested languages, or maybe that wouldn't really buy much. (Managing
the encoding of the expected-file might be tricky if you got too
ambitious about that.)

Just as proof-of-concept, this is approximately what I think we
should do to begin with.

The main thing that's likely wrong here is that I just manually
shoved a new entry into src/backend/po/es.po. I suspect that
the .po-extraction machinery would fail to pick up that string
because it's in src/test/regress/regress.c. We could hack it
to do that, or we could put the test function into some backend
file. I don't have much sense of which would be cleaner.

Lesser loose ends: I didn't bother fleshing out the test message
to cover all of the likely PRI* cases, and my Spanish probably
sucks. I'm also unsure if this will work as-is on Windows;
are the LC_MESSAGES settings the same there?

regards, tom lane

On Thu, Nov 20, 2025 at 11:23 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm also unsure if this will work as-is on Windows;
are the LC_MESSAGES settings the same there?

Bilal [CC'd], have you ever looked into gettext support for Windows
CI? I think we'd need at least msgfmt.exe, libintl.{dll,lib,h}
installed on the image, though I have no clue which
distribution/package/whatever would be appropriate. I assume a script
in pg-vm-images[1]https://github.com/anarazel/pg-vm-images/tree/main/scripts would need to install that, once we pick one. Does
anyone happen to know where EDB's installer pipeline pulls gettext
from?

[1]: https://github.com/anarazel/pg-vm-images/tree/main/scripts

I wrote:

The main thing that's likely wrong here is that I just manually
shoved a new entry into src/backend/po/es.po. I suspect that
the .po-extraction machinery would fail to pick up that string
because it's in src/test/regress/regress.c. We could hack it
to do that, or we could put the test function into some backend
file. I don't have much sense of which would be cleaner.

Oh, better idea about that: let's make regress.so have its own
translation domain. This allows testing the TEXTDOMAIN mechanism
as well as the basics, and it keeps the patch pretty self-contained.

I was amused to see that "make update-po" was able to fill in
translations for all of the pre-existing ereport's in regress.c.
I guess they all had duplicates somewhere else? But I take no
credit or blame for any of those translations.

The other loose ends remain.

regards, tom lane

Hi,

On Thu, 20 Nov 2025 at 01:44, Thomas Munro <thomas.munro@gmail.com> wrote:

On Thu, Nov 20, 2025 at 11:23 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I'm also unsure if this will work as-is on Windows;
are the LC_MESSAGES settings the same there?

Bilal [CC'd], have you ever looked into gettext support for Windows
CI? I think we'd need at least msgfmt.exe, libintl.{dll,lib,h}
installed on the image, though I have no clue which
distribution/package/whatever would be appropriate. I assume a script
in pg-vm-images[1] would need to install that, once we pick one. Does
anyone happen to know where EDB's installer pipeline pulls gettext
from?

[1] https://github.com/anarazel/pg-vm-images/tree/main/scripts

Yes, I was working on that some time ago and I was able to enable NLS
(and many other dependencies) on Windows CI image by using the
dependencies from Dave Page's winpgbuild repository [1]https://github.com/dpage/winpgbuild. I was going
to share these changes but some other things came up and this one got
delayed. I am planning to return to this again soon.

As an example, I re-generated the Windows CI image and tested it with
VS 2019 [2]https://cirrus-ci.com/task/4655787001249792 and VS 2022 [3]https://cirrus-ci.com/task/5786281818456064. There are 3 tests failed on both but they
are not related to NLS. A portion of configure output:

[07:40:23.149] External libraries
[07:40:23.149] bonjour : NO
[07:40:23.149] bsd_auth : NO
[07:40:23.149] docs : YES
[07:40:23.149] docs_pdf : NO
[07:40:23.149] gss : YES 1.22.1
[07:40:23.149] icu : YES 77.1
[07:40:23.149] ldap : YES
[07:40:23.149] libcurl : NO
[07:40:23.149] libnuma : NO
[07:40:23.149] liburing : NO
[07:40:23.149] libxml : YES 2.13.9
[07:40:23.149] libxslt : YES 1.1.43
[07:40:23.149] llvm : NO
[07:40:23.149] lz4 : YES 1.10.0
[07:40:23.149] nls : YES
[07:40:23.149] openssl : YES 3.0.18
[07:40:23.149] pam : NO
[07:40:23.149] plperl : YES 5.42.0
[07:40:23.149] plpython : YES 3.10
[07:40:23.149] pltcl : NO
[07:40:23.149] readline : NO
[07:40:23.149] selinux : NO
[07:40:23.149] systemd : NO
[07:40:23.149] uuid : YES 1.6.2
[07:40:23.149] zlib : YES 1.3.1
[07:40:23.149] zstd : YES 1.5.7

[1]: https://github.com/dpage/winpgbuild
[2]: https://cirrus-ci.com/task/4655787001249792
[3]: https://cirrus-ci.com/task/5786281818456064

--
Regards,
Nazir Bilal Yavuz
Microsoft

On Wed, Nov 19, 2025 at 3:13 PM Thomas Munro <thomas.munro@gmail.com> wrote:

I was also curious to know if the nearby floating point formatting
kludge added by commit f1885386 was still needed today. CI passes
without it, and the standard is pretty clear: "The exponent always
contains at least two digits, and only as many more digits as
necessary to represent the exponent". I didn't look too closely at
the fine print, but that text was already present in C89 so I guess
MSVCRT just failed to conform on that point.

We can also drop HAVE_BUGGY_STRTOF for MinGW. This passes on CI.

That'd leave only Cygwin with HAVE BUGGY_STRTOF. Perhaps they have
fixed their implementation[1]https://github.com/cygwin/cygwin/commit/fb01286fab9b370c86323f84a46285cfbebfe4ff? Here's an experimental patch to drop
all remnants, which could be used to find out. No Windows/Cygwin
here. Hmm, what if we just commit it anyway? If their strtof() is
still broken and someone out there is running the tests and sees this
test fail, why shouldn't they take that up with libc at this stage?

[1]: https://github.com/cygwin/cygwin/commit/fb01286fab9b370c86323f84a46285cfbebfe4ff

Thomas Munro <thomas.munro@gmail.com> writes:

That'd leave only Cygwin with HAVE BUGGY_STRTOF. Perhaps they have
fixed their implementation[1]? Here's an experimental patch to drop
all remnants, which could be used to find out. No Windows/Cygwin
here. Hmm, what if we just commit it anyway? If their strtof() is
still broken and someone out there is running the tests and sees this
test fail, why shouldn't they take that up with libc at this stage?

Hmm, we could get rid of the whole resultmap mechanism ...

regards, tom lane

On Sun, Nov 23, 2025 at 4:25 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

That'd leave only Cygwin with HAVE BUGGY_STRTOF. Perhaps they have
fixed their implementation[1]? Here's an experimental patch to drop
all remnants, which could be used to find out. No Windows/Cygwin
here. Hmm, what if we just commit it anyway? If their strtof() is
still broken and someone out there is running the tests and sees this
test fail, why shouldn't they take that up with libc at this stage?

Hmm, we could get rid of the whole resultmap mechanism ...

Yeah. I thought I'd see what blowback my
if-Cygwin-strtof()-really-is-still-broken-they-should-fix-it argument
attracted before spending the time to nuke all those lines too.
Here's that patch. We could always revert resultmap we found a new
reason to need it, but I hope we wouldn't.

On 24.11.25 00:03, Thomas Munro wrote:

On Sun, Nov 23, 2025 at 4:25 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thomas Munro <thomas.munro@gmail.com> writes:

That'd leave only Cygwin with HAVE BUGGY_STRTOF. Perhaps they have
fixed their implementation[1]? Here's an experimental patch to drop
all remnants, which could be used to find out. No Windows/Cygwin
here. Hmm, what if we just commit it anyway? If their strtof() is
still broken and someone out there is running the tests and sees this
test fail, why shouldn't they take that up with libc at this stage?

Hmm, we could get rid of the whole resultmap mechanism ...

Yeah. I thought I'd see what blowback my
if-Cygwin-strtof()-really-is-still-broken-they-should-fix-it argument
attracted before spending the time to nuke all those lines too.
Here's that patch. We could always revert resultmap we found a new
reason to need it, but I hope we wouldn't.

These patches look sensible to me.

Maybe wait a bit to see if Andrew can manually reproduce the issue one
way or the other on Cygwin.

Otherwise, I'd say go for it.