BUG #16920: Can't compile PostGIS with MingW64 against PostgreSQL 14 head

PG Bug reporting form <noreply@postgresql.org> writes:

I pulled PostgreSQL 14 earlier today to troubleshoot an issue we were having
and discovered I can not compile PostGIS with Mingw64 against PostgreSQL 14
head.

Hmm, our mingw64 buildfarm animals aren't detecting any particular
problem here. Is PostGIS doing anything unusual with injecting
special symbols via -D switches? Or maybe including something else
before postgres.h? The symptoms look like a header-inclusion-order
problem ...

regards, tom lane

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Thursday, March 11, 2021 10:02 AM
To: r@pcorp.us
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16920: Can't compile PostGIS with MingW64 against
PostgreSQL 14 head

PG Bug reporting form <noreply@postgresql.org> writes:

I pulled PostgreSQL 14 earlier today to troubleshoot an issue we were
having and discovered I can not compile PostGIS with Mingw64 against
PostgreSQL 14 head.

Hmm, our mingw64 buildfarm animals aren't detecting any particular
problem here. Is PostGIS doing anything unusual with injecting special
symbols via -D switches? Or maybe including something else before
postgres.h? The symptoms look like a header-inclusion-order problem ...

regards, tom lane

The order in one of the problem files looks like this:
Starting:
https://git.osgeo.org/gitea/postgis/postgis/src/branch/master/postgis/lwgeom
_in_gml.c#L50

#include <libxml/tree.h>
#include <libxml/parser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>

#include "postgres.h"
#include "executor/spi.h"
#include "utils/builtins.h"

#include "../postgis_config.h"
#include "lwgeom_pg.h"
#include "liblwgeom.h"
#include "lwgeom_transform.h"

Note I didn't have issue compiling PostgreSQL under mingw64 (no change) - I
need to double-check the output, but I think all tests passed.
What extensions does the buildfarm animals test?

Thanks,
Regina

"Regina Obe" <lr@pcorp.us> writes:

From: Tom Lane [mailto:tgl@sss.pgh.pa.us]

Hmm, our mingw64 buildfarm animals aren't detecting any particular
problem here. Is PostGIS doing anything unusual with injecting special
symbols via -D switches? Or maybe including something else before
postgres.h? The symptoms look like a header-inclusion-order problem ...

The order in one of the problem files looks like this:
Starting:
https://git.osgeo.org/gitea/postgis/postgis/src/branch/master/postgis/lwgeom
_in_gml.c#L50

#include <libxml/tree.h>
#include <libxml/parser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>

#include "postgres.h"
#include "executor/spi.h"
#include "utils/builtins.h"

#include "../postgis_config.h"
#include "lwgeom_pg.h"
#include "liblwgeom.h"
#include "lwgeom_transform.h"

Our usual recommendation is to include postgres.h before all else;
is there a really good reason to be pulling in libxml headers before
that?

Having seen this, I now guess that your issue is happening because
one of those headers has already included <sys/stat.h> before
win32_port.h tries to do so. That makes our attempt to relabel the
system's "struct stat" as "struct microsoft_native_stat" not work.
In general, the portability hacks that postgres.h does for some
platforms aren't reliable if any system headers have already been
pulled in.

regards, tom lane

Our usual recommendation is to include postgres.h before all else; is

there a

really good reason to be pulling in libxml headers before that?

Having seen this, I now guess that your issue is happening because one of
those headers has already included <sys/stat.h> before win32_port.h tries

to

do so. That makes our attempt to relabel the system's "struct stat" as

"struct

microsoft_native_stat" not work.
In general, the portability hacks that postgres.h does for some platforms
aren't reliable if any system headers have already been pulled in.

regards, tom lane

I don't think there is a reason to include libxml first. Thanks for the
help. I'll revise those to see if it fixes the issue without messing with
the Postgres include files.

Thanks,
Regina

Our usual recommendation is to include postgres.h before all else; is
there a really good reason to be pulling in libxml headers before that?

Having seen this, I now guess that your issue is happening because one
of those headers has already included <sys/stat.h> before win32_port.h
tries to do so. That makes our attempt to relabel the system's
"struct stat" as "struct microsoft_native_stat" not work.
In general, the portability hacks that postgres.h does for some
platforms aren't reliable if any system headers have already been pulled

in.

regards, tom lane

I don't think there is a reason to include libxml first. Thanks for the

help. I'll

revise those to see if it fixes the issue without messing with the

Postgres

include files.

Thanks,
Regina

Thanks again Tom. That seemed to do the trick.

I was mistaken though that in the postgresql regress I do get a lot of
failures like failure on stats.

But many of these errors seemed to be the same as what is on the buildfarm

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=walleye&amp;dt=2021-03-0
8%2021%3A35%3A41

My output looks like below. I also have some crashers on PostGIS upgrade
regress, but I need to recheck those on our debian bot to see if it's
PostgreSQL 14 or Mingw specific.

============== creating temporary instance ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 58080 with PID 328
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test tablespace ... ok 594 ms
parallel group (20 tests): name char int2 pg_lsn text float4 oid boolean
int4 money txid varchar int8 regproc bit uuid float8 enum numeric rangetypes
boolean ... ok 252 ms
char ... ok 203 ms
name ... ok 201 ms
varchar ... ok 266 ms
text ... ok 237 ms
int2 ... ok 228 ms
int4 ... ok 249 ms
int8 ... ok 258 ms
oid ... ok 236 ms
float4 ... ok 232 ms
float8 ... ok 301 ms
bit ... ok 280 ms
numeric ... ok 494 ms
txid ... ok 237 ms
uuid ... ok 285 ms
enum ... ok 290 ms
money ... ok 225 ms
rangetypes ... ok 520 ms
pg_lsn ... ok 191 ms
regproc ... ok 231 ms
parallel group (20 tests): point numerology time path line macaddr8 circle
date macaddr timetz lseg strings interval inet tstypes polygon box
multirangetypes timestamp timestamptz
strings ... ok 295 ms
numerology ... ok 221 ms
point ... ok 188 ms
lseg ... ok 286 ms
line ... ok 268 ms
box ... ok 419 ms
path ... ok 218 ms
polygon ... ok 399 ms
circle ... ok 273 ms
date ... ok 270 ms
time ... ok 208 ms
timetz ... ok 266 ms
timestamp ... ok 588 ms
timestamptz ... ok 617 ms
interval ... ok 267 ms
inet ... ok 273 ms
macaddr ... ok 252 ms
macaddr8 ... ok 250 ms
tstypes ... ok 267 ms
multirangetypes ... ok 408 ms
parallel group (10 tests): comments unicode expressions xid horology
misc_sanity geometry type_sanity regex opr_sanity
geometry ... ok 184 ms
horology ... ok 176 ms
regex ... ok 328 ms
type_sanity ... ok 191 ms
opr_sanity ... ok 413 ms
misc_sanity ... ok 173 ms
comments ... ok 100 ms
expressions ... ok 109 ms
unicode ... ok 98 ms
xid ... ok 132 ms
test create_function_1 ... ok 212 ms
test create_type ... ok 82 ms
test create_table ... ok 502 ms
test create_function_2 ... ok 71 ms
parallel group (5 tests): copydml insert insert_conflict copyselect copy
copy ... FAILED (test process exited with exit
code 2) 222 ms
copyselect ... FAILED (test process exited with exit
code 2) 220 ms
copydml ... FAILED (test process exited with exit
code 2) 170 ms
insert ... FAILED (test process exited with exit
code 2) 168 ms
insert_conflict ... FAILED (test process exited with exit
code 2) 171 ms
parallel group (3 tests): create_operator create_misc create_procedure
create_misc ... FAILED (test process exited with exit
code 2) 52 ms
create_operator ... FAILED (test process exited with exit
code 2) 50 ms
create_procedure ... FAILED (test process exited with exit
code 2) 55 ms
parallel group (5 tests): create_index create_view index_including_gist
create_index_spgist index_including
create_index ... FAILED (test process exited with exit
code 2) 47 ms
create_index_spgist ... FAILED (test process exited with exit
code 2) 64 ms
create_view ... FAILED (test process exited with exit
code 2) 53 ms
index_including ... FAILED (test process exited with exit
code 2) 62 ms
index_including_gist ... FAILED (test process exited with exit
code 2) 50 ms
parallel group (16 tests): create_cast roleattributes hash_func errors
select drop_if_exists create_function_3 create_am create_aggregate
typed_table vacuum inherit updatable_views constraints triggers
infinite_recurse
create_aggregate ... ok 312 ms
create_function_3 ... ok 279 ms
create_cast ... ok 175 ms
constraints ... FAILED (test process exited with exit
code 2) 364 ms
triggers ... FAILED (test process exited with exit
code 2) 365 ms
select ... FAILED 241 ms
inherit ... FAILED (test process exited with exit
code 2) 355 ms
typed_table ... ok 304 ms
vacuum ... FAILED (test process exited with exit
code 2) 343 ms
drop_if_exists ... ok 246 ms
updatable_views ... FAILED (test process exited with exit
code 2) 346 ms
roleattributes ... ok 174 ms
create_am ... FAILED 277 ms
hash_func ... ok 170 ms
errors ... ok 170 ms
infinite_recurse ... FAILED (test process exited with exit
code 2) 350 ms
test sanity_check ... FAILED 442 ms
parallel group (20 tests): select_distinct_on select_implicit random case
select_having delete select_distinct namespace prepared_xacts select_into
hash_index union update join btree_index subselect transactions portals
aggregates arrays
select_into ... FAILED (test process exited with exit
code 2) 416 ms
select_distinct ... ok 408 ms
select_distinct_on ... ok 175 ms
select_implicit ... ok 197 ms
select_having ... ok 340 ms
subselect ... FAILED (test process exited with exit
code 2) 483 ms
union ... FAILED (test process exited with exit
code 2) 426 ms
case ... ok 216 ms
join ... FAILED (test process exited with exit
code 2) 421 ms
aggregates ... FAILED (test process exited with exit
code 2) 477 ms
transactions ... FAILED (test process exited with exit
code 2) 470 ms
random ... ok 205 ms
portals ... FAILED (test process exited with exit
code 2) 467 ms
arrays ... FAILED (test process exited with exit
code 2) 473 ms
btree_index ... FAILED (test process exited with exit
code 2) 456 ms
hash_index ... FAILED (test process exited with exit
code 2) 389 ms
update ... FAILED (test process exited with exit
code 2) 396 ms
delete ... ok 302 ms
namespace ... ok 363 ms
prepared_xacts ... ok 366 ms
parallel group (20 tests): security_label drop_operator lock tablesample
init_privs password collate object_address matview replica_identity gin
spgist privileges generated identity gist rowsecurity brin join_hash
groupingsets
brin ... FAILED (test process exited with exit
code 2) 578 ms
gin ... FAILED (test process exited with exit
code 2) 575 ms
gist ... FAILED (test process exited with exit
code 2) 574 ms
spgist ... FAILED (test process exited with exit
code 2) 572 ms
privileges ... FAILED (test process exited with exit
code 2) 568 ms
init_privs ... ok 442 ms
security_label ... ok 195 ms
collate ... ok 475 ms
matview ... FAILED (test process exited with exit
code 2) 551 ms
lock ... ok 420 ms
replica_identity ... FAILED (test process exited with exit
code 2) 553 ms
rowsecurity ... FAILED (test process exited with exit
code 2) 551 ms
object_address ... ok 533 ms
tablesample ... ok 416 ms
groupingsets ... FAILED (test process exited with exit
code 2) 542 ms
drop_operator ... ok 197 ms
password ... ok 409 ms
identity ... FAILED (test process exited with exit
code 2) 531 ms
generated ... FAILED (test process exited with exit
code 2) 527 ms
join_hash ... FAILED (test process exited with exit
code 2) 527 ms
parallel group (14 tests): dbsize collate.icu.utf8 tidscan tsrf
tidrangescan tid async alter_operator alter_generic misc incremental_sort
create_table_like misc_functions sysviews
create_table_like ... ok 499 ms
alter_generic ... ok 287 ms
alter_operator ... ok 247 ms
misc ... FAILED 296 ms
async ... ok 229 ms
dbsize ... ok 137 ms
misc_functions ... FAILED 519 ms
sysviews ... ok 519 ms
tsrf ... ok 178 ms
tid ... ok 182 ms
tidscan ... ok 164 ms
tidrangescan ... ok 171 ms
collate.icu.utf8 ... ok 130 ms
incremental_sort ... FAILED 282 ms
parallel group (6 tests): collate.linux.utf8 amutils psql_crosstab psql
rules stats_ext
rules ... FAILED 549 ms
psql ... FAILED 467 ms
psql_crosstab ... ok 222 ms
amutils ... FAILED 207 ms
stats_ext ... FAILED 1181 ms
collate.linux.utf8 ... ok 189 ms
test select_parallel ... FAILED 1486 ms
test write_parallel ... ok 184 ms
test insert_parallel ... ok 1059 ms
parallel group (2 tests): subscription publication
publication ... FAILED (test process exited with exit
code 2) 118 ms
subscription ... FAILED (test process exited with exit
code 2) 114 ms
parallel group (17 tests): select_views dependency tsdicts foreign_key
tsearch foreign_data portals_p2 functional_deps bitmapops indirect_toast
combocid advisory_lock guc xmlmap window cluster equivclass
select_views ... FAILED (test process exited with exit
code 2) 107 ms
portals_p2 ... FAILED (test process exited with exit
code 2) 129 ms
foreign_key ... FAILED (test process exited with exit
code 2) 126 ms
cluster ... FAILED (test process exited with exit
code 2) 140 ms
dependency ... FAILED (test process exited with exit
code 2) 108 ms
guc ... FAILED (test process exited with exit
code 2) 127 ms
bitmapops ... FAILED (test process exited with exit
code 2) 121 ms
combocid ... FAILED (test process exited with exit
code 2) 118 ms
tsearch ... FAILED (test process exited with exit
code 2) 111 ms
tsdicts ... FAILED (test process exited with exit
code 2) 109 ms
foreign_data ... FAILED (test process exited with exit
code 2) 108 ms
window ... FAILED (test process exited with exit
code 2) 117 ms
xmlmap ... FAILED (test process exited with exit
code 2) 110 ms
functional_deps ... FAILED (test process exited with exit
code 2) 103 ms
advisory_lock ... FAILED (test process exited with exit
code 2) 104 ms
indirect_toast ... FAILED (test process exited with exit
code 2) 99 ms
equivclass ... FAILED (test process exited with exit
code 2) 112 ms
parallel group (6 tests): json_encoding jsonpath_encoding json jsonpath
jsonb jsonb_jsonpath
json ... ok 193 ms
jsonb ... FAILED (test process exited with exit
code 2) 281 ms
json_encoding ... ok 184 ms
jsonpath ... ok 206 ms
jsonpath_encoding ... ok 181 ms
jsonb_jsonpath ... FAILED (test process exited with exit
code 2) 274 ms
parallel group (18 tests): conversion prepare limit plancache returning xml
sequence truncate rowtypes with polymorphism domain alter_table largeobject
copy2 temp rangefuncs plpgsql
plancache ... ok 411 ms
limit ... FAILED 305 ms
plpgsql ... FAILED (test process exited with exit
code 2) 499 ms
copy2 ... FAILED (test process exited with exit
code 2) 497 ms
temp ... FAILED (test process exited with exit
code 2) 495 ms
domain ... FAILED (test process exited with exit
code 2) 492 ms
rangefuncs ... FAILED (test process exited with exit
code 2) 488 ms
prepare ... ok 211 ms
conversion ... ok 202 ms
truncate ... FAILED (test process exited with exit
code 2) 477 ms
alter_table ... FAILED (test process exited with exit
code 2) 477 ms
sequence ... FAILED (test process exited with exit
code 2) 461 ms
polymorphism ... FAILED (test process exited with exit
code 2) 464 ms
rowtypes ... FAILED (test process exited with exit
code 2) 460 ms
returning ... ok 399 ms
largeobject ... FAILED (test process exited with exit
code 2) 457 ms
with ... FAILED (test process exited with exit
code 2) 447 ms
xml ... FAILED (test process exited with exit
code 2) 415 ms
parallel group (9 tests): hash_part partition_info reloptions explain
partition_aggregate indexing partition_join tuplesort partition_prune
partition_join ... ok 1331 ms
partition_prune ... FAILED 1591 ms
reloptions ... ok 308 ms
hash_part ... FAILED 247 ms
indexing ... ok 1166 ms
partition_aggregate ... ok 1053 ms
partition_info ... ok 284 ms
tuplesort ... ok 1371 ms
explain ... ok 300 ms
parallel group (2 tests): event_trigger oidjoins
event_trigger ... ok 206 ms
oidjoins ... ok 218 ms
test fast_default ... ok 262 ms
test stats ... FAILED 34525 ms
============== shutting down postmaster ==============

=========================
91 of 204 tests failed.

"Regina Obe" <lr@pcorp.us> writes:

I was mistaken though that in the postgresql regress I do get a lot of
failures like failure on stats.
But many of these errors seemed to be the same as what is on the buildfarm
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=walleye&amp;dt=2021-03-0
8%2021%3A35%3A41

walleye's owners have so far failed to extract any useful data about why
it's been crashing. If you could provide stack traces from some of those
crashes, it'd be much appreciated.

regards, tom lane

On Thu, Mar 11, 2021 at 7:03 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

walleye's owners have so far failed to extract any useful data about why
it's been crashing. If you could provide stack traces from some of those
crashes, it'd be much appreciated.

+1. I cannot reproduce the error with a setup similar to Walleye (Windows
10 MinGW64 8.1.0 x86_64).

Regards,

Juan José Santamaría Flecha

"Regina Obe" <lr@pcorp.us> writes:

I was mistaken though that in the postgresql regress I do get a lot of
failures like failure on stats.
But many of these errors seemed to be the same as what is on the
buildfarm
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=walleye&amp;dt=202
1-03-0
8%2021%3A35%3A41

walleye's owners have so far failed to extract any useful data about why

it's

been crashing. If you could provide stack traces from some of those

crashes,

it'd be much appreciated.

regards, tom lane

Okay the crash on PostGIS against PostgreSQL 14 does seem to be mingw
related since our debian bot passes with flying colors.
Both have postgresql built with --enable-cassert

(PG 13 is fine for mingw64 BTW also cassert enabled)

On my mingw64 gcc 8.1.0 (cassert enabled, not sure that matters) -- in
PostGIS regress - it starts crashing with this query:

SELECT topology.ST_AddIsoEdge('sqlmm_topology', 1, 2, 'LINESTRING(0 0, 10 0,
5 5, 5 -5)');

Hopefully the back trace is of some help -- I'm going to try to formulate a
similar one without PostGIS involvement.

As far as the PostgreSQL tests, I'm not sure how to run the regress tests
standalone to grab a stack trace. Suggestions welcome.

Thread 1 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 22272.0x49ec]
0x00007ffe75f5f23f in ntdll!RtlLeaveCriticalSection ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
(gdb) bt
#0 0x00007ffe75f5f23f in ntdll!RtlLeaveCriticalSection ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#1 0x00007ffe75fd4c0e in ntdll!memset () from C:\WINDOWS\SYSTEM32\ntdll.dll
#2 0x00007ffe75fbc872 in ntdll!__C_specific_handler ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#3 0x00007ffe75fd1fef in ntdll!.chkstk () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#4 0x00007ffe75f60939 in ntdll!RtlUnwindEx ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#5 0x00007ffe74002eed in msvcrt!_setjmpex ()
from C:\WINDOWS\System32\msvcrt.dll
#6 0x00000000008d929a in pg_re_throw () at elog.c:1776
#7 0x00000000008d911d in errfinish (filename=<optimized out>,
lineno=<optimized out>, funcname=0x6b025560 <__func__.114531>
"pg_error")
at elog.c:589
#8 0x000000006afc91d3 in pg_error (fmt=<optimized out>, ap=<optimized out>)
at lwgeom_pg.c:250
#9 0x000000006afdfb03 in lwerror (fmt=0x0) at lwutil.c:197
#10 0x000000006b000c16 in lwt_AddIsoEdge (topo=topo@entry=0x83629b0,
startNode=startNode@entry=1, endNode=endNode@entry=2,
geom=geom@entry=0x7a303b8) at lwgeom_topo.c:794
#11 0x000000006afc70d2 in ST_AddIsoEdge (fcinfo=0x7a2caa8)
at postgis_topology.c:3567
#12 0x00000000005f7604 in ExecInterpExpr (state=0x7a2c9c0,
econtext=0x7a2c6c0, isnull=<optimized out>) at execExprInterp.c:1209
#13 0x000000000062f23f in ExecEvalExprSwitchContext (isNull=0x509ef9f,
econtext=0x7a2c6c0, state=0x7a2c9c0)
at ../../../src/include/executor/executor.h:315
#14 ExecProject (projInfo=0x7a2c9b8)
at ../../../src/include/executor/executor.h:349
#15 ExecResult (pstate=<optimized out>) at nodeResult.c:136
#16 0x00000000005fc0ca in ExecProcNode (node=0x7a2c5a8)
at ../../../src/include/executor/executor.h:247
#17 ExecutePlan (execute_once=<optimized out>, dest=0x7a2b2e8,
direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>,
operation=CMD_SELECT, use_parallel_mode=<optimized out>,
planstate=0x7a2c5a8, estate=0x7a2c370) at execMain.c:1531
#18 standard_ExecutorRun (queryDesc=0x8338990, direction=<optimized out>,
count=0, execute_once=<optimized out>) at execMain.c:350
#19 0x00000000007a08cb in PortalRunSelect (portal=0x79bcc70,
portal@entry=0x7fffffff, forward=forward@entry=true,
count=count@entry=0,
dest=dest@entry=0x7a2b2e8) at pquery.c:912
#20 0x00000000007a20cc in PortalRun (portal=0x7fffffff,
portal@entry=0x79bcc70, count=0, count@entry=2147483647,
isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true,
dest=dest@entry=0x7a2b2e8, altdest=altdest@entry=0x7a2b2e8,
qc=qc@entry=0x509f2d0) at pquery.c:756
#21 0x000000000079d6ab in exec_simple_query (
query_string=0x79fd1e0 "SELECT topology.ST_AddIsoEdge('sqlmm_topology',
1, 2, 'LINESTRING(0 0, 10 0, 5 5, 5 -5)');") at postgres.c:1173
#22 0x000000000079ee40 in PostgresMain (argc=argc@entry=1,
argv=argv@entry=0x509f980, dbname=<optimized out>,
username=<optimized out>) at postgres.c:4327
#23 0x0000000000708608 in BackendRun (port=0x509f7c0, port=0x509f7c0)
at postmaster.c:4464
#24 SubPostmasterMain (argc=argc@entry=3, argv=argv@entry=0x5176af0)
at postmaster.c:4977
#25 0x000000000093cb13 in main (argc=3, argv=0x5176af0) at main.c:186

On Thu, Mar 11, 2021 at 7:03 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

walleye's owners have so far failed to extract any useful data about why
it's been crashing. If you could provide stack traces from some of those
crashes, it'd be much appreciated.

+1. I cannot reproduce the error with a setup similar to Walleye (Windows 10 MinGW64 8.1.0 x86_64).

Regards,

Juan José Santamaría Flecha

I posted in separate note a bug trace I did get with PostGIS.
Did you build PostgreSQL with --enable-cassert?
I repulled PostgreSQL latest and recompiled -- not with --enable-cassert.

All the PostgreSQL tests pass without the --enable-cassert and my PostGIS topology crash goes away too.

There also seems to have been changes in code base since last I compiled that seemed to have helped a bit. Recompiling again with cassert - I get 1 fewer failure. I didn't check to see which test started passing.

Was 91 of 204 tests failing before - /messages/by-id/000401d716a0$1ed0fc70$5c72f550$@pcorp.us

Now down to

89 of 204 tests failed, 1 of these failures ignored.

============== running regression test queries ==============
test tablespace ... ok 556 ms
parallel group (20 tests): boolean char name varchar text int2 uuid txid int4 bit float8 float4 oid int8 money enum pg_lsn regproc numeric rangetypes
boolean ... ok 395 ms
char ... ok 382 ms
name ... ok 367 ms
varchar ... ok 351 ms
text ... ok 328 ms
int2 ... ok 308 ms
int4 ... ok 291 ms
int8 ... ok 272 ms
oid ... ok 254 ms
float4 ... ok 235 ms
float8 ... ok 215 ms
bit ... ok 198 ms
numeric ... ok 366 ms
txid ... ok 157 ms
uuid ... ok 139 ms
enum ... ok 156 ms
money ... ok 100 ms
rangetypes ... ok 359 ms
pg_lsn ... ok 88 ms
regproc ... ok 81 ms
parallel group (20 tests): strings numerology point lseg macaddr line interval box path timetz time date circle polygon inet macaddr8 tstypes multirangetypes timestamp timestamptz
strings ... ok 444 ms
numerology ... ok 431 ms
point ... ok 417 ms
lseg ... ok 402 ms
line ... ok 377 ms
box ... ok 359 ms
path ... ok 342 ms
polygon ... ok 321 ms
circle ... ok 295 ms
date ... ok 272 ms
time ... ok 252 ms
timetz ... ok 232 ms
timestamp ... ok 428 ms
timestamptz ... ok 464 ms
interval ... ok 128 ms
inet ... ok 111 ms
macaddr ... ok 82 ms
macaddr8 ... ok 82 ms
tstypes ... ok 97 ms
multirangetypes ... ok 224 ms
parallel group (10 tests): geometry horology comments expressions type_sanity unicode misc_sanity xid regex opr_sanity
geometry ... ok 217 ms
horology ... ok 201 ms
regex ... ok 321 ms
type_sanity ... ok 156 ms
opr_sanity ... ok 378 ms
misc_sanity ... ok 136 ms
comments ... ok 70 ms
expressions ... ok 72 ms
unicode ... ok 65 ms
xid ... ok 88 ms
test create_function_1 ... ok 61 ms
test create_type ... ok 82 ms
test create_table ... ok 518 ms
test create_function_2 ... ok 52 ms
parallel group (5 tests): copydml insert insert_conflict copyselect copy
copy ... FAILED (test process exited with exit code 2) 222 ms
copyselect ... FAILED (test process exited with exit code 2) 202 ms
copydml ... FAILED (test process exited with exit code 2) 158 ms
insert ... FAILED (test process exited with exit code 2) 139 ms
insert_conflict ... FAILED (test process exited with exit code 2) 107 ms
parallel group (3 tests): create_misc create_operator create_procedure
create_misc ... FAILED (test process exited with exit code 2) 45 ms
create_operator ... FAILED (test process exited with exit code 2) 40 ms
create_procedure ... FAILED (test process exited with exit code 2) 39 ms
parallel group (5 tests): index_including index_including_gist create_view create_index_spgist create_index
create_index ... FAILED (test process exited with exit code 2) 574 ms
create_index_spgist ... ok 445 ms
create_view ... FAILED 358 ms
index_including ... ok 311 ms
index_including_gist ... ok 317 ms
parallel group (16 tests): create_aggregate create_function_3 create_cast constraints triggers roleattributes updatable_views drop_if_exists vacuum typed_table inherit select errors hash_func create_am infinite_recurse
create_aggregate ... FAILED (test process exited with exit code 2) 336 ms
create_function_3 ... FAILED (test process exited with exit code 2) 316 ms
create_cast ... FAILED (test process exited with exit code 2) 299 ms
constraints ... FAILED (test process exited with exit code 2) 298 ms
triggers ... FAILED (test process exited with exit code 2) 241 ms
select ... FAILED (test process exited with exit code 2) 221 ms
inherit ... FAILED (test process exited with exit code 2) 207 ms
typed_table ... FAILED (test process exited with exit code 2) 195 ms
vacuum ... FAILED (test process exited with exit code 2) 140 ms
drop_if_exists ... FAILED (test process exited with exit code 2) 107 ms
updatable_views ... FAILED (test process exited with exit code 2) 89 ms
roleattributes ... FAILED (test process exited with exit code 2) 75 ms
create_am ... FAILED 159 ms
hash_func ... ok 103 ms
errors ... ok 76 ms
infinite_recurse ... ok 213 ms
test sanity_check ... FAILED 472 ms
parallel group (20 tests): select_into select_distinct select_distinct_on delete update hash_index btree_index arrays portals random transactions aggregates join case union subselect select_having select_implicit namespace prepared_xacts
select_into ... FAILED (test process exited with exit code 2) 464 ms
select_distinct ... FAILED 450 ms
select_distinct_on ... FAILED 435 ms
select_implicit ... ok 419 ms
select_having ... ok 399 ms
subselect ... FAILED (test process exited with exit code 2) 377 ms
union ... FAILED (test process exited with exit code 2) 356 ms
case ... FAILED (test process exited with exit code 2) 308 ms
join ... FAILED (test process exited with exit code 2) 284 ms
aggregates ... FAILED (test process exited with exit code 2) 254 ms
transactions ... FAILED (test process exited with exit code 2) 224 ms
random ... failed (ignored) (test process exited with exit code 2) 210 ms
portals ... FAILED (test process exited with exit code 2) 184 ms
arrays ... FAILED (test process exited with exit code 2) 153 ms
btree_index ... FAILED (test process exited with exit code 2) 132 ms
hash_index ... FAILED (test process exited with exit code 2) 93 ms
update ... FAILED (test process exited with exit code 2) 62 ms
delete ... FAILED (test process exited with exit code 2) 46 ms
namespace ... FAILED (test process exited with exit code 2) 41 ms
prepared_xacts ... FAILED (test process exited with exit code 2) 37 ms
parallel group (20 tests): init_privs security_label lock collate spgist replica_identity tablesample password drop_operator object_address identity matview rowsecurity generated gist privileges join_hash groupingsets brin gin
brin ... FAILED (test process exited with exit code 2) 858 ms
gin ... FAILED (test process exited with exit code 2) 851 ms
gist ... FAILED (test process exited with exit code 2) 817 ms
spgist ... ok 681 ms
privileges ... FAILED (test process exited with exit code 2) 764 ms
init_privs ... ok 378 ms
security_label ... ok 358 ms
collate ... ok 532 ms
matview ... FAILED (test process exited with exit code 2) 671 ms
lock ... ok 372 ms
replica_identity ... ok 527 ms
rowsecurity ... FAILED (test process exited with exit code 2) 597 ms
object_address ... ok 553 ms
tablesample ... ok 458 ms
groupingsets ... FAILED (test process exited with exit code 2) 520 ms
drop_operator ... ok 467 ms
password ... ok 388 ms
identity ... FAILED (test process exited with exit code 2) 434 ms
generated ... FAILED (test process exited with exit code 2) 408 ms
join_hash ... FAILED (test process exited with exit code 2) 369 ms
parallel group (14 tests): alter_generic alter_operator misc async dbsize tsrf tid tidscan collate.icu.utf8 tidrangescan incremental_sort misc_functions create_table_like sysviews
create_table_like ... ok 504 ms
alter_generic ... FAILED 265 ms
alter_operator ... ok 249 ms
misc ... FAILED 247 ms
async ... ok 231 ms
dbsize ... ok 212 ms
misc_functions ... FAILED 415 ms
sysviews ... ok 397 ms
tsrf ... ok 145 ms
tid ... ok 133 ms
tidscan ... ok 138 ms
tidrangescan ... ok 126 ms
collate.icu.utf8 ... ok 86 ms
incremental_sort ... FAILED 194 ms
parallel group (6 tests): psql_crosstab amutils collate.linux.utf8 psql rules stats_ext
rules ... FAILED 595 ms
psql ... ok 353 ms
psql_crosstab ... ok 92 ms
amutils ... ok 84 ms
stats_ext ... FAILED 1104 ms
collate.linux.utf8 ... ok 61 ms
test select_parallel ... FAILED 1765 ms
test write_parallel ... ok 200 ms
test insert_parallel ... ok 1058 ms
parallel group (2 tests): subscription publication
publication ... FAILED (test process exited with exit code 2) 142 ms
subscription ... FAILED (test process exited with exit code 2) 121 ms
parallel group (17 tests): select_views portals_p2 foreign_key cluster functional_deps xmlmap window foreign_data tsdicts tsearch combocid bitmapops guc dependency advisory_lock indirect_toast equivclass
select_views ... FAILED (test process exited with exit code 2) 311 ms
portals_p2 ... FAILED (test process exited with exit code 2) 290 ms
foreign_key ... FAILED (test process exited with exit code 2) 275 ms
cluster ... FAILED (test process exited with exit code 2) 259 ms
dependency ... FAILED (test process exited with exit code 2) 241 ms
guc ... FAILED (test process exited with exit code 2) 215 ms
bitmapops ... FAILED (test process exited with exit code 2) 202 ms
combocid ... FAILED (test process exited with exit code 2) 186 ms
tsearch ... FAILED (test process exited with exit code 2) 154 ms
tsdicts ... FAILED (test process exited with exit code 2) 127 ms
foreign_data ... FAILED (test process exited with exit code 2) 111 ms
window ... FAILED (test process exited with exit code 2) 91 ms
xmlmap ... FAILED (test process exited with exit code 2) 65 ms
functional_deps ... FAILED (test process exited with exit code 2) 49 ms
advisory_lock ... FAILED (test process exited with exit code 2) 42 ms
indirect_toast ... FAILED (test process exited with exit code 2) 40 ms
equivclass ... FAILED (test process exited with exit code 2) 40 ms
parallel group (6 tests): json_encoding jsonpath_encoding jsonpath json jsonb_jsonpath jsonb
json ... ok 203 ms
jsonb ... FAILED (test process exited with exit code 2) 233 ms
json_encoding ... ok 66 ms
jsonpath ... ok 92 ms
jsonpath_encoding ... ok 64 ms
jsonb_jsonpath ... FAILED (test process exited with exit code 2) 158 ms
parallel group (18 tests): plancache limit copy2 temp prepare conversion rangefuncs domain returning polymorphism rowtypes sequence xml with alter_table truncate largeobject plpgsql
plancache ... ok 609 ms
limit ... ok 594 ms
plpgsql ... FAILED (test process exited with exit code 2) 800 ms
copy2 ... ok 489 ms
temp ... ok 472 ms
domain ... ok 551 ms
rangefuncs ... ok 427 ms
prepare ... ok 372 ms
conversion ... ok 352 ms
truncate ... FAILED (test process exited with exit code 2) 576 ms
alter_table ... FAILED (test process exited with exit code 2) 482 ms
sequence ... ok 423 ms
polymorphism ... ok 372 ms
rowtypes ... ok 359 ms
returning ... ok 228 ms
largeobject ... FAILED (test process exited with exit code 2) 364 ms
with ... FAILED (test process exited with exit code 2) 323 ms
xml ... FAILED (test process exited with exit code 2) 256 ms
parallel group (9 tests): hash_part reloptions partition_info explain partition_aggregate indexing partition_join tuplesort partition_prune
partition_join ... ok 1257 ms
partition_prune ... FAILED 1455 ms
reloptions ... ok 168 ms
hash_part ... FAILED 129 ms
indexing ... ok 1068 ms
partition_aggregate ... ok 966 ms
partition_info ... ok 146 ms
tuplesort ... ok 1269 ms
explain ... ok 160 ms
parallel group (2 tests): event_trigger oidjoins
event_trigger ... ok 208 ms
oidjoins ... ok 219 ms
test fast_default ... ok 264 ms
test stats ... FAILED 33865 ms

"Regina Obe" <lr@pcorp.us> writes:

There also seems to have been changes in code base since last I compiled that seemed to have helped a bit. Recompiling again with cassert - I get 1 fewer failure. I didn't check to see which test started passing.

In cases like this where some of the backends are actually crashing,
you can't draw very many conclusions from the number of tests that
are reported to fail. A crash will take out any concurrently-running
tests, so there's already some variability depending on whether any
concurrent scripts manage to finish before the one that crashed.
Plus, there are likely to be follow-on "failures" from later tests
that expect earlier tests to have left specific database state behind,
which they didn't because they failed to finish.

Counting the number of actual core dumps might provide some useful
data about whether things are better or worse.

regards, tom lane

Counting the number of actual core dumps might provide some useful data
about whether things are better or worse.

regards, tom lane

Yah forgot about that.

Okay the first core dump happens in the

Copyselect test:

-- this statement
copy (select t into temp test3 from test1 where id=3) to stdout;

The backtrace of that looks like this:

#0 0x00007ffe76032136 in ntdll!RtlRaiseStatus ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#1 0x00007ffe75f60bb3 in ntdll!RtlUnwindEx ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#2 0x00007ffe74002eed in msvcrt!_setjmpex ()
from C:\WINDOWS\System32\msvcrt.dll
#3 0x00000000008da31a in pg_re_throw () at elog.c:1776
#4 0x00000000008da19d in errfinish (filename=<optimized out>,
filename@entry=0xa291a0 <BinarySignature+24> "copyto.c",
lineno=lineno@entry=490,
funcname=funcname@entry=0xa29980 <__func__.112127> "BeginCopyTo")
at elog.c:589
#5 0x0000000000577f08 in BeginCopyTo (pstate=pstate@entry=0x7969920,
rel=rel@entry=0x0, raw_query=raw_query@entry=0x7969a38,
queryRelId=queryRelId@entry=0, filename=0x0, is_program=false,
attnamelist=0x0, options=0x0) at copyto.c:488
#6 0x00000000005712f8 in DoCopy (pstate=0x7969920, stmt=0x79e13f8,
stmt_location=0, stmt_len=63, processed=0x509ef38) at copy.c:306
#7 0x00000000007a4363 in standard_ProcessUtility (pstmt=0x79e14e0,
queryString=0x79e0950 "copy (select t into temp test3 from test1 where
id=3) to stdout;", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0,
dest=0x79e17c0, qc=0x509f2d0) at utility.c:724
#8 0x00000000007a15e8 in PortalRunUtility (portal=0x79a84d0,
pstmt=0x79e14e0, isTopLevel=<optimized out>,
setHoldSnapshot=<optimized out>, dest=0x79e17c0, qc=0x509f2d0)
at pquery.c:1159
#9 0x00000000007a213a in PortalRunMulti (portal=0x79a84d0,
portal@entry=0x7fffffff, isTopLevel=true, isTopLevel@entry=false,
setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x79e17c0,
altdest=0x79e17c0, altdest@entry=0x509f2d0, qc=0x509f2d0,
qc@entry=0x935b00 <getopt_long>) at pquery.c:1305
#10 0x00000000007a2feb in PortalRun (portal=0x7fffffff,
portal@entry=0x79a84d0, count=0, count@entry=2147483647,
isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true,
dest=dest@entry=0x79e17c0, altdest=altdest@entry=0x79e17c0,
qc=qc@entry=0x509f2d0) at pquery.c:779
#11 0x000000000079e72b in exec_simple_query (
query_string=0x79e0950 "copy (select t into temp test3 from test1 where
id=3) to stdout;") at postgres.c:1173
#12 0x000000000079fec0 in PostgresMain (argc=argc@entry=1,
argv=argv@entry=0x509f980, dbname=<optimized out>,
username=<optimized out>) at postgres.c:4327
#13 0x0000000000708858 in BackendRun (port=0x509f7c0, port=0x509f7c0)
at postmaster.c:4464
#14 SubPostmasterMain (argc=argc@entry=3, argv=argv@entry=0x1f6ae0)
at postmaster.c:4977
#15 0x000000000093db93 in main (argc=3, argv=0x1f6ae0) at main.c:186

I'll go thru the other tests to see which ones are crashes and provide a bt
of those too if you find it useful.
Anyway at a glance they all seem to be on tests designed to throw an error.

Thanks,
Regina

"Regina Obe" <lr@pcorp.us> writes:

The backtrace of that looks like this:

#0 0x00007ffe76032136 in ntdll!RtlRaiseStatus ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#1 0x00007ffe75f60bb3 in ntdll!RtlUnwindEx ()
from C:\WINDOWS\SYSTEM32\ntdll.dll
#2 0x00007ffe74002eed in msvcrt!_setjmpex ()
from C:\WINDOWS\System32\msvcrt.dll
#3 0x00000000008da31a in pg_re_throw () at elog.c:1776
#4 0x00000000008da19d in errfinish (filename=<optimized out>,
filename@entry=0xa291a0 <BinarySignature+24> "copyto.c",
lineno=lineno@entry=490,
funcname=funcname@entry=0xa29980 <__func__.112127> "BeginCopyTo")
at elog.c:589

...

I'll go thru the other tests to see which ones are crashes and provide a bt
of those too if you find it useful.
Anyway at a glance they all seem to be on tests designed to throw an error.

It does look suspiciously like something is broken in our attempt to
longjmp back to the server's outermost loop. We had a similar report
last fall, also about MinGW:

/messages/by-id/CA+BEBhvHhM-Bn628pf-LsjqRh3Ang7qCSBG0Ga+7KwhGqrNUPw@mail.gmail.com

and I don't think we ever figured out what was happening there either.

(FWIW, walleye's owners report that the failure goes away when they
change their compiler options to more debug-friendly settings. Hmm.)

regards, tom lane

"Regina Obe" <lr@pcorp.us> writes:

The backtrace of that looks like this:

#0 0x00007ffe76032136 in ntdll!RtlRaiseStatus ()
#3 0x00000000008da31a in pg_re_throw () at elog.c:1776

...

I'll go thru the other tests to see which ones are crashes and provide
a bt of those too if you find it useful.
Anyway at a glance they all seem to be on tests designed to throw an

error.

It does look suspiciously like something is broken in our attempt to

longjmp

back to the server's outermost loop. We had a similar report last fall,

also

about MinGW:

/messages/by-id/CA+BEBhvHhM-Bn628pf-
LsjqRh3Ang7qCSBG0Ga%2B7KwhGqrNUPw%40mail.gmail.com

and I don't think we ever figured out what was happening there either.

(FWIW, walleye's owners report that the failure goes away when they change
their compiler options to more debug-friendly settings. Hmm.)

regards, tom lane

What is considered a more debug-friendly setting?

My configure looks like this:
./configure --prefix=${PROJECTS}/postgresql/rel/pg${PG_VER}\
--build=${MINGHOST} \
--with-pgport=8447 --enable-cassert --enable-debug \
--with-zlib --with-libxml

If I take out the --enable-cassert switch, all tests pass and no crashers.

As far as crashers go, they all seem to be crashing at the

0x00000000008da31a in pg_re_throw () at elog.c:1776

Line
FWIW:

These are the crashers:

# CRASHERS
copyselect - crash #3 0x00000000008da31a in pg_re_throw () at elog.c:1776
copydml - crash - copy (insert into copydml_test default values) to stdout;
#3 0x00000000008da31a in pg_re_throw () at elog.c:1776

create_index - crash
test vacuum ... FAILED #crash
test transactions ... FAILED (test process exited with exit
code 2) 174 ms
test subscription #crash
test foreign_data ... FAILED (test process exited with exit
code 2) 191 ms
test jsonb_jsonpath ... FAILED (test process exited with exit
code 2) 150 ms
test xml ... FAILED (test process exited with exit
code 2) 142 ms

#These ones aren't crashing but failing regression when --enable-cassert is
on

test select ... FAILED 102 ms
test inherit ... FAILED 2024 ms
test sanity_check ... FAILED
test join ... FAILED 948 ms
test aggregates ... FAILED 6478 ms
test portals ... FAILED 515 ms
test arrays ... FAILED 261 ms
test brin ... FAILED 515 ms
test groupingsets ... FAILED 370 ms
test misc_functions ... FAILED 302 ms
test rules ... FAILED 1264 ms
test psql ... FAILED 406 ms
test amutils ... FAILED 58 ms
test select_parallel ... FAILED 1962 ms

test limit ... FAILED 99 ms
test alter_table ... FAILED 4693 ms
test rowtypes ... FAILED 266 ms

test stats ... FAILED 34207 ms

I don't think I've recompiled PG13 that recently so I wonder if my build is
too old to see the PG13 ones you linked to. I'll try rebuilding that to see
if I see similar issues.

Thanks,
Regina

"Regina Obe" <lr@pcorp.us> writes:

(FWIW, walleye's owners report that the failure goes away when they change
their compiler options to more debug-friendly settings. Hmm.)

What is considered a more debug-friendly setting?

Apparently they don't see a crash with CFLAGS="-ggdb -Og -g3"
(as opposed to the default -O2 possibly with -g).

As far as crashers go, they all seem to be crashing at the
0x00000000008da31a in pg_re_throw () at elog.c:1776

Cool, thanks for confirming that. At least now we know
what it is we don't know.

regards, tom lane

"Regina Obe" <lr@pcorp.us> writes:

(FWIW, walleye's owners report that the failure goes away when they
change their compiler options to more debug-friendly settings. Hmm.)

What is considered a more debug-friendly setting?

Apparently they don't see a crash with CFLAGS="-ggdb -Og -g3"
(as opposed to the default -O2 possibly with -g).

As far as crashers go, they all seem to be crashing at the
0x00000000008da31a in pg_re_throw () at elog.c:1776

Cool, thanks for confirming that. At least now we know what it is we

don't

know.

regards, tom lane

I just tested latest PG13 stable branch with same --configure as I used for
PG14 and all tests pass even with --enable-cassert.
So I guess something changed in master.

I thought maybe there was some overriding of some of the CFLAGS but the
configure.log of both PG13 and PG14 show the same

CFLAGS='-O2 -pipe -fno-ident
-I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include
-I/c/mingw810/prerequisites/x86_64-zlib-static/include
-I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include'

"Regina Obe" <lr@pcorp.us> writes:

I just tested latest PG13 stable branch with same --configure as I used for
PG14 and all tests pass even with --enable-cassert.
So I guess something changed in master.

Don't suppose I could interest you in "git bisect"-ing that?

regards, tom lane

"Regina Obe" <lr@pcorp.us> writes:

I just tested latest PG13 stable branch with same --configure as I used

for

PG14 and all tests pass even with --enable-cassert.
So I guess something changed in master.

Don't suppose I could interest you in "git bisect"-ing that?

regards, tom lane

Sure you can :)
I'll go thru the commits and let you know what I find.
At a glance looks like walleye is suggesting it happened in this commit

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=85d94c5

"Regina Obe" <lr@pcorp.us> writes:

At a glance looks like walleye is suggesting it happened in this commit
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=85d94c5

That one only touched libpq not the backend, but yeah, walleye's history
suggests the one immediately before it (3174d69fb9) might've been the
culprit. Which makes little sense, because that had about nothing to do
with error handling. So I'm interested to see if you can reproduce the
conclusion that that's where it broke.

regards, tom lane

"Regina Obe" <lr@pcorp.us> writes:

I just tested latest PG13 stable branch with same --configure as I

used for

PG14 and all tests pass even with --enable-cassert.
So I guess something changed in master.

Don't suppose I could interest you in "git bisect"-ing that?

regards, tom lane

Sure you can :)
I'll go thru the commits and let you know what I find.
At a glance looks like walleye is suggesting it happened in this commit

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=85d94c5

Okay wasn't that commit, but close.

My bisect shows this commit as the culprit

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=3174d69fb

Sadly it's a huge commit so not even sure where to start troubleshooting it.

Hope that helps,
Regina