FATAL: could not send end-of-streaming message to primary: no COPY in progress
Hi,
If you shut down a primary that has a streaming standby, the standby still
ends up with this:
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline 1 at ...
FATAL: could not send end-of-streaming message to primary: no COPY in
progress
Repro is just primary plus streaming standby, then stop the primary. The
attached TAP test does that.
Thomas reported this in 2016 [1]/messages/by-id/CAEepm=1fEZ7Dt2r-HEp71BL8Bf+HzLmU6gN-yTgH4u7zhxEcXQ@mail.gmail.com. Fujii sketched two ways out [2]/messages/by-id/CAHGQGwHvzV2J0QodA8x1xCx3CbaBmJTveQeoLFzX8hq5G25jEA@mail.gmail.com: either
have walsender wait for CopyDone after "COPY 0", or stop walreceiver from
sending end-of-copy once it already saw that. Kyotaro rejected the first
[3]: /messages/by-id/20160420.171830.207678798.horiguchi.kyotaro@lab.ntt.co.jp
pursued. Andres later called the FATAL itself a bug [4]/messages/by-id/20200323194914.novyhczbkqvk4j5x@alap3.anarazel.de. Commit
5fd56b9f5b4
improved the related disconnect messaging, but this path is still there.
ISTM Fujii's second option is still the right fix. On shutdown, physical
walsender already ends the COPY with CommandComplete and exits. receive()
consumes that, then endstreaming unconditionally calls PQputCopyEnd(), and
libpq quite reasonably complains that there is no COPY in progress.
Timeline switch is different: there the server waits for the client's
CopyDone, so the existing path remains correct.
The attached patch just remembers that CommandComplete already ended the
COPY, and skips PQputCopyEnd() in that case. No walsender change.
Does this still seem worth fixing?
[1]: /messages/by-id/CAEepm=1fEZ7Dt2r-HEp71BL8Bf+HzLmU6gN-yTgH4u7zhxEcXQ@mail.gmail.com
/messages/by-id/CAEepm=1fEZ7Dt2r-HEp71BL8Bf+HzLmU6gN-yTgH4u7zhxEcXQ@mail.gmail.com
[2]: /messages/by-id/CAHGQGwHvzV2J0QodA8x1xCx3CbaBmJTveQeoLFzX8hq5G25jEA@mail.gmail.com
/messages/by-id/CAHGQGwHvzV2J0QodA8x1xCx3CbaBmJTveQeoLFzX8hq5G25jEA@mail.gmail.com
[3]: /messages/by-id/20160420.171830.207678798.horiguchi.kyotaro@lab.ntt.co.jp
/messages/by-id/20160420.171830.207678798.horiguchi.kyotaro@lab.ntt.co.jp
[4]: /messages/by-id/20200323194914.novyhczbkqvk4j5x@alap3.anarazel.de
/messages/by-id/20200323194914.novyhczbkqvk4j5x@alap3.anarazel.de
Regards,
Stepan Neretin
Attachments:
t259254_10002-Add-TAP-test-for-primary-shutdown-end-of-streaming-F.patchtext/x-patch; charset=US-ASCII; name=0002-Add-TAP-test-for-primary-shutdown-end-of-streaming-F.patchDownload+32-1
0001-Avoid-FATAL-when-primary-ends-streaming-with-COPY-0.patchtext/x-patch; charset=US-ASCII; name=0001-Avoid-FATAL-when-primary-ends-streaming-with-COPY-0.patchDownload+15-1
On Tue, Nov 17, 2026 at 2:58 AM Stepan Neretin <slpmcf@gmail.com> wrote:
The attached patch just remembers that CommandComplete already ended the
COPY, and skips PQputCopyEnd() in that case. No walsender change.Does this still seem worth fixing?
Yes, I think.
Thanks for the patch! I haven't read it yet, but when I ran CI with it,
I observed that subscription/029_on_error failed with the following error:
436/436 postgresql:subscription / subscription/029_on_error
ERROR 257.82s exit status 29
――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
stderr:
# poll_query_until timed out executing this query:
# SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN
('r', 's');
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
# Last pg_subscription_rel contents:
# 16392|16384|f|
# die: timed out waiting for subscriber to synchronize data at
/__w/postgresql/postgresql/src/test/subscription/t/029_on_error.pl
line 130.
(test program exited with status code 29)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Regards,
--
Fujii Masao
On Tue, Nov 17, 2026 at 1:55 PM Fujii Masao <masao.fujii@gmail.com> wrote:
On Tue, Nov 17, 2026 at 2:58 AM Stepan Neretin <slpmcf@gmail.com> wrote:
The attached patch just remembers that CommandComplete already ended the
COPY, and skips PQputCopyEnd() in that case. No walsender change.Does this still seem worth fixing?
Yes, I think.
Thanks for the patch! I haven't read it yet, but when I ran CI with it,
I observed that subscription/029_on_error failed with the following error:436/436 postgresql:subscription / subscription/029_on_error
ERROR 257.82s exit status 29
――――――――――――――――――――――――――――――――――――― ✀
―――――――――――――――――――――――――――――――――――――
stderr:
# poll_query_until timed out executing this query:
# SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN
('r', 's');
# expecting this output:
# t
# last actual query output:
# f
# with stderr:
# Last pg_subscription_rel contents:
# 16392|16384|f|
# die: timed out waiting for subscriber to synchronize data at
/__w/postgresql/postgresql/src/test/subscription/t/029_on_error.pl
line 130.(test program exited with status code 29)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Regards,
--
Fujii Masao
Hi Fujii,
Thanks for checking. The 029_on_error failure was real.
receive() is also used for tablesync COPY TO STDOUT, so CommandComplete
there set copyEnded, and the later catchup stream reused that connection
and skipped CopyDone. v2 clears the flag in startstreaming().
029_on_error and 061_primary_shutdown_streaming both pass locally with
this.
Attached v2.
Regards,
Stepan Neretin
Attachments:
t259254_3v2-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchDownload+17-1
v2-0002-Add-TAP-test-for-primary-shutdown-during-streaming.patchtext/x-patch; charset=US-ASCII; name=v2-0002-Add-TAP-test-for-primary-shutdown-during-streaming.patchDownload+32-1
Hello
1.
Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during streaming
This should also have a meson.build edit so the meson builds also run the new test
2. Stopping the primary and waiting a bit gives this on the standby:
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline 1 at 0/030000B0.
FATAL: could not receive database system identifier and timeline ID from the primary server: no connection to the server
After the early return walreceiver waits for new orders and then runs IDENTIFY_SYSTEM on the closed connection. The test passes because it only checks that the old message is gone.
3. On master, a clean publisher shutdown hits the same error in the logical apply worker, and that disables a disable_on_error subscription:
LOG: data stream from publisher has ended
ERROR: could not send end-of-streaming message to primary: server closed the connection unexpectedly
LOG: subscription "sub" has been disabled because of an error
With the patch only the first line remains and the subscription stays enabled. This doesn't seem like a bad thing, but probably should be mentioned at least in the commit message, and maybe tested too?
4. Partial WAL segment archived on archive_mode=always standbys
Since endstreaming no longer fails, walreceiver continues into the end-of-timeline block in WalReceiverMain(), which closes the current segment and calls XLogArchiveNotify() on it, even though the segment is incomplete.
Setup:
* standby with archive_mode = always and archive_command = 'test ! -f arch/%f && cp %p arch/%f'
* stop the primary
* wait a few seconds
* start it again
* run pg_switch_wal() a few times:
master: archived while primary down: (none)
pg_stat_archiver: 5|000000010000000000000007|0|
patched: archived while primary down: 000000010000000000000003
pg_stat_archiver: 1|000000010000000000000003|6|000000010000000000000003
5. Would something like this work instead?
libpqrcv_endstreaming():
+ *next_tli = 0;
+ if (PQstatus(conn->streamConn) == CONNECTION_BAD)
+ return;
walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI):
+ if (primaryTLI == 0)
+ proc_exit(0);
On Wed, Nov 18, 2026 at 4:54 AM Zsolt Parragi <zsolt.parragi@percona.com>
wrote:
Hello
1.
Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during
streaming
This should also have a meson.build edit so the meson builds also run
the new test2. Stopping the primary and waiting a bit gives this on the standby:
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline 1 at 0/030000B0.
FATAL: could not receive database system identifier and timeline ID
from the primary server: no connection to the serverAfter the early return walreceiver waits for new orders and then runs
IDENTIFY_SYSTEM on the closed connection. The test passes because it
only checks that the old message is gone.3. On master, a clean publisher shutdown hits the same error in the
logical apply worker, and that disables a disable_on_error
subscription:LOG: data stream from publisher has ended
ERROR: could not send end-of-streaming message to primary: server
closed the connection unexpectedly
LOG: subscription "sub" has been disabled because of an errorWith the patch only the first line remains and the subscription stays
enabled. This doesn't seem like a bad thing, but probably should be
mentioned at least in the commit message, and maybe tested too?4. Partial WAL segment archived on archive_mode=always standbys
Since endstreaming no longer fails, walreceiver continues into the
end-of-timeline block in WalReceiverMain(), which closes the current
segment and calls XLogArchiveNotify() on it, even though the segment
is incomplete.Setup:
* standby with archive_mode = always and archive_command = 'test ! -f
arch/%f && cp %p arch/%f'
* stop the primary
* wait a few seconds
* start it again
* run pg_switch_wal() a few times:master: archived while primary down: (none)
pg_stat_archiver: 5|000000010000000000000007|0|
patched: archived while primary down: 000000010000000000000003
pg_stat_archiver:
1|000000010000000000000003|6|0000000100000000000000035. Would something like this work instead?
libpqrcv_endstreaming(): + *next_tli = 0; + if (PQstatus(conn->streamConn) == CONNECTION_BAD) + return; walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI): + if (primaryTLI == 0) + proc_exit(0);
Hi Zsolt,
Thanks for the careful review. Agreed on points 1 to 4. I missed those
follow ons in v2. On point 5 I like the direction, but CONNECTION_BAD
alone does not cover the original report.
After COPY 0, libpq has left COPY while PQstatus() can still be
CONNECTION_OK, so PQputCopyEnd() still fails with no COPY in progress.
That is why copyEnded stays. CONNECTION_BAD is a useful complement for
an abrupt close, so I kept that too.
v3 also does proc_exit(0) in physical walreceiver when endstreaming
reports next_tli == 0. Timeline switch still sends a non zero next TLI
and keeps waiting mode. Clean shutdown does not, and falling through
was what caused the IDENTIFY_SYSTEM FATAL and the incomplete segment
archive under archive_mode=always.
Both TAP tests are in meson.build now, and they cover the endstreaming
FATAL, IDENTIFY_SYSTEM, disable_on_error, incomplete segment archive,
and reconnect after restart.
Attached.
Regards,
Stepan Neretin
Attachments:
t259254_5v3-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patchtext/x-patch; charset=US-ASCII; name=v3-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patchDownload+169-1
v3-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchtext/x-patch; charset=US-ASCII; name=v3-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchDownload+30-1
Hi Stepan,
Thanks for updating v3.
V3 skips the final CopyDone, but after COPY 0 both receivers leave the
inner receive loop and can try to send feedback before reaching that
check(WalReceiverMain -> XLogWalRcvSendReply, LogicalRepApplyLoop ->
send_feedback). If feedback is due, the send can fail with no COPY in
progress. We could add copyEnded to libpqrcv_send to guard it, though
the callers could still mark the feedback as sent for skipping..
For HEAD, I wonder if it would be clearer for walrcv_receive() to
return two distinct values when the stream ends. With two return
values, the function signature is intact and both callers would keep
the existing handshake and send CopyDone after PGRES_COPY_IN. After
PGRES_COMMAND_OK, the two could skip further sends and finish their
local cleanup. This seems to change more and touch both loops, so I
see it as an option for HEAD.
Additionally,
PQstatus(conn->streamConn) == CONNECTION_BAD
Do we need this check in libpqrcv_endstreaming()?
copyEnded already covers COPY 0. If it is false, the check could turn
an unexpected disconnect into a clean exit. Is there a caller path
that needs it?
--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
On Sun, Nov 22, 2026 at 2:01 PM Xuneng Zhou <xunengzhou@gmail.com> wrote:
Hi Stepan,
Thanks for updating v3.
V3 skips the final CopyDone, but after COPY 0 both receivers leave the
inner receive loop and can try to send feedback before reaching that
check(WalReceiverMain -> XLogWalRcvSendReply, LogicalRepApplyLoop ->
send_feedback). If feedback is due, the send can fail with no COPY in
progress. We could add copyEnded to libpqrcv_send to guard it, though
the callers could still mark the feedback as sent for skipping..For HEAD, I wonder if it would be clearer for walrcv_receive() to
return two distinct values when the stream ends. With two return
values, the function signature is intact and both callers would keep
the existing handshake and send CopyDone after PGRES_COPY_IN. After
PGRES_COMMAND_OK, the two could skip further sends and finish their
local cleanup. This seems to change more and touch both loops, so I
see it as an option for HEAD.Additionally,
PQstatus(conn->streamConn) == CONNECTION_BAD
Do we need this check in libpqrcv_endstreaming()?
copyEnded already covers COPY 0. If it is false, the check could turn
an unexpected disconnect into a clean exit. Is there a caller path
that needs it?--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
Hi,
Thanks for the review. Agreed on both points.
After COPY 0, both receivers leave the inner loop and can still send
feedback before endstreaming. If a status update is due,
PQputCopyData() fails with no COPY in progress. v5 skips those sends
on the caller side once the stream has ended, instead of guarding
libpqrcv_send(), so we do not mark feedback as sent when nothing was
written. Local flush still happens via XLogWalRcvFlush(dying=true).
A second return value from walrcv_receive() for COMMAND_OK vs COPY_IN
looks clearer for HEAD. I would keep this fix smaller and
backpatchable, and leave that as a follow-up.
CONNECTION_BAD in endstreaming() is not needed. copyEnded already
covers COPY 0, and an abrupt disconnect normally ERROR's in receive()
before we get there. With next_tli == 0 doing proc_exit(0), that check
could also turn an unexpected disconnect into a quiet exit. Removed in
v5.
Attached.
Regards,
Stepan Neretin
Attachments:
t259254_7v5-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patchtext/x-patch; charset=US-ASCII; name=v5-0002-Add-TAP-tests-for-primary-publisher-shutdown-during-.patchDownload+169-1
v5-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchtext/x-patch; charset=US-ASCII; name=v5-0001-Fix-FATAL-when-primary-ends-streaming-with-COPY-0.patchDownload+48-7
Attached.
I haven't looked at the test yet.
+ * end of streaming.
+ */
+ if (conn->copyEnded)
+ {
+ conn->copyEnded = false;
+ *next_tli = 0;
+ return;
+ }
BTW, we can move *next_tli = 0; to the start of
libpqrcv_endstreaming() to save one extra assignment.
--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
On Sun, Nov 22, 2026 at 9:08 PM Xuneng Zhou <xunengzhou@gmail.com> wrote:
Attached.
I haven't looked at the test yet.
+ * end of streaming. + */ + if (conn->copyEnded) + { + conn->copyEnded = false; + *next_tli = 0; + return; + }BTW, we can move *next_tli = 0; to the start of
libpqrcv_endstreaming() to save one extra assignment.--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.
Hi,
Thanks. Agreed. v6 moves *next_tli = 0 to the start of
libpqrcv_endstreaming(). Nothing else changed.
Attached.
Regards,
Stepan Neretin