Fix resource leaks in pg_receivewal on LZ4 close failures

Started by Fujii Masao11 days ago12 messageshackers
Jump to latest
#1Fujii Masao
masao.fujii@gmail.com

Hi,

I'd like to propose the attached patch to fix file descriptor and
memory leaks when LZ4 compression finalization or the final write
fails. Without this fix, repeated failures can cause these resources
to accumulate.

Thoughts?

Regards,

--
Fujii Masao

Attachments:

t258466_1
v1-0001-pg_receivewal-Fix-resource-leaks-and-error-report.patchapplication/octet-stream; name=v1-0001-pg_receivewal-Fix-resource-leaks-and-error-report.patchDownload+16-7
#2Chao Li
li.evan.chao@gmail.com
In reply to: Fujii Masao (#1)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Nov 11, 2026, at 17:48, Fujii Masao <masao.fujii@gmail.com> wrote:

Hi,

I'd like to propose the attached patch to fix file descriptor and
memory leaks when LZ4 compression finalization or the final write
fails. Without this fix, repeated failures can cause these resources
to accumulate.

Thoughts?

Regards,

--
Fujii Masao
<v1-0001-pg_receivewal-Fix-resource-leaks-and-error-report.patch>

I have a question about the fix in dir_close(). In the current code, r = close(df->fd);, then if r == 0, it will do some operations based on method, such as unlink tmppath. But this patch does:
```
(void) close(df->fd);
goto cleanup;
```

Which bypasses the operations for successful close(df->fd), is that intentional?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#3Fujii Masao
masao.fujii@gmail.com
In reply to: Chao Li (#2)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Wed, Nov 11, 2026 at 7:52 PM Chao Li <li.evan.chao@gmail.com> wrote:

Which bypasses the operations for successful close(df->fd), is that intentional?

Yes. Previously, if compression finalization or the final write failed,
the function returned -1 immediately, skipping rename, unlink, and
sync. So the patch just preserves that behavior while releasing
the resources.

Regards,

--
Fujii Masao

#4Michael Paquier
michael@paquier.xyz
In reply to: Fujii Masao (#1)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Wed, Nov 11, 2026 at 06:48:51PM +0900, Fujii Masao wrote:

I'd like to propose the attached patch to fix file descriptor and
memory leaks when LZ4 compression finalization or the final write
fails. Without this fix, repeated failures can cause these resources
to accumulate.

Thoughts?

That's worth a backpatch to me all the way down to me, where this has
been introduced. This stuff could be run in a loop in a service-like
fashion, bloating the fds on repeated failures.
--
Michael

#5Chao Li
li.evan.chao@gmail.com
In reply to: Fujii Masao (#3)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Nov 11, 2026, at 23:55, Fujii Masao <masao.fujii@gmail.com> wrote:

On Wed, Nov 11, 2026 at 7:52 PM Chao Li <li.evan.chao@gmail.com> wrote:

Which bypasses the operations for successful close(df->fd), is that intentional?

Yes. Previously, if compression finalization or the final write failed,
the function returned -1 immediately, skipping rename, unlink, and
sync. So the patch just preserves that behavior while releasing
the resources.

Regards,

--
Fujii Masao

It’s true that the previous code returned -1 immediately on LZ4F_isError, causing the fd leak that this patch fixes. But I think that also cause a temporary file leak. In writeTimeLineHistoryFile(), there is a call:
```
/*
* If we fail to make the file, delete it to release disk space
*/
stream->walmethod->ops->close(f, CLOSE_UNLINK);
```

This call is intended to delete a temporary file, so I don’t think an LZ4F error should prevent that deletion. Maybe fix this leak as well?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#6Xuneng Zhou
xunengzhou@gmail.com
In reply to: Fujii Masao (#3)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Wed, Nov 11, 2026 at 11:56 PM Fujii Masao <masao.fujii@gmail.com> wrote:

On Wed, Nov 11, 2026 at 7:52 PM Chao Li <li.evan.chao@gmail.com> wrote:

Which bypasses the operations for successful close(df->fd), is that intentional?

Yes. Previously, if compression finalization or the final write failed,
the function returned -1 immediately, skipping rename, unlink, and
sync. So the patch just preserves that behavior while releasing
the resources.

The fix LGTM. I am also a bit curious why unlink is also skipped
before though it is unrelated to the cur patch.

--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.

#7Fujii Masao
masao.fujii@gmail.com
In reply to: Chao Li (#5)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Thu, Nov 12, 2026 at 9:36 AM Chao Li <li.evan.chao@gmail.com> wrote:

This call is intended to delete a temporary file, so I don’t think an LZ4F error should prevent that deletion. Maybe fix this leak as well?

Maybe yes, seems we can attempt to unlink the file on error when
CLOSE_UNLINK is specified. This should also cover close() and gzclose()
failures, not just LZ4 finalization or final write failures? If so, I think
a separate patch would be better, since this changes cleanup behavior
beyond the LZ4 resource leak fix.

Regards,

--
Fujii Masao

#8Chao Li
li.evan.chao@gmail.com
In reply to: Fujii Masao (#7)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Nov 19, 2026, at 15:19, Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Nov 12, 2026 at 9:36 AM Chao Li <li.evan.chao@gmail.com> wrote:

This call is intended to delete a temporary file, so I don’t think an LZ4F error should prevent that deletion. Maybe fix this leak as well?

Maybe yes, seems we can attempt to unlink the file on error when
CLOSE_UNLINK is specified. This should also cover close() and gzclose()
failures, not just LZ4 finalization or final write failures? If so, I think
a separate patch would be better, since this changes cleanup behavior
beyond the LZ4 resource leak fix.

Sounds fair.

Would you prepare for a follow up patch? Or if you want me to do so?

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#9Fujii Masao
masao.fujii@gmail.com
In reply to: Chao Li (#8)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Thu, Nov 19, 2026 at 4:42 PM Chao Li <li.evan.chao@gmail.com> wrote:

Would you prepare for a follow up patch? Or if you want me to do so?

Please feel free to work on that :)

Regards,

--
Fujii Masao

#10Chao Li
li.evan.chao@gmail.com
In reply to: Fujii Masao (#9)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Nov 19, 2026, at 20:37, Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Nov 19, 2026 at 4:42 PM Chao Li <li.evan.chao@gmail.com> wrote:

Would you prepare for a follow up patch? Or if you want me to do so?

Please feel free to work on that :)

Sure, will do.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

#11Fujii Masao
masao.fujii@gmail.com
In reply to: Chao Li (#10)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Thu, Nov 19, 2026 at 9:45 PM Chao Li <li.evan.chao@gmail.com> wrote:

Sure, will do.

Thanks!

I think we can commit my proposed patch independently of the file unlink
cleanup patch, so I've pushed it first.

Regards,

--
Fujii Masao

#12Chao Li
li.evan.chao@gmail.com
In reply to: Fujii Masao (#11)
Re: Fix resource leaks in pg_receivewal on LZ4 close failures

On Nov 20, 2026, at 19:49, Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Nov 19, 2026 at 9:45 PM Chao Li <li.evan.chao@gmail.com> wrote:

Sure, will do.

Thanks!

I think we can commit my proposed patch independently of the file unlink
cleanup patch, so I've pushed it first.

Regards,

--
Fujii Masao

Yea, I’m waiting for this patch pushed. I’ll work on a follow up patch next week.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/